Run browser tests after every deploy with GitHub Actions
A small set of plain-language release checks that run on each deployment, fail the job on a broken journey, and keep the replay next to the commit that broke it.

After the deploy, not before it#
Unit and integration tests run before you ship. They cannot tell you whether the deployed app works: the environment variables, the database, the email provider and the CDN only meet in production. That is why teams keep a post-deploy checklist, and why one piece on CI puts it bluntly: manual checks after deploy work until they don't, and stop working the moment you deploy more than once a day.
The automated version of that checklist is a short browser suite that runs right after each deploy against the address that was just deployed.
Pick three journeys#
Start small. For a typical SaaS product, the docs suggest:
- A test user can sign in and see the right workspace.
- A member can create a record and find it after reloading.
- An editor can publish something and see the published version.
Write each one as plain text in a Testsy project, run it from the app until it passes reliably, then wire it in. The release smoke starter is a reviewed example to edit.
The workflow#
Create a key with Run tests and read results in Settings → API keys and save it as the secret TESTSY_AI_API_KEY. The app's Connect page shows this workflow with your project filled in:
# .github/workflows/testsy-ai.yml
name: Testsy
on:
deployment_status:
workflow_dispatch:
jobs:
qa:
if: github.event_name == 'workflow_dispatch' || github.event.deployment_status.state == 'success'
runs-on: ubuntu-latest
steps:
- name: Run Testsy tests
env:
TESTSY_AI_API_KEY: ${{ secrets.TESTSY_AI_API_KEY }}
TESTSY_AI_URL: https://app.testsy.dev
TARGET: ${{ github.event.deployment_status.environment_url }}
run: |
curl -fsSL "$TESTSY_AI_URL/cli/testsy-ai.mjs" -o testsy-ai.mjs
# On a deployment event the tests run against that deploy's address (a preview or staging of the same site).
node testsy-ai.mjs run --project p_your_project --junit testsy-ai-junit.xml ${TARGET:+--url "$TARGET"}
- uses: actions/upload-artifact@v4
if: always()
with: { name: testsy-ai-junit, path: testsy-ai-junit.xml }It runs on a successful deployment event (or by hand), passes the deployment's address so website tests start there, waits for every verdict, and writes a JUnit file that is kept even when the step fails. There is also a published action, uses: testsy-ai/testsy-action@v1, that does the same with less to copy; CI lists its inputs.
Decide what blocks a release#
By default the job fails when a run fails or does not finish:
| Value | The run | Fails the build by default |
|---|---|---|
failed | A check failed. | Yes |
error | Did not finish because of a problem on our side. Not counted. | Yes |
blocked | The site stopped the tester, for example a bot wall. Not counted. | No |
inconclusive | The tester could not tell for sure. | No |
A blocked or inconclusive run is not a pass, even when the default lets the job succeed. For a gate that needs a conclusive answer, add --fail-on failed,error,blocked,inconclusive,cancelled. Choose deliberately and write the choice down; removing a verdict from the gate to make a broken release green defeats the point.
When it fails#
Each run records the commit, branch and job, and on GitHub a table of results goes into the job summary. Each started run also prints a watch link anyone can open without an account, live and afterwards. Open the failed one and read the failure card: the check, what was expected, what was seen, and the step to watch it fail.

Fix it, deploy, and the same workflow reruns the same tests. Keep the failed result; it is the record of what broke and when.
Grow it slowly#
- Add email coverage when sign-up matters: Test sign-ups that send an email or SMS code.
- Run the same tests on each pull request's preview: Test every Vercel preview before you merge.
- Add a test for every production bug you fix: one bug, one test.
Every run counts toward your plan. Three tests on each deploy, five deploys a day, is fifteen runs a day.
This post is part of Release testing for product teams.
Filed underWebReleases and regressionsCI, schedules and agents