BLOG
BlogGuides

Run browser tests after every deploy with GitHub Actions

A small set of plain-language release checks that run on each deployment, fail the job on a broken journey, and keep the replay next to the commit that broke it.

After the deploy, not before it#

Unit and integration tests run before you ship. They cannot tell you whether the deployed app works: the environment variables, the database, the email provider and the CDN only meet in production. That is why teams keep a post-deploy checklist, and why one piece on CI puts it bluntly: manual checks after deploy work until they don't, and stop working the moment you deploy more than once a day.

The automated version of that checklist is a short browser suite that runs right after each deploy against the address that was just deployed.

Pick three journeys#

Start small. For a typical SaaS product, the docs suggest:

  • A test user can sign in and see the right workspace.
  • A member can create a record and find it after reloading.
  • An editor can publish something and see the published version.

Write each one as plain text in a Testsy project, run it from the app until it passes reliably, then wire it in. The release smoke starter is a reviewed example to edit.

The workflow#

Create a key with Run tests and read results in Settings → API keys and save it as the secret TESTSY_AI_API_KEY. The app's Connect page shows this workflow with your project filled in:

# .github/workflows/testsy-ai.yml
name: Testsy
on:
  deployment_status:
  workflow_dispatch:
jobs:
  qa:
    if: github.event_name == 'workflow_dispatch' || github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - name: Run Testsy tests
        env:
          TESTSY_AI_API_KEY: ${{ secrets.TESTSY_AI_API_KEY }}
          TESTSY_AI_URL: https://app.testsy.dev
          TARGET: ${{ github.event.deployment_status.environment_url }}
        run: |
          curl -fsSL "$TESTSY_AI_URL/cli/testsy-ai.mjs" -o testsy-ai.mjs
          # On a deployment event the tests run against that deploy's address (a preview or staging of the same site).
          node testsy-ai.mjs run --project p_your_project --junit testsy-ai-junit.xml ${TARGET:+--url "$TARGET"}
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: testsy-ai-junit, path: testsy-ai-junit.xml }

It runs on a successful deployment event (or by hand), passes the deployment's address so website tests start there, waits for every verdict, and writes a JUnit file that is kept even when the step fails. There is also a published action, uses: testsy-ai/testsy-action@v1, that does the same with less to copy; CI lists its inputs.

Decide what blocks a release#

By default the job fails when a run fails or does not finish:

ValueThe runFails the build by default
failedA check failed.Yes
errorDid not finish because of a problem on our side. Not counted.Yes
blockedThe site stopped the tester, for example a bot wall. Not counted.No
inconclusiveThe tester could not tell for sure.No

A blocked or inconclusive run is not a pass, even when the default lets the job succeed. For a gate that needs a conclusive answer, add --fail-on failed,error,blocked,inconclusive,cancelled. Choose deliberately and write the choice down; removing a verdict from the gate to make a broken release green defeats the point.

When it fails#

Each run records the commit, branch and job, and on GitHub a table of results goes into the job summary. Each started run also prints a watch link anyone can open without an account, live and afterwards. Open the failed one and read the failure card: the check, what was expected, what was seen, and the step to watch it fail.

A failed run: the check that failed, what was expected and what was seen, and the steps to reproduce it.
A failed run: the check that failed, what was expected and what was seen, and the steps to reproduce it.

Fix it, deploy, and the same workflow reruns the same tests. Keep the failed result; it is the record of what broke and when.

Grow it slowly#

Every run counts toward your plan. Three tests on each deploy, five deploys a day, is fifteen runs a day.

This post is part of Release testing for product teams.

Filed underWebReleases and regressionsCI, schedules and agents