Run SaaS smoke tests in GitHub Actions

Check signup, task persistence and publishing after deployment with Testsy, GitHub Actions, result links and a deliberate CI failure policy.

Choose a small release suite#

Use a Testsy project for the few customer journeys that must work after deployment. For a small SaaS product, useful first checks are:

• A test user can sign in and see the right workspace.
• A member can create a task and find it after reloading.
• An editor can publish a synthetic document and see the published version.

Start with an editable release smoke test, review its plan, and run it against your staging app before adding CI. Use synthetic records and dedicated test accounts. A shared staging workspace can accumulate data between runs, so make the test's record and cleanup rules explicit.

The deployed address must be reachable from Testsy. A GitHub job's localhost is not the hosted browser's localhost. Deploy your preview or staging build first, then check that environment.

Create a run-only API key#

In Settings → API keys, create a key with Run tests and read results. Save it as the GitHub Actions secret QA_AI_API_KEY. Do not paste it into the workflow file or log it.

Open the Testsy project and copy its p_ project ID. Its Integrations page provides the workflow with that project already filled in.

Add the workflow after deployment#

This shared example runs on a successful deployment event or a manual dispatch. Replace p_your_project with your project ID. Confirm that the saved tests point to the deployment you intend to check: triggering the workflow does not automatically rewrite their start addresses to a GitHub preview URL.

# .github/workflows/qa-ai.yml
name: Testsy
on:
  deployment_status:
  workflow_dispatch:
jobs:
  qa:
    if: github.event_name == 'workflow_dispatch' || github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - name: Run Testsy tests
        env:
          QA_AI_API_KEY: ${{ secrets.QA_AI_API_KEY }}
          QA_AI_URL: https://app.testsy.dev
        run: |
          curl -fsSL "$QA_AI_URL/cli/qa-ai.mjs" -o qa-ai.mjs
          node qa-ai.mjs run --project p_your_project --junit qa-ai-junit.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: qa-ai-junit, path: qa-ai-junit.xml }

If your deployment uses a different event, place the Testsy step after the deploy step in that workflow, or adapt the trigger to your deployment provider. Avoid starting the same suite twice from overlapping triggers.

The command runs the project's saved tests, waits for their results and writes JUnit output. The workflow retains that file even when the Testsy step fails. Run links and CI origin information help connect a result to its commit and workflow job.

Decide what blocks a release#

The default CLI policy fails for failed and error. A blocked or inconclusive run is not a pass, even if that default policy allows the CI job to succeed.

For a release gate that requires a conclusive result, add --fail-on failed,error,blocked,inconclusive,cancelled to the existing command:

node qa-ai.mjs run --project p_your_project --junit qa-ai-junit.xml --fail-on failed,error,blocked,inconclusive,cancelled

This also treats a cancelled run as a failed gate. Tests that cannot start still require investigation, regardless of the selected verdict policy. Choose the policy deliberately. Do not remove an inconvenient verdict from the gate merely to make a broken release green.

Investigate the exact failure#

Open the Testsy result linked from the job. Compare the expected value with the observed one and inspect the replay at the failed step.

If the app discarded the new task after reload, investigate its save behavior. If a bot challenge or sign-in barrier prevented the check, fix access or test-account setup. If the run was inconclusive, look for missing evidence or an ambiguous requirement. An infrastructure error needs a Testsy investigation, not a change to the product's expected behavior.

After a supported fix is deployed, rerun the same test and keep the original failed result. The CLI's exit codes distinguish verdict failures, setup problems and timeouts. Tests that could not start must be reviewed too.

Extend only after the first checks are useful#

Add email-verification coverage when signup is important to your release. Use the CLI guide for selected test IDs and other CI providers, and app builds from CI for supported app artifacts.