Automate

Run tests in CI

Run a project's tests after every deploy, fail the build when a check fails, and keep the result with the commit. One file, no dependencies, any CI.

Before you start#

Create an API key. Settings → API keys → Create key, with "Run tests and read results". It is shown once.
Store it as a secret named QA_AI_API_KEY in your CI (GitHub: Settings → Secrets and variables → Actions; GitLab: a masked CI/CD variable).
Find your project id. It starts with p_ and is in the address of the project page, or run node qa-ai.mjs projects. The app's Integrations page shows these snippets with your project filled in.

Add the workflow#

Run it after your deploy finishes, so the tests see the new version: on GitHub, on deployment_status, or as the last step of your deploy job.

Integrations: the workflow with your project already filled in.

# .github/workflows/qa-ai.yml
name: Testsy
on:
  deployment_status:
  workflow_dispatch:
jobs:
  qa:
    if: github.event_name == 'workflow_dispatch' || github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - name: Run Testsy tests
        env:
          QA_AI_API_KEY: ${{ secrets.QA_AI_API_KEY }}
          QA_AI_URL: https://app.testsy.dev
        run: |
          curl -fsSL "$QA_AI_URL/cli/qa-ai.mjs" -o qa-ai.mjs
          node qa-ai.mjs run --project p_your_project --junit qa-ai-junit.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: qa-ai-junit, path: qa-ai-junit.xml }

# .gitlab-ci.yml
include:
  - remote: 'https://app.testsy.dev/integrations/qa-ai.gitlab-ci.yml'

qa-ai:
  extends: .qa-ai
  stage: test        # or a stage after your deploy
  variables:
    QA_AI_PROJECT: p_your_project

export QA_AI_API_KEY=qa_…            # from API keys
export QA_AI_URL=https://app.testsy.dev
curl -fsSL "$QA_AI_URL/cli/qa-ai.mjs" -o qa-ai.mjs

node qa-ai.mjs run --project p_your_project                 # every test in the project
node qa-ai.mjs run --test t_abc,t_def --junit out.xml   # chosen tests
node qa-ai.mjs projects
node qa-ai.mjs tests --project p_your_project

The GitLab template takes these variables: QA_AI_PROJECT, QA_AI_TESTS (test ids instead of a whole project), QA_AI_MODE (auto), QA_AI_TIMEOUT (45m), QA_AI_FAIL_ON (failed,error) and QA_AI_BUILD (app builds: files or ab_ ids, separated by spaces). Results appear in the merge request's test report.

Results stay with the commit. Each run records the CI, commit, branch and job and links back to the job. On GitHub a table of results is added to the job's summary. The JUnit file shows each test as passed or failed, with the link to its proof.

When the build fails#

By default the build fails when a run fails or does not finish. Change it with --fail-on (GitLab: QA_AI_FAIL_ON):

Value	The run	Fails the build by default
`failed`	A check failed.	Yes
`error`	Did not finish because of a problem on our side. Not counted.	Yes
`blocked`	The site stopped the tester, for example a bot wall. Not counted.	No
`inconclusive`	The tester could not tell for sure.	No

Exit codes#

Code	Meaning
`0`	Every run passed, or none ended in a `--fail-on` verdict.
`1`	A run ended in a `--fail-on` verdict.
`2`	A setup or API problem: a missing key, an unknown project, no runs started.
`3`	Timed out (`--timeout`, default 45m). The JUnit file is still written.

App builds#

Upload the build your pipeline just made and run the app's tests on it: see App builds from CI.