Run plain-language browser tests in GitLab CI
Include Testsy's GitLab template, point it at your project or a review app, and fail the pipeline on a broken journey, with JUnit results and watch links in the job.

The setup#
Three things, from the CI docs:
- Create an API key in Testsy with Run tests and read results (Settings → API keys). It is shown once.
- Add it as a masked CI/CD variable named
TESTSY_AI_API_KEY. - Find your project id: it starts with
p_and is in the project page's address.
Then add the template to .gitlab-ci.yml:
# .gitlab-ci.yml
include:
- remote: 'https://app.testsy.dev/integrations/testsy-ai.gitlab-ci.yml'
testsy-ai:
extends: .testsy-ai
stage: test # or a stage after your deploy
variables:
TESTSY_AI_PROJECT: p_your_projectThe app's Connect page shows this with your project filled in.
The variables#
The template is configured with variables on the job:
| Variable | What it does |
|---|---|
TESTSY_AI_PROJECT | The project whose tests run |
TESTSY_AI_TESTS | Test ids, instead of a whole project |
TESTSY_AI_MODE | auto (default), cold or warm |
TESTSY_AI_TIMEOUT | How long to wait, 45m by default |
TESTSY_AI_FAIL_ON | Verdicts that fail the job, failed,error by default |
TESTSY_AI_TARGET_URL | Start website tests at another address, such as a review app |
TESTSY_AI_BUILD | App builds to test: files, https links or ab_ ids |
Review apps#
GitLab gives each merge request's review app its address in CI_ENVIRONMENT_URL. Pass it, and the same saved tests run against the review app instead of production:
TESTSY_AI_TARGET_URL: $CI_ENVIRONMENT_URLThe run explores from that address (a learned path belongs to the saved one), and the result shows the address it ran against. Sign-in accounts belong to the test, so a review app that needs different credentials needs its own test.
What the job shows#
Each run records the pipeline, commit, branch and job, and links back to the job. The JUnit file lists each test as passed or failed with a link to its proof, so GitLab's test report shows them in the merge request. Each started run prints a watch link that anyone can open without an account, while it runs and afterwards; paste it in the merge request for reviewers.
When it fails#
The job fails on the verdicts in TESTSY_AI_FAIL_ON. A run that could not start fails it too (exit code 2), so a missing key or an exhausted plan does not pass silently. Open the result from the job log and read the failure card: the check, expected and seen, and the step where it failed.
App builds in the same pipeline#
For iPhone and Android apps, TESTSY_AI_BUILD takes the build your pipeline just made, uploads it unless that exact file is already there, and runs the app's tests on it. The template's lookup job can skip the build entirely when the native fingerprint has not changed: see app builds from CI.
This post is part of Release testing for product teams.
Filed underWebCI, schedules and agents