# Run plain-language browser tests in GitLab CI

> Include Testsy's GitLab template, point it at your project or a review app, and fail the pipeline on a broken journey, with JUnit results and watch links in the job.

By Jason Nguyen, Founder, Testsy. Published 2026-11-21. https://testsy.dev/blog/run-plain-language-tests-in-gitlab-ci/

## The setup

Three things, from [the CI docs](https://testsy.dev/docs/ci/):

1. Create an API key in Testsy with **Run tests and read results** (**Settings → API keys**). It is shown once.
2. Add it as a masked CI/CD variable named `TESTSY_AI_API_KEY`.
3. Find your project id: it starts with `p_` and is in the project page's address.

Then add the template to `.gitlab-ci.yml`:

```yaml
# .gitlab-ci.yml
include:
  - remote: 'https://app.testsy.dev/integrations/testsy-ai.gitlab-ci.yml'

testsy-ai:
  extends: .testsy-ai
  stage: test        # or a stage after your deploy
  variables:
    TESTSY_AI_PROJECT: p_your_project
```

The app's **Connect** page shows this with your project filled in.

## The variables

The template is configured with variables on the job:

| Variable | What it does |
| --- | --- |
| `TESTSY_AI_PROJECT` | The project whose tests run |
| `TESTSY_AI_TESTS` | Test ids, instead of a whole project |
| `TESTSY_AI_MODE` | `auto` (default), `cold` or `warm` |
| `TESTSY_AI_TIMEOUT` | How long to wait, 45m by default |
| `TESTSY_AI_FAIL_ON` | Verdicts that fail the job, `failed,error` by default |
| `TESTSY_AI_TARGET_URL` | Start website tests at another address, such as a review app |
| `TESTSY_AI_BUILD` | App builds to test: files, https links or `ab_` ids |

## Review apps

GitLab gives each merge request's review app its address in `CI_ENVIRONMENT_URL`. Pass it, and the same saved tests run against the review app instead of production:

```yaml
TESTSY_AI_TARGET_URL: $CI_ENVIRONMENT_URL
```

The run explores from that address (a learned path belongs to the saved one), and the result shows the address it ran against. Sign-in accounts belong to the test, so a review app that needs different credentials needs its own test.

## What the job shows

Each run records the pipeline, commit, branch and job, and links back to the job. The JUnit file lists each test as passed or failed with a link to its proof, so GitLab's test report shows them in the merge request. Each started run prints a [watch link](https://testsy.dev/docs/watch-links/) that anyone can open without an account, while it runs and afterwards; paste it in the merge request for reviewers.

## When it fails

The job fails on the verdicts in `TESTSY_AI_FAIL_ON`. A run that could not start fails it too (exit code 2), so a missing key or an exhausted plan does not pass silently. Open the result from the job log and read the failure card: the check, expected and seen, and the step where it failed.

## App builds in the same pipeline

For iPhone and Android apps, `TESTSY_AI_BUILD` takes the build your pipeline just made, uploads it unless that exact file is already there, and runs the app's tests on it. The template's lookup job can skip the build entirely when the native fingerprint has not changed: see [app builds from CI](https://testsy.dev/docs/app-builds-ci/).

This post is part of [Release testing for product teams](https://testsy.dev/blog/release-testing-for-product-teams/).
