BLOG
BlogGuides

Run plain-language browser tests in GitLab CI

Include Testsy's GitLab template, point it at your project or a review app, and fail the pipeline on a broken journey, with JUnit results and watch links in the job.

The setup#

Three things, from the CI docs:

  1. Create an API key in Testsy with Run tests and read results (Settings → API keys). It is shown once.
  2. Add it as a masked CI/CD variable named TESTSY_AI_API_KEY.
  3. Find your project id: it starts with p_ and is in the project page's address.

Then add the template to .gitlab-ci.yml:

# .gitlab-ci.yml
include:
  - remote: 'https://app.testsy.dev/integrations/testsy-ai.gitlab-ci.yml'

testsy-ai:
  extends: .testsy-ai
  stage: test        # or a stage after your deploy
  variables:
    TESTSY_AI_PROJECT: p_your_project

The app's Connect page shows this with your project filled in.

The variables#

The template is configured with variables on the job:

VariableWhat it does
TESTSY_AI_PROJECTThe project whose tests run
TESTSY_AI_TESTSTest ids, instead of a whole project
TESTSY_AI_MODEauto (default), cold or warm
TESTSY_AI_TIMEOUTHow long to wait, 45m by default
TESTSY_AI_FAIL_ONVerdicts that fail the job, failed,error by default
TESTSY_AI_TARGET_URLStart website tests at another address, such as a review app
TESTSY_AI_BUILDApp builds to test: files, https links or ab_ ids

Review apps#

GitLab gives each merge request's review app its address in CI_ENVIRONMENT_URL. Pass it, and the same saved tests run against the review app instead of production:

TESTSY_AI_TARGET_URL: $CI_ENVIRONMENT_URL

The run explores from that address (a learned path belongs to the saved one), and the result shows the address it ran against. Sign-in accounts belong to the test, so a review app that needs different credentials needs its own test.

What the job shows#

Each run records the pipeline, commit, branch and job, and links back to the job. The JUnit file lists each test as passed or failed with a link to its proof, so GitLab's test report shows them in the merge request. Each started run prints a watch link that anyone can open without an account, while it runs and afterwards; paste it in the merge request for reviewers.

When it fails#

The job fails on the verdicts in TESTSY_AI_FAIL_ON. A run that could not start fails it too (exit code 2), so a missing key or an exhausted plan does not pass silently. Open the result from the job log and read the failure card: the check, expected and seen, and the step where it failed.

App builds in the same pipeline#

For iPhone and Android apps, TESTSY_AI_BUILD takes the build your pipeline just made, uploads it unless that exact file is already there, and runs the app's tests on it. The template's lookup job can skip the build entirely when the native fingerprint has not changed: see app builds from CI.

This post is part of Release testing for product teams.

Filed underWebCI, schedules and agents