BLOG
BlogGuides

Run browser tests from any CI with one file

CircleCI, Bitbucket, Jenkins, Buildkite or Azure Pipelines: one dependency-free script runs your saved tests, writes JUnit and exits non-zero on a failure.

No plugin needed#

Not every team runs GitHub Actions or GitLab. Testsy's command-line tool is one JavaScript file with no dependencies, so it runs anywhere with Node 18 or later:

curl -fsSL https://app.testsy.dev/cli/testsy-ai.mjs -o testsy-ai.mjs
export TESTSY_AI_API_KEY=qa_…   # from a secret, never in the file
node testsy-ai.mjs run --project p_your_project --junit testsy-ai-junit.xml

It recognises GitHub Actions, GitLab, CircleCI, Bitbucket, Buildkite, Azure Pipelines and Jenkins, and sends the commit, branch, job and job link with each run, so a result points back to the build that started it.

What run does#

It starts every test in the project (or the ones you name with --test), then checks every 15 seconds until each has a verdict, printing each result as it lands. Each start sends an idempotency key and requests are retried on network errors, server errors and rate limits, so a retry never starts a run twice.

The useful flags:

Flag
--url <address>Website tests start at this address: a preview, review app or staging
--junit <file>One test case per run, with the link to its proof
--fail-onVerdicts that make it exit 1; failed,error by default
--timeoutHow long to wait; 45 minutes by default
--build <file or link>App tests run this build
--no-waitStart the runs, print their ids and exit

Exit codes your CI understands#

CodeMeaning
0Every run passed, or none ended in a --fail-on verdict
1A run ended in a --fail-on verdict
2A setup problem: missing key, unknown project, a test that could not start
3Timed out; the JUnit file is still written

A missing result never makes the suite pass: a run that has no verdict is reported as an error.

Where to put it#

After the deploy step, so the tests see the new version. Pass the deployed address with --url if it differs from the saved one. Keep the JUnit file as an artifact, and publish it with your CI's test report feature so failures show in the build page.

Reading a failure#

The tool prints each run's verdict and a link. Runs started this way also come with a watch link that opens the run live and its result afterwards, without a Testsy account, for whoever is on call. The result shows the check that failed, what was expected and what was seen, with a replay.

This post is part of Release testing for product teams.

Filed underWebCI, schedules and agents