Run and read results
Verdicts and what counts
What each result means, which ones count toward your plan, and which fail a CI build.
The verdicts#
| Verdict | What happened | Counted |
|---|---|---|
| Passed | Every check passed. | Yes |
| Failed | A check failed. | Yes |
| Blocked | The site stopped the tester, for example with a bot wall, so some checks could not be done. | No |
| Inconclusive | The tester could not tell for sure. | Yes |
| Did not finish | The run stopped because of a problem on our side. Run it again. | No |
| Stopped | You stopped it before it finished. | No |
While a run is going it is Waiting (for a free slot or a free device), Running or Stopping.
Blocked runs#
A blocked run usually means the site refuses traffic from cloud browsers. Allow our static addresses in your firewall or bot protection: see Network and allow-listing. The home page offers Allow our browsers next to blocked runs.
In CI#
By default a CI build fails when a run fails or does not finish:
| Value | The run | Fails the build by default |
|---|---|---|
failed | A check failed. | Yes |
error | Did not finish because of a problem on our side. Not counted. | Yes |
blocked | The site stopped the tester, for example a bot wall. Not counted. | No |
inconclusive | The tester could not tell for sure. | No |
Change it with --fail-on (command line) or QA_AI_FAIL_ON (GitLab). See Run tests in CI.
Flaky tests#
A project's Flaky filter shows tests whose last run passed but that failed, were blocked or were inconclusive in one of their last 14 runs. Each test row shows a strip of its last 14 runs.