BLOG
BlogGuides

Flaky tests, and what a test verdict should be allowed to say

A test that passes and fails on the same code teaches people to ignore it. Why a verdict needs room to say blocked or inconclusive, and how to find flaky tests.

Pass, fail, and everything else#

Most test tools have two answers. Reality has more. The site's bot protection stopped the browser. The page never finished loading. The value on the screen could be read two ways. A two-answer tool has to squeeze those into pass or fail, and either choice is wrong: a false fail teaches people to rerun until green, and a false pass hides a bug.

Currents, which reports on Playwright suites, cites a survey of 335 developers and testers that found the main cost of unreliable tests is the gradual loss of confidence in their results. Once a team expects red to mean "probably nothing", a real failure gets the same treatment.

Four honest outcomes#

Testsy's verdicts, from the docs:

VerdictWhat happenedCounted
PassedEvery check passed.Yes
FailedA check failed.Yes
BlockedThe site stopped the tester, for example with a bot wall.No
InconclusiveThe tester could not tell for sure, and says why.Yes

A blocked run is not your bug and not our success, so it is not counted against your plan. An inconclusive run is the tester declining to guess. We published one: a two-person test whose steps were done out of order, where the verifier refused to pass it and said exactly why.

Find flaky tests early#

A project's Flaky filter shows tests whose last run passed but that failed, were blocked or were inconclusive in one of their last 14 runs. Each test row shows a strip of its last 14 runs, so a test that flips stands out before anyone learns to ignore it.

The Runs page: every run with its verdict, filtered by verdict, type and where it was started.
The Runs page: every run with its verdict, filtered by verdict, type and where it was started.

Common causes, and fixes#

  • The check is vague. "The dashboard loads correctly" can be judged differently each time. Name the value: "the Revenue card shows a number". See write a test without writing code.
  • The data changes. A test that expects "3 projects" fails when another test adds one. Create the data the test needs inside the test, with names unique to it.
  • The page is slow sometimes. Testsy waits for what the step needs rather than a fixed time; if a page sometimes takes longer than the run's limit, that is worth knowing about your app. Slow pages without sleeps shows how.
  • Bot protection blocks some runs and not others. Allow our static addresses.

In CI, decide what a gate needs#

By default a pipeline fails on failed runs and runs that did not finish. For a release gate that requires a conclusive answer, include blocked and inconclusive in --fail-on. Whatever you choose, do not drop a verdict from the gate to make a release green.

This post is part of Release testing for product teams.

Filed underWebReleases and regressionsVerdicts and proof