# Making an AI tester prove its verdicts

> An agent that drives a browser can say "passed" about almost anything. These are the rules we built Testsy on so that every verdict comes with its evidence.

By Jason Nguyen, Founder, Testsy. Published 2026-11-20. https://testsy.dev/blog/making-an-ai-tester-prove-its-verdicts/

## The failure we designed against

An AI agent that can click through a website will tell you a test passed. The hard part is knowing what that sentence rests on.

Here is the case we keep coming back to. A page has an **Export report** button. The tester clicks it. The page says "Export started: report.csv" and a file named `report.csv` arrives. Every visible sign says the export works.

The file contains an HTML page titled "500 Internal Server Error".

![The page after the click: "Export started: report.csv". Nothing on screen says anything is wrong.](https://testsy.dev/blog/img/planted-bug-page.webp)

![What the tester found when it opened the file: named .csv, content is an HTML error page.](https://testsy.dev/blog/img/planted-bug-file.webp)

A tester that judges from the screen passes this test. So does a tester that checks the file's name. You can open [the real run](https://app.testsy.dev/examples/csv-export) of this case on our practice site: it failed, and the reason quotes what was inside the file. That run took 77 seconds.

A wrong pass is the worst result a testing tool can produce. A wrong fail costs you ten minutes. A wrong pass ships the bug with a green tick on it. So the rules below all push one way: when the evidence is missing, the answer is not "passed".

## Rule 1: the expectation is fixed before the run

A test is steps and checks. Before a test is saved, the plan is reviewed: a second pass confirms that every check is something a tester could judge by looking at the screen, and flags the ones that are not. "The database saved the task" is not a check a browser can decide. "After reloading, the task still reads Release smoke sample" is.

Once the run starts, the checks do not change. The tester cannot reword a check into one that is easier to satisfy, and a check that turns out to be impossible is reported that way.

## Rule 2: every check records the value it was decided on

A check is never just green or red. The result stores the value the tester read and the region of the screen it was read from, linked to the step where it happened.

In [the passing example](https://app.testsy.dev/examples/publishing), an editor publishes an article and the public site must show the same title. The check does not say "titles match". It says the dashboard showed "Green desks for everyone" with status published, the public page listed the same title, and the text read from the page agrees with what was rendered. If you disagree with the verdict, you can see what to argue with.

![A passed run: each check with the value it was decided on and the part of the screen it was read from.](https://testsy.dev/docs/img/run-checks-light.webp)

[Recording: The review video of that run: publish an article, then find the same title on the public site.](https://testsy.dev/blog/video/publishing-run.mp4)

This is also what makes a failure useful. "Expected Release smoke sample, saw Untitled, at step 5" is something you can hand to a developer or paste into a coding agent.

## Rule 3: the one that acts is not the one that judges

The part of the tester that clicks and types wants to finish the journey. If it also graded the result, it would grade itself kindly.

So judging is separate. The checks are decided on the recorded evidence: the screens, the values read, the files downloaded. What the judging concludes is not fed back to the acting part while it works, so the tester cannot steer toward whatever would make a check pass.

## Rule 4: files are read, not trusted by name

A download is evidence only if someone opens it. The tester reads the bytes: the CSV export above failed because the content was HTML, whatever the filename claimed. The same goes for a confirmation email. "We sent you an email" on the screen does not establish that one arrived, so a test with a [mailbox](https://testsy.dev/docs/mailbox-phone-endpoints/) waits for the real message and the result shows it.

## Rule 5: "could not tell" is a verdict

Most tools have two answers. Testsy has four that matter:

| Verdict | What it means |
| --- | --- |
| Passed | Every check passed, each on a recorded value. |
| Failed | A check did not meet what was expected. |
| Blocked | The site stopped the tester, for example with a bot wall. Not counted against your plan. |
| Inconclusive | The tester could not establish the answer from the evidence. |

![The Runs page: every run with its verdict, and what was seen for the ones that did not pass.](https://testsy.dev/docs/img/runs-light.webp)

Blocked and Inconclusive exist so that neither gets rounded up to Passed or down to Failed. A bot wall is not a bug in your checkout. A check the tester could not see is not a check that passed. The full list, including runs that stop on our side, is in [Verdicts and what counts](https://testsy.dev/docs/verdicts/).

In CI the default is that a failed run fails the build. Whether a blocked or inconclusive run should also stop a release is your call, and it is one flag. We would rather you make that choice than have us hide it.

## Rule 6: when the app breaks, say so

A button that throws an error and does nothing looks, on screen, like a button that is slow. The tester watches for errors from the site's own code and for an app that crashes or stops responding. When one breaks a step, the run fails and the result shows the error, the step before it and the stack trace. Errors from ads, analytics and chat widgets are ignored.

## What this costs

These rules are not free, and you should know the price.

- **Time.** A run usually takes 2 to 8 minutes the first time. Reading values, recording regions and judging afterwards is slower than a single pass that clicks and declares. After a first passing run, the tester replays the path it learned, so later passing runs are quicker.
- **More results that are not green.** An honest Inconclusive is annoying when you wanted a pass. It usually means the check needs to name something visible.
- **A pass is narrow.** It covers that journey, on that device, in that environment. It does not say the rest of the app works.

## What is still hard

- **Sites that are not public.** The tester reaches your site from the internet. A site on a private network or on your laptop cannot be tested yet.
- **Bot protection.** Some sites refuse cloud browsers. We report those runs as Blocked and publish the [addresses to allow](https://testsy.dev/docs/network/), but we cannot test a site that will not let the tester in.
- **Checks about things nobody can see.** If the only proof is in your database, the test needs a second place where the result becomes visible, or an [endpoint](https://testsy.dev/docs/mailbox-phone-endpoints/) the tester may call.

If you want to judge the result format before trusting any of this, [both example runs](https://testsy.dev/examples/) are open without an account: the checks, the values, the replay and the video.
