# Five AI testers, one planted bug: what each verdict rested on

> We gave the same tests to five AI testing tools and to our own, one with a bug planted in a download. One tool passed it and one excused it. Here is what we saw.

By Jason Nguyen, Founder, Testsy. Published 2026-11-20. https://testsy.dev/blog/five-ai-testers-one-planted-bug/

## Read this first

Testsy publishes this post, and Testsy is one of the tools in it. Treat it accordingly.

We ran these tests ourselves in October and November 2026, on free trials of the hosted tools, so coverage is uneven: one tool ran 14 of the cases, another only 5, because that is where its free runs ended. Every tool has shipped changes since, ours included. This is a record of what we saw on those days, with the method, so you can repeat the part that matters on whatever you use today.

We are not naming the hosted tools. The point is the patterns, and you can check for each of them in any tool in a few minutes. The one tool we do name is [e2e](https://e2e.com), because it is open source and we ran it on our own machine with our own model key.

## The method

- **The same words everywhere.** Tests written as plain-English requirements: 8 on public websites (an encyclopedia, a code host, two government services, three shops and a travel site), 5 on our own practice site, and a few in iPhone and Android apps. Each requirement was pasted as written, with no hints for any tool.
- **A known answer.** One practice-site test has a bug planted in it and must come back failed. The rest should pass wherever the site lets the tester in.
- **One cold run.** No retries, no tuning between runs, except where we say so.
- **Scored on the verdict and on what it rested on:** was the verdict right, and did the result show the values the test asked for.
- **One exclusion.** A tool whose terms forbid benchmarking was left out.

That gave 38 scored runs across five hosted tools, three runs of the 13 web tests on e2e, and our own runs of the same tests.

## The planted bug

The test is two sentences:

```text
On the Customer exports page, click Export report (beta).
Verify that the downloaded report is a valid CSV file containing customer data.
```

The page behaves the way a real bug does. The button works. The page says "Export started: report.csv". A file named `report.csv` arrives. The file contains an HTML page titled "500 Internal Server Error".

![The page after the click. Every visible sign says the export worked.](https://testsy.dev/blog/img/planted-bug-page.webp)

![The downloaded file, opened: named report.csv, detected as HTML, titled "500 Internal Server Error".](https://testsy.dev/blog/img/planted-bug-file.webp)

The only way to get this right is to open the file.

| Tester | Verdict | What it rested on |
| --- | --- | --- |
| Testsy | Failed (correct) | Read the file: named report.csv, content is an HTML error page with no customer data. [The run is public](https://app.testsy.dev/examples/csv-export). |
| Tool A, hosted, plain-English steps | Failed (correct) | Right verdict. It had no way to read a file: in the same run it also passed the page's other export without opening it. |
| Tool B, hosted agent with a code sandbox | Failed (correct) | Right verdict; its stated reason was that no file had been captured. On the working export it did read the CSV and compared a value in it with the page. |
| Tool C, hosted, plans tests from a crawl | Blocked (wrong) | Could not get the download and reported the test as blocked by the environment. The bug was excused. |
| Tool D, hosted, turns a test into code | Passed (wrong) | Replaced "is a valid CSV" with a check that the status text appears, then saved that as the test. The saved test replays green. |
| e2e, open source, run by us | Failed (correct), three runs of three | Could not open the file and failed for that reason. The right verdict, but it could not verify the working export either. |

![Our result for this test: expected a valid CSV with customer data, saw an HTML error page, judged at the step where the file was opened.](https://testsy.dev/docs/img/run-failed-light.webp)

Two tools got the verdict wrong, in opposite directions. The other three got it right without having seen what was in the file. That second group is the one to worry about: on the day the export is fixed, a tester that never opens the file has no way to notice.

Tool D shows why this matters most where tests are saved. The weaker check was not a one-off misreading. It was written into the saved test, so every later run would have been green too.

## Five ways a pass can be worth less than it looks

Across all the runs, the wrong or weak results fell into a few shapes.

**1. The check is quietly replaced by an easier one.** Besides the export, Tool D changed "verify it is paused" to a check for the text "Status: Ended", and on the code host it searched for a different issue number from the one it had recorded, then reported that step as passed.

**2. The values you asked for are not in the result.** Several tests say "record the price", "record the author". Tool C passed the encyclopedia test with none of the five values it was asked to record, and Tool A passed a long travel-site journey without the property name or the cancellation terms. A pass with no values cannot be checked by a person.

**3. A wrong value sits inside a passed check.** On a government mortgage calculator, Tool C passed with the 25-year total wrong: it had reported the 30-year figure.

**4. A weaker check turns the same page into a pass.** On one shop, the delivery panel showed "Sorry, we can't load this info". Tool B waited for the panel's content and failed the test. Tool C checked only the panel's header and passed all three of its tests.

**5. A last look overrules a correct run.** In an Android calendar app, a fifth tool (hosted, phones only) completed the task and then failed it, because its final check looked only at the last screen. Adding "finally go back to the month view" to the test made the same run pass.

One more, which is not about passes: **time caps hidden by loops.** Tool D spent three of its ten sessions in a loop until its ten-minute limit, scrolling, toggling play and pause, or re-sorting a list, and ended with no verdict.

## Where the others beat us

A comparison where the author wins everything is not worth reading. These are the places we lost.

- **Speed.** e2e finished a web test in about 30 to 50 seconds. Ours took about 3 minutes at the time. We have been working on speed since, without dropping the checks described above, but that gap was real.
- **Getting in the door.** A travel site refused our browser, and a furniture shop did on some days. Tool A completed the travel journey in under 11 minutes, and Tool B passed the furniture shop with every value recorded. We have since changed how our browsers reach sites; on those days we were blocked, and the result said so.
- **Evidence, in Tool B's case.** It was the only other tool that reliably recorded values, read downloaded files, made its own test files in code and asked before re-running. Its reports were the best of the hosted tools.
- **Phones ready in seconds.** The phone-only tool had a device and the app ready in four or five seconds, with a live view you could tap during the run.

## How e2e scored, and why that is unfair to it

Pasted the whole requirement as one goal with one closing check, e2e got 7, 6 and 7 of the 13 web tests right in three runs. Scored the same way, our own runs were right on 11 or 12 of the 13, depending on the day.

That is not how e2e is meant to be used. It is a framework for developers, who would write several small goals with a precise check after each. Written that way, its replay worked and a repeat run took 14 seconds. Its misses on our set were mostly things a developer would add: it had no way to read a downloaded file and no test files to upload. If your tests live in your repository and a developer maintains them, those numbers undersell it.

## Check your own tool in ten minutes

You do not need our test set. Three probes cover most of what we found.

1. **Plant a file that lies.** Our practice page is public: [the Customer exports page](https://api.testsy.dev/lab/exports.html). Paste the two-sentence test above into your tool. Failed is the right answer, and the reason should mention what is inside the file.
2. **Ask it to record values.** Add "record the price and the product code" to a test you already have. Then look for those two values in the result. If the test passes and they are not there, the pass rests on something else.
3. **Ask for a check it cannot make.** "Verify the confirmation email arrived", in a tool with no mailbox. The honest answers are "could not verify" or a failure. A pass means the check was swapped for a sentence on the screen.

Run them on Testsy too. [Both of our example runs](https://testsy.dev/examples/) are open without an account, and the rules our tester follows are in [Making an AI tester prove its verdicts](https://testsy.dev/blog/making-an-ai-tester-prove-its-verdicts/).

## Limits of this comparison

- Free trials, so few runs per tool and no repeats for most. A single run can be unlucky.
- Hosted tools reach sites from different networks. Some results say more about a network than about a tester.
- We wrote the tests and the practice site, and we built our tester against failures like these. A set written by someone else would find our weak spots faster.
- The hosted tools were run in October 2026 and e2e in early November 2026. Check current versions before deciding anything.
