# What an AI tester cannot check for you

> A browser test proves what a user can see and do. It does not audit security, load or code. An honest list of what Testsy covers, and what still needs a person.

By Jason Nguyen, Founder, Testsy. Published 2026-11-21. https://testsy.dev/blog/what-an-ai-tester-cannot-check-for-you/

## Why this post exists

It is tempting to treat any test that passes as "the app is fine". A browser test checks one thing well: what a person sees and can do in your app. Knowing where that stops is how you avoid trusting a green result for something it never looked at.

## What Testsy does check

- **Journeys a user can take:** signing up, signing in, creating and editing records, uploads and downloads, checkout in a test mode, settings.
- **What the screen shows:** each check is judged on visible values, kept with a crop of where it was read.
- **Messages a run receives:** the confirmation email, the reset link, a code, on [its own mailbox or phone number](https://testsy.dev/docs/mailbox-phone-endpoints/).
- **Errors on the way:** script errors and failed requests are recorded with the run; app crashes come with the stack trace.
- **What a second person sees,** in a separate browser, when the test names two people.

## What it does not check

**Security.** It does not audit your database rules, find exposed keys, or try requests outside your app's screens. These are the most serious failures in apps built with AI: one write-up cites a scan where [170 of 1,645 Lovable apps exposed user data](https://www.indianic.com/resources/vibe-coded-app-broke-in-production) through missing row-level security, and another reports that [45% of AI-generated code fails basic security tests](https://www.upsilonit.com/blog/vibe-coding-limitations). A browser test can show that user B does not *see* user A's data on the page; it cannot show the API refuses to send it. Get a security review.

**Load and speed.** One run is one user. It will not tell you what happens with a thousand at once.

**Your code.** The tester does not read or review it. It only sees the result.

**Pixel-perfect design.** Checks are about values and states ("the total reads $36.00"), not a visual diff against a mockup.

**Accessibility compliance.** The tester reads the page, but a pass is not an accessibility audit.

**Sites it cannot reach.** An app only on your laptop or a private network cannot be tested yet. A site behind a bot wall may stop the run; that run is **Blocked** and not counted against your plan. [Network and allow-listing](https://testsy.dev/docs/network/) explains how to let it in.

## When it is not sure, it says so

A test tool that always answers pass or fail will sometimes guess. Testsy has four honest outcomes: **Passed**, **Failed**, **Blocked** (the site stopped the tester) and **Inconclusive** (it could not tell for sure, with the reason). An inconclusive result is a prompt to make the test clearer, not a pass. See [verdicts](https://testsy.dev/docs/verdicts/).

![The Runs page: every run with its verdict, filtered by verdict, type and where it was started.](https://testsy.dev/docs/img/runs-light.webp)

## How to use it well

Use Testsy for what a user would notice, after every change and on a schedule. Pair it with:

- a security review of your database rules and keys before launch and after changes to them,
- your host's monitoring for uptime and speed,
- a person, now and then, who uses the app with fresh eyes.

This post is part of [Testing an app you built with AI](https://testsy.dev/blog/test-an-app-you-built-with-ai/).
