BLOG
BlogGuides

Fix one bug without breaking another: one bug, one test

AI-built apps fall into a loop where each fix breaks something that worked. Keep a saved test for every bug you fix and run them all after each change.

The loop#

You ask the agent to fix the checkout. It does. The next day sign-in is broken. You ask it to fix sign-in, and the checkout breaks again.

Founders who build with AI describe this more than any other problem. A rescue agency's write-up names four causes that compound: the context window fills up, the same prompt does not return the same code, the agent has no map of what depends on what, and it forgets constraints you gave it earlier. It cites a study in which 75.76% of coding tasks produced no two identical outputs across five runs. Another guide on why vibe-coded apps keep breaking describes a "fix" that passes the one test the agent can see and silently breaks three callers.

The fix that works is old and boring.

One bug, one test#

Every time a bug is fixed, write down how to show it is fixed, and keep it. A guide on regressions in AI-built apps suggests asking your agent or developer a single question: show me the automated test that fails if this exact bug ever comes back. If there is none, the bug can come back with nobody noticing.

For a non-technical builder the problem is that "an automated test" means code. With Testsy it means the sentence you would have said to a friend:

Add "Blue mug" to the cart, open the cart and go to checkout.
Check the order total reads "$18.00".

Save it under the bug's name. Now the bug has a test.

Run all of them after every change#

A test that is never run again protects nothing. A project's Run all starts every saved test; the project page groups them by their last result, so a test that used to pass and now fails stands out.

A project: tests grouped by their last result, with Run on each and Run all for the project.
A project: tests grouped by their last result, with Run on each and Run all for the project.

Make it a habit after each change the agent makes:

  1. The agent changes the code.
  2. You (or the agent, over MCP) press Run all.
  3. Anything that broke shows as failed, with what was expected and what was seen.

The Claude Code plugin includes a skill called testsy-regression-from-bug for exactly this: turn a bug you describe into a saved test, so the agent cannot quietly reintroduce it.

What a caught regression looks like#

Our public example is a CSV export that used to work. After a change, the file it downloaded was an HTML error page with a .csv name. The page said the export had started, so a quick look would pass it.

The saved test opened the file and failed on the check, showing what it expected and what it saw. Open the example result to see it.

Keep the list short and real#

  • Write each test about one thing a user would notice: a total, a name, a page that opens.
  • Name the visible value in the check. "Checkout works" cannot be judged; "the total reads $18.00" can. Write a test without writing code has more on wording.
  • Run them on a schedule as well, so a change outside the code (a plugin, a setting, a provider) is caught too.

Each test run counts as one run on your plan. Five tests after each change is five runs.

This post is part of Testing an app you built with AI.

Filed underWebAI-built appsReleases and regressions