Let Codex check its own work in a real browser
Add Testsy to Codex with one command. The agent drafts a plain-language test, runs it against your app, reads every check and reruns after its fix.

The gap Codex cannot close alone#
Codex is good at changing code and running the commands you give it. What it cannot do on its own is look at your deployed app the way a customer does: open the page, click, wait, read what is on the screen. So "the tests pass" is the strongest thing it can tell you, and the tests are often ones it wrote.
Anthropic, writing about its own agents, describes the same pattern: the agent marked features complete without proper testing until it was told to test in a browser as a user would. The fix is to give the agent a tester that does exactly that.
Add it with one command#
Create an API key in Testsy (Settings → API keys), store it as TESTSY_AI_API_KEY, then:
codex mcp add testsy --url https://app.testsy.dev/mcp --bearer-token-env-var TESTSY_AI_API_KEYThe skills (testsy-check-release, testsy-investigate-run, testsy-regression-from-bug) are in testsy-ai/testsy-integrations under skills/. They tell the agent to keep the original expectation, read the evidence, and tell a real failure from a test that could not finish.
Codex asks before it starts a run or changes a test. For unattended use with codex exec and approvals off, the MCP docs give the setting that approves Testsy's tools.
A session, step by step#
"Check that the invoice page shows the right total after adding two items" becomes:
plan_testdrafts the steps and checks from that sentence. Codex shows you the plan; you answer any question it asks, such as which sign-in account to use.create_testsaves it;run_teststarts it and returns a watch link.wait_runreturns the moment the run ends, with each check, its outcome, the reason and the value it was decided on ("Total: $42.00").- A failed check comes with what was expected and what was seen. Codex reads it, fixes the code, and calls
run_teston the same test again.

In CI as well#
The same saved tests run from your pipeline after each deploy, with no agent involved: the command-line tool is one file with no dependencies, writes JUnit, and exits 1 when a check fails. So the test Codex wrote on Tuesday guards every deploy after it.
Before you let it loose#
- Point runs at staging and use made-up details: a run can create accounts and change records.
- Use the narrowest key that works. "Run tests and read results" is enough for an agent that only runs saved tests.
- Every run counts toward your plan. The free trial has 10.
This post is part of Testing an app you built with AI.
Filed underWebAI-built appsCI, schedules and agents