Test a SaaS release from Claude Code

Use Claude Code and Testsy to turn a SaaS release checklist into a saved browser test, inspect the evidence, and rerun after a fix.

Start with one release risk#

Your coding agent has finished a change. Check the behavior a customer needs next: signing in, creating a task, publishing a document, or exporting a report. Choose one complete journey with a clear expected result.

This guide uses a task app: a signed-in member creates a task, reloads the page, and sees the saved task. Use a staging workspace and synthetic data that you are authorized to change. The start address must be reachable from Testsy's hosted browser; an address that works only on your laptop is not enough.

Open an editable release smoke test. Replace the example details with your app's address, a saved test account and the action that matters to your release. You can also inspect the public example reports first.

Connect Claude Code#

Create a Testsy API key in Settings → API keys with permission to create and edit tests. Store it in your local environment as QA_AI_API_KEY; keep it out of source control and shared transcripts. A run-only key is sufficient when you only run existing tests.

claude mcp add --transport http qa-ai https://app.testsy.dev/mcp --header "Authorization: Bearer $QA_AI_API_KEY"

Restart or reconnect the client if it does not discover the tools. The server uses the same workspace and permissions as the API key. See MCP setup and the tool reference for connection details.

Ask for a reviewed test#

Give the agent a small, explicit job. Replace the bracketed parts before using this prompt:

Use Testsy to check the deployed task app at [staging URL].
Use project [project name or ID] and the saved test account [credential name].

The member should be able to create a task named "Release smoke sample",
see it in the task list, reload, and still see that task with its saved title.
Do not change other tasks or invite real people.

Draft the steps and checks with plan_test and show me the plan before saving.
Keep the expected behavior separate from what the browser later observes.

The agent should select the correct project, draft the plan, and ask about missing details. Add the account through the app's Sign-in accounts settings rather than pasting its password into the conversation.

Check that the plan tests your requirement. A plan that cannot determine whether the task persisted needs a clearer check before it runs.

Run it and inspect the result#

Once the plan is approved, ask the agent to save and run it. It can use create_test, run_test, and get_run. Wait for a final result; a queued job or a plan is not a completed check.

Read the verdict, each check's reason, the observed values and the replay. A task that appears immediately but disappears after reload should fail the persistence check. Keep that failure visible.

• Passed: the completed test met its checks. It covers this journey and environment, not every feature in your app.
• Failed: a check did not meet the expected behavior. Inspect the relevant screenshot and step before deciding on a fix.
• Blocked: the tester could not complete an action because of an access barrier or a tester limitation. Resolve that obstacle before relying on the test.
• Inconclusive: the evidence did not establish a trustworthy answer. Clarify the check or investigate the missing evidence.
• Error: Testsy or its infrastructure did not finish the run. Keep it separate from an application failure.

Fix, deploy, then rerun#

Use the run's Copy fix prompt to bring the failed check and supporting evidence into your coding session. Review the proposed code change, deploy it to the environment the test uses, and rerun the same saved test.

Ask the agent to summarize the original failure and the new result with links to both. Do not weaken the original expected behavior to obtain a pass. A passing rerun is evidence about the deployed fix; it does not authorize a merge or production release.

When the check becomes part of every release, connect it to GitHub Actions or a schedule.