BLOG
BlogGuides

Run tests in production without polluting your data

Tests against production leave accounts, orders and sign-ups in your numbers. Mark test data so it is easy to filter and clean, and keep the runs that only read separate.

The cost of testing the real thing#

Staging is not production: different mail settings, different keys, different data. So teams run a few tests against production, nightly or after each deploy. Then the side effects arrive:

  • Sign-ups and orders that inflate the dashboard the founders read every morning.
  • Welcome emails, onboarding sequences and sales alerts sent for test accounts.
  • Records that leak into public places. One team found that 1,071 of 1,190 cards waiting in their moderation queue were left by automated tests, twice.

None of this is a reason to stop testing production. It is a reason to plan for the data.

1. Make test data easy to recognise#

Pick one marker and use it everywhere:

  • Email addresses. A run's own mailbox always has an address at inbox-test-engine.testsy.dev, so every account a test signs up with can be filtered by that domain. Adding +something before the @ reaches the same mailbox when a test needs more than one address.
  • Names. Put a prefix in what the test creates: "TEST Blue mug order", "TEST First project".
  • Stored accounts. Keep a few named test accounts on your own domain, for example qa-customer@yourcompany.com, for tests that need an existing user.

Then filter them out where people read numbers: analytics, the admin dashboard, the CRM. Stripe's design is the model that write-up points to: every Stripe object carries a livemode flag, so anything downstream can tell live from test.

2. Split the tests that write from the tests that read#

Most of what you want to know about production can be checked without creating anything:

  • Signing in with a stored account and seeing the dashboard.
  • Search, product pages, pricing, the help centre.
  • An existing order's page and its PDF.

Run those nightly (how). Keep the ones that create (sign-up, checkout, invitations) fewer, and run them on deploys rather than every hour.

3. Clean up after the run#

  • In the test itself. End the test the way a user would tidy up: "Delete the project TEST First project and check it is gone from the list." The delete is then tested too.
  • With an endpoint. A run can call into your own system through an endpoint you define, for example cleanup_test_user, with your token kept in an encrypted header the tester never sees. Endpoint calls are on paid plans.
  • On a schedule of your own. A nightly job that deletes accounts on the test domain older than a day.

4. Do not test money with real money#

Payments in production need real cards and real refunds. Test checkout on a staging site with your provider's test mode (Stripe's test cards), and in production only check that the checkout page loads and shows the right price.

5. Keep test traffic out of analytics#

Runs block ads and trackers by default (test options), which keeps most test visits out of tracker-based analytics. Look at your own analytics after the first runs to confirm it for your setup. Switch trackers on only for a test that checks the tracking itself, such as a cookie banner.

This post is part of Release testing for product teams.

Filed underWebReleases and regressionsCI, schedules and agents