BLOG
BlogGuides

Test both sides of a feature flag before you flip it

A flag doubles the app you ship: on and off. Bugs hide in the side nobody tested. Two stored accounts, one in the rollout and one out, run the same journey each.

The side nobody tested#

Feature flags let you ship code switched off and turn it on later, for some users first. They also mean two versions of the app run in production at once, and most test suites only check one of them.

The incidents people write up are about the other one:

  • A Circle engineering post-mortem on a real customer-facing incident: the flag did not prevent the problem because of wrong assumptions about the flag-off case, and manual testing did not cover the flag turned off. Their first lesson: treat off as the default case.
  • A tester on the Ministry of Testing describes the suite validating happy paths in a single flag state, while staging and production had flags set differently.
  • A team that deleted a flag that had been off for a year turned a feature back on for the group it once targeted, where it collided with another feature.

Two accounts, one journey#

The simplest way to cover both sides in the browser: two stored sign-in accounts, one inside the rollout and one outside it. Most flag tools can target a user or an email address, so put qa-flag-on@yourcompany.com in the rollout and leave qa-flag-off@yourcompany.com out.

Then write the journey once per account. The checks differ only where the flag changes what a user sees.

Sign in as the account "flag-on".
Open Billing and check the new "Usage" tab is shown.
Open the Usage tab and check this month's runs are listed.
Create an invoice and check it appears in the list.
Sign in as the account "flag-off".
Open Billing and check there is no "Usage" tab.
Create an invoice and check it appears in the list.

The last line of each test is the same on purpose: the shared part of the journey must work in both versions, and that is where the "off" bugs hide.

When to run them#

  • Before you widen the rollout. Run both; flipping a flag is a release, even with no deploy.
  • After every deploy while the flag exists (with GitHub Actions): new code lands in both versions.
  • When you remove the flag. Delete the "off" test only after the flag's code is gone, and run the "on" test as the new default.

If the flag lives in the address#

Some teams switch a flag with a query parameter or a preview deploy. Then one test can run against two addresses: run_test (or the CLI) accepts a start address for a single run, which is how preview deploys are tested, and the saved test keeps its own.

What this does not cover#

Two accounts test two states. Three flags make eight, and flags that interact need a decision about which combinations matter; test those, and keep flags few and short-lived. The rest is unit tests in your code with each flag value forced, which a browser test does not replace.

This post is part of Release testing for product teams.

Filed underWebSign-in and codesReleases and regressionsCI, schedules and agents