# One mobile test, six OS versions: what changed on screen

> We ran the same plain-language tests on iOS 17, 18 and 27 and on Android 10, 12 and 16. The prompts differed on every version. The test text never changed.

By Jason Nguyen, Founder, Testsy. Published 2026-11-20. https://testsy.dev/blog/one-mobile-test-on-every-os-version/

## The phone on your desk is one phone

A team tests on the phones it owns. Usually that is this year's iPhone, on the latest iOS, and whatever Android phone someone had in a drawer. Its users are on everything else.

The differences between OS versions are rarely in your own screens. They are in the parts the phone draws around them: permission prompts are reworded, a simple alert becomes a full sheet, a permission that did not exist is suddenly required. Your app's code runs the same line on every version and gets a different experience on each.

Test scripts take this badly. A script that taps a button labelled "OK" breaks on the version where the button says "Allow". So teams either maintain a script per version or stop running the old versions.

We ran two of the tests from this series again on older versions, without changing a word.

## The same permission test on iOS 17, 18 and 27

The test is the one from [Test what your app does when the user says no to a permission](https://testsy.dev/blog/test-permission-prompts-in-a-mobile-app/):

```text
Open Permissions. Request Location and allow it while using the app,
then request Camera and deny it, then request Microphone and deny it,
then request Contacts and allow it.
Verify Location: granted, Camera: denied, Microphone: denied
and Contacts: granted.
```

The picture at the top of this page is the moment the app asks for contacts, on three versions of iOS:

- **iOS 17**: a small alert, "QA Lab Would Like to Access Your Contacts", with **Don't Allow** and **Allow**.
- **iOS 18**: a full sheet asking "How do you want to share contacts?", with **Select Contacts** and **Share All 6 Contacts**.
- **iOS 27**: the same question, restyled.

"Allow contacts" meant tapping **Allow** on one and **Share All 6 Contacts** on the other two. The test says "request Contacts and allow it". On each version the tester read what the phone offered and picked the choice that means allow.

The camera prompt changed too, in its wording and its look:

![The camera prompt on iOS 17 ("QA Lab Would Like to Access the Camera") and on iOS 27 ("Allow QA Lab to access your camera?"). The test said "request Camera and deny it" both times.](https://testsy.dev/blog/img/versions-camera-frames.webp)

All of these runs passed, with the same four lines on the app's screen at the end.

| Runs on | Result | Time | The run |
|---|---|---|---|
| iPhone, iOS 27 | Passed | 2 min 54 s | [open](https://app.testsy.dev/share/89_g1Bw9YckRIBj4-yXzgq2G2leFeH1I) |
| iPhone, iOS 18 | Passed | 1 min 52 s | [open](https://app.testsy.dev/share/0vgWpC9POsvAaFdv_vYjVDRaMmu8cC0W) |
| iPhone, iOS 17 | Passed | 1 min 39 s | [open](https://app.testsy.dev/share/Ulhhth7_ExY9-dh8xip8pSD6iCqgWBqr) |
| iPad, iOS 27 | Passed | 2 min 31 s | [open](https://app.testsy.dev/share/JdZzAbxuy6w5aH1Y-15GLivndErfGu-J) |

The fourth run is the same test on an iPad. The prompts sit in the middle of a much larger screen, and the test text is still the same.

![The same test on an iPad with iOS 27: the camera prompt, and the four statuses at the end.](https://testsy.dev/blog/img/versions-ipad-frames.webp)

![The iOS 17 result. The header says what it ran on.](https://testsy.dev/blog/img/versions-ios17-result.webp)

## The same notification test on Android 16, 12 and 10

The test is the one from [Test that a notification opens the right screen](https://testsy.dev/blog/test-that-a-notification-opens-the-right-screen/):

```text
Open Notifications and tap Notify in 5 seconds,
allowing notifications if the phone asks.
When the Order shipped notification arrives, open it and verify
the app shows Opened from notification: Order shipped.
```

Here the difference is not how a prompt looks. It is whether there is one. Android has required apps to ask before sending notifications [since version 13](https://developer.android.com/develop/ui/views/notifications/notification-permission). Before that, there was nothing to ask.

![Left: Android 16 asks "Allow QA Lab to send you notifications?". Right: an Android 10 tablet, where no prompt appeared and the notification opened the app all the same.](https://testsy.dev/blog/img/versions-android-frames.webp)

The test covers both with five words: "if the phone asks". On Android 16 the phone asked and the tester tapped **Allow**. On Android 12 and on the Android 10 tablet it did not ask, and the check passed with the reason written out: no prompt appeared, and none is required on that version.

| Runs on | Prompt | Result | Time | The run |
|---|---|---|---|---|
| Android phone, Android 16 | Asked, allowed | Passed | 1 min 16 s | [open](https://app.testsy.dev/share/9Ucbmwo3xCzQLI5AlPpcN5Pg3mk4YsEa) |
| Android phone, Android 12 | None | Passed | 1 min 13 s | [open](https://app.testsy.dev/share/UqV28JyE8O9HYYAGwOxJS-FVd8QKK9XZ) |
| Android tablet, Android 10 | None | Passed | 1 min 33 s | [open](https://app.testsy.dev/share/PZCiNFWYvoXjeFeHYZLr3W5J_sdSrKTs) |

![The Android 10 tablet result. The tablet ran the test on its side, which is how Android tablets start.](https://testsy.dev/blog/img/versions-android10-tablet-result.webp)

## What we changed between runs

Nothing in the test. Each extra run was started with a different device or OS version and the same saved test. In the dashboard that is the **Runs on** line under the app. From CI it is one flag:

```text
testsy-ai run --project <p_id> --os-version "iOS 17"
testsy-ai run --project <p_id> --device tablet
```

The versions on offer are iOS 27, 26, 18, 17 and Android 16 to 10, each on a phone or a tablet.

One thing you will notice if you open the runs: the number of checks differs, 10 on iOS 27, 6 on iOS 18 and 13 on iOS 17. We saved these tests as plain text, so each run split the text into checks its own way. All of them verified the same four statuses. If you want the identical checklist on every run, review the plan once when you create the test and save it. The tester then follows that checklist as written.

## Which versions to run

You do not need every test on every version. A workable split:

- **Every build**: your main journeys on the latest iOS and the latest Android.
- **Every release**: the same journeys on the oldest version you support on each side, and on a tablet.
- **When something touches the phone**: any test with a permission, a notification or a share sheet, on one old and one new version. Those are the screens that differ.

## Run it

Upload a build, choose **iPhone app** or **Android app** in **New test**, and change **Runs on**. [Android and iOS apps](https://testsy.dev/docs/apps/#phone-or-tablet-and-the-os-version) lists the devices and versions, and [CI](https://testsy.dev/docs/ci/) shows the flags.

This post is one of a series: [The hardest parts of mobile app testing, each one run for real](https://testsy.dev/blog/the-hardest-parts-of-mobile-app-testing/).
