Build a Mobile Test Strategy by Fidelity and Cost

A common way a mobile suite fails is quietly expensive. Forty Appium scenarios cover sign-in on a rented cloud phone. They take most of an hour, a timeout somewhere is routine, and the run that finally goes green used a debuggable build against a mocked backend. The biometric prompt was never shown, because the cloud phone had no enrolled face. The build that shipped was minified. So the suite spent the most expensive runtime available proving things a JVM could have proved in seconds, and still never touched the two risks that only a real device and a release binary carry.

The opposite mistake looks disciplined and is just as blind: one emulator image, one API level, every instrumented test pointed at it, and "compatibility" ticked on the release checklist.

Both come from choosing a runtime per test suite rather than per assertion. What follows is a way to place each check by the platform behaviour it has to see, using a sign-in journey as the running example.

Fidelity is a property of where the test runs

Android's testing guidance gives the word a precise meaning, and it is worth adopting: fidelity "refers to the similarity of the test runtime environment to the production environment." It is a property of the runtime, not of how many screens a test touches. A one-line test of a SQLite query run on a physical phone is high-fidelity. A thirty-step journey on Robolectric is not.

That definition splits every placement decision into two questions. What is the least faithful runtime in which the assertion can still fail for the right reason? And what does it cost to run there on every change?

Android lays out a five-layer model (unit, component, feature, application, release candidate) and a rule of thumb for choosing between them: "consider the lowest layer of the pyramid that can give the team the right level of feedback." Apple's overview makes the same trade from the other side, calling UI tests "the ultimate indicator your app works the way you expect" while warning that they take longer and that "various app variables" can fail the same test. Neither platform treats the pyramid as a quota. Android notes outright that camera or media apps "might depend on specific hardware features" that change its shape.

What each mobile runtime can see

The host JVM, or a Swift Testing target. Android's local tests "execute on your development machine or a server" and are "usually small and fast." They see your logic and nothing of the framework. On Apple platforms, Xcode 16 and later ship Swift Testing for unit tests, which can parallelise tests in-process.

Robolectric. A simulated Android inside the JVM that Google maintains and says most Google apps "rely heavily on." It runs Espresso and Compose tests unchanged when they sit in the test source set. Its limits are documented: it "doesn't have a screen like an emulator does," some APIs are partial, and system UI such as edge-to-edge and picture-in-picture, or unsupported features like WebView, still need a device test. Android's own sign-in example places the whole sign-in dialog, an application-level test, on Robolectric. Most teams would put that on an emulator by reflex.

An emulator or simulator running the real binary. This is where the app process, the manifest, navigation, launch arguments and real rendering come in. Apple is blunt about the ceiling: simulators "don't replicate the performance or features of a physical device," and for hardware-specific frameworks the instruction is to handle the unavailable case in code and, "to test the feature itself, run your code on a physical device."

A physical device. The only place performance numbers, sensors, camera access and vendor OS behaviour mean anything. Push is a good example of why: Apple documents that notification delivery "depends on a device's battery, network connectivity, and other factors," and that APNs "might deliver the notification immediately, defer delivery, make multiple attempts to deliver, or discard it." No simulator run gives you evidence about that.

One asymmetry deserves attention before any tooling decision. Espresso and Compose tests run inside the app's process. Espresso waits until the message queue is empty and every registered idling resource is idle, and Compose tests advance "using a virtual clock," so they "don't run in real time." Tools that drive from outside the process cannot see any of that. UI Automator, which Appium's Android driver sits on, is documented as testing "from outside of the app's process," and its element lookups wait up to a default 10,000 ms. That is a weakness for speed and flakiness and a strength for one purpose: the same page says it "lets you test release versions with minification applied."

One sign-in journey, four runtimes

The journey: email and password, an optional biometric sign-in on return visits, and a server that can answer with success, wrong credentials, a locked account, or a demand for a second factor. Here is where each piece of it belongs.

Token policy and validation on the host

Whether a stored session needs refreshing is arithmetic over two instants. It fails for exactly one reason, a wrong comparison, and it needs a controllable clock, not a device.

class SessionPolicy(
    private val clock: Clock,
    private val skew: Duration = Duration.ofSeconds(30),
) {
    fun needsRefresh(expiresAt: Instant): Boolean =
        !clock.instant().plus(skew).isBefore(expiresAt)
}

class SessionPolicyTest {
    private val now = Instant.parse("2026-10-12T09:00:00Z")
    private val policy = SessionPolicy(Clock.fixed(now, ZoneOffset.UTC))

    @Test
    fun refreshesWhenExpiryFallsInsideTheSkewWindow() {
        assertTrue(policy.needsRefresh(now.plusSeconds(29)))
    }

    @Test
    fun keepsATokenThatOutlivesTheSkewWindow() {
        assertFalse(policy.needsRefresh(now.plusSeconds(31)))
    }
}

The iOS side of the same rule set is where Swift Testing earns its place. Parameterised arguments turn a list of malformed addresses into separate, individually reported cases:

import Testing
@testable import SignIn

@Test("malformed addresses never enable submission",
      arguments: ["", "ada", "ada@", "@example.test", "ada@example"])
func rejectsMalformedAddress(_ input: String) {
    #expect(EmailRule.validate(input) == .invalid)
}

Apple's XCTest page sets the boundary for this framework precisely. A target can hold both Swift Testing and XCTest tests, but "don't mix API from the two frameworks in the same test," and XCTest remains the framework for UI tests and performance tests. So new unit tests go to Swift Testing, and the UI layer stays on XCTest with XCUIAutomation.

The screen against a fake auth service

This is the layer most often skipped, and skipping it is how validation ends up in Appium. Compare the two ways of checking that the server's "account locked" response produces the right message.

Driven through a device, this needs a built APK, an installed UiAutomator2 server, a session, and a backend that can be told to lock an account:

// Appium via WebdriverIO: minutes of setup to observe one string
it('shows the lockout message', async () => {
  await $('~email').setValue('[email protected]');
  await $('~password').setValue('correct horse');
  await $('~sign-in').click();
  await expect($('android=new UiSelector().textContains("15 minutes")')).toBeDisplayed();
});

The same behaviour as a Compose test, with the auth API and the biometric check replaced by fakes you control:

@RunWith(AndroidJUnit4::class)
class SignInScreenTest {
    @get:Rule val compose = createComposeRule()

    private val auth = FakeAuthApi()
    private val biometrics = FakeBiometrics(enrolled = false)

    @Test
    fun lockedAccountShowsRetryTimeAndHidesBiometricOffer() {
        auth.nextResult = AuthResult.Locked(retryAfter = Duration.ofMinutes(15))
        compose.setContent { SignInScreen(SignInViewModel(auth, biometrics)) }

        compose.onNodeWithTag("email").performTextInput("[email protected]")
        compose.onNodeWithTag("password").performTextInput("correct horse")
        compose.onNodeWithTag("sign-in").performClick()

        compose.onNodeWithText("Try again in 15 minutes").assertIsDisplayed()
        compose.onNodeWithTag("use-biometrics").assertDoesNotExist()
    }
}

Placed in src/test, this runs on Robolectric on every commit. Moved to src/androidTest, the same file runs on an emulator. The biometric fake is the point of interest: the fallback path when nothing is enrolled, the cancel path and the lockout path are all state in your view model, and all of them are testable here. What is not testable here is the system prompt itself, which belongs to the OS. The later article on permission and system dialogs in this series covers driving that UI.

The SwiftUI equivalent is a host-side test of the view model behind the screen with the same kind of fake injected. On either platform, the discipline is architectural before it is about testing. Android's fundamentals page names the cost of skipping it: classes that cannot be unit-tested "might have to be covered by bigger integration tests or UI tests."

The real binary on an emulator or simulator

Now the question changes from "does the screen behave" to "does the app, as built, get there". Navigation from launch, dependency wiring, configuration changes, a stub backend reached over a real socket. On Android, Gradle managed devices let the build own the emulator lifecycle. They need API level 27 or higher, restore a clean snapshot between runs, and can run a named group in parallel:

android {
    testOptions {
        managedDevices {
            localDevices {
                create("phoneApi30") {
                    device = "Pixel 2"
                    sdkVersion = 30
                    systemImageSource = "aosp"
                }
                create("tabletApi34") {
                    device = "Nexus 9"
                    sdkVersion = 34
                    systemImageSource = "aosp"
                }
            }
            groups {
                create("signInApp") {
                    targetDevices.add(devices["phoneApi30"])
                    targetDevices.add(devices["tabletApi34"])
                }
            }
        }
    }
}

# On CI runners without hardware rendering, the docs require the GPU flag.
./gradlew signInAppGroupDebugAndroidTest \
  -Pandroid.testoptions.manageddevices.emulator.gpu=swiftshader_indirect

Which device profiles belong in that group is a separate decision, and part four of this series is about making it from usage data rather than habit. For now, two images is already a different claim from one.

On iOS the equivalent is an XCUITest target pointed at a simulator, with the backend chosen at launch:

import XCTest

final class SignInAppTests: XCTestCase {
    func testLockedAccountShowsRetryTime() {
        let app = XCUIApplication()
        app.launchEnvironment["AUTH_BASE_URL"] = "http://127.0.0.1:8089"
        app.launchEnvironment["AUTH_SCENARIO"] = "locked-15m"
        app.launch()

        app.textFields["email"].tap()
        app.textFields["email"].typeText("[email protected]")
        app.secureTextFields["password"].tap()
        app.secureTextFields["password"].typeText("correct horse")
        app.buttons["sign-in"].tap()

        XCTAssertTrue(app.staticTexts["Try again in 15 minutes"]
            .waitForExistence(timeout: 5))
    }
}

One detail in Apple's launchEnvironment reference catches people who reuse an application proxy across steps: you can modify the variables after launch, but such changes "don't affect the current launch session," and only apply on the next launch. Set the scenario before launch(), or relaunch.

The release build on physical devices

Android defines a release candidate test by its binary: the same as an application test "except that the application binary is minified and optimized," run "as close to production as possible without exposing the app to public user accounts or public backends." This is where an out-of-process driver pays for itself: UI Automator's documentation ties testing from outside the process directly to testing minified release versions, natively on Android. Appium does it for both platforms from one client, at a price.

{
  "platformName": "Android",
  "appium:automationName": "UiAutomator2",
  "appium:app": "/builds/signin-release-minified.apk",
  "appium:udid": "R5CX10ABCDE",
  "appium:disableWindowAnimation": true,
  "appium:uiautomator2ServerLaunchTimeout": 60000
}

Appium is a WebDriver server that delegates each command to a platform driver. The UiAutomator2 driver proxies most commands to a server it installs on the device, and some to ADB. The XCUITest driver runs a second WebDriver implementation, WebDriverAgent, on the device, which the driver "builds and manages" for you, a process the Appium docs themselves describe as "a pain" that "involves the use of Xcode." Session start-up has its own budget: the UiAutomator2 server launch timeout defaults to 30,000 ms. What you buy is one suite in one language across both platforms, running against the binary you ship. Keep this tier to the few journeys that justify it: sign-in with a real staging account, biometric sign-in on a device with an enrolled credential, and whatever the store rejection or support tickets say breaks. The next article in the series takes apart the driver and plugin boundaries in detail.

Placement table for the sign-in journey

The trigger column follows Android's example infrastructure plan, which runs unit and component tests on every commit, feature tests pre-merge, application tests post-merge, and release candidate tests nightly and before a release.

Nightly runs need a named decision

Android illustrates the inefficient strategy with a top-heavy shape whose device tests run only nightly, so that a bug "might not" be caught "until the nightly or weekly end-to-end tests run," and whose cost is days rather than minutes. Moving tests to nightly, the same page adds, can shorten CI "but you could also prolong the feedback loop."

The failure that hides behind this is organisational. A nightly device suite that fails and blocks nothing is a report nobody reads. Android's guidance asks for a strategy document that "assigns responsibilities for various test categories." For mobile, make that concrete per row: which risk, which layer, which runtime, what it gates and who answers when it is red.

- risk: session silently expires mid-journey
  layer: unit
  runtime: host JVM + Swift Testing
  gates: merge
  owner: identity-client team

- risk: minified build breaks sign-in against staging
  layer: release-candidate
  runtime: physical devices, release build
  devices: see device-matrix.md
  gates: store submission
  owner: mobile release captain (rota)
  on_red: release blocked; triage within one working day

If a row cannot name what it gates, it is monitoring, and it should be scheduled and staffed as monitoring, not counted as a quality gate.

AI tooling is making the top of the stack cheaper to explore. Xcode 27's release notes add agents that "can now boot simulators, install and launch apps, synthesize touch events, and capture screenshots." That is useful for inventorying flows and finding the states a suite is missing. It does not change which runtime can observe which behaviour, and it should not choose your device portfolio; usage data and platform risk do that.

Move three sign-in checks down this week

List every test in your sign-in area with the runtime it uses today. For each one on a device or emulator, write down the platform behaviour it observes that a lower runtime could not. Where the answer is "none," move it: validation to the host, message mapping to a screen test with a fake auth API. Three moves is enough to start, and the runtime saved is the budget for what comes next.

Then add the test you are almost certainly missing: one sign-in on the minified release build, on a physical device, against staging. Record the build variant, device model and OS version in the run output, because "passed on release" without those three fields cannot be checked later.

If you need deterministic staging accounts and form inputs for that run, the test data generator produces repeatable values you can commit alongside the scenario.

Questions about mobile test placement

Is Robolectric trustworthy enough for UI behaviour tests?

For state and behaviour, Android's own guidance says yes: "the fidelity needed for it is more than enough," with the example of a Compose test checking that the UI changed after a click. Treat it as insufficient for pixel-accurate screenshots, system UI, and WebView. And do not reach for Robolectric shadows to unit-test logic; the documentation calls that a "last resort" for legacy code.

Should one Appium suite replace Espresso and XCUITest?

Only at the tier where one cross-platform suite against release binaries is the goal. Below that, in-process frameworks synchronise with the app and run faster. Appium also cannot support every WebDriver command on every platform; its docs give cookies as a command that makes no sense in native apps. Use it where out-of-process is the requirement, not the default.

Can a simulator run prove a performance regression is fixed?

No. Apple's run-destination guidance says simulators do not replicate device performance, and Device Hub's overview directs "investigating performance issues" to physical devices. A simulator trace can point at a suspect; the confirmation needs hardware. The performance article later in the series works through measuring a journey rather than a function.

Does every feature need a test at every layer?

No, and Android says so directly: "not all features have to be tested in every category." A settings toggle might need one screen test and nothing else. The layers are places a test can go, not boxes each feature must fill.

Primary references

• Android Developers — Testing strategies. The definition of fidelity as "the similarity of the test runtime environment to the production environment"; the five-layer model with its scope, network, execution, build-type and lifecycle table; release candidate tests run on a "minified and optimized" binary; the sign-in example that places the sign-in dialog on Robolectric; the example trigger plan (every commit, pre-merge, post-merge, nightly, pre-release); the warning that nightly cadence can "prolong the feedback loop"; and that "not all features have to be tested in every category."
• Android Developers — Fundamentals of testing Android apps. Local versus instrumented tests, that local tests run on "your development machine or a server," and that untestable classes "might have to be covered by bigger integration tests or UI tests."
• Android Developers — Robolectric strategies. That Robolectric has no screen, supports some APIs partially, is relied on heavily by most Google apps, has sufficient fidelity for UI behaviour tests, is a "last resort" for unit testing, and still leaves edge-to-edge, picture-in-picture and WebView to device tests.
• Android Developers — Scale your tests with build-managed devices. API level 27 minimum, snapshot restore to a clean state, device groups run in parallel, the task naming pattern, and the swiftshader_indirect GPU flag for servers without hardware rendering.
• Android Developers — Espresso. The synchronisation conditions Espresso waits for before each action: an empty message queue, no running AsyncTask, and idle idling resources.
• Android Developers — Synchronize your Compose tests. That Compose tests synchronise with the UI by default and advance "using a virtual clock," so they do not run in real time.
• Android Developers — Write automated tests with UI Automator. That UI Automator tests "from outside of the app's process," which lets it test minified release builds; the modern API introduced in 2.4; and the default 10,000 ms element wait.
• AndroidX Test release notes. Current stable versions: Espresso 3.7.0, core 1.7.0.
• Android Gradle plugin release notes. AGP 9.4 as the current release at the time of checking.
• Apple Developer — Testing (Xcode). Swift Testing for unit tests from Xcode 16, XCTest with XCUIAutomation for UI tests, the pyramid recommendation, and UI tests as "the ultimate indicator" that take longer and can fail for "various app variables."
• Apple Developer — XCTest. That a target may hold Swift Testing and XCTest tests but you should not mix their APIs in one test, and that XCTest remains the framework for UI and performance tests.
• Apple Developer — Swift Testing. Parameterised test functions and in-process parallelisation.
• Apple Developer — XCUIApplication.launchEnvironment. That changes made after launch "don't affect the current launch session" and apply on the next launch.
• Apple Developer — Running your app on simulated or physical devices. That simulators "don't replicate the performance or features of a physical device," and that to test a hardware-specific feature itself you run on a physical device.
• Apple Developer — Device Hub. Device Hub as the manager for simulated and physical devices, and physical devices for hardware-dependent features and performance investigation.
• Apple Developer — Testing notifications using the Push Notification Console. That delivery depends on battery, connectivity and other factors, and that APNs may deliver, defer, retry or discard a notification.
• Apple Developer — Xcode 27 Release Notes. Xcode 27 with Swift 6.4, and agents that can boot simulators, launch apps, synthesise touches and capture screenshots.
• Appium — How does Appium work?. Appium as an implementation of the WebDriver protocol that delegates platform behaviour to drivers, and that some commands such as cookies cannot be supported in native apps.
• Appium — Intro to Appium drivers. The UiAutomator2 driver's reliance on UiAutomator2, ADB and a helper app, and the XCUITest driver's two-part design with WebDriverAgent running on the device, built and managed through Xcode.
• Appium UiAutomator2 driver README. Command proxying to the on-device UiAutomator2 server and ADB, the 30,000 ms default server launch timeout, and the disableWindowAnimation capability. Driver 8.7.0.
• Appium 3.8.0 release. The current stable Appium server at the time of checking, alongside the 4.0.0 beta line.