What Is Android Test Orchestrator?

Google released Android Testing Support Library 1.0 recently. After reading the overview, I'm a little confused with Android Test Orchestrator. It said Typically, AndroidJUnitRun

Solution 1:

After researching the issue a bit I can provide the following answers:

Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.

As mentioned, AndroidJUnitRunner runs on the same instrumentation process so basically your tests are running state-full, this might wreak havoc if your tests have some sort of dependency on the process state. In Android test orchestrator each test is run in its own process so dependencies aren't an issue.

if one test crashes, it prevents the remainder of the test suite from running

The crash in question here is a crash of the process, not the activity/application. You can test this by inserting in one your tests System.exit(0); Typically, this will stop the whole test run while in Android test orchestrator the tests continue as expected.

For completeness, Android Test Orchestrator runs pm clear after each test.

This is an oversight of google and has been retracted from the official documentation as can be observed here.

Basically, The advantages of using Android test orchestrator is all about the separate process for each test which improves stability and ensures full execution of tests.

Solution 2:

Android Test Orchestrator is a tool which allows you to run each of your app’s tests within its own invocation of Instrumentation. This means that each test (method annotated with @Test ) will be run on a separate instance of AndroidJUnitRunner.

What issues does it solve? When dealing with UI tests we identified 2 major problems which occur from time to time when run on CI or locally:

1. Occasional crashes which stop the entire test suite.
2. Test overlap.

Solution 3:

Orchestrator can prevent crash to interrupt the whole test, eg. native crash. But it may slow the tests