środa, 9 lutego 2011

Running a headless emulator

Knowing that headless system is a system designed to be run without a monitor you might
wonder if this really makes sense to have an emulator running without a possibility to have a look what is going on there :).

That might sound crazy but think of the continuous integration like hudson. Hudson will run the test(s) to check the stability of the build and generates reports afterwards. No one will be interested in sitting in front of the screen connected to hudson to check if the test are running ok and when the test fails. This can be obtained from the reports or the test logs and repeated (hopefully) on developers machines with the debugger connected.

Also when the UI test fails they will be discarded from the screen so you might not be even able to spot where the failure occurred.

Finally to run emulator you need X server running, a DISPLAY variable exported so might be too much of setup for simple continuous integration server.

In order to run emulator in headless mode issue a command

emulator @Android2.2 -no-window

And that is explained in help window

emulator -help
-no-window disable graphical window display

wtorek, 8 lutego 2011

Randomizing the order of the tests

One of JUnit best practices say that one should not assume the order in which tests in a test case are run. It is assumed that method will be executed in arbitrary order.
Therefore the having dependencies between the tests is discouraged.

JUnit uses reflection to find the test cases and it is not guaranteed that the order will be the same in different VM(s). But, you do not change your vm so often so the order of the test remains usually persistent across the lifetime of the project.

I have seen the tests that people do make some assumptions about the order and one test "prepares" the data or environment for the next. And the result is that test may fail if executed separately.

The dependencies might not be intentional at all, and just be the result of the way the test were ordered and no one realizes that they exist.

In the project I am right now we wanted to ensure that there are no dependencies between the tests and the solution was to intentionally make the order unpredictable. The problem was to find a way to randomize the test cases.

In android (2.2) source code when you look at
you will see that
is using something called a test grouping that is constructed with
and later adds a test(s) from a testCases list.

I am not sure in which order is this list populated but I am guessing the order can be predicted. As a result we will get a almost constant sequence of the test(s) unless some new test(s) are introduced.

So looking at the method android.test.InstrumentationTestRunner.onCreate(Bundle)
showed that the set of test is created inside the method using a local instance of TestSuiteBuilder and is assigned to the AndroidTestRunner instance. A class instance but the getter getAndroidTestRunner() everytime return new instance of the runner.

Recently I became more comfortable (& skilled) in java reflection and at first thought I expected to be able to get to that field with reflection. The problem was that onCreate just starts the test just before it completes. So you might not be able to randomize or you will randomize when the test are executed.

What is more with reflection you will have an ugly code, with lot of exception catching and without compiler checks so error prone and hard to maintain. Consequently, is very fragile and if the field name will change it will fail badly. Last but not least is a common belief the reflection is very slow and even though I had not measured it I share the same opinion.

The reflection might be used between onCreate and onStart but I figured out that we can save the runner used in onCreate and in onStart or start shuffle the test(s).

The result is below
 package net.retsat1.starlab.testrandomizer;  
import java.util.Collections;
import java.util.List;
import junit.framework.TestCase;
import android.test.AndroidTestRunner;
import android.test.InstrumentationTestRunner;
* A test runner that each time runs the set of test executes them in random order.
* This might be useful to eliminate the dependencies between the test(s)
* @author marek.defecinski
public class RandomizingTestRunner extends InstrumentationTestRunner {
private AndroidTestRunner androidTestRunner;
* We hijack this method to make some modifications of the order of tests
public void onStart() {
List<TestCase> testCases = androidTestRunner.getTestCases();
* When the {@link AndroidTestRunner android runner} is created in {@link InstrumentationTestRunner#onCreate(android.os.Bundle)} we save it here.
* The same method at the end has a start method which we override to randomize the order of the test
protected AndroidTestRunner getAndroidTestRunner() {
androidTestRunner = super.getAndroidTestRunner();
return androidTestRunner;

Hope someone will find it useful.