Advanced Testing Tips for Android, Part 1

I’m currently developing an Android app and have been reading up on Android testing. Here are some tips that I’ve come across and found useful, as well as some ideas of my own.

Emulator vs Device

Testing on a real device is generally recommended over using the Android emulator for various reasons:

  • You should test as close as possible to the user environment.
  • A device is a more realistic environment for acceptance testing.
  • The emulator is slow, and often buggy. I’ve encountered issues with various versions of the emulator, when trying to test things like orientation changes, location services, etc.

While I agree that you should do most of your testing on real devices, sometimes testing on the emulator can be a useful addition.

Advantages of Testing on the Emulator

1. You can increase memory and heap for an AVD (Android Virtual Device).

This may be necessary if you have a lot of tests to run together or have complicated tests (e.g. if using recursion). Of course devices are limited to how much memory they have installed and could run out of memory when running test suites.

2. For testing with particular configurations, e.g system settings such as GPS.

You can setup an AVD with GPS on and another exactly the same except with GPS off (once configured, they would be saved as snapshots). Then you could have annotations in the test cases for which tests to run in which environment.

If you were testing this with a device, then you would need to remember to configure the settings before the tests are run. This may be alright for manual testing, but a major hassle for automated testing (for instance, if you’re using Continuous Integration practices).

In fact, a tool like the Jenkins Android Emulator plugin could be configured to start the appropriate AVD in the emulator when required (or even create one based on a set of configurations).

3. The emulator is free.

Of course using the emulator is cheaper than buying a truckload of devices, especially with so many Android version and screen sizes out there. You can use the emulator to fill in the gaps for whichever configurations you don’t have devices for (is it worth buying separate devices to test Android version 2.3 vs 2.33, for instance).

Testing on a Rooted Device

Most advice about using rooted devices for testing seems to be, DON’T. Most of the time this is good advice, as you want your test environment to be as close to what your customers are using as much as possible.

However you might find it sometimes useful to run some of your tests on a rooted device. In particular, being able to have read and write access to otherwise secure directories can be useful.

Some examples of this would be:

  • to copy a test database onto the device
  • copying junit report files off the device for archiving
  • doing a full backup of various device configurations, to restore later for testing

Some more tips later in part 2 of this article.

Using a custom ServiceLoader in Android

For those looking for a way to create modular and extendible code using some sort of ‘plugin’ architecture, one of the possibilities is to use the ServiceLoader class.

A service provider is a factory for creating all known implementations of a particular class or interface S. The known implementations are read from a configuration file in META-INF/services/.

Here are some examples of it’s usage in Java:
Creating Extensible Applications With the Java Platform

What I was after was use it in Android, and to be able to put the service implementation classes in their own libray project or jar file, so they could be used independently in a modular fashion. A possible use of this structure is to have different ‘plugins’ for different versions of an app.

There are other similar java plugin frameworks out there, but I thought I’d stick with something that already exists in Java rather than learning yet another API.

However there are some issues with using ServiceLoader in Android, mostly having to do with where to put the configuration file that contains the list of plugin implementation classes. Unfortunately ServiceLoader is hard-coded to look for the file at ‘META-INF/services/’, which makes it hard if we want to make it work in Android.

This article ‘Using serviceloader on android‘ gives some ideas about how use it with Ant. However I wanted a solution that could also be used with Eclipse.

Customizing ServiceLoader

Because ServiceLoader is a final class, it cannot be subclassed but we can still customize it by making our own copy from the Java SDK source code. For example, let’s get a copy of ServiceLoader.java from JDK 1.6 source, and rename it to CustomServiceLoader.

Looking at the source, the first thing we notice is that the location for the config file is hard-coded:

private static final String PREFIX = "META-INF/services/";

Then here is where that location is used (it is only used when hasNext() is called on the iterator due to the lazy initialization pattern):

public boolean hasNext() {
  if (nextName != null) {
  return true;
  }
  if (configs == null) {
    try {
      String fullName = PREFIX + service.getName();
      if (loader == null)
        configs = ClassLoader.getSystemResources(fullName);
      else
        configs = loader.getResources(fullName);
    } catch (IOException x) {
      fail(service, "Error locating configuration files", x);
    }
  } ...

There are several ways we could customize this class to make it easier to use with Android.

Option 1: Replace the hard-code PREFIX field with your own location

So in our custom class, we could just change the location for the PREFIX constant to point to somewhere else in our classpath. This is the quickest way, but suffers from the same inflexibility as the original ServiceLoader.

private static final String PREFIX = "com.yourdomain.services/";

Since the location is being looked up internally using getResources() or getSystemResources() from the classloader, all we have to do is to make sure the configuration file is in the classpath somewhere, and then tell CustomServiceLoader where to search for it.

Another idea for this option was to do it the ‘Android’ way and use the AssetManager to find the configuration file in the assets directory. This would mean passing the AssetManager as an extra parameter in the public API methods for CustomServiceLoader.
However the Android build process will only take into account the assets directory in the main project and ignore the assets directory in library projects, so the file would have to go in the main project. I prefer the configuration to exist in the library along with it’s code so that it could function as an independent unit, so I did not use the assets folder.

Option 2: Pass the location

Rather than hard-coding the location, we could just pass the location as an additional parameter when you want to load the services. This would mean changing the method signatures to pass the location parameter through (or store it) until it is needed. Hence instead of calling …

CustomServiceLoader.load(MyService.class)

you would call …

CustomServiceLoader.load(MyService.class, "com.yourdomain.services/")

I did not end up using this option, since it would have required more substantial changes to the original class.

This is one of the caveats regarding copying source code and changing it, and why I generally dislike doing this. If you make any significant changes to it, then there is always the chance you will introduce bugs into the code (therefore requires more testing!). Also if the original code base changes, your code will not include those changes.

Making it Modular for Multiple Plugins

The above solutions would work if you only had one plugin (i.e. a jar or project containing some service implementation classes and a configuration file), but what if you wanted to be able to handle any number of plugins. My idea was to have each plugin as a independent unit, each with it’s own configuration file.
Then the app would load all the plugin classes it found and aggregate them to make all the service implementations available.

Why would you want to do this? Well, many Android apps have multiple versions, i.e. a paid vs a free version, or a lite vs a pro version. So you could have 1 plugin for the lite version offering a limited number of services.
Then for the pro/paid version, you could either replace the plugin with another version that offers more services or just add those extra services as addition plugins.

It would require a bit more work on CustomServiceLoader to allow it to handle this situation.

This means that, for instance, if the location you want is at ‘com.yourdomain.services’, then if you have 2 plugin libraries or jars with their configuration files at that same location, then the apk will build since it would consider them to be duplicates:

Error generating final archive: Found duplicate file for APK: com.yourdomain.services/com.yourdomain.plugin.MyService

Let’s have a look at the code again:

private static final String PREFIX = "com.yourdomain.services/";

...

public boolean hasNext() {
  if (nextName != null) {
  return true;
  }
  if (configs == null) {
    try {
      String fullName = PREFIX + service.getName();
      if (loader == null)
        configs = ClassLoader.getSystemResources(fullName);
      else
        configs = loader.getResources(fullName);
    } catch (IOException x) {
      fail(service, "Error locating configuration files", x);
    }
  } ...

The problem is that the code is only looking at 1 location as specified by the PREFIX constant, but having the same location in multiple plugin libraries will break the build process due to duplicate file problem.
We can put the configuration files in different locations in each plugin library, but then how does CustomServiceLoader find them?

I’m still working on this, but got some ideas:
1. Add some code to be able to handle wildcards in the location.
e.g.
Plugin1 would have the location of it’s configuration file at ‘com.yourdomain.services1/’.
Plugin2 would have the location at ‘com.yourdomain.services2/’.
We could pass is a wildcard location, such as ‘com.yourdomain.services*’.

In the Spring Framework, there are classes such as ResourcePatternResolver and PathMatchingResourcePatternResolver that does something like this. This is what enables Spring to find ApplicationContext.xml in different locations. This seems to be the most flexible way of doing it, but requires more work.

2. Another way would be to use option 2 above (passing the location as a parameter), but instead of just 1 location, pass in multiple locations (in a Collection or array) and aggregate the results of the getResources() calls.
This is easier, but far from ideal, as you have to hard-code the locations of the configuration file in each plugin library somewhere.

Tips on handling orientation changes in Android 2

Here are some tips on how to handle and test orientation changes in Android 2

This post applies to Android 2.x, since onRetainNonConfigurationInstance() is deprecated in Android 3.x.

In my previous post I showed one method I use to test orientation changes. The standard way of handling orientation change is to save the results of expensive operations
in onRetainNonConfiguration(), and then when the activity is recreated that data can be re-used.

From the Android developer documentation

Finally, remember that onRetainNonConfigurationChange() should be used only to retain data that is expensive to load. Otherwise, keep it simple and let Android do everything.

And here.

This function is called purely as an optimization, and you must not rely on it being called

This means that onRetainNonConfigurationChange() is supposed to be an optimization, so the activity should be able to recreate the data if required

Now here is a list some things that I will usually save in onRetainNonConfigurationChange().

Expensive operation results – just like the documention says. After all you don’t want your app to be re-doing things like database queries, network calls or complex
calculations every time the user flips the phone.

AsyncTask – there are some situations where a running AsyncTask will be saved, so that it can continue it’s processing after the orientation change (rather than starting the task again). Makes sense since one of reasons to use an AsyncTask is to handle long-running operations.

Intermediate data – I was using some fields in the activity to hold some (non-expensive) data, so I didn’t bother to save it because I thought it was trivial. The result was NullPointerExceptions because that data was required either in the onCreate() or later on, e.g. onDialogCreate()when I wanted to display that data in a message. So you can either save this data, re-create it in onCreate() or at least do a null check before using it.

Maps – if the activity displays a map (e.g. using a MapActivity), then it might be more user friendly to save things like the zoom level, satellite view option, and the centre point of the map.  This is so that the user has a similar map view after the orientation changes.

Some other tips not related to saving state

Dialogs– if you are doing your own dialog handling, instead of letting the activity do it by using onCreateDialog() and showDialog(), then you need to handle the situation
where the dialog is displayed when the orientation change occurs. I generally let the activity do the hard lifting, except maybe in cases where the life cycle of the dialog doesn’t
match the activity, e.g. a progress dialog for an AsyncTask.
Generally in my unit tests, I will follow each dialog display with an orientation change to check that afterwards:
1. the dialog is still displayed
2. if the dialog shows data, that the data hasn’t changed

Layouts – if you have alternate layouts for portrait and landscape, goes without saying that you need to test both! This includes things like testing the order of focus for the views, etc. since the views may be in different positions in the different layouts.
This is the reason that in my previous post: Test Android orientation changes with Robotium I ran the test methods multiple time starting in both portrait and landscape mode.

Some more testing tips

  • Some versions of the emulator may have bugs that show up when your test invokes an orientation change, e.g. the 2.3 version will allow your app to change normally from portrait to landscape, but then it has problems when you try to change back to portrait again. The test will still run OK, the will not look right in the emulator window.
  • If you are using the emulator and have some Robotium or instrumentation test code to click on an item in a list by index, e.g. solo.clickInList(1). Then the orientation change may cause some test failures when going into landscape mode.
    This is because fewer items being displayed in the list in landscape than in portrait mode, of course this is only a problem is there are more items in the list than can be displayed in the view or ‘page’.
    See if you can use solo.clickOnText() instead with the option to allow scrolling if required.

If you can think of anything else that might be useful, please let me know!

Test Android orientation changes with Robotium

One of the most painful things I found while learning Android development was handling orientation changes (who thought flipping your phone would cause so much trouble).

This is because the activities are being destroyed and recreated every time the phone is changing between portrait and  landscape mode. Here I’m assuming that it is being handled ‘properly’ by saving state data (e.g. in onRetainNonConfigurationInstance() ) and then reloading it when the activity is created.

After this saving and reloading code is in place, then it needs to be unit tested, and here I’ve found Robotium to be very useful in running instrumentation tests with orientation changes.

An example

This is just one way that I’ve found useful, here is an example – take this simple test to simulate the user entering some text and clicking some buttons:

public void testValidateText()
{
  EditText targetText = solo.getEditText(0);
  solo.enterText(targetText, "hello");

  // click on button to bring up dialog
  solo.clickOnButton("Validate");

  // wait to dialog to pop up
  getInstrumentation().waitForIdleSync();

  // click on button in dialog
  solo.clickOnButton("OK");
}

For the sake of simplicity, I’ve left out the Robotium setup code and the asserts (which would normally be the point of the testing!).

Ring the (orientation) changes

In the next version of this test, I will:

  • rename the test method to something that doesn’t start with ‘test’, because Android used JUnit 3, this will stop this method from being run in the test case
  • add some parameters to the test method as flags for doing orientation change
  • add to code in the method to change the orientation
  • write some wrapper methods to call our test method
public void testValidateText()
{
  doTestValidateText(ActivityInfo.SCREEN_ORIENTATION_PORTRAIT, false);
}

public void testValidateTextWithOrientationChange()
{
  doTestValidateText(ActivityInfo.SCREEN_ORIENTATION_PORTRAIT, true);
}

public void testValidateTextInLandscapeWithOrientationChange()
{
  solo.setActivityOrientation(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE);
  doTestValidateText(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE, true);
}

public void doTestValidateText(int currentOrientation, boolean changeOrientation)
{
  int orientation = currentOrientation;

  EditText targetText = solo.getEditText(0);
  solo.enterText(targetText, "hello");

  orientation = changeOrientation(solo, changeOrientation, orientation);

  // click on button to bring up dialog
  solo.clickOnButton("Validate");

  orientation = changeOrientation(solo, changeOrientation, orientation);

  // wait to dialog to pop up
  getInstrumentation().waitForIdleSync();

  orientation = changeOrientation(solo, changeOrientation, orientation);

  // click on button in dialog
  solo.clickOnButton("OK");
}

So now when the test case is run, the original test method will be run in 3 ways:

  • testValidateText() will run the test as before
  • testValidateTextWithOrientationChange() will run the test, but after every user action will change the orientation with the line
    orientation = changeOrientation(solo, changeOrientation, orientation);
  • testValidateTextInLandscapeWithOrientationChange() will also run the test with orientation changes, but starting off in landscape mode

Here is the code for the orientation change.


public int getNextOrientation(int currentOrientation)
{
  if (currentOrientation == ActivityInfo.SCREEN_ORIENTATION_PORTRAIT)
  {
    return ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE;
  }
  else
  {
    return ActivityInfo.SCREEN_ORIENTATION_PORTRAIT;
  }
}

public int changeOrientation(Solo solo, int currentOrientation)
{
  int nextOrientation = getNextOrientation(currentOrientation);
  solo.setActivityOrientation(nextOrientation);

  return nextOrientation;
}

public int changeOrientation(Solo solo, boolean changeOrientation, int orientation)
{
  int newOrientation = orientation;
  if (changeOrientation)
  {
    newOrientation = changeOrientation(solo, orientation);
  }
  return newOrientation;
}

So basically the wrapper methods just invoke the original test method with a flag for the orientation that it starts in, and whether to do orientation changes during the test or not.

In the next post, I will list some problems in my code that I exposed with this testing, and other issues with coding and testing for orientation changes.