At XING we are around 30 Android developers that are contributing to the same app.
In the past six months, we invested a lot of time in making our espresso tests execution on our Jenkins CI server way more stable. In this article, I would like to share our story of how we got to stable espresso tests.
We rely heavily on running all of our unit tests and UI tests on each pull request to ensure that they are constantly green and no tests are broken.
While running unit tests from a terminal is quite simple, if you use tools like robolectric, different architecture patterns and run them solely on the JVM, it’s quite a complex problem to run your espresso or other tests (like calabash or appium) that require a device to run from a command line interface.
Running tests on real devices
One solution would be to run these tests on real devices that are connected to the CI instance directly, but depending on your CI solution (we use Jenkins) this might be a problem. Also, this will not scale well if you have multiple people creating or updating pull requests at the same time. Another problem is, that you probably would want to run on real devices that most of your users use. For XING that would be:
Most used devices for the XING Android App (July 2018)
As you can tell from the chart above you want to connect Samsung devices to your CI server and run the tests on there. Good luck with that. It will be tough just to unlock all of these devices before running the tests. We had a similar set up in the past using a nice tool called OpenSTF that we integrated with our Jenkins but it was causing nothing but pure, sweet pain, when using it there.
Another problem of the many we had was that we need to maintain potential adb failures that would occur regularly.
In addition to that we started to see problems with power supply on our device farm as soon as we started adding more devices to it. Suddenly, devices would switch off or discharge at random.
We saw the failures, weren’t happy about them and decided we need something else because this approach just wouldn’t work and introduced way too much flakiness, where tests would randomly fail or not even start because the phone crashed.
Using AVDs / emulators to run espresso tests
Previously, we had also tried to run the espresso tests on AVDs (Android Virtual Devices), which made the tests more consistent, but introduced the problem of having to manage everything around them.
Creating them, starting them, cleaning before running tests on them and killing them once they are done. While it seemed to be more stable in the beginning ,we now had to maintain a lot of bash scripts that tried to execute the necessary steps one after another which were really hard to write in the first place and almost impossible to change if we had to.
In addition to that, the whole system was really vulnerable to inconsistencies, like early exits of tests or other unexpected events that could occur at any time.
Our team and the XING Android community said that time has come to put an end to this insanity and find a proper solution to our problem.
We looked at different technologies that could help us to solve our problem and finally ended up with Docker which is a nice tool to run things really isolated within a so-called container on almost any machine.
The idea was to create a docker container that has all Android dependencies that you need to build and run an Android app in it and that boots up a single AVD whenever it is started. So the rough plan for our CI was to:
- Start the docker container on our CI
2. Include the project folder of our Android App in the container
3. Build the app inside the container
4. Run the tests inside the container
5. Return the results back to our Jenkins machine that hosted the docker container
Of course it would still require some sort of scripting to do that so we decided to take it even further and create a fastlane plugin that will handle all of these things for us.
That’s when we built mango:
mango is able to do all of the above mentioned work for you with just a simple fastlane task that you configure in your project:
desc "Run espresso tests on docker images"
lane :espresso_tests do
run_dockerized_task(
container_name: "espresso_container",
docker_image: "joesss/mango-docker:latest",
container_timeout: 120,
android_task: "./gradlew connectedAndroidTest",
post_actions: "adb logcat -d > logcat.txt",
pull_latest_image: true
)
endAs you can see you just need to include the mango plugin using:
fastlane add_plugin mango
And you’ll get access to the run_dockerized_taskaction in your Fastfile.
In this action you just need to specify a few things like the docker_image, container_name and the android_task and you should be ready to go.
Alongside with this fastlane plugin we also published our Dockerfiles that we are using for ourselves and have proven to be working fine for our needs.
Though the tests are now around 99% stable on each of the pull requests there are some nitpicks that you need to take care about if you want to use it on your CI:
- You have to use
fastlane - We use ubuntu machines as our hosts for the docker containers. We experienced some problems if you try to run Qemu/kvm to emulate an Android CPU when running on macOS
We are currently using 8x 24-Core machines connected to our Jenkins.
This allows us to run 160 docker tasks in parallel without exceeding the capabilities of our systems. This enabled us to run all unit tests (~7000), all integration (~600) and all of our espresso UI tests (~250) tests on every commit we do in around 15 minutes without having huge queues.
Through this, we get super fast feedback and make sure that all of our tests are run as early as possible because new code reaches the actual users.
So if you want to get started using mango or try it yourself just go ahead and do so 🙂 If you hit any walls create an issue and ask for help there.
What do you think about our solution? How did you solve this problem for your team? Please make sure to let us know in the comments below!
Thanks for reading
