APPIUM, XCUITEST, ESPRESSO, AND MORE

When it comes choosing what mobile testing tool is right for you, there is a huge array of options, each with different strengths and weaknesses.

WHY SHOULD YOU DO MOBILE AUTOMATION IN THE FIRST PLACE?

It’s clear that anyone making apps should test them. Failure to catch bugs or regressions can cost companies thousands of dollars a day, and releasing broken apps can frustrate and alienate your users. The cost of a bug in a mobile app can be much higher than a similar bug on a web app. Not only could a bug immediately lead to poor reviews and thus a lowered install rate, but on many mobile app marketplaces, getting a fixed version out could take days or even weeks. But why should you automate your mobile app testing? The reasons for doing automated mobile testing are arguably the same as for traditional web development.

SPEED OF TESTING

While manual testing can be useful, it is a slow and human resource-heavy process. By automating your testing, you can run a suite of tests that would take a manual tester hours to complete in minutes or seconds. Speeding up your testing can allow you to expand your test coverage so you can be more confident that you’re releasing bug-free code.

SCOPE OF TESTING

Not only is manual testing slow, but there is a limit to how many user flows or scenarios can be tested realistically. Part of the insight of the Agile software development methodology is that testing should be done as close to code changes as possible. Ideally, this means that your app should be tested on every single commit, not least because a failing test can be tied to a small unit of code change, rather than potentially being related to dozens or hundreds of changesets that went out with a release. Also ideally, all tests should be run, including those for parts of the app which haven’t been touched by the code change. This is because in complex applications it is often not possible to know beforehand what the interaction effects of a code change will be, and the best way to be sure there is no extraneous bug introduced is to run through the entire suite of tests. Needless to say, manually testing every possible case on every single commit is impossible, especially when you consider the number of platforms to be tested. Leveraging a mobile automation framework is at least part of the answer to all of these problems. Even with automation, however, scaling up your testing can be difficult. End-to-end tests are expensive and, to ensure the testsuite completes in a reasonable amount of time, you’ll need to run them at a high concurrency. If you choose a cloud testing platform like Sauce Labs, you can scale instantly without needing to buy or install more servers or purchase more VM licenses.

COST OF TESTING

Automated testing can save you time and money, since you can spend fewer resources on manual testing. Add to that the cost savings of using Sauce’s cloud platform instead of maintaining in-house infrastructure, and automated mobile testing can be a cost-effective solution for ensuring you’re releasing bug-free apps.

SOME OPTIONS FOR MOBILE TESTING TOOLS

Mobile testing on Sauce Labs is powered by Appium, an open source mobile automation framework for testing native, hybrid, and web apps. However, there are many different options for mobile testing, all with different philosophies and approaches. We’ll discuss Appium’s philosophy and why we choose to support it, as well as explore some of the other tools available and the differences.

Appium

APPIUM’S PHILOSOPHY

We believe that Appium’s approach to mobile automation makes it a strong mobile testing tool that will continue to improve. With the mobile marketplace fragmented between OS platforms and app types, there is need for a truly cross-platform mobile automation tool, and Appium aims to be that tool.

APPIUM’S PHILOSOPHY FOLLOWS FOUR PRINCIPLES:

1. Test the same app you submit to the marketplace. You shouldn’t need to compile third party apps or frameworks into your app’s code that needs to be stripped from the app before submitting it to the app store. The reason for this is simple – if you test one version of your app and submit another version, there is a risk for bugs you didn’t foresee being released into the marketplace. Therefore, to minimize this risk, we believe it is prudent to test the same app you release.
2. Write your tests in any framework, using any language. People choose which language to write in for many different reasons, and many even write in multiple languages in a given day. This means that any tool that limits what languages and frameworks it supports will be limiting to those using it. Appium strives to support as many languages and frameworks as possible, to give developers the most flexibility.
3. Use a standard automation specification and API. Developing standards for specifications and API benefits everyone in the open source ecosystem. There are already existing specifications and standards for web testing tools, and Appium’s philosophy is that mobile automation tools should reuse and adapt these existing standards to determine what the standards for mobile testing tools should be.
4. Build a large and thriving open-source community effort. For both the health of the tool and the benefit of anyone wanting to test, it’s a good idea to encourage a robust community to support and improve the project. Appium strives to build a vibrant community on the belief that that community will further the technology for everyone.

It’s because of this approach that Sauce Labs has made the effort to support Appium as a mobile automation framework. We believe that this philosophy translates to a well-thought out vision that drives the tool forward. We also believe that this philosophy sets Appium apart from other open source frameworks. To learn more about Appium’s vision moving forward, read our article Appium and the Fallacy of the Zero Sum Game.

BENEFITS OF APPIUM

• Appium is one of the only options for cross-platform testing. In other words, if you have the same app running on both iOS and Android, you can use Appium to run your tests on both platform with minimal or no code changes. This allows significant reuse of code.
• As we mentioned above, Appium allows you to use any programming language for your tests, and can be mixed into any custom test framework you like. Appium gives you more freedom in how to set up your testing than any alternative project.
• Appium provides a stable API on top of a crazy automation space. The underlying vendor-provided automation tools change their API on what seems like an annual basis. Appium covers over these differences and gives you the same WebDriver-based interface as the years go on, making maintaining your tests easy as Appium upgrades the underlying technology.
• If you care about hybrid or web testing, Appium provides the most natural vocabulary for switching between native app content and web content, since it leverages the WebDriver protocol for both.

Sauce Labs is committed to the health of the open source ecosystem, and they believe Appium is a great option for those looking for a mobile testing tool. Sign up for a free trial to start testing.

XCUITest

XCUITest is the official UI testing tool from Apple. XCUITests are bundled with your application and run via Xcode. XCUITest is only for iOS testing, and tests must be written in Swift or Objective-C. Some benefits are:

• Stay within the Xcode environment
• Use Swift or Objective-C alongside your app code
• Take advantage of Apple’s services around XCUITest

Espresso

Espresso is the newest official UI testing tool from Google (superseding both UiAutomator and UiAutomator2). Espresso has a completely different architecture, where UI tests live alongside the app code, and can even have access to shared code. Espresso is only for testing Android apps, and Espresso tests must be written in Java or Kotlin. Some of the benefits of Espresso are:

• Speed: Espresso was designed from the ground up for speed of automation
• Reliability: Espresso has built-in support for ensuring automation does not continue until a view is in an idle state. This avoids a common class of automation bugs where the test tries to continue to the next step but the application is not yet ready
• Stay within the Android Studio environment
• Take advantage of Google’s services around Espresso

Earl Grey

Earl Grey is Google’s answer to XCUITest, and is an iOS UI Automation framework which also happens to be open source! (However, using Earl Grey does report anonymous statistics back to Google). Earl Grey is for iOS only, and tests must be written in Objective-C or Swift. The main benefit of Earl Grey is that it brings Espresso’s synchronization features to iOS testing, to once again ensure that the automation is not trying to do something in the app while the app is busy.

Detox

Detox is a JavaScript-based test framework which, like Appium, aims to be cross-platform (though Android support is not yet fully shipped at the time of writing). It is meant to be used as a Node.js library in your test code, so tests must be written in JavaScript. Detox is also fully open source. Some of its benefits are:

• Write cross-platform tests
• Like Espresso and Earl Grey, Detox tries to ensure your app is in an idle state before allowing automation to continue
• Underlying technology is precisely Espresso and Earl Grey, so it builds off of Google’s good work

Choosing a Framework

Given the above list of options, it can be overwhelming to know where to start. At the end of the day, all of these technologies allow the work of automated testing to proceed. There are too many specific pros and cons of each framework to list them here. In general, a good choice of framework involves asking questions about how a team functions and who writes the tests, more than simply looking at a technical feature list. For example, if your testing is organized by a QA team that likes to write Ruby code, then Appium will be the best choice. Similarly, if your needs are for testing a cross-platform hybrid app, Appium is the obvious winner. If, on the other hand, you are testing a single-platform native app on Android, and your developers are writing the code, it may be wise to explore Espresso, because of the tight integration made possible between the test and app code. At Sauce Labs, they believe in Appium’s vision of one automation protocol for all platforms, but recognize that in the real world, different tools are appropriate for different scenarios.

Sauce Labs – Scalable Mobile Testing

Sauce Labs makes automated testing awesome. Their cloud-based platform helps developers test native & hybrid mobile and web applications across 800+ browser / OS platforms, including iOS, Android & Mac OS X. Sauce supports Selenium, Appium and popular JavaScript unit testing frameworks, and integrates with all of the top programming languages, test frameworks and CI systems. With built-in video recording and screenshots of every test case, debugging tools, and secure tunneling for local or firewalled testing, Sauce makes running, debugging and scaling test suites quick and easy.