t’s essential to run our tests in parallel to perform more of them in a tighter window, get feedback earlier and release faster. This is especially true for user interface tests which tend to be the most time consuming and flaky group. And Apple has heard our prayers! During the last WWDC the company presented one of the most useful features regarding testing – possibility of running concurrent XCTest sessions on multiple iOS/tvOS devices.         From Xcode 9 and later we are able not only to run the same test on different devices simultaneously, but also different set of tests on different devices. Basically Apple introduced test sharding. Wonder what is that?           Say you have two iPhone X devices and 10 tests. Now you can split your test suite into 2 shards (parts) and run 5 tests per each device. In theory it should decrease run time by factor of n, where n is the number of shards. More about this concept can be found in one of my previous posts on Parallel Android Tests.

Parallel tests on Simulators

      Let’s start from easier approach of parallel tests – running them on Simulators. The first question we need to ask ourselves is – how many devices we want to run tests on? You’ll be amazed once you know how many concurrent sessions on Simulators were allowed by Apple. According to xcodebuild logs – it’s limitless! Awesome, huh?         The simplest way to run iOS tests on Simulators consists of such steps:

• Creating additional n UI test schemes
• Splitting existing test suite between created schemes
• Creating additional n iOS Simulators (optional, if you want to run on different device models)
• Running parallel tests on n simulators differentiating them by names/UDIDs

Creating test targets

          In Xcode 9 creating UI Test Targets is rather easy. If you already have UI Test Target, all you need to do is to duplicate it n times. If not, check out my XCUITest Essentials post, where I covered the basics of creating UI Test scheme.

Splitting tests

          After we created additional schemes we have to split our entire test suite between those. To do that we need to open Scheme Settings by navigating to Product > Scheme > Manage Schemes. Now we have to select created schemes one by one and disable/enable needed tests to shard them between targets.

Cloning Simulators

            Since we want to run different tests on the same device type in parallel, we need to create additional simulators of the same model and iOS version. For instance, we want to run our tests on 3 devices of type iPhone X (iOS 11.0). This would mean that 2 additional (excluding default one) devices should be created in our system. We could do that in Simulators menu under Window > Devices and Simulators. By pressing + we can set new device’s model and iOS versions along with paired watchOS device.

Parallel Test Run

          As I mentioned before there were two options for running iOS/tvOS tests in parallel – same test+different devices and different tests+same device.

Same tests on different devices

To run same tests on different devices we would need to include multiple destination flags for each device we want to run test scheme on into xcodebuild test command: In this case our test scheme SimpleCalculatorUITests will be executed on iPhone X and iPhone 8 simultaneously.

Different tests on same device (test sharding)

          That’s where previously created schemes and additional simulators come in handy. To run different test targets on the same device type we would need to invoke xcodebuild for each of them, setting different simulator for run as destination value: In this case three different test schemes will be executed on 3 devices of iPhone X (iOS 11.0) type.

Parallel test class run

            Similarly to how we executed test targets in parallel, we are able to run test classes (or even methods) in parallel. -only-testing flag will help us in doing that. All we need to do is to specify test class and test scheme it belongs to: In the example above MinusTest and ResetTest classes will be run on iPhone X device, AdditionTest will be executed oniPhone X 2 and MultiplyTest one on iPhone X 3. And all of that will be done in parallel!

Auto splitting of tests

              Previous approach is great, but it’s not perfect. We can not specify which tests to run manually every time, and even if we could, continuous integration systems are not smart enough to do something like that every time. So the next improvement I came across was auto splitting of test classes equally between available simulators: devices is the array which contains names for all simulators we want to run tests on. First, we add xcodebuild string to each of the array elements. Second, we search for all the test class files under the path path_to_test_classes by given pattern (in my case it’s *Test.swift), extracting filename from each found file path string.         Then we iterate through those filenames and split them across the devices we have in devices one by one. In case we reached the end of array, index would be reset and test splitting would continue from the first device in list. In the end we add test command to each array element and join them in one cmd string. And finally, we evaluate string as a shell command.        

Note: this script should be wrapped into shell script file and put under the project folder.

Headless test run

        In Xcode 9 Apple’s gone even further by allowing running parallel tests in headless mode. Now we don’t have to start simulators beforehand anymore and xcodebuild won’t do that implicitly either. In theory it should decrease execution time and save some system resources, particularly video ones.

Parallel tests on Real Devices

            The process of running parallel tests on real iOS/tvOS devices is more or less similar to one on simulators. But instead of specifying platform, name and OS we would need to specify device’s UDID:

Splitting tests between all Devices

            I have good news for you – we can reuse auto splitting scriptfor test execution on real devices. The only thing we’d need to change is devices array creation before executing the script:       At first we will collect all UDIDs of devices matching iPhoneinto one string type with the help of system_profiler MacOS utility. Then we will split the string into device array. After that we can use previously implemented auto splitting script.     Leveraging this approach we could run our tests on all available (connected) at the moment devices and it should make our test run strategy truly scalable and robust.      

BECOME IOS SOFTWARE TEST ENGINEER WITH CODEFITNESS!

Sign up to The Ultimate Mobile Test Automation Bootcamp (5 seats left)

• We are not typical bootcamp. We teach through day-to-day work tasks using Agile approach. From day one you will start automating real use cases of a real startup-like iOS app. Thus you will gain real experience.
• You will learn programming via hands-on task completion — focusing you on topics that are necessary to complete your task
• You will work in pair just like at real-world job. Pair programming is proven to be very effective and used heavily industry wide.
• You will experience best software development practices:
  • Each test case code will be committed to common GitHub repository via PullRequest. We will teach everything about Git and GitHub from scratch
  • Upon successful code review and merge, your test code will be executed on Jenkins — your work will be run along with existing tests thus will be checked if your newly automated test is not breaking existing tests. This practice is called pre-merged testing and in recent years has become industry standard
• We will start each session with methodical approach on how to solve most asked Interview Programming problems thus will prepare you for coding interview screening