Parallel tests on Simulators
Let’s start from easier approach of parallel tests – running them on Simulators. The first question we need to ask ourselves is – how many devices we want to run tests on? You’ll be amazed once you know how many concurrent sessions on Simulators were allowed by Apple. According toxcodebuild
logs – it’s limitless! Awesome, huh? The simplest way to run iOS tests on Simulators consists of such steps:- Creating additional n UI test schemes
- Splitting existing test suite between created schemes
- Creating additional n iOS Simulators (optional, if you want to run on different device models)
- Running parallel tests on n simulators differentiating them by names/UDIDs
Creating test targets
In Xcode 9 creating UI Test Targets is rather easy. If you already have UI Test Target, all you need to do is to duplicate it n times. If not, check out my XCUITest Essentials post, where I covered the basics of creating UI Test scheme.
Splitting tests
After we created additional schemes we have to split our entire test suite between those. To do that we need to open Scheme Settings by navigating to Product > Scheme > Manage Schemes. Now we have to select created schemes one by one and disable/enable needed tests to shard them between targets.
Cloning Simulators
Since we want to run different tests on the same device type in parallel, we need to create additional simulators of the same model and iOS version. For instance, we want to run our tests on 3 devices of typeiPhone X (iOS 11.0)
. This would mean that 2 additional (excluding default one) devices should be created in our system. We could do that in Simulators menu under Window > Devices and Simulators. By pressing + we can set new device’s model and iOS versions along with paired watchOS device.
Parallel Test Run
As I mentioned before there were two options for running iOS/tvOS tests in parallel – same test+different devices and different tests+same device.Same tests on different devices
To run same tests on different devices we would need to include multipledestination
flags for each device we want to run test scheme on into xcodebuild test
command:xcodebuild \
-scheme SimpleCalculatorUITests \
-destination 'platform=iOS Simulator,name=iPhone X,OS=11.0' \
-destination 'platform=iOS Simulator,name=iPhone 8,OS=11.0' \
test
SimpleCalculatorUITests
will be executed on iPhone X
and iPhone 8
simultaneously.
Different tests on same device (test sharding)
That’s where previously created schemes and additional simulators come in handy. To run different test targets on the same device type we would need to invokexcodebuild
for each of them, setting different simulator for run as destination
value:xcodebuild \
-scheme SimpleCalculatorUITests \
-destination 'platform=iOS Simulator,name=iPhone X,OS=11.0' \
test & \
xcodebuild \
-scheme "SimpleCalculatorUITests copy" \
-destination 'platform=iOS Simulator,name=iPhone X 2,OS=11.0' \
test & \
xcodebuild \
-scheme "SimpleCalculatorUITests copy 2" \
-destination 'platform=iOS Simulator,name=iPhone X 3,OS=11.0' \
test &
iPhone X (iOS 11.0)
type.
Parallel test class run
Similarly to how we executed test targets in parallel, we are able to run test classes (or even methods) in parallel.-only-testing
flag will help us in doing that. All we need to do is to specify test class and test scheme it belongs to:xcodebuild \
-scheme SimpleCalculatorUITests \
-destination 'platform=iOS Simulator,name=iPhone X,OS=11.0' \
-only-testing:SimpleCalculatorUITests/MinusTest \
-only-testing:SimpleCalculatorUITests/ResetTest \
test & \
xcodebuild \
-scheme SimpleCalculatorUITests \
-destination 'platform=iOS Simulator,name=iPhone X 2,OS=11.0' \
-only-testing:SimpleCalculatorUITests/AdditionTest \
test & \
xcodebuild \
-scheme SimpleCalculatorUITests \
-destination 'platform=iOS Simulator,name=iPhone X 3,OS=11.0' \
-only-testing:SimpleCalculatorUITests/MultiplyTest \
test &
MinusTest
and ResetTest
classes will be run on iPhone X
device, AdditionTest
will be executed oniPhone X 2
and MultiplyTest
one on iPhone X 3
. And all of that will be done in parallel!Auto splitting of tests
Previous approach is great, but it’s not perfect. We can not specify which tests to run manually every time, and even if we could, continuous integration systems are not smart enough to do something like that every time. So the next improvement I came across was auto splitting of test classes equally between available simulators:
#!/usr/bin/env bash
devices=("platform=iOS Simulator,name=iPhone X,OS=11.0"
"platform=iOS Simulator,name=iPhone X 2,OS=11.0"
"platform=iOS Simulator,name=iPhone X 3,OS=11.0")
test_scheme_name='SimpleCalculatorUITests'
for (( i=0; i<${#devices[@]}; i++ ));
do
devices[$i]="xcodebuild
-scheme $test_scheme_name
-destination '"${devices[$i]}"'"
done
i=0
path_to_test_classes='SimpleCalculatorUITests/PageObject/Tests'
for entry in "$path_to_test_classes"/*Test.swift
do
if (( i == ${#devices[@]} )); then
i=0
fi
name="${entry##*/}"
name="${name%.*}"
devices[$i]=${devices[$i]}"
-only-testing:$test_scheme_name/$name"
((i++))
done
cmd=''
for (( i=0; i<${#devices[@]}; i++ ));
do
cmd=${cmd}${devices[$i]}" test & "
done
echo ${cmd}
eval ${cmd}
devices
is the array which contains names for all simulators we want to run tests on. First, we add xcodebuild
string to each of the array elements. Second, we search for all the test class files under the path path_to_test_classes
by given pattern (in my case it’s *Test.swift
), extracting filename from each found file path string. Then we iterate through those filenames and split them across the devices we have in devices
one by one. In case we reached the end of array, index would be reset and test splitting would continue from the first device in list. In the end we add test
command to each array element and join them in one cmd
string. And finally, we evaluate string as a shell command. Note: this script should be wrapped into shell script file and put under the project folder.
Headless test run
In Xcode 9 Apple’s gone even further by allowing running parallel tests in headless mode. Now we don’t have to start simulators beforehand anymore andxcodebuild
won’t do that implicitly either. In theory it should decrease execution time and save some system resources, particularly video ones.https://youtu.be/KbcsmR-I-pk
Parallel tests on Real Devices
The process of running parallel tests on real iOS/tvOS devices is more or less similar to one on simulators. But instead of specifyingplatform
, name
and OS
we would need to specify device’s UDID:xcodebuild \
-scheme SimpleCalculatorUITests \
-destination 'id=${UDID_1}' \
test & \
xcodebuild \
-scheme "SimpleCalculatorUITests copy" \
-destination 'id=${UDID_2}' \
test &
Splitting tests between all Devices
I have good news for you – we can reuse auto splitting scriptfor test execution on real devices. The only thing we’d need to change isdevices
array creation before executing the script:#!/usr/bin/env bash
device_type='iPhone'
devicesString=$(system_profiler SPUSBDataType |
grep -A 11 -w "${device_type}" |
grep "Serial Number" |
awk '{ print $3 }')
devices=(${devicesString// / })
for (( i=0; i<${#devices[@]}; i++ ));
do
devices[$i]="id=${devices[$i]}"
done
iPhone
into one string type with the help of system_profiler
MacOS utility. Then we will split the string into device array. After that we can use previously implemented auto splitting script. Leveraging this approach we could run our tests on all available (connected) at the moment devices and it should make our test run strategy truly scalable and robust. BECOME IOS SOFTWARE TEST ENGINEER WITH CODEFITNESS!
Sign up to The Ultimate Mobile Test Automation Bootcamp (5 seats left)- We are not typical bootcamp. We teach through day-to-day work tasks using Agile approach. From day one you will start automating real use cases of a real startup-like iOS app. Thus you will gain real experience.
- You will learn programming via hands-on task completion — focusing you on topics that are necessary to complete your task
- You will work in pair just like at real-world job. Pair programming is proven to be very effective and used heavily industry wide.
- You will experience best software development practices:
- Each test case code will be committed to common GitHub repository via PullRequest. We will teach everything about Git and GitHub from scratch
- Upon successful code review and merge, your test code will be executed on Jenkins — your work will be run along with existing tests thus will be checked if your newly automated test is not breaking existing tests. This practice is called pre-merged testing and in recent years has become industry standard
- We will start each session with methodical approach on how to solve most asked Interview Programming problems thus will prepare you for coding interview screening