The Problem
Occasionally when running tests you may see unexpected behavior due to A/B testing (a.k.a. split testing) of the application you’re working with. In order to keep your tests running without issue we need a clean way to opt-out of these split tests.A quick primer on A/B testing
Split testing is a simple way to experiment with an application’s features to see which changes lead to higher user engagement. A simple example would be testing variations of an e-mail landing page to see if more people sign up. In such a split test there would be the control (how the application looks and behaves now) and variants (e.g., 2 or 3 changes that could include changing text or images on the page, element positioning, color of the submit button, etc.). Once the variants are configured, they are put into rotation, and the experiment starts. During this experiment each user will see a different version of the feature and their engagement with it will be tracked. Split tests live for the length of the experiment or until a winner is found (e.g., tracking indicates that a variant converted higher than the control). If no winner is found, new variants may be created and another experiment tried. If a winner is found, then the winning variant becomes the new control and the feature gets updated accordingly.A Solution
Thankfully there are some standard opt-out mechanisms built into A/B testing platforms. They tend to come in the form of an appended URL or forging a cookie. Let’s dig in with an example of each approach with a popular A/B testing platform, Optimizely.An Example
Our example page is from the-internet and can be seen here. There are three different variants of the page that are available, each with different heading text:- Control:
A/B Test Control - Variation 1:
A/B Test Variation 1 - Opt-out:
No A/B Test
org.junit.After, etc.), driving the browser with Selenium (e.g., org.openqa.selenium.WebDriver, etc.), and matchers for our assertions (e.g., org.hamcrest.CoreMatchers, etc.)) and start our class with some setup and teardown methods.// filename: ABTestOptOut.java
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.By;
import org.openqa.selenium.Cookie;
import org.openqa.selenium.firefox.FirefoxDriver;
import static org.hamcrest.CoreMatchers.*;
import static org.hamcrest.MatcherAssert.assertThat;
public class ABTestOptOut {
WebDriver driver;
@Before
public void setUp() throws Exception {
driver = new FirefoxDriver();
}
@After
public void tearDown() throws Exception {
driver.quit();
}
// ...
// filename: ABTestOptOut.java
// ...
@Test
public void WithCookieAfterVisitingPage() {
driver.get("http://the-internet.herokuapp.com/abtest");
String headingText = driver.findElement(By.tagName("u")).getText();
assertThat(headingText, startsWith("A/B Test"));
driver.manage().addCookie(new Cookie("optimizelyOptOut", "true"));
driver.navigate().refresh();
headingText = driver.findElement(By.cssSelector("u")).getText();
assertThat(headingText, is("No A/B Test"));
}
// ...
"A/B Test". After that we add the opt-out cookie, refresh the page, and then confirm that we are no longer in the A/B test group by checking the heading text again. We could also load the opt-out cookie before navigating to the page.// filename: ABTestOptOut.java
// ...
@Test
public void WithCookieBeforeVisitingPage() {
driver.get("http://the-internet.herokuapp.com");
driver.manage().addCookie(new Cookie("optimizelyOptOut", "true"));
driver.get("http://the-internet.herokuapp.com/abtest");
assertThat(driver.findElement(By.cssSelector("u")).getText(), is("No A/B Test"));
}
// ...
// filename: ABTestOptOut.java
// ...
@Test
public void WithOptOutUrl() {
driver.get("http://the-internet.herokuapp.com/abtest?optimizely_opt_out=true");
driver.switchTo().alert().dismiss();
assertThat(driver.findElement(By.cssSelector("u")).getText(), is("No A/B Test"));
}
}
?optimizely_opt_out=true we achieve the same outcome as before. Keep in mind that this approach triggers a JavaScript alert, so we have to switch to and dismiss it (e.g., driver.switchTo().alert().dismiss();) before performing our check.Expected Behavior
When you save this file and run it (e.g.,mvn clean test from the command-line) here is what will happen:- Open the browser
- Opt-out of the split tests (either by cookie or appended URL)
- Visit the split testing page
- Grab the header text
- Confirm that the session is opted out of the split tests
- Close the browser
