Splitting Traffic for A/B Testing

Traffic splitting enables you to give different waterfalls, or ad experiences, to a fraction of your users.

Understand Traffic Splitting

Traffic splitting means that when devices make ad requests to one ad unit, you can choose to redirect some or all of those devices to a second ad unit, without making any changes to your app or to the MoPub SDK. When preceded by careful setup of both ad units, redirecting ad requests from one ad unit to another is a way to test different waterfalls and determine which setup maximizes ARPDAU.

In this article, we refer to the original ad unit from which you divert traffic as ad unit A or side A, and the cloned ad unit (containing changes only to the attributes you want to test) as ad unit B or side B.

Uses for Traffic Splitting

  • Split traffic for A/B testing: Traffic splitting is most commonly used for A/B testing of different waterfall configurations, and is especially useful in maximizing the potential of Advanced Bidding.

  • Test large changes by splitting traffic: Without traffic splitting, if you wanted to test the effects of significant changes to your waterfall (for instance, “flattening” your waterfall to a single priority), it would require a lot of time-consuming work in the Publisher UI. If the results were not beneficial, you’d have to invest the same time and effort to revert your changes.

    With traffic splitting, you can set up your ‘B’ ad unit with your changes, and MoPub can then send 100% of users from ad unit A to ad unit B. This allows the original waterfall (that of ad unit A) to keep its state, and in case of negative results, you can delete the split and restore all traffic to the waterfall for ad unit A.

Traffic is Split at the Device Level

Traffic is split based on the device ID, using the IDFA when available. In the absence of an IDFA, we use the MoPub ID, a device ID that is unique to that user in that app.

Waterfalls Must Be Independently Measurable

Splitting traffic causes users to experience different waterfalls. You must set up your waterfalls carefully, ensuring that the revenue results we collect for ad units A and B be as accurate as possible.

This is easy for Marketplace impressions because we know the exact amount for which each impression is sold. For networks, on the other hand, we use an estimate based on either Auto CPM or a CPM you have input yourself. These CPMs must be as accurate as possible. Network CPMs in A/B-tested waterfalls should be either:

  • Fixed CPM both in the MoPub UI and in the network’s setting: Fixed CPMs mean that the network has configured their placement to fill if and only if they are willing to pay the fixed amount for the impression.

    OR

  • Auto CPM with different placement IDs in each ad unit: This means that if a network observes different performance in the different sides of the test, we can capture that difference in our results.

Splitting Traffic

A Subset of Users Will Experience Both A and B

Note that a small portion of users will not not reliably fall permanently into both A or B, but will possibly switch each day. This is because we use the device ID to determine which bucket a user should be in, so in the cases where there is no IDFA we must rely on the MoPub ID. On older SDKs, the MoPub IDs are reset every 24 hours, so there’s a chance that these users will begin to fall into a different bucket. In a typical case, this is a small fraction of users, so we should still be able to observe differences in overall results.

Set Up Traffic Splitting

The high-level steps for splitting traffic are:

  1. Identify the running ad unit against which you will test (ad unit A).
  2. Create ad unit B, making sure it is identical to ad unit A, except for the changes you want to test. Verify that the waterfalls for ad units A and B are identical in every detail except the details you’re testing.
  3. Activate the split. The split will take effect immediately.

All network line items must be either:

  • fixed CPM on the network side, and have static CPMs on the MoPub side that reflect that,

    OR

  • enabled with Auto CPM, using unique line items, and unique network placement IDs within them.

    (For unsupported networks that have neither an Auto CPM or a flat CPM model, make unique line items with unique network placements, and try to update your CPMs regularly.)

Create or Edit a Traffic Split in the Publisher UI

To set up an A/B test within an app, start with your running ad unit (ad unit A) and create a traffic split from it, specifying the test ad unit to which you want to direct traffic (ad unit B), and how much traffic to send there.

  1. In the Publisher UI Apps tab, select the app for your A/B testing.

  2. In the App Details page, click on the running ad unit from which you want to divert traffic (this will be your ad unit A).

  3. In the top of the Ad Unit Details page for ad unit A, expand the drop menu next to Edit Ad Unit and select Edit/Create Traffic Split.

    Splitting Traffic

  4. In the Traffic split page, indicate the split amount, and specify the test ad unit to which you want to direct the remaining traffic (this will be your ad unit B). Ad unit B must have the same ad format as ad unit A. You can specify how much traffic to send from ad unit A to B, choosing one of three values: 0%, 50%, or 100%.

    Splitting Traffic

    Click Create split. The split takes effect immediately.

For existing splits, you can use this page to either change the portion of traffic being split, or delete the split. If you delete a split, 100% of traffic reverts to ad unit A.

Interpret A/B Test Results

The target metric for A/B testing depends on the test type, as summarized in this table:

Test Type Primary Metric Secondary Metrics
Waterfall optimization ARPDAU (Internal - ARPreqDAU) eCPM
Imp/DAUs
Advanced Bidding ARPDAU  
Latency Analysis Waterfall Latency Latency (Demand)
Timeout
Fill Rate (Demand)
Show Rate (Demand)
Render Rate

For definitions of these metrics, refer to our MoPub Analytics Platform Performance Metrics definitions.

Look for a difference of at least 5% in order to be confident that the winner is actually having a positive effect. That is, with a large enough sample size (we recommend 85K per bucket), you should see a 5% increase in ARPDAU from ad unit A (original) to ad unit B (test).

If you have fewer users per bucket, you need a larger increase in ARPDAU to conclude positive results. For example, for 30K users, look for a 10% increase in ARPDAU; for 50K users, look for approximately a 7.5% increase in ARPDAU.

Best Practices for Traffic Splitting

  • Start with an A/A test: We recommended that you keep your ad unit B setup identical to ad unit A for a week to ensure that performance between the two is comparable (this is known as an A/A test). Then make the desired changes to ad unit B and assess performance over time.
  • Replicate, then alter: Replicate your side A waterfall exactly, then make changes to it on the B side. The two waterfalls should be identical except for the changes you want to measure.
  • Careful verification: Once you have set up the waterfall on your B side, check that its network revenue and impressions shown on the Mopub platform match the data on the network reporting side.
  • Sample size: We recommend at least 85K unique devices in each bucket.
  • Test duration: We recommend that you run each test for a minimum of two weeks after calibrating the network placement IDs, which may take up to a week.
  • Controlled environment: Make sure that no changes are made to the side A waterfall during testing to ensure meaningful results.
  • Network placement IDs: Pay very close attention to the ad unit IDs/placement IDs for networks. There should be new placement IDs for networks to ensure that waterfalls are independently measurable; otherwise, the results will not be fully accurate.
  • Perform sanity checks: Do both buckets have the same number of users? Does this hold over time?
  • MoPub Analytics: Use MoAnalytics to check for ARPDAU changes and other metrics.

A/B Tests Provide Directional Rather Than Direct Results

Even when we are as careful as possible, the results of A/B tests should be considered “directional,” rather than exact. A subset of users may fall into different buckets each day. To run true A/B tests, the entire ad tech ecosystem would need to keep their As and Bs independent. We try to do this with networks by assuming that they will treat two placement IDs within the same app independently. In most cases, networks do not do this.


Last updated March 30, 2021

TWITTER, MOPUB, and the Bird logo are trademarks of Twitter, Inc. or its affiliates. All third party logos and trademarks included are the property of their respective owners.

© 2021 MoPub (a division of Twitter, Inc.)