Using A/B testing in mediation

Maximize your mediation performance by A/B testing different mediation group configurations #A/Btests #waterfalladsources #biddingadsources

Use A/B tests to make informed decisions about how to set up your ad sources in mediation, which can help you increase your earnings. 

How A/B tests works

A/B tests let you evaluate 2 variations of a mediation group and compare the results. Use an existing mediation group and create a variation of it to see which performs better. The test splits your app’s users between your existing mediation group setup and the variation, so that their performance can be measured side by side.

You can change your mediation group set up in Variant B by:

• Adding or removing ad sources to see their impact on your mediation group
• Using a different waterfall order by changing the manual eCPM value of ad sources
• Enabling or disabling optimization on waterfall ad sources
• Changing the number of times an ad source is called

Note: The test must receive at least 10,000 ad requests before AdMob can determine a result.

Click to view an example test

The test runs for 90 days, but you can stop at any time. Based on the results of the test, you can decide to use the original mediation group setup (Variant A) or the variation (Variant B). 

Once you start the A/B test, you won’t be able to make changes to Variant A or Variant B until the test is stopped or has ended. This helps keep the test fair and accurate.

Set up an A/B test

Complete the following steps:

1. Sign in to your AdMob account at https://apps.admob.com.
2. Click Mediation in the sidebar.
3. Next to the mediation group that you want to create an A/B test for, click Set up A/B test. 
4. Enter a name for your test. 
5. Set the user allocation. You can choose 50%, 10%, or 1%. This determines the percentage of your users that Variant B will receive. For example, if you set the user allocation to 10%, Variant A (your existing mediation group setup) will receive 90% of your app’s users, while Variant B will receive 10%. 
6. Change the ad source configuration in Variant B. You can:
   • Add or remove ad sources to see their impact on your mediation group
   • Change the order of your waterfall by changing the manual eCPM value
   • Enable or disable optimization on waterfall ad sources
   • Change the number of times an ad source is called
7. To ensure accurate reporting, we recommend using unique ad unit mappings when using the same ad source in Variant A and B. For example, use a mapping that isn't already used in variant A.
   • Click Edit in the Ad unit mapping column. In most cases, the mapping details are available in your account with the ad network. 
   • Learn more about adding waterfall ad sources and bidding ad sources. 
8. Click Save.
9. Click Start test. 
   Once you start the A/B test, you won’t be able to make changes to the mediation group, including Variant A or Variant B, until the test is stopped or has completed. Any change while the experiment is running may impact the results. This means you can’t edit ad units, location targeting, etc. 
10. Click Start in the confirmation box, and the test will begin in a few minutes.  

By default, the test will run for 90 days but you can choose to take action and end the test based on the data available. Learn how to analyze your A/B test & take action.

Best practices

• Choose ad units that have enough traffic for the test to determine a result. A test must receive at least 10,000 ad requests before AdMob can determine a result. 
• Change only one setting at a time so you can more accurately attribute any performance difference to a particular change.
• Run the test for at least two weeks. More data helps us more confidently estimate which variant will perform best.
• Consider using different mappings in Variant A and B. This helps keep test data and reporting accurate for your waterfall ad sources.