Data is updated regularly in your mediation group page. At any point during the 90 days, you can make a decision based on the data based on the data to end the test and commit to a variant. Committing to a variant means that you’re choosing to use one of the variants as your mediation group configuration for 100% of your users.
To access and monitor your A/B test:
- Sign in to your AdMob account at https://apps.admob.com.
- Click Mediation in the sidebar.
- Select the mediation group with the A/B test you want to view.
To help you analyze the results of your A/B test, AdMob reports the scaled monthly earnings of Variant A and Variant B. Scaled monthly earnings is an estimate of how much you’ll earn in 28 days if you were to apply this variant to 100% of your users. This provides a fair comparison and a more accurate picture of how each variant is doing.
Check the scaled monthly earnings difference at the top of the A/B test page to understand the magnitude of difference between the variants.
You’ll also see the estimated earnings, eCPM, and match rate of Variant A and Variant B, which will be updated throughout the test. This data also shows how each ad source performs in each variant so that you can easily compare the performance of an ad source across both variants.
Understanding the results
Once a test begins, AdMob compares the scaled monthly earnings in both variants to determine which variant will perform best. AdMob shows the leading variant in the A/B test page once it has determined that a variant has performed consistently and substantially better than the other variant. We’ll also provide a recommended action: “Keep original” (if variant A is performing better) or “Apply variant” (if variant B is performing better).
We’ll also show how confident AdMob is in the leading variant by providing an estimated percent chance (for example, “99% (estimated) chance that Variant A will perform better than B”).
AdMob determines the leading variant and the estimated percent chance using the following criteria:
- How consistently the leading variant performs better than the other.
- The magnitude of the difference between both variants.
- The volume of data available. We recommend choosing an ad unit with substantial traffic to increase the amount of data available during the test. Note that the test must receive at least 10,000 ad requests in order for AdMob to determine the results.
A higher consistency and a larger difference in performance, as well as a high volume of data, helps AdMob increase the estimated percent chance.
If any of these criteria aren’t met, AdMob will not be able to determine a leading variant. In this case, you’ll see no conclusive results in the A/B test page. However, you’ll still be able to view the metrics from the test for each variant.
When the user allocation is set to less than 50%, the data for Variant B is a scaled calculation. This means the data is a fair comparison because AdMob calculates the data for Variant B as if it were receiving an equal user split.
For example, if the user allocation for Variant B is set to 10%, the estimated earnings will be scaled by 9. This allows you to do a fair comparison with Variant A (which has a user allocation of 90%). So if Variant B earns $10, the scaled calculation of Variant B’s earnings is $90.
Take action
We recommend letting your test run for at least two weeks before taking action. You can commit to a variant at any time during the 90 days. Committing to a variant means that you’re choosing to use one of the variants as your mediation group configuration for 100% of your users. Consider the difference in estimated earnings and the percent estimated chance when you decide to commit to a variant.
You can choose to keep Variant A or apply Variant B:
- To select Variant A, click Keep. This means your original mediation group configuration is retained.
- To select Variant B, click Apply. This means that we apply the settings of the variation to your account.
If you don’t commit to a variant by the end of the 90 days, the test stops and Variant A is used on 100% of your users, even if it wasn’t the leader of the test.
You have 30 days after the experiment ends to choose a variant before the test expires and Variant B is deleted.