About experiments

Running an experiment allows you to compare one of your ad settings against a variation of that setting to see which performs better. Experiments help you to make informed decisions about how to configure your ad settings, and can help you to increase your earnings. Here are some examples of the types of experiments that you can run:

  • Compare the performance of text-only ad units to ad units that have both text and display enabled
  • Analyze the effect of changing the color of ad text on ad performance
  • Determine the impact on your revenue of using ad serving, sensitive and/or general category blocks

There are two ways to create experiments. You can either:

Life of a typical experiment

A typical experiment that you create yourself, follows these steps:

1. Preparing for your experiment

Before you create your experiment it’s a good idea to decide exactly what you’d like to test in the experiment.

  • Ad units: All ad type and text ad style settings are available for experiments. If there are multiple changes that you would like to try (e.g., change the text ad border color and background color), you may want to run two different experiments - first an experiment for the border color change and then an experiment for the background color change.

    Note that the following ad units are not eligible for use in experiments:
  • Allow & block ads: The following ad blocking settings are available for experiments:

Note that there are some limitations on the number of experiments that you can have running at any one time:
  • You can either run ad unit settings experiments (maximum one experiment per ad unit) or a single global “Allow & block ads” experiment, but you can’t run both types of experiment at the same time.

2. Creating your experiment

When you create an experiment you:

  • Select the original ad setting that you want to compare the experiment variation against.
  • Select which settings you’d like to change for the variation.
  • Decide whether you want Google to optimize the traffic split for your experiment. We highly recommend that you let Google optimize the traffic split for you, as this will maximize the efficiency of your experiment.
    Learn how Google optimizes the traffic split for experiments
    We use a multi-armed bandit approach to optimize the traffic split for experiments. Each day, we take a fresh look at your experiment to see how the original and the variation performed, and we adjust the amount of traffic that each receives going forward. If one appears to be doing well it gets more traffic, and if one is clearly underperforming it gets less. The adjustments we make are based on a statistical formula that considers sample size and performance metrics together, so we can be confident that we’re adjusting for real performance differences and not just random chance. As the experiment progresses, we learn more about the original and the variation, and so do a better job in choosing the best-performing variation. Experiments based on multi-armed bandits are typically much more efficient than "classical" A-B experiments based on statistical-hypothesis testing. For more detailed information, see Multi-armed bandit experiments in the Google Analytics Help Center.

The “create experiment” page guides you through each step, and provides you with more information. When you’re finished setting up your experiment, you can start it.

3. Monitoring your experiment

Once your experiment is up and running, it’s up to you to monitor its progress.

After 24 hours, you should see data in both the:

4. Choosing the winner of your experiment

Over time, you should see either the original ad setting or the variation outperform the other. To help you decide the right time to choose your winner, we calculate a confidence score for the two settings. This score indicates how likely the original or the variation is to be the better performing (i.e., the highest earning) setting in the long term.

We recommend that you wait until one of the scores reaches at least 95% (indicated by five blue dots) before you choose that setting as the winner.
  • If you choose the original as the winner, then the settings of the original are retained.
  • If you choose the variation as the winner, then we apply the settings of the variation to your account.

In either case, we stop splitting your traffic and collecting data, and your experiment ends.

If you have further questions about experiments, see the Experiments FAQ.
Was this article helpful?