Life of a Typical Experiment

The amount time an experiment takes depends on how much time you invest in evaluating your current pages, designing alternative test pages, and monitoring the incoming data. Content Experiments collects data for a minimum of three days and a maximum of three months.

1. Preparing for Your Experiment

Before you take any action in Content Experiments, you need to:

  • Decide on the overall goal of your experiment (e.g., increase traffic, sell more widgets)
  • Optionally, set up a corresponding goal in Analytics
  • Choose the page you want to test
  • Figure out which elements on that page you want to test
  • Create several versions of the page
  • Make all test versions of the page live

2. Setting Up Your Experiment

When you set up your experiment, you:

  • Select the Analytics goal or metric you want to use
  • Choose what percentage of your traffic you want to include
  • Specify which original page and which variations of the original you want to include
  • Add experiment code to your original page
  • Verify that the Analytics tracking code is installed on the original and all variation pages

The wizard guides you through each step, and provides you with more detailed information. When you’re finished setting up your experiment, you can start it then, or save it and start it later.

3. Letting Your Experiment Run

During your experiment, the percentage of traffic that you specified sees either your original page or one of your variation pages. Content Experiments collects information about how well each page encourages users to accomplish the goal or you chose or how well it improves the metric you chose, and then calculates which pages are performing best, how they compare to your original page, and how certain you can be that the variation in performance is not due to chance.

If one variation is performing consistently better than the others, Content Experiments diverts more traffic to that variation. Read more about the statistical model we use.

After your experiment has been running for a few days, you can see data both on the experiment list and in the experiment report.

If you modify an experiment to reduce the percentage of users who are included, then any users who have already been exposed to an experiment page continue to see that page when they visit your site.

If you modify an experiment to disable a variation, any users who were exposed to that variation are then served the original page when they visit your site. Also, data for the disabled variation is not included in calculations for the experiment.

Experiments can run up to three months. If, however, there is one page that is clearly outperforming the others, the experiment stops early. After your experiment ends, all users see the original page until you replace it.

If you stop an experiment, you cannot restart it.

4. Evaluating Your Experiment and Deciding how to Respond

If your experiment has a page that is clearly better than the others, the obvious response is to replace the original page with the one that performs better. You can then run additional experiments to see whether you can improve upon that new page.

If your experiment does not produce a page that is clearly better than the others, you might want to revise your experiment or launch a new one. For example, if you did not get enough traffic to your pages to get a statistically significant result, you could try the same experiment again but increase the percentage of users who see your experiment. Or, you might run a follow-up experiment where you test other more obvious differences in the pages.

Was this helpful?
How can we improve it?