Run an Experiment

Prepare

Identify Your Business Objective

Before you start creating different test versions of your web pages, define your objective in the experiment. For example:

  • Increase revenue for product sales on your site.

  • Raise awareness of your brand and drive traffic to your brick-and-mortar stores.

  • More pageviews for content pages so you can justify your rates to advertisers.

Clarify How Your Site can Meet that Objective

Once your objective is clear, determine how you want your website to help you meet that objective.

For example, if your objective is to raise awareness of a new car model and drive traffic to your dealerships, you might have a goal path like the following:

Landing page > Specifications > Build your own > Search dealer inventory > Request a quote

In this example, you want a user to see the landing page for the new model, check out the specifications, create his or her own version of the car given the available options, search dealer inventory for matching stock, and then request a quote from a local dealer.

If your objective is to improve a metric other than goal completion, like decreasing bounce rate or increasing session duration, design your variation pages with those objectives in mind.

Set Up Your Analytics Goal (Optional)

Create an Analytics goal, with a goal path that corresponds to the pages you've identified, and with a URL or event goal that corresponds to the action you want users to take on your site (go to a page or engage in an event).

Following the example of a new car model, the landing, specifications, build, and search-inventory pages make up your goal path. The request-a-quote page is your goal.

With a longer goal path like this, you have different options:

  • Test the entire path from landing page to goal page. Users may not take a linear path from your landing page to your goal page, and instead return from the path pages to your landing page, and then eventually to your goal. If a linear progression through the pages is less important than ensuring an eventual goal completion, you can set up your Analytics goal to include all the steps, and then test the entire path in one experiment.

    If you design your test this way, you can also use the Goal Flow report to see the various loop-backs and skipped pages in your goal path.

  • Test each segment of your path. If a linear progression through your goal path is important, you can run successive experiments for each stage of the path. For example, set up a goal and corresponding experiment to test which version of the landing page works best to get users to the specifications page; and then a second goal and experiment to see which version of the specifications page works best to get users to the build page; and so on.

Choose Your Original (Test) Page

Start by testing the first page in the path you want users to follow. In our example of a new car model, start with the landing page for the new model as your original page. Users might arrive here from your overall brand page, search results, an online ad, or because you advertised the URL in an automobile magazine.

If you're running successive experiments to test path segments, then the original (test) page is different for each experiment: landing page > specifications page; specifications page > build page; and so on.

Design Variations of Your Original Page

You can include up to 10 variations of a page in an experiment.

Using your original page as the baseline, you can create the following kinds of variations:

  • Images: Find out which type of graphics incite the best user response: for example, photos of people (e.g., attractive models, well known spokespersons), product photos, stylized images of your products, simple icons.

  • Text: Find out what kind of textual approach works best: straightforward, technical, casual, comic. Is shorter copy more effective than longer copy? Does a phrase like Learn more generate more progress toward a goal than a phrase like Click here.

  • Image or Text, or Image & Text: Discover whether users are more influenced by images than by text, or whether they prefer a combination of both.

  • Too much, too little, just right: Do users respond best to a page saturated with design elements, numerous options and calls to action, a page that includes only the minimum amount of information necessary to move them along the goal path, or is the sweet spot somewhere in between?

  • Emphasizing one product feature over another: If you're marketing a new car to driving enthusiasts, are they more concerned with horsepower and torque, or are chassis design and handling the stronger incentives?

Prepare Your Pages for the Experiment

Once you have created all the variations of your original page, you then need to:

Decide what Percentage of Users to Include in the Experiment

There are three considerations to make when you choose what percentage of your site’s users to include in your experiment:

  • The volume of traffic to your conversion page
  • The number of variation pages in your experiment
  • The riskiness of your experimental changes

The lower the volume of traffic to your conversion page, and/or the more variation pages you include in your experiment, the longer it takes to get statistically significant results. Under these circumstances, you can increase the percentage of users who see your experiment in order to get results more quickly. You have the option to increase the percentage of users while an experiment is running.

If your experiment includes any significant departures from business as usual, for example, changes to a longstanding logo or slogan, then you can include fewer of your users in the experiment to test the initial reaction but limit any possible negative side effects. You have the option to remove a variation of a page while an experiment is running.