An A/B test is a randomized experiment using two or more variants of the same web page (A and B). Variant A is the original and variant B through n each contain at least one element that is modified from the original.
Before you can start testing, you need something to test on. Rather than randomly testing items on your homepage or adapting your checkout flow, start small. Test a change to a Call To Action (CTA), change the color of a button, or remove an extraneous form field. Once you're comfortable creating variants and experiments, you can expand the scope of your testing.
Create a hypothesis
Before creating your first experience you need to identify a problem, then create a hypothesis (backed up by data, of course) about what you can change to improve it.
What's the problem that you want to solve? Have conversions dropped off? Have traffic patterns changed? Have your demographics shifted? A close examination of trends in your Google Analytics behavior reports is a great place to start.
Once you've identified a problem, assemble a team within your organization and solicit their opinion about the cause of the problem. Use feedback from this cross-functional team to form your hypothesis, an educated guess that you'll validate or invalidate with experimentation.
"Changing the color of the 'Add to cart' button from blue to green will increase revenue by 10 percent."
After you've identified a problem (low conversions), and worked with your team on a hypothesis (changing the button color) you're ready to test your hypothesis on your website.
Create an A/B test
In this experiment, we’ll test changing the color of a button:
- Go to your Optimize Account (Main menu > Accounts).
- Click on your Container name to get to the Experiments page.
- Click CREATE EXPERIMENT.
- Enter an Experiment name (up to 255 characters).
- Enter an Editor page URL (the web page you'd like to test).
- Click A/B test.
- Click CREATE.
Use a fully resolved URL in the Editor page URL field. Redirects aren't supported here.
The variants card
The top of the experiment page includes the variants card. This where you'll create the specific changes to your web page that you wish to test, called variants. You can create as many variants as you wish to test against your original page (the Editor page in Optimize).
Create a variant
To get started, click CREATE VARIANT (bottom right), enter a variant name, then click ADD. Repeat this process to create additional variants. When finished, you'll see a list of your new variants on the variants card.
To start making changes. click anywhere in the variant row (which will say "0 changes"). This will launch the Optimize visual editor – an overlay on top of your editor page consisting of two components: the app bar (at the top of the page) and the editor panel (floating in the lower right).
- Click on any web page element you wish to edit (e.g. a button).
- Use the editor panel to make a change (e.g. change the button color).
- Click SAVE.
- Continue making edits as necessary.
- Click DONE.
Learn more about the visual editor.
All variants are weighted equally by default in Optimize. A visitor who is included in your experiment has an equal chance of seeing any of your variants. If you want to direct more, less, or even all of your traffic to a specific variant, you can adjust your variant weights on the experiment details page.
Learn more about variant weighting.
The configuration card
Configure your experiment objectives on the OBJECTIVES tab:
- Select a Google Analytics view.
- Select a Primary Objective.
- (Optional) Click + ADD AN OBJECTIVE to add an objective. Learn more.
- Add a description and hypothesis.
- Click SAVE.
Configure who and when to target on the TARGETING tab.
Who to target
The Who section of the Targeting tab is used to select the visitors to whom you wish to target your variant. Enter a numeric value (in tenths of a percent) or use the slider to specify the percentage of your visitors to include in your experiment.
When to target
The When section of the Targeting tab determines where the experiment is shown. When is evaluated each time a user visits the experiment page. Use the rules to set where your experiment is seen. To create a targeting rule, click AND, then select a targeting rule type.
You must create either a URL or path targeting rule to start your experiment. The quickest way to get started is to create a URL matches rule with the same Editor page URL that you used when creating your experiment.
Optimize includes the following targeting rule types, which you can read more about in the following articles:
- URL targeting
- Google Analytics Audiences targeting
- Behavior targeting
- Geo targeting
- Technology targeting
- First-party cookie targeting
- Query parameter targeting
- Data layer variable targeting
Learn more about targeting.
Start your experiment
Click START EXPERIMENT. When the status field says "running," your experiment is live on the web. Most updates happen within a minute.
How long should your experiment run?
Keep an experiment running until at least one of these conditions has been met:
- Two weeks have passed, to account for cyclical variations in web traffic during the week.
- At least one variant has a 95 percent probability to beat baseline.
Experience management options
Optimize provides several ways to manage your experiences, whether in draft, running, or ended status. This article explains how to search inside containers and how to access functions like edit, copy, stop, and archive.
To monitor a running experiment or see the results of a concluded experiment, click the Reporting tab at the top of the experiment detail page. The report is broken down into a series of cards that contain data about your experiment, including its status and how your variants perform against your objectives.
In addition to the reports included in Optimize, you can also see Optimize reports in Google Analytics. Sign in to Google Analytics, select the Reporting tab and select Behavior > Experiments in the report navigation. Learn more about Optimize reports.
In this video we'll show you how to create a new A/B test in Optimize in three easy steps, including how to create a variant, target an audience, and choose an objective.