Glossary of terms

The language, lexicon and lingo of testing.

In order to get the most out of Optimize, it’s important to understand a few concepts and terms.

In this article:

A/B test

(A/B/n test, bucket test, split test)

A randomized experiment using two or more variants of the same web page (A and B). Variant A is the original and variant B through n each contain at least one element that is modified from the original. Learn more about how to create an A/B test.

Baseline conversion rate

A modeled conversion rate attributed to the variant based on the percentage of sessions that resulted in a conversion (i.e. the experiment objective being met).

Combination

In a multivariate experiment, which contains multiple sections, combinations are the number of variants in each section multiplied by each other. See the example in Multivariate test. In a multivariate test with multiple sections, a combination is the experience created from each section's variants. For example, headline 1 with hero image A.

Container

A container holds all of the configuration information for a website’s experiments. Typically, you set up one container per Analytics 360 property (which should correspond to one web domain). The container snippet (JavaScript code) needs to be added to the website pages being tested.

Editor page

The web page used to create variants. Note that this may differ from your URL targeting rules.

Estimated conversion rate

(Modeled conversion rate, Projected conversion rate)

Based on the data to date, this is the conversion rate you can expect to see in the future. Because Optimize takes into account the day of week and other sections into its statistical simulations, this projected rate may differ from the historical rate measured so far.

Experiment

An experiment is used to test variants of web pages to determine which is most effective at achieving an objective. Examples include A/B, redirect, and multivariate experiments.

Experiment sessions

Any session that has seen an experiment variant, and all future sessions for that user for the duration of the experiment (whether they see the variant again or not). Learn more about sessions in the Analytics 360 help center.

Fractional factorial experiment

Only a subset of combinations is served in order to find results quickly. These experiments sacrifice the detailed interaction analysis available with full-factorial experiments for greater speed in finding results.

Full factorial experiment

Every combination of a multivariate experiment is served to users, to understand all interactions between variants. These generally take longer to reach significance but also allow for detailed measurement of interactions, as well as offering the ability to serve the winning combination.

Improvement

For a given objective, the difference in conversion rate - measured as a percentage - between the variant and the baseline.

Leader

The variant that maximizes the objective.

Multivariate test

(MVT, compound test)

An experiment that tests two or more sections to understand their effects on each other. For example, variants of a headline can be tested at the same time as variants of a hero image. Instead of showing which page variant is most effective (as in an A/B test), a multivariate test identifies the most effective combination of variants. Rather than the two or three page variants found in simple A/B tests, multivariate tests frequently test multiple variants of multiple page elements simultaneously. Learn more about how to create a multivariate test.

Objective

The website functionality you wish to optimize.

Original

Your current web page, prior to any modification.

Probability to beat baseline

The probability that a given variant will result in a conversion rate better than the original's conversion rate. Note that with an original and one variant, the variant's Probability to beat baseline starts at 50 percent (which is just chance).

Probability to be best

The probability that a given variant performs better than all other variants.

Redirect destination

A separate and unique web page used as a variant in a redirect experiment.

Redirect test

(Split URL test)

A redirect test is a type of A/B test that allows you to test separate web pages against each other. In redirect tests variants are identified by URL or path instead of an element(s) on the page. Redirect tests are useful when you want to test two very different landing pages, or a complete redesign of a page. Learn more about how to create a redirect test.

Section

(Factor, Element)

A single element of a web page (e.g. a headline, image, or button) that is modified to create variants. An A/B test only contains one section (with one or more variants); in a multivariate experiment, multiple sections are tested at the same time.

Variant

(Variation, Level)

A variant can be anything from a change to a single element, or changes to multiple elements, or a totally different page in an experiment. In an A/B experiment the unit of variant can be a web page or an element of a web page. In a multivariate experiment you have multiple variant of each section.

Was this article helpful?
How can we improve it?