How optimization effectiveness is measured
Optimization uses a control group and an optimized group to measure the performance lift (for the Performance module) or eCPM lift (for the Revenue module) that you can achieve by optimizing your ad delivery. Users who visit your site are automatically placed into either the optimized or control group at random, ensuring that the only difference between them is whether they receive optimized ads. Typically, 80% of users will be in the optimized group, and 20% in the control group. However, you may see a lower percentage of optimized impressions if you’re overbooked or using as fast as possible delivery.
Performance lift measures the increase in CTR as a result of optimization, and eCPM lift measures the increase in eCPM as a result of optimization, with the added benefit of providing freed up impressions to your network. Learn more about the differences between revenue and performance optimization.
Using a control group allows you to track performance lift by comparing ad delivery to the control group with delivery to an optimized group. Any external factors are removed so that the only difference between the control group and the optimized group is that users in the optimized group receive ads delivered using the optimization algorithm. Ads delivered to the control group run in parallel to the optimized ads. Through this test methodology, you can be confident that your results are consistent and accurately attributed to the optimization algorithms, and not other outside factors.
Understanding lift metrics
When you view an optimization report, the Lift column displays four different possible outcomes for each order and line item, for each applicable module type:
Positive: The CTR or eCPM has increased. This is the desired outcome of optimization. If the amount of lift is low, it’s likely that the ad is behind schedule or overbooked.
0%: There was a small, but statistically insignificant increase in CTR or eCPM as a result of optimization. Why do I see zero or negative lift for my ads?
Negative: There was a statistically significant decrease in CTR or eCPM. Why do I see zero or negative lift for my ads?
N/A: There isn’t enough data in the control and/or optimized groups. Each line item requires a minimum number of clicks (10 in the control group, 40 in the optimized group) in order to calculate lift. Why is optimization data unavailable for some of my line items?
Sometimes an order can experience positive lift while one or more individual line items within it shows lift that is negative or statistically insignificant (N/A).
If an order contains line items that are N/A, the order’s overall lift is calculated based on the remaining line items, taken as a weighted average. For example, if an order contains two line items where the lift for line item A is 150% and for line item B it is N/A, the lift for the order itself will be 150%.
Line items with 0% lift are averaged together with the other line items in the same order and weighted by the number of optimized impressions served. For example, if an order contains two line items where the lift for line item A is 150% and for line item B it is 0%, the average lift for the order will be 75% (assuming both line items served evenly).
You might also notice a discrepancy in the number of impressions for line items versus the aggregate number of impressions for the order.