Test your custom bidding algorithm

Testing your new custom bidding algorithm helps you evaluate its performance and make adjustments to optimize your return on investment:

• Custom bidding testing sandbox: You can use the testing sandbox to validate your algorithm before assigning it to a live campaign for the first time.  
• Custom bidding A/B experiments: You can improve your algorithm once you assign it to a live campaign to continue optimizing as you gather live impression data.

Once you assign your algorithm to a campaign, you can continue to track and optimize the performance of your algorithm by creating and evaluating reports.

Topics in this article

• Validate with the testing sandbox
• Optimize custom bidding with experiments
• Track custom bidding performance with reports

Validate with the testing sandbox

You can use the custom bidding test sandbox to estimate:

• the performance of your custom bidding script or custom bidding goal
• if your algorithm is likely to meet the minimum data requirements
• if your algorithm performs as you expected

When you use the custom bidding sandbox, it tests your custom bidding algorithm using a maximum sample of 10,000 historical impressions from the last 30 days from your chosen advertiser. This helps you determine if your custom bidding algorithm is estimated to perform well or if you need to make adjustments. You can also use the sandbox to test different bidding strategies and compare the results of different custom bidding algorithms.

Sample size

You can improve accuracy when testing your algorithm by having a large and diverse dataset by checking the minimum data requirements.

If your sample size doesn’t meet the minimum data requirements, you can still test your algorithm but it may estimate lower performance which can change as more data becomes available.

You can select one of the following views when evaluating the results of your test:

• All impressions in sample: View all impressions to understand the overall distribution of the estimated impression values. 
• Only conversions: View only impressions with conversions. This view includes all conversion events in your advertiser based on the default attribution model. Useful when conversions are part of your campaign’s goals.
• Only clicks: View only impressions with clicks. Useful when clicks are part of your campaign’s goals.

The following insights are available:

Insight	Description
# of impressions in sample	Shows the total number of impressions included in the sample within the last 30 days. Includes the percent of impressions that: Impr. with errors: Shows impressions that resulted in an error when evaluated by your algorithm. Not scored: Shows the percent of impressions excluded by your algorithm. Impr. has no value: Shows the percent of impressions with a negative or zero score. These are impressions that didn’t meet your algorithm’s criteria and aren’t estimated to be relevant to your campaign goals. Positively scored impr: Shows the percent of impressions with a positive score, estimating these impressions to be valuable and relevant to your campaign goals. Example: Impressions in a sample You can use the number of impressions to estimate how much of the total impressions are valuable. Impressions in sample: 10,000 Not scored: 5% Positive scores: 70% Impression with no value: 25% In this example, out of a sample size of 10,000 impressions: 5% of video impressions were not scored to focus your algorithm to exclude video impressions when training.  This means that 70% of impressions are estimated to be valuable and relevant to your campaign’s goals. While 25% don’t meet your algorithm’s criteria and aren't estimated to be relevant to your campaigns goals.
Impr. Value/Cost	Measures how effective your algorithm will be in generating impressions relevant to your campaign’s goals. The value is calculated by dividing the total value of impressions with the total cost of the sample. Example: Impression value/cost You can use the impression value over cost to estimate the estimated return on investment from your custom bidding algorithm. Total impression value: 1000 Total campaign cost: $500 Impression value/cost: 2 This means that for every dollar spent on the campaign, your algorithm is expected to generate 2 times as much value generated as it costs, calculated as 2 divided by 1.
Score distribution	Shows score distribution and groups impressions by score range. Shows individual scores if there are more than 10 unique scores. The score metrics are also shown in a table and you can download the .csv file to view the individual score that makes up the score distribution.  Includes the count for: Unique scores: The number of unique score values defined by your algorithm. The scores defined by your algorithm are distributed across the sample group during testing.  Max score: The maximum score value defined by your custom bidding algorithm. Min score: The minimum score value defined by your custom bidding algorithm. Example: Score distribution You can use the score distribution to troubleshoot execution errors. For example: If your custom bidding algorithm defines parameters for what impressions score a value of 1, 2, 3, 4, and 5. But the score distribution graph only shows 3 unique scores. This may indicate an execution error and you may want to check your custom bidding algorithm. Note: Your custom bidding algorithm may produce score values outside the values that’s shown using training data once assigned to a live campaign.

Optimize custom bidding with experiments

You can set up an experiment to compare an algorithm with different algorithms to determine which one performs better. 

For example:

You can compare the total custom impression value of a new algorithm with other algorithms to determine which one generates more impressions. 

A/B testing is a powerful tool that can help you improve your algorithms. Learn more about how to set up an experiment.

Track custom bidding performance with reports

You can create offline reports and view in-line metrics to evaluate how your custom bidding algorithm performs toward your KPI and cost.

You can use the following available metrics in your reports to evaluate custom bidding performance:

• Impressions with custom value
• Impressions with positive custom value
• Total impression custom value
• % impressions with positive custom value
• Estimated total cost for impressions with custom value (advertiser currency)
• Estimated cost per mile (CPM) for impressions with custom value (advertiser currency)
• Impression custom value / cost
• Impression value per 1000 impressions

You can view the full descriptions and learn more about metrics in reports.

Create an offline report

You can set up an offline report by choosing the Custom bidding report template, which includes the algorithm and algorithm ID dimensions.

You can select the impression custom value/cost and other available metrics to evaluate your algorithm's performance toward your KPIs given the cost.

Since an advertiser can have more than one algorithm at a given time, you can use the filters section to only include the specific algorithm you want to evaluate. Learn more about offline reporting.

Reporting time frame

Try to choose a date range when generating a report. For example: The last 7 days or the flight days of the campaign.

Algorithms need to be active for 2 or more days to generate data for reports. Reports don’t include data from the current and previous day due to the time it takes to record and score impressions.

View in-line custom bidding metrics

You can view performance metrics for all insertion orders and line items that use custom bidding.

To view in-line custom bidding metrics:

1. Go to Campaigns.
2. Select the Insertion orders.
3. Choose an insertion order.
4. Use the drop arrow above the table and select Performance view.
5. In Performance view, scroll the table to the right to view the Custom Bidding column.

To create a custom view that displays more custom bidding metrics:

1. Go to Campaigns.
2. Select the Insertion orders.
3. Choose an insertion order.
4. Use the drop arrow above the table and select Create new custom view.
5. Under the Custom Bidding section, select the metrics you want to include.
6. Select Apply.