YouTube brand safety: Description of methodology

Google is seeking Media Rating Council accreditation stating that YouTube in-stream video ads adhere to the industry standards for content level brand safety processes and controls. This applies to YouTube in-stream video inventory purchased through Google Ads, Display & Video 360, and YouTube Reserve services.

The industry guidelines were developed in an effort coordinated by the Interactive Advertising Bureau (IAB), the Media Rating Council (MRC), the American Association of Advertising Agencies (4A’s), and the Association of National Advertisers (ANA). YouTube was audited against these guidelines by an independent third-party auditor engaged by the MRC.

About Google’s YouTube brand safety accreditation audit 

The accreditation audit is focused on YouTube’s brand safety processes and controls applied to YouTube’s in-stream video ads. This relates to:

  • The measurement and reporting of digital in-stream video ad impressions and the related viewability metrics across desktop, mobile web and mobile application environments net of general invalid traffic (GIVT), sophisticated invalid traffic (SIVT), and brand unsafe content across Google Ads, Display & Video 360, and YouTube Reserve services
  • Reporting of brand safety error rate at the YouTube platform level

The following ad types & device types are in-scope for this accreditation:

  • Desktop, mobile app, and mobile web
  • Skippable in-stream ads
  • Non-skippable in-stream video ads
  • Bumper ads

Exclusions from the accreditation audit

The following are excluded from the audit:

  • All non-YouTube and Google video partners inventory
  • Over-the-top devices (for example, connected TVs)
  • Non-in-stream ads (for example, Masthead and Discovery ad formats)
  • Performance optimization targeting tools (for example, topic classifiers, geolocation, keywords, or audiences)
  • Specific YouTube channel targeting (or exclusion)

Brand safety methodology

Google’s YouTube brand safety video classification methodology encompasses policies that determine what content is permitted on YouTube and eligible to monetize via ads, technology to analyze the tremendous amount of video on the platform and a team of human raters to augment those automated classifications. 

The initial safety layer is the YouTube Community Guidelines, which are out of scope of this accreditation audit but provide common-sense rules about what is allowed on YouTube. Before their content is considered for monetization on YouTube, creators must first meet the application criteria for the YouTube Partner Program

When a video is uploaded to YouTube, the video’s attributes (such as visual data, audio data, comments, and other metadata) are automatically analyzed by our machine learning models to classify the video against our advertising policies (for example, Advertiser-Friendly Guidelines) and topic taxonomy. We also collect creator-provided ratings as another input into our monetization decisioning. Only content meeting our monetization policies will be eligible to show ads. 

In addition to automated classification, many of our videos undergo manual review by our trained third party raters to determine ad eligibility. The millions of videos the raters evaluate train our machine-learning technology to identify similar videos in the future. The geographically distributed extended workforce ensures a global perspective in decisioning and training, and our policy is developed while consulting specialists with language and cultural expertise. Additionally, all content available in Google Preferred is manually reviewed prior to showing any ads.  

We provide advertisers with additional suitability settings (Google Ads and YouTube Reserve and Display & Video 360) to help them exclude content (for example, inventory types, live streams, or content suitable only for mature audiences) that, while in compliance with our policies, may not fit an advertiser’s brand or business. All advertisers are defaulted to only show ads against content that meets the Advertiser-Friendly Guidelines, which can be seen mapped to the 4A’s Brand Safety Floor here. Advertisers are advised to apply exclusions to their campaign prior to the campaign launch. Advertisers can escalate specific exclusions through their Account Managers or by using this form.

Google’s YouTube brand safety models utilize supervised algorithms to replicate the methods performed by Google’s human raters to classify YouTube content, using the same available video content features, in a scalable manner with a high degree of accuracy. Our brand safety machine learning models are updated daily on fixed model architecture versions in order to utilize fresh training data from videos newly reviewed by our manual reviewers, and we additionally launch new classifier architectures and reclassify the corpus every few months. Machine learning classifiers analyze and provide monetization decisions for all YouTube videos, which decisions are visible to the creators in the YouTube Partner Program (as detailed here). We apply the same monetization policies to all YouTube videos regardless of whether on YouTube.com or embedded, and all ads purchased through Google Ads, YouTube Reserve, or Display & Video 360 abide by those decisions. We automatically ensure age-appropriate suggestions in adjacency environments (for example, Watch Next recommendations). Google also employs additional controls specifically for analyzing live streaming video content.

Reporting 

YouTube, applicable as per the scope above, is 99% brand safe as defined by the 4A’s Brand Safety Floor framework (BSFF), a framework created in collaboration with several industry members, including platforms and advertisers. Our brand safety error rate, which is under 1%, fully encompasses content considered unsafe by the BSFF.  This section provides an overview of our brand safety error rate at platform level.

YouTube’s existing monetization policy organization matches the BSFF’s categories closely, enabling our brand safety error rate measurement. We refine and iterate on policy in the standard course. 

The brand safety error rate, indicating how often unsafe content is incorrectly monetized, is calculated as follows:

  • Brand safety error rate = # of impressions on unsafe content / # total impressions

We take 1000 impression-weighted random samples a day (for 5 days a week) from across all ad impressions on YouTube.  We then calculate the brand safety error rate as a 60-day average across all 60,000 impressions.

Each impression is associated with one video, which is human reviewed by trained raters and given a Brand Safety decision.

Because the brand safety error rate is low (under 1%), we see variance in the day-to-day of the brand safety error rate value from each set of 1000 impressions. While volatile day-to-day, these measurements can be aggregated into a moving average that provides a robust estimate of our brand safety error rate, with 95% confidence and error margins between 5%-10% relative to the measured value.

With this in mind, the number of days used for the moving average is identified by finding the sample size needed for a 95% confidence interval with margins of error ± 5%-10% of the measured brand safety error rate value:

 

For 95% confidence on a metric like our brand safety error rate (where p̂=1% and p =1%), a z-score of 1.96 is used. For margins of error to be 5%-10% of the measured value (meaning a measurement averaging 1% has a confidence interval of 0.90%-1.10%, or 1% ± 0.10%) we need n = 60,000 samples. We find smaller samples have too large a margin of error: for example, even with n = 30,000 samples we see a confidence interval 1% ± 0.11%, exceeding the 5-10% window.

Known limitations of brand safety systems

Google believes the below limitations to be insignificant:

  • User data deletion and privacy. Google places the utmost value on stewarding user data. As a result, when a video is deleted from YouTube (including when a user deletes their Google Account) or a video is marked as private on YouTube, we are not able to include it for measurement purposes.
  • Ad campaigns can be configured with a wide variety of settings, targeting, and exclusion criteria. Narrowly defined campaigns may show some variability in brand safety error rates. This variance has shown to be insignificant. Google conducts periodic studies at lower granularity to ensure that the platform-wide error rate is truly representative of customer expectations.
  • New trends. Google’s YouTube brand safety policies prohibit ads from running on violative content. As YouTube’s video library evolves, our systems adapt to newly found types of content and our policies improve to reduce ambiguities.
  • Sampling for human review. Due to the sheer volume of videos uploaded to YouTube on a regular basis, we use a robust sampling methodology detailed above that is representative of the full population. 

Client notification

While we endeavor to minimize the platform-wide brand safety error rate at all times, we will notify clients in the event that it should exceed 1% by using the notification area at the top of this Help Center article. The notification shall remain in place until the error rate is reliably below 1%. 

For the past 12 months, the platform-wide brand safety error rate was greater than 1% on the following days:

  • None

Business partner qualification

YouTube platform level ad policies apply to all parties. Learn more about the ad policies for advertisers. Additionally YouTube partner program guidelines governs which channels are eligible for monetization. Google maintains strong quality processes for onboarding and continuous quality monitoring of the third party human raters.

Any substantive changes in our above stated brand safety methodology will be communicated as part of this help center article, in addition to any regular methodology changes being communicated as part of the ad buying platform (Google Ads/YouTube Reserve and Display & Video 360).