Batch upload identifiers to audience segments

Only available in Google Ad Manager 360.

 

Learn about bulk uploading identifiers with Google Cloud. #audiencesegments

This article has been updated in alignment with upcoming changes to Audience Solutions.

To adhere to the EU user consent policy and continue creating audience segments globally, publishers using batch upload must make updates to pass the required consent signals to Ad Manager by March 2024.

You must contact Publisher Support or Sales to activate Audience Solutions. They need to activate provided lists, which will result in a delay before publishers begin to see their audience segment(s).

You can create a file of identifiers and upload it into Google Ad Manager via the Google Cloud. Identifiers or IDs available for upload include cookies, mobile advertising IDs (AdID or IDFA), or PPIDs. Uploading IDs via the Google Cloud saves time if you need to add many of them at once.

Cookies, PPIDs, and device advertising IDs must have been part of an Ad Manager ad request before they can be added to a segment. You can download lists of encrypted identifiers in Data Transfer reports (Google Ad Manager 360 only), or retrieve raw identifiers from other systems (for example, raw device advertising IDs from your mobile application's logs).

The maximum life of PPIDs for Batch Upload is 180 days after the last user activity, or, the last time it was seen in serving. For device ids, the maximum is 540 days after creation, in other words, 540 days after the first time the device was seen in serving.

Note that using both event-ping based population method (for example, register event, pre-population) and cookie-upload based population method (that is, provided CSV lists) may generate conflicts where removed PPIDs may not end up appearing as completely excluded.

This article covers the following topics:

• Prerequisites
• Create the upload file
• Deleting identifiers in bulk
• Upload the file
• Batch uploaded identifier expiration
• Best practices for batch uploads

Prerequisites

Before you can upload the identifiers, work with your account manager to:

• Locate and submit your network code. Find your network code in Ad Manager under Admin, then Global settings, and then Network code.
• Ensure that you've created an active first-party audience segment with which you want to associate the identifiers. If you don't want this segment to collect users using your own inventory, select "Publisher Managed" as the Population method.
  
  Expect to wait up to 24 hours from the time of segment creation to the time when you can perform a batch upload.
• Create a Google Group that contains all Google Accounts that will have access to upload and view files.

Once your account manager has completed your setup, you'll be ready to create the file of identifiers you want to upload.

Publishers who are new to uploading identifiers may encounter failures in the process. This often happens because they upload identifiers that have not been passed in ad requests prior to the upload. In this instance, they see the following message in their .fail files: 
"Could not find the <type of identifier> in our server, retry is not helpful".

Create the upload file

Required consent tag

All publishers must make updates to pass required consent signals to Ad Manager.

To confirm that consent has been gathered, add the tag process_consent in the header of any uploaded files. Rows should contain values for this header column in order to conform with CSV standards, but they will not affect processing; we recommend an empty string. Any files that are missing this phrase will be assumed to not have the proper consent from users and will fail to be processed by Google.

This update is required:

• For all users, regardless of location, as a technical requirement.
• For EU/EEA users, to adhere to the EU user consent policy and continue creating audience segments for users in the European Economic Area (EEA).

About identifiers

The batch upload process supports several different identifier types, and it's important that types of identifiers are uploaded to the segment using the correct upload file format. These identifiers fall into two categories:

• Encrypted identifiers: anything obtained from Google systems
• Raw identifiers: obtained from an external system or source

Encrypted identifiers are uploaded using the cookie_encrypted file format, while raw identifiers are uploaded in a type-specific upload file format.

Encrypted identifiers and PPIDs are network-specific. Therefore, batch upload of PPIDs or encrypted identifiers obtained from one network can't be uploaded to another network. That is, a user identifier obtained from the Data Transfer report of network A can't be batch uploaded to network B. Device identifiers, on the other hand, are not network-specific.

The following table summarizes the available identifiers and how to upload:

Identifier	Description	Source	Upload as
Cookie	Cookie identifiers	Data transfer UserId	cookie_encrypted
		Raw (external source)	N/A (cookie must be encrypted to upload)
PPID	Publisher provided identifiers	Data transfer PublisherProvidedID	cookie_encrypted
		Raw (external source)	ppid
IDFA	Identifier for advertising	Data transfer UserId	cookie_encrypted
		Raw (external source)	cookie_idfa
AdID	Android TV identifiers	Data transfer UserId	cookie_encrypted
		Raw (external source)	cookie_adid
Roku ID	Roku identifiers	Data transfer UserId	cookie_encrypted
		Raw (external source)	cookie_rida
tvOS ID	Apple TV identifiers	Data transfer UserId	cookie_encrypted
		Raw (external source)	cookie_tvos
Vizio IFA	Vizio identifiers	Data transfer UserId	cookie_encrypted
		Raw (external source)	cookie_vida
Samsung TIFA	Samsung identifiers	Data transfer UserId	cookie_encrypted
		Raw (external source)	cookie_tifa

Using a text editor, create a plain text file and save it using any file name. You can save the file using any file extension or no extension at all.

Google Cloud file names are publicly visible. Do not include confidential information in your file names.

File content requirements

• Ensure that the contents of the file are comma-separated.
• The first line must read:
  • For encrypted cookies and encrypted identifiers obtained from Data Transfer reports (Google Ad Manager 360 only): cookie_encrypted,list_id,process_consent
  • For raw Apple IDFAs: cookie_idfa,list_id,process_consent
  • For raw Android AdIDs: cookie_adid,list_id,process_consent
  • For raw PPIDs: ppid,list_id,process_consent
  • For raw Roku IDs: cookie_rida,list_id,process_consent
  • For raw tvOS IDs: cookie_tvos,list_id,process_consent
• Each subsequent line should contain an identifier, a comma, and the ID of the audience segment with which the identifier should be associated
• Each line should end with the unix line ending type linefeed (LF), NOT carriage return (CR). Depending on the method you use to produce the upload file, the line ending type is set in your text editor or in the program that produces the file.
• The encrypted identifiers are available to publishers through a Data Transfer report (Google Ad Manager 360 only). You can get the audience segment ID by signing into your Google Ad Manager account and selecting Inventory, then Audience segments.
• The "process_consent" column should be the final column in the upload file.
• Cells in the "process_consent" column may be left blank. However, it is important to include the column with the "process_consent" value set as the header.
• Batch upload files cannot be larger than 1GB in size. Files larger than this might fail to process correctly.

Example file content for encrypted cookie IDs:

cookie_encrypted,list_id,process_consent
ScpJKu-yV8je93qkd32MOA,3153490,
w2gsrUcwxF-OiJTRmQswQA,3153490,

 

Example file content for raw AdIDs:

cookie_encrypted,list_id,process_consent
ScpJKu-yV8je93qkd32MOA,3153490,
w2gsrUcwxF-OiJTRmQswQA,3153490,

Deleting identifiers in bulk

You can use the same upload file to delete identifiers in bulk by adding a column called delete. For each identifier you want to delete from the segment, enter a 1 in the delete column and for each ID you want to keep in the segment, enter a 0 in the delete column.

You must ensure that every line in the file contains either a 1 or a 0 in the delete column. Otherwise, the upload will fail.

In the example below, the first cookie is kept in the segment, but the second cookie is disassociated from the segment with id=3153490.

cookie_encrypted,list_id,delete,process_consent
ScpJKu-yV8je93qkd32MOA,3153490,0,
w2gsrUcwxF-OiJTRmQswQA,3153490,1,

In the following example, the first IDFA is disassociated from the segment, but the second IDFA is kept in the segment.

cookie_idfa,list_id,delete,process_consent
ScpJKu-yV8je93qkd32MOA,3153490,1,
w2gsrUcwxF-OiJTRmQswQA,3153490,0,

Each line must contain as many comma-separated fields as are present in the header. Note that the identifiers never contain commas or other characters inconsistent with CSV format.

Keep in mind that it can take several hours for Ad Manager to process identifier uploads, whether you're adding or deleting them. When you add an identifier, you need to wait for it to be processed before you can delete it with a subsequent upload.

Upload the file

Once you've created the file, you'll need to upload it to a Google Storage bucket (subfolders are allowed). There are three ways you can access Ad Manager cloud storage buckets. In order of complexity:

• On the web: Visit https://console.developers.google.com/storage/browser/gdfp_cookieupload_[Ad Manager network code]/
• gsutil is a Python-based command-line tool that provides Unix-like commands for interacting with the storage bucket. Bucket authentication is abstracted and handled automatically.
• The Google Cloud Storage API is a full-featured API for manipulating the storage bucket, available through JSON or XML RESTful web interfaces. API client libraries are available for many popular programming environments, including Java, JavaScript, Python, and Objective-C. This approach is most useful if you need to manipulate the storage buckets programmatically to integrate with a Google App Engine app or a Java web app.

Within 24 hours, a log file will be generated in your Google Storage account with the same name as your file and a suffix indicating whether all of the identifiers in the file were processed successfully.

For example, if your file is named 1234_20130115_1, then the generated file will be named 1234_20130115_1.success or 1234_20130115_1.fail. It lists the number of identifiers processed successfully, with detailed error messages for identifiers that could not be processed. Download this log file for more detailed status information.

The results filename will appear in the following folder of your Google Storage account:

https://console.developers.google.com/storage/gdfp_cookieupload_result_[Ad Manager network code]/

After upload, the successfully processed identifiers in your file are treated as members of their associated segments when targeting inventory.

The .fail or .success file only indicates whether the file was parsed successfully; it doesn't provide meaningful information about the net increase/decrease in list size. The file is only named .success if every single identifier was uploaded successfully. However, in most cases there's at least one failure, so the output file contains the .fail suffix.

The upload and log files will be deleted after 60 days.

Batch uploaded identifier expiration

Identifiers uploaded as part of a Batch Upload operation obey the expiration settings that have been configured on the audience segment through the Ad Manager UI.

If the audience segment is configured with an expiration value of X days and the identifier does not satisfy the segment membership criteria within X days after upload, that identifier will expire from the audience segment after X days. Each upload operation resets the segment membership and expiration countdown for the identifiers that are uploaded.

Best practices for batch uploads

File structure

Minimize the number of files used for uploads. Note that a single file can be used to upload identifiers into multiple lists, since the list id is specified for each line in the file.

Upload throughput

We request you adhere to the following limits when uploading files:

• Upload no more than 500 files per 24 hour window
• Up to 1 GB per batch upload file
• Up to 5 GB in total batch upload files per 24 hour window

Incremental/complete uploads

• To avoid unnecessary processing, we recommend using an incremental (delta) upload of identifiers as a routine practice. You should add or delete only the identifiers whose membership in the segment has changed since the last upload, rather than uploading the full segment identifier list.
• On a less frequent basis (no more than once per week, and still subject to the upload throughput limits above), you may run a full upload of all segment identifier data. This full upload will refresh the segment membership of any members who had previously aged out of the audience segment.
  • You should be tracking the "freshness" of identifiers through some other system, and you should have some way to exclude identifiers from the refresh upload if these identifiers have been abandoned (are not expected to be seen again in serving).

Processing time

Expect to wait up to 24 hours from the time of batch upload to the time when all batch changes are reflected in serving. In other words, upload your files at least 24 hours ahead of when you need the segments in serving.

Additionally, expect to wait up to 48 hours from the time of batch upload before the changes are visible in the Ad Manager user interface.