Important: Starting in early November 2024, access to Ad Manager storage buckets for Data Transfer begins migrating from Google Groups to a new Ad Manager user role permission.
For details on the migration timeline and next steps, visit Migrate Data Transfer access from Google Groups to Ad Manager Permissions.
Google Cloud Storage is a separate Google product that Ad Manager uses as a data repository for Data Transfer reports and batch-uploaded audience cookie IDs.
Methods
There are three ways you can access Ad Manager cloud storage buckets. In order of complexity:
- On the web: Visit
https://console.developers.google.com/storage/gdfp-[Ad Manager network code]. - gsutil is a Python-based command-line tool that provides Unix-like commands for interacting with the storage bucket. Bucket authentication is abstracted and handled automatically.
- The Google Cloud Storage API is a full-featured API for manipulating the storage bucket, available through JSON or XML RESTful web interfaces. API client libraries are available for many popular programming environments, including Java, JavaScript, Python, and Objective-C. This approach is most useful if you need to manipulate the storage buckets programmatically to integrate with a Google App Engine app or a Java web app.
This article provides details on the Google Cloud Storage API. Web and gsutil access are easier to manage, so we recommend exploring these methods first. They are fully documented on the Google Cloud Storage developer site.
Manage access to your Ad Manager storage buckets
New process for Data Transfer storage access (available in November 2024)
Starting in November 2024, you can begin migrating users that need access to Data Transfer storage buckets from Google Groups to a new Ad Manager user role permission. Users in the Administrator role will be able to access the storage buckets by default. For other users, you can add them to a user role with the new "View data transfer buckets" permission. Learn more about the migration.
Old process for Data Transfer storage access (available until January 2025)
Note: For Data Transfer storage access, we encourage you to review the migration phases and take steps before January 2025.
Storage buckets used by Ad Manager are included in a Google-owned cloud storage project. Ad Manager storage buckets don't appear under your own project list in the Google Developers Console.
You will need to work with your Google representative to set up features that depend on Ad Manager storage buckets. They will provide a Google group for you to use to manage your storage buckets.
If you’re the administrator of an existing G Suite group, you can provide the name of your G Suite Google group to your representative as part of the feature activation process. They will work with you to activate any feature that uses Ad Manager storage buckets using your existing group.
If you manage more than one Ad Manager network with access to Data Transfer, you should communicate this to your Google representative so that network management can be consolidated. One group will be used to manage all of the Data Transfer buckets across your networks.
Use the Google Cloud Storage API
If you've determined that API access is best for your needs, we recommend that you configure a Google Cloud Storage service account.
Configure a Google Cloud Storage service account
When accessing the storage bucket via the API, it's preferable to configure a service account rather than running in the context of a user. Service accounts simplify application development by using a private key for authentication instead of a dynamically generated OAuth token. To configure a service account:
-
Go to the Google Developer Console.
-
Create a new project (or select an existing project) to be the parent of your application and click into it.
-
(Optional) If you plan to copy files from the Ad Manager storage bucket to your own Google Cloud Storage account, click Billing & settings to add a billing source to your project.
-
Create a new client ID:
-
Click APIs & Services > Credentials.
-
Click Create new Client ID.
-
Select Service account as your application type, then click Create Client ID.
-
The email address generated takes the format
[unique-id]@developer.gserviceagccount.com. Copy and save it to add to your Google group. -
Click Generate new P12 key. The file is saved to your computer. Use this key in the applications you develop to access the API, as shown in the code example below.
-
- Add the email address to the Google group you created to manage access to your Ad Manager storage buckets. Click Direct add members instead of 'invite' to add the service account. If you don't have access, ask the Google group administrator to add the address on your behalf.
Google provides code samples and libraries for Google Cloud Storage. The following Java example for reading a file from an Ad Manager cloud storage bucket shows how the components you've configured when setting up the service account might factor into your code:
-
Project name: Name of the Google Cloud Storage project.
-
Service account email address: Email address you generated.
-
.p12 key file: File you downloaded.
-
Bucket name: Google provides this name when you activate a feature that uses Ad Manager cloud storage buckets.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.security.GeneralSecurityException;
import java.util.Collections;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.storage.Storage;
import com.google.api.services.storage.model.StorageObject;
public class GcsApiTest {
/**
* The name of the project under which the service account was created.
*
* This information is displayed under the Google Developers Console.
*/
private static final String PROJECT_NAME = "project name";
/**
* Developer Email address of the service account.
*
* This email is generated upon creating a Service Account Client ID in your
* Google Developers Console, and can be retrieved from the Credentials
* page. This email must also be added to the Google Group used to control
* access to the storage bucket.
*/
private static final String SERVICE_ACCOUNT_EMAIL = "service account email address";
/**
* Bucket to use for storage operations.
*
* The name of this bucket was provided to you by your Account
* Manager. It likely has a name similar to "gdfp-12345678" or
* "gdfp_cookieupload_12345678", depending on the Ad Manager add-on you're using.
*/
private static final String BUCKET_NAME = "bucket name";
/**
* Google Cloud Storage OAuth 2.0 scope for read/write. This should
* correspond to the access rights that your Google Group has for the
* bucket, and you cannot request access rights that are not granted to the
* Group (which may be read_only).
*/
private static final String STORAGE_SCOPE =
"https://www.googleapis.com/auth/devstorage.read_write";
/**
* Path to the key.p12 file providing access to the bucket.
*
* This file is created when the service client ID is created. If you don't
* have this file, you will need to generate a new client p12 key from the
* Google Developers Console.
*/
private static final String KEY_P12 = "path to .p12 key file";
/** HTTP transport. */
private HttpTransport httpTransport;
private Storage storage;
// constructor, sets up credentials and storage objects
public GcsApiTest() {
File p12File = new File(KEY_P12);
try {
httpTransport = GoogleNetHttpTransport.newTrustedTransport();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(JacksonFactory.getDefaultInstance())
.setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
.setServiceAccountScopes(
Collections.singleton(STORAGE_SCOPE))
.setServiceAccountPrivateKeyFromP12File(p12File).build();
storage = new Storage.Builder(httpTransport,
JacksonFactory.getDefaultInstance(), credential)
.setApplicationName(PROJECT_NAME).build();
} catch (GeneralSecurityException | IOException e1) {
e1.printStackTrace();
System.exit(1);
}
}
/**
* Method to return the name of the first file in the bucket.
*
* @return the name of the file, or null if the bucket is empty
* @throws IOException
*/
public String GetFirstFile() throws IOException {
Storage.Objects.List listObjects = storage.objects().list(BUCKET_NAME);
listObjects.setMaxResults(5L);
com.google.api.services.storage.model.Objects objects = listObjects
.execute();
// empty bucket?
if (null == objects.getItems() || objects.getItems().isEmpty()) {
System.out.println("Bucket \"" + BUCKET_NAME
+ "\" empty or invalid.");
return null;
}
StorageObject object = objects.getItems().get(0);
System.out.println("First object in bucket: \"" + object.getName()
+ "\".");
return object.getName();
}
/**
* Method to download the specified file from the storage bucket
*
* @param filename
* Name of the file that should be downloaded.
* @throws IOException
*/
public void DownloadFile(String filename) throws IOException {
Storage.Objects.Get getObject = storage.objects().get(BUCKET_NAME,
filename);
OutputStream os = new FileOutputStream(filename, true);
getObject.getMediaHttpDownloader().setDirectDownloadEnabled(true);
getObject.executeMediaAndDownloadTo(os);
System.out.println("File \"" + filename + "\" downloaded.");
}
/**
* Main method to execute the different tests.
*
* @param args
*/
public static void main(String[] args) {
GcsApiTest gcsApiTest = new GcsApiTest();
try {
String filename = gcsApiTest.GetFirstFile();
gcsApiTest.DownloadFile(filename);
} catch (IOException e) {
System.out.println(e.getMessage());
}
}
}
Throttling and concurrent connections
There is no predefined limit on concurrent connections. However, to avoid abuse, Google throttles Data Transfer fetch requests.
