Access Data Transfer storage buckets

Access to Data Transfer storage buckets is managed through user roles in Google Ad Manager. Once you have the necessary user role permission, you can use the method of your choice to access your Data Transfer files. 

Jump to a section of the article:

• About user roles for the permission
• View the permission in a role
• Methods to access storage buckets
• Use the Google Cloud Storage API

About user roles for the permission

To access the storage buckets, you need to be in a role with the "View data transfer buckets" permission enabled.  

• Administrator user roles: The permission is enabled by default for the built-in Administrator user role.
• Custom user roles: The permission can be added to custom user roles. 

To make changes to user roles, you need to be an Administrator on your network, or have the "Edit users, roles, and teams in Admin tab" permission. Note that a user can only be associated with one role at a time.

For more details, visit Manage user role membership.

View the permission in a role

You can review the role a user is currently in and add or remove the permission as needed. 

1. Sign in to Google Ad Manager.
2. Click Admin, then Access & authorization.
3. Click Roles, and then the name of a role. 
4. Under "Reporting," locate the "View data transfer buckets" permission.
   
5. (Optional) To enable the permission for the role, check the box.

Methods to access storage buckets

Google Cloud Storage is a separate Google product that Ad Manager uses as a data repository for Data Transfer reports and batch-uploaded audience cookie IDs.

This article is intended to help you use standard Google Cloud Storage technologies to interface with Ad Manager's specific cloud storage setup. However, your primary reference for Google Cloud Storage is the Google Cloud Storage developer site.

There are three ways you can access Ad Manager cloud storage buckets. In order of complexity:

• On the web: Visit https://console.developers.google.com/storage/gdfp-[Ad Manager network code].
• gsutil is a Python-based command-line tool that provides Unix-like commands for interacting with the storage bucket. Bucket authentication is abstracted and handled automatically.
• The Google Cloud Storage API is a full-featured API for manipulating the storage bucket, available through JSON or XML RESTful web interfaces. API client libraries are available for many popular programming environments, including Java, JavaScript, Python, and Objective-C. This approach is most useful if you need to manipulate the storage buckets programmatically to integrate with a Google App Engine app or a Java web app.

This article provides details on the Google Cloud Storage API. Web and gsutil access are easier to manage, so we recommend exploring these methods first. They are fully documented on the Google Cloud Storage developer site.

Use the Google Cloud Storage API

If you've determined that API access is best for your needs, we recommend that you configure a Google Cloud Storage service account.

Configure a Google Cloud Storage service account

When accessing the storage bucket via the API, it's preferable to configure a service account rather than running in the context of a user. Service accounts simplify application development by using a private key for authentication instead of a dynamically generated OAuth token. To configure a service account:

1. Go to the Google Developer Console.

2. Create a new project (or select an existing project) to be the parent of your application and click into it.

3. (Optional) If you plan to copy files from the Ad Manager storage bucket to your own Google Cloud Storage account, click Billing & settings to add a billing source to your project.

4. Create a new client ID:

   1. Click APIs & Services and then Credentials.

   2. Click Create new Client ID.

   3. Select Service account as your application type, then click Create Client ID.

   4. The email address generated takes the format [unique-id]@developer.gserviceagccount.com. Add the service account user to your Ad Manager network, and make sure they’re in a role with the "View data transfer buckets" permission. 

   5. Click Generate new P12 key. The file is saved to your computer. Use this key in the applications you develop to access the API, as shown in the code example below.

5. Users with the "View data transfer buckets" permission will have access to your Ad Manager storage buckets.

Code example

Google provides code samples and libraries for Google Cloud Storage. The following Java example for reading a file from an Ad Manager cloud storage bucket shows how the components you've configured when setting up the service account might factor into your code:

• Project name: Name of the Google Cloud Storage project.

• Service account email address: Email address you generated.

• .p12 key file: File you downloaded.

• Bucket name: Google provides this name when you activate a feature that uses Ad Manager cloud storage buckets.

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.security.GeneralSecurityException;
import java.util.Collections;

import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.storage.Storage;
import com.google.api.services.storage.model.StorageObject;

public class GcsApiTest {
    /**
     * The name of the project under which the service account was created.
     * 
     * This information is displayed under the Google Developers Console.
     */
    private static final String PROJECT_NAME = "project name";

    /**
     * Developer Email address of the service account.
     * 
     * This email is generated upon creating a Service Account Client ID in your
     * Google Developers Console, and can be retrieved from the Credentials
     * page. 
     */
    private static final String SERVICE_ACCOUNT_EMAIL = "service account email address";

    /**
     * Bucket to use for storage operations.
     * 
     * The name of this bucket was provided to you by your Account
     * Manager. It likely has a name similar to "gdfp-12345678" or
     * "gdfp_cookieupload_12345678", depending on the Ad Manager add-on you're using.
     */
    private static final String BUCKET_NAME = "bucket name";

    /**
     * Google Cloud Storage OAuth 2.0 scope for read/write. This should
     * correspond to the access rights for your "View data transfer buckets"  
     * permission  for the bucket, and you cannot request access rights that are 
     * not granted via the permission.
     */
    private static final String STORAGE_SCOPE = 
        "https://www.googleapis.com/auth/devstorage.read_write";

    /**
     * Path to the key.p12 file providing access to the bucket.
     * 
     * This file is created when the service client ID is created. If you don't
     * have this file, you will need to generate a new client p12 key from the
     * Google Developers Console.
     */
    private static final String KEY_P12 = "path to .p12 key file";

    /** HTTP transport. */
    private HttpTransport httpTransport;
    private Storage storage;

    // constructor, sets up credentials and storage objects
    public GcsApiTest() {
	File p12File = new File(KEY_P12);

	try {
	    httpTransport = GoogleNetHttpTransport.newTrustedTransport();

	    GoogleCredential credential = new GoogleCredential.Builder()
		    .setTransport(httpTransport)
		    .setJsonFactory(JacksonFactory.getDefaultInstance())
		    .setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
		    .setServiceAccountScopes(
			    Collections.singleton(STORAGE_SCOPE))
		    .setServiceAccountPrivateKeyFromP12File(p12File).build();
	    storage = new Storage.Builder(httpTransport,
		    JacksonFactory.getDefaultInstance(), credential)
		    .setApplicationName(PROJECT_NAME).build();
	} catch (GeneralSecurityException | IOException e1) {
	    e1.printStackTrace();
	    System.exit(1);
	}
    }

    /**
     * Method to return the name of the first file in the bucket.
     * 
     * @return the name of the file, or null if the bucket is empty
     * @throws IOException
     */
    public String GetFirstFile() throws IOException {
	Storage.Objects.List listObjects = storage.objects().list(BUCKET_NAME);
	listObjects.setMaxResults(5L);
	com.google.api.services.storage.model.Objects objects = listObjects
		.execute();

	// empty bucket?
	if (null == objects.getItems() || objects.getItems().isEmpty()) {
	    System.out.println("Bucket \"" + BUCKET_NAME
		    + "\" empty or invalid.");
	    return null;
	}

	StorageObject object = objects.getItems().get(0);
	System.out.println("First object in bucket: \"" + object.getName()
		+ "\".");
	return object.getName();
    }

    /**
     * Method to download the specified file from the storage bucket
     * 
     * @param filename
     *            Name of the file that should be downloaded.
     * @throws IOException
     */
    public void DownloadFile(String filename) throws IOException {
	Storage.Objects.Get getObject = storage.objects().get(BUCKET_NAME,
		filename);
	OutputStream os = new FileOutputStream(filename, true);
	getObject.getMediaHttpDownloader().setDirectDownloadEnabled(true);
	getObject.executeMediaAndDownloadTo(os);

	System.out.println("File \"" + filename + "\" downloaded.");
    }

    /**
     * Main method to execute the different tests.
     * 
     * @param args
     */
    public static void main(String[] args) {
	GcsApiTest gcsApiTest = new GcsApiTest();

	try {
	    String filename = gcsApiTest.GetFirstFile();
	    gcsApiTest.DownloadFile(filename);
	} catch (IOException e) {
	    System.out.println(e.getMessage());
	}
    }
}

Throttling and concurrent connections

There is no predefined limit on concurrent connections. However, to avoid abuse, Google throttles Data Transfer fetch requests.

Read the Google Cloud Storage section of the official Google Code blog for the latest news, and post questions to the Google Cloud Storage Discussion Forum. Add "Bug" or "Feature Request" to the subject line as appropriate. For further Google Cloud Storage questions, see the Google Cloud Storage FAQ.