Penetration Testing Guidelines

Summary

Penetration tests requested by Google must fulfill the requirements listed below. Note that requirements may differ between vendors due to special agreements.

Penetration Test and Report Requirements – Report the result of the penetration test in writing and include:
- Appropriate contextual information
- The findings and their risk severity
Assessment Duration and Dates – Plan an appropriate total testing effort (in person days) according to the number and scale of systems in scope. Penetration tests of a small scope can take only a few days, while a large scope can require multiple weeks. The penetration test must have been performed in the past 12 months.
Scope of the Assessment – Scope the penetration test to include all infrastructure used to provide services to Google. If there is a security dependency (i.e. the infrastructure used to provide services to Google depends on other systems for its security), those systems need to be included in the scope of testing.
Attribution Information – Entrust an appropriately qualified third-party provider with the penetration test. The quality between providers can vary significantly. Google recommends carefully comparing providers.
Assessment Methodology – Employ manual methods for identifying security vulnerabilities when penetration testing. Manually verifying automated results is not sufficient. Additionally, perform penetration tests using both unauthenticated and authenticated (i.e. the testers must be provided with credentials to the application, if an authentication exists) methodologies.
Documentation of Findings – When reporting findings, assign a risk severity and include technical details to allow Google to independently assess the risk and impact of the finding.
Remediation Planning / Confirmation – Remediate findings with a critical or high risk severity, these should not be risk-accept. In addition to the penetration test report, also share a remediation plan. For findings that have already been addressed, include a remediation confirmation.

Adherence to a security standard, automated vulnerability scanning/testing, and Bug Bounty and Vulnerability Rewards Programs are not sufficient replacements for penetration test reports.

Google’s Requirements for Penetration Testing and Reporting

The following subsections describe the requirements set out by Google for penetration testing and the penetration testing report that you will submit to Google during the VSA process.

Some resources providing background information:

For an understanding of Google’s definition of penetration testing, see Different Types of Security Tests.
For an understanding of why Google requires third-party testing, see The Importance of Third-Party Testing.
For an understanding of the difference between compliance and penetration testing, see Compliance and Penetration Testing.

Penetration Test and Report Requirements

Ensure that the penetration test report is complete. If the information you share leaves some points unanswered, there may be gaps in the quality of testing that was performed that will need to be clarified. This can result in delays to the approval process.

To ensure your report is complete, verify that it answers these key questions clearly before submitting it to Google:

Was the penetration test performed within the past 12 months?
Was the total testing effort in line with the number of systems which were tested?
Was the penetration test scoped to include all systems used to provide services to Google?
Was the penetration test performed by a third-party provider?
Did the penetration test include manual methods of analysis for identifying security vulnerabilities?
Was the penetration test performed using suitable authenticated test accounts?
Have findings with high and critical risk severity been addressed (not risk-accepted)?
- Where the report doesn't cover remediation, provide a separate remediation plan / confirmation document.

If the penetration testing report does not cover ALL of these points, please provide an attestation letter from the penetration testing provider that clearly responds to these questions in full.

Assessment Duration and Dates

Capture the dates on which the assessment was conducted, as well as the testing effort (in person days).

If the penetration test is part of a periodic assessment, include the dates of the previous and upcoming assessments. This information enables Google to understand how recently the penetration test was conducted as well as the regularity of testing.

Scope of the Assessment

A well-defined scope must include application-level testing of any application related to the processing or storage of data in the service provided to Google.

As a consequence, a good penetration test report should include the following elements when describing the scope:

A well-defined scope definition that demonstrates sufficient coverage of the various threat models the application is subjected to.
If there were areas that could not be assessed for whatever reason, mention this in the scope definition as well.
If the penetration test consists of multiple areas, indicate the testing duration devoted to each area.
For areas where active penetration testing is explicitly not permitted (i.e. select third-party SaaS services), include a configuration review.

Following these points helps to reduce any ambiguity on whether a particular area was sufficiently covered during the assessment. It also helps Google identify any residual risks that need a closer examination after a penetration test.

Attribution Information

The report must include the name of the security company involved in the assessment.

If you need support in picking a qualified penetration testing provider, see Penetration Testing Provider Selection.

Assessment Methodology

Google requires that penetration testers use an assessment methodology with the following characteristics:

Covers both authenticated and unauthenticated testing if the asset has an authenticated section. A comparison of the two approaches can be found at Unauthenticated vs. Authenticated Testing.
Enforces that the majority of the work is performed manually. Automated testing or manual verification of automated testing does not meet this requirement. For more information read Manual vs Automated Testing.

A good penetration test report should clearly outline the assessment methodology followed during the assessment and call out if the assessment was performed authenticated and manually. Ideally, the penetration testers follow an industry-recognized standard that is relevant to the environment (i.e. OWASP, PTES, or OSSTMM). Finally, the report should also include a section outlining the tools used during the assessment.

Documentation of Findings

The section on findings is the core of a penetration test report. Unfortunately, the structure and information provided in this section may vary greatly between different penetration test providers.

Make sure your penetration testing provider includes the following information in this section:

An overview table of all findings, listed by title and indicating the risk severity
For each finding, detailed documentation covering:
- Risk severity of the finding.
- The finding’s impact from a business and technical perspective.
- Clear, concise steps on how to reproduce the finding.
- Screenshots or other form of evidence confirming the finding (if applicable).
- Recommendations on how to resolve the finding. These recommendations should be tailored to the application and environment, avoid generic recommendations.

If this is part of a retest, the report should provide a history of findings which persisted from previous assessments. Please include information about why remediation efforts were not effective. If remediation was not carried out, business owners should provide supporting information behind that decision.

Remediation Planning / Confirmation

Your penetration test should always conclude with a remediation plan. This is not something that your penetration test provider creates, but is instead part of your action plan regarding the identified findings.

For findings with critical and high severity that have not been mitigated at the time of sharing your penetration test report, create a remediation plan that outlines:

The expected remediation date.
The planned remediation action and how to prevent future regression of identified findings.
Considered residual risk of findings that could only be partially remediated.

For findings with critical and high severity that have been mitigated at the time of sharing your penetration test report, create a remediation confirmation statement containing:

Confirmation that all findings with critical and high severity have been remediated (as opposed to being risk-accepted).
Information about residual risk for findings that could only be partially remediated.
Supporting information from business owners on how they intend to prevent regressions of identified findings (e.g. unit test, SDLC, and so on).

Annex

The topics here are supplements to the information provided above, and are linked to from the appropriate locations.

Bug Bounty and Vulnerability Reward Programs

Bug bounty programs can provide useful input into a mature security program as long as they are properly scoped and managed. Many companies choose to run security programs that offer rewards for reported bugs or security issues, including the Google Vulnerability Reward Program.

The quality of these programs varies based on a number of factors, including scope, exclusions, repeatability, reward, interest, program visibility, etc. As such, it is very hard to measure their overall quality and coverage, which are key indicators of a high quality penetration test. Based on these considerations, Google is not able to accept reports from bug bounty programs or providers as a replacement for a third-party penetration test.

Different Types of Security Tests

The below table describes the most common types of security testing that may be required from partners working with Google.

	Vulnerability Scan	Static Source Code Analysis	Penetration Test	Source Code Audit
Method	Automated	Automated	Manual¹	Manual
Purpose	Identify known vulnerabilities based on pre-configured signatures.	Identify a specific set of vulnerabilities in source code.	In-depth, manual security assessment of a defined scope.	Very in-depth review of a single application/product.
Benefits	Fast way to identify misconfigurations and missing security updates. Since these scans are automated, they can be run frequently and complement the vulnerability management process very well.	Fast way to identify vulnerabilities newly introduced into code.	Can identify previously unknown vulnerabilities. Very low false positive rate. Can find subtle logic errors that lead to security issues. Can follow vulnerability chains (exploiting vulnerabilities to identify further issues). Humans are still best at assessing risks.	Can identify pretty much all types of vulnerabilities and security weaknesses in an application's code. Also able to identify design issues and recommend improvements.
Limitations	Vulnerabilities in custom or uncommon software cannot be identified. Only finds vulnerabilities a signature has been created for. False positive rate is often very high.	Still in its infancy. Usually reports a very high number of false positives, making it hard to identify real findings. Limited support for discovery of logic errors (which includes entire classes of vulnerabilities, such as authentication bypass, some types of privilege escalation, etc.)	Time-intensive and expensive. Quality highly dependent on the provider. Scoping often dictates the quality of the test.	Very time-intensive and expensive.
VSA/IPA Requirements²	Attestation of Quarterly scanning	N/A	Annual third-party penetration test report	N/A
Example Providers³	Nessus, Nexpose, Qualys, Trustwave App Scanner, …	Fortify, Veracode, CheckMarx, Whitehat, …	NCC Group, Bishop Fox, Leviathan Security, …

¹ Penetration testing that is marketed as Automated penetration testing or as validated scans (e.g. Whitehat Business Logic Assessments) do not meet the threshold to be accepted as manual penetration testing under this definition.

² Please check your Information Protection Addendum (IPA) for further guidance on these requirements for your specific project. Where additional types of testing are required, these will be clearly requested as part of the IPA and VSA process.

³ This list includes security companies that are known to provide these services, and does not constitute a recommendation on the quality or coverage provided.

The Importance of Third-Party Testing

Although many companies have internal teams capable of performing detailed penetration testing, it is important to include third-party testing as part of your testing program.

Third parties often approach testing with a different mindset than an internal team, and as a result are more likely to examine processes and functions without being prejudiced by previous uses of the application. In some cases, internal testers can be quick to discount the presence of certain bug classes due to familiarity with the product, deployment style, or existing security mechanisms that should be in place to protect against such attacks.

In contrast, a third-party penetration tester can provide an unbiased and honest look at an application, something that is often lacking with internal teams. This is equally important when performing annual penetration testing, and it is a best practice to rotate testing teams or third-party providers every 2-3 years to ensure a fresh view on the application.

Compliance and Penetration Testing

Compliance with common security standards (e.g. ISO 27001, NIST, PCI-DSS) may require you to perform specific types of security scanning and testing on a regular basis. Although these types of testing are an important part of a well-managed and mature security and privacy program, they may not in themselves meet Google's requirements.

A common example is PCI-DSS, which requires penetration testing to be performed as part of the compliance requirements. Simply having PCI Compliance does not however mean that the penetration testing that was performed to obtain that compliance meets Google's standards. Specifically, the requirements regarding the scope of systems to test will likely differ between the two approaches. Google also specifically calls out validity periods (12 months) for penetration testing, and calls for testing to be performed in an authenticated and manual fashion.

As you fill the Vendor Questionnaire, please indicate any valid certifications received such as ISO 27001, SOC2 etc. A Google Security Engineer will advise whether they can be accepted as alternatives, which will require you to provide a link or artifact for our compliance records.

Penetration Testing Provider Selection

Not every company offering penetration testing services automatically provides high-quality results. This poses a challenge if you have no expertise internally to select a suitable service provider.

Here are a few tips and recommendations on how to find a good penetration testing provider:

Companies that specialize in security services often provide better penetration tests than companies that "also" do pentests in addition to entirely unrelated services.
Sometimes even security companies use penetration tests to pitch their "main" product. Picking a company where the main products are related to security consulting (including pentesting) is often a safer bet.
Most highly qualified penetration testing providers run a blog where they provide technical details of security vulnerabilities and research they have recently conducted. It also helps to look for "CVEs", that is, vulnerabilities, in standard software that the company identified in the course of their work.
Presentations at industry conferences such as BlackHat, DEF CON, Hack-in-the-Box, CanSec West, etc. are usually a good indicator that the company has the required expertise and experience.
Even within qualified companies, expertise between individual people can vary significantly. To get the most out of a penetration test, request senior security consultants, ideally folks credited with having found security vulnerabilities (CVEs), or who have presented at industry conferences.

Unauthenticated vs. Authenticated Testing

Google testing requirements mandate third-party penetration testing that includes authenticated tests. This ensures that all areas of the application are tested equally to discover issues that may otherwise not be easily identified without full access to the application.

	Unauthenticated Testing	Authenticated Testing
Definition	Testers have no knowledge of the environment under test. No credentials are provided to the testers.	Testers have full access to information about the platform being tested. This often includes accounts (including administrative users), and access to discuss functionality with developers during the testing process.
Level of detail	Low	Medium/High
Benefits	Simulates casual attackers and automated attacks on the first line of defense within an application. Does not require a high level of expertise to perform, no knowledge of the business logic is required.	Detail-orientated testing, including business-logic-related issues. More likely to find issues that would otherwise not be discovered during code audit, unauthenticated testing, or automated scanning alone. Can be useful in testing monitoring and alerting capabilities when used in coordination with an internal IR/Monitoring team exercise.
Limitations	Often misses issues that require knowledge of the application or infrastructure, or require a minimum level of access as a user.	Can take additional resources from the development team, as well as testers. Often requires more experienced testers due to the increased complexity and type of issues.
VSA/IPA Requirements	Not accepted	Annually

Limiting the scope of a penetration test to unauthenticated users may result in only a small subset of issues being discovered, compared to a more comprehensive test where testers have credentials to access the application itself. A limited test can find issues that a casual external attacker would discover, but this kind of test often overlooks more detailed attacks and functionality that can be abused by existing users of the system.

Specifically, an authenticated user might be able to access information that should not be user-accessible, or have the ability to perform actions that should be restricted to administrative users. With the increase of multi-tenant systems, attacks from legitimate users of the system, either directly, or due to hijacked accounts (via weak passwords, phishing, malware, etc…), are seen as a threat that is often overlooked when planning the scope and methodology of a penetration test.

Manual vs Automated Testing

Google testing requirements mandate manual discovery of security relevant issues as part of an authenticated penetration test. This ensures that complex logic errors and unknown vulnerabilities are discovered by experienced penetration testers.

	Manual Testing	Automated Testing
Definition	Penetration testers perform manual checks of the security controls in place and attempt to bypass or circumvent them in the process of identifying vulnerabilities.	Security scanner software is launched against the asset in scope. The scanner works through a predefined set of known vulnerabilities or malicious payloads to test if the asset is vulnerable.
Level of detail	Medium/High	Low
Benefits	Very low false positive rate. Allows for identification of previously unknown vulnerabilities and logic errors that lead to security issues. Testers can exploit vulnerabilities in a chain to identify further issues.	Fast way to identify misconfigurations and missing security updates, as well as known vulnerabilities. Can be used to complement a vulnerability management program.
Limitations	The quality of the testing depends on the service provider and the skills of the individual penetration testers. Manual testing is also time-intensive and expensive. Defining the right scope is crucial.	Vulnerabilities in custom software cannot be identified at all or only in generic cases. The false positive rate is often very high.
VSA/IPA Requirements	Annual third-party penetration test report	Attestation of Quarterly scanning

Regular automated vulnerability scans of your network, servers, and workstations are a valuable part of a mature security program. The results help your security team identify areas where patching against known vulnerabilities and misconfigurations has not occurred.

However, automated vulnerability scans alone do not meet Google’s requirements as they are automated, commonly unauthenticated, and focus on known security misconfiguration and vulnerabilities in common software, instead of being tailored to the needs of your organization.

Validated Automated Security Scans

It is often argued that this type of scan is a combination of the manual and automated methodologies. However, the operation model often consists of automated discovery and manual validation. This improves the false-positive rate, but apart from that this approach still shares the same limitations as automated security scanning and doesn’t bring any of the benefits of manual testing. As such, validated automated scans are not considered a sufficient replacement for manual testing

Was this helpful?

How can we improve it?