Weighting is designed to remove bias from a survey sample and make the results more closely represent the target population. For example, if the percentage of female respondents in a survey is less than that of the target population, we increase the contribution of the female responses by a factor (a weight) so that females are representatively proportionate to the target population in the weighted results.
While surveys are running, Google Surveys dynamically targets respondents to match the demographics of the internet population using their inferred demographics (age, gender, and geography). In the U.S., we use estimates for the national internet population from the U.S. Census Bureau’s 2015 Current Population Survey (CPS) Computer and Internet Use Supplement. The dynamic targeting uses the joint distribution of those dimensions, for example the percentage of 65+ year-old females in Wyoming. This dynamic targeting approach means that surveys are fairly close to the target distribution after running, and the amount of weighting needed is minimal.
After the survey has gathered responses, weights are applied to each response to match the breakdowns of age, gender, and region to those demographic breakdowns in the internet population. Google Surveys uses an iterative raking process to calculate weights for each demographic dimension. The demographic dimensions used for weighting are listed on the question info card. The weights applied after the fact will match the marginal distributions, meaning the percentage of females, the percentage of 65+ year olds, and the percentage of people in the Western US, but not the joint distribution of female and 65+ and the Western US out of the total population. Weights are applied to the first question only. Weighted results are calculated by normalizing the weights available in each question as respondents screen out, drop off in the survey, or are filtered out in a survey report.
If weights are available, we show weighted data by default. If you prefer to see unweighted data, you can turn on Raw counts on your survey results page. When a survey is still gathering responses, it may not have enough responses for the weights to reliably match the demographics of the internet population. In those cases, we still show the weighted data by default, but there is a warning in the survey report to indicate that the weights may be unreliable. Similarly, when you filter a survey's results down to a subpopulation, like 18-24 year-olds excluding other ages, the warning message may show even if weighting was reliable for the survey prior to filtering.
Note that we cannot calculate weights for respondents who have unknown demographics for age, gender, or geography. Responses with unknown demographics are excluded when weighting is applied. Turning on Raw counts displays these responses with unknown demographics.
Please note that we don't weight the following types of surveys at this time:
- Surveys with any demographic targeting (age, gender, geography) that completed before August 19, 2017
- Surveys targeted to your website with Website Satisfaction
- Surveys targeted to an audience panel, such as students or small-business owners