Weighting is designed to remove bias from a survey sample and make the results more closely represent the target population. For example, if the percentage of female respondents in a survey is less than that of the target population, we will increase the contribution of the female responses by a factor (a weight) so that females are representatively proportionate to the target population in the weighted results.
While surveys are running, Google Surveys dynamically targets respondents to match the demographics of the internet population using their inferred demographics (age, gender, and geography). The dynamic targeting uses the joint distribution of those dimensions, for example the percentage of 65+ year-old females in Wyoming. This dynamic targeting approach means that surveys are fairly close to the target distribution after running, and the amount of weighting needed is minimal.
After the survey has gathered responses, weights are applied to each response to match the breakdowns of age, gender, and region to those demographic breakdowns in the internet population. Google Surveys uses an iterative raking process to calculate weights for each demographic dimension. The weights applied after the fact will match the marginal distributions, meaning the percentage of females, the percentage of 65+ year olds, and the percentage of people in the Western US, but not the joint distribution of female and 65+ and the Western US out of the total population. Weights are applied to the first question only. Weighted results are calculated by normalizing the weights available in each question as respondents screen out, drop off in the survey, or are filtered out in a survey report.
If weights are available, we will show weighted data by default. If you prefer to see unweighted data, you can turn on Raw counts on your survey results page. When a survey is still gathering responses, it may not have enough responses for the weights to reliably match the demographics of the internet population. In those cases, we will still show the weighted data by default, but there will be a warning in the survey report to indicate that the weights may be unreliable. Similarly, when you filter a survey's results down to a subpopulation, like 18-24 year-olds excluding other ages, the warning message may show even if weighting was reliable for the survey prior to filtering.
Note that we cannot calculate weights for respondents who have unknown demographics for age, gender, or geography. Responses with unknown demographics will be excluded when weighting is applied. Turning on Raw counts will display these responses with unknown demographics.
Please note that we don't weight the following types of surveys at this time:
- Surveys that completed before October 15, 2016
- Surveys with any demographic targeting (age, gender, geography) - coming soon!
- Surveys targeted to your website with Website Satisfaction
- Surveys targeted to an audience panel, such as students or small business owners