You can verify if a web crawler accessing your server really is Googlebot (or another Google user-agent). This is useful if you're concerned that spammers or other troublemakers are accessing your site while claiming to be Googlebot. Google doesn't post a public list of IP addresses for webmasters to whitelist. This is because these IP address ranges can change, causing problems for any webmasters who have hard-coded them, so you must run a DNS lookup as described next.
To verify Googlebot as the caller:
- Run a reverse DNS lookup on the accessing IP address from your logs, using the
- Verify that the domain name is in either googlebot.com or google.com
- Run a forward DNS lookup on the domain name retrieved in step 1 using the
hostcommand on the retrieved domain name. Verify that it is the same as the original accessing IP address from your logs.
> host 188.8.131.52 184.108.40.206.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. > host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 220.127.116.11
> host 18.104.22.168 22.214.171.124.in-addr.arpa domain name pointer rate-limited-proxy-66-249-90-77.google.com. > host rate-limited-proxy-66-249-90-77.google.com rate-limited-proxy-66-249-90-77.google.com has address 126.96.36.199