Witness an Increase in your ROI
Unlock higher rankings, quality traffic, and amplified conversions through tailored award-winning SEO strategies.
This is important news for those who use Reverse DNS lookup to verify Googlebot. Google has published a list of all IP addresses used by Googlebot to crawl and discover websites for Google Search.
Where can I find the IP list?
The list of Googlebot IP addresses has been released in two JSON files.
- JSON 1 – This list will help you identify Googlebot by matching the crawler’s IP address with the IP addresses in the JSON file.
- JSON 2 – For all other Google crawlers, you can compare the crawler’s IP address against the complete list of Google IP addresses in the JSON file.
JSON can be a bit overwhelming for everyone to understand. Especially when it comes to the JSON files that include the subnet masks. If you’re having difficulties reading the IP address in the JSON file, we recommend calculating the IP ranges using this tool.
Why do I need Googlebot IP Addresses?
Crawlers scraping their website is something that no website owner appreciates. This could also lead to sensitive information being compromised. Crawlers with malicious intent can scrape your entire website and steal all of your data. You’re at a high risk of a data breach as a result of this.
However, for SEO, you need to allow the Googlebot agent to crawl your website to reward your SEO efforts in Google Search. Having an entire list of Googlebot IP addresses ensures that you can verify whether the agent requesting to crawl your website is indeed the Googlebot.
There are two methods that site owners can use to verify Googlebot:
- Manual – This can be done using the command-line tools. You can make use of 1 v 1 lookups. This works for small scale lookups
- Automated – If you have a large list of IP addresses you want to look up, command-line won’t be the ideal solution. You can use an automated solution to look up all requested IP addresses against a list of verified Googlebot addresses.
This is where the full list of IP addresses used by the Googlebot comes in handy.
Key Takeaway
A website may go down or even crash due to fraudulent and spammy bots sending a large number of requests. Reverse DNS lookup and other methods need to be used from time to time to ensure that only useful bots are crawling your website.
Blocking all the IP addresses isn’t a solution that fits well as you might end up blocking the Googlebot. This will have a significant impact on your Search Engine Optimization performance. You can tell if a request is from Googlebot if you have a list of verified Googlebot IP addresses.
