|
Getting your Trinity Audio player ready...
|
Understanding What is GPTBot and its functionality:
OpenAI recently published information about GPTBot, a web crawler that utilizes consumed crawl data to answer AI-generated queries in ChatGPT. This bot uses the collected data to improve its knowledge of AI content generation. It is further known for its capacity to gather personally identifiable information (PII) to enhance its models on data understanding.
While researching this topic, we came across a thread on Webmasters where a user had raised concern over GPTBot while stating “Just had over 1,000 hits from this bot, hitting individual pages. As it happens my site automatically served an Error 403 for each hit because the bot is not in my whitelist, nor did it pass the ‘human’ test.”
OpenAI, in one of its recent documents, has shared details on how to block or disallow GPTBot from accessing your website.
OpenAI Shared its GPTBot User Agent Token and String:
User-agent token: GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt:
User-agent: GPTBot
Disallow: /
To allow GPTBot to access only parts of your site you can add the GPTBot token to your site’s robots.txt like this:
User-agent: GPTBot
Allow: /directory-1/
Disallow: /directory-2/
IP Addresses published by OpenAI which are used to crawl websites:
20.15.240.64/28
20.15.240.80/28
20.15.240.96/28
20.15.240.176/28
20.15.241.0/28
20.15.242.128/28
20.15.242.144/28
20.15.242.192/28
40.83.2.64/28
You can disallow GPTBot from crawling your website content in any way by using the same protocol we follow to block other search engine bots. Many search engine companies are also looking for other alternatives to block the crawling of other AI bots using robots.txt.
Popular Searches
