Understanding What is GPTBot and its functionality:
OpenAI recently published information about GPTBot, a web crawler that utilizes consumed crawl data to answer AI-generated queries in ChatGPT. This bot uses the collected data to improve its knowledge of AI content generation. It is further known for its capacity to gather personally identifiable information (PII) to enhance its models on data understanding.
While researching this topic, we came across a thread on Webmasters where a user had raised concern over GPTBot while stating “Just had over 1,000 hits from this bot, hitting individual pages. As it happens my site automatically served an Error 403 for each hit because the bot is not in my whitelist, nor did it pass the ‘human’ test.”
OpenAI, in one of its recent documents, has shared details on how to block or disallow GPTBot from accessing your website.
OpenAI Shared its GPTBot User Agent Token and String:
User-agent token: GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt:
User-agent: GPTBot
Disallow: /
To allow GPTBot to access only parts of your site you can add the GPTBot token to your site’s robots.txt like this:
User-agent: GPTBot
Allow: /directory-1/
Disallow: /directory-2/
IP Addresses published by OpenAI which are used to crawl websites:
20.15.240.64/28
20.15.240.80/28
20.15.240.96/28
20.15.240.176/28
20.15.241.0/28
20.15.242.128/28
20.15.242.144/28
20.15.242.192/28
40.83.2.64/28
You can disallow GPTBot from crawling your website content in any way by using the same protocol we follow to block other search engine bots. Many search engine companies are also looking for other alternatives to block the crawling of other AI bots using robots.txt.
Popular Searches
List of Search Engines | Top Google Searches | Importance of Digital Marketing | Importance of Website | Youtube SEO Tools | Types of SEO |Website Structure | Benefits of SEO | Cloaking |Google Sandbox | SEO Friendly Website | Blog Commenting for SEO | Server Side Rendering Vs. Client Side Rendering | Youtube Trends | Types of Sitemaps | Social Bookmarking |Off Page SEO Checklist | HTTP Status Codes | Vanity URL | SEO Vs. PPC | Best SEO Blogs | Benefits of LinkedIn Ads | Keyword Density | How to Use Keywords in Blog Posts | Website Migration | Digital Marketing Types | Search Engine Optimization | Canonical Tags | On Page SEO | What is Off Page SEO | Link Building for SEO | Image Optimization | SEO Company in Boston | Dallas SEO | SEO Company Houston
