What is OpenAI’s GPTBot and Everything You Need to Know About It

What is OpenAI’s GPTBot and Everything You Need to Know About It

Witness an Increase in your ROI

Unlock higher rankings, quality traffic, and amplified conversions through tailored award-winning SEO strategies.

    Getting your Trinity Audio player ready...
    0
    (0)

    Understanding What is GPTBot and its functionality:

    OpenAI recently published information about GPTBot, a web crawler that utilizes consumed crawl data to answer AI-generated queries in ChatGPT. This bot uses the collected data to improve its knowledge of AI content generation. It is further known for its capacity to gather personally identifiable information (PII) to enhance its models on data understanding.

    AI vs Human - Infidigit

    While researching this topic, we came across a thread on Webmasters where a user had raised concern over GPTBot while stating “Just had over 1,000 hits from this bot, hitting individual pages. As it happens my site automatically served an Error 403 for each hit because the bot is not in my whitelist, nor did it pass the ‘human’ test.”

    OpenAI, in one of its recent documents, has shared details on how to block or disallow GPTBot from accessing your website.

    OpenAI Shared its GPTBot User Agent Token and String: 

    User-agent token: GPTBot

    Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

    To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt:

    User-agent: GPTBot

    Disallow: /

    To allow GPTBot to access only parts of your site you can add the GPTBot token to your site’s robots.txt like this:

    User-agent: GPTBot

    Allow: /directory-1/

    Disallow: /directory-2/

    IP Addresses published by OpenAI which are used to crawl websites:

    20.15.240.64/28

    20.15.240.80/28

    20.15.240.96/28

    20.15.240.176/28

    20.15.241.0/28

    20.15.242.128/28

    20.15.242.144/28

    20.15.242.192/28

    40.83.2.64/28

    You can disallow GPTBot from crawling your website content in any way by using the same protocol we follow to block other search engine bots. Many search engine companies are also looking for other alternatives to block the crawling of other AI bots using robots.txt.

    How useful was this post?

    0 / 5. 0

    Leave a Comment


    Secrets to be the first on search, right in your inbox.

    Subscribe to our newsletter and get carefully curated SEO news, articles, resources and inspiration on-the-go.

    Share this article

    OpenAI's GPTBot - Infidigit

    What is OpenAI’s GPTBot and Everything You Need to Know About It