Open AI Launched GPTBot as A Web Crawler

Earlier this month, OpenAI, as the provider platform for ChatGPT, launched a web crawler called GPTBot to access information on the internet and provide it on their platform. How do search engines respond, and how can you limit the crawler's access to your content? Find the answers below.

 

What is GPTBot?

In a move to advance artificial intelligence (AI) technology, OpenAI has introduced GPTBot, an advanced web crawler designed to enhance the capabilities of highly anticipated AI models like GPT-4 and GPT-5.

GPTBot plays a crucial role in AI development. This web crawler can be identified by the unique user agent token "GPTBot" and the following user agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

 

Function and Limitations of GPTBot

In OpenAI's documentation, GPTBot will be used carefully to scan and collect data from web pages, with a promise to contribute to the advancement of future AI models.

They also ensure data source quality and ethics through strict filtering processes. For example, on websites that require payment and involve the collection of personally identifiable information (PII) or contain text that violates established policies, GPTBot will be systematically filtered.

 

Controlling GPTBot Interactions on Websites

Through the webmaster world discussion forum, many webmasters have noticed frequent visits from GPTBot to their websites. This has made them more cautious and prompted them to take actions like disallowing the robot.

If you believe this AI robot should not gather information from your site, here's how to limit it.

Implement specific directives in your site's robots.txt file. If you want to completely prevent GPTBot from accessing any part of your site, include the command:

 

User-agent: GPTBot

Disallow: /

 

By adding the above command, you have effectively told GPTBot not to crawl any part of your site.

However, if you want to grant specific access to GPTBot to certain parts of your site, you can implement commands like this:

 

User-agent: GPTBot

Allow: /directory-1/

Disallow: /directory-2/

 

In this scenario, GPTBot is allowed to access content in "directory-1," but explicitly prevented from crawling "directory-2." This control allows you to customize GPTBot's access according to your site's preferences and priorities.

 

Search Engine Responses

In response to OpenAI's move, Google seems to be considering alternative measures to prevent harmful GPTBot crawlers. In its blog, Google announced its intention to develop a complementary protocol to robots.txt, which has been in use for over three decades. 

The motivation behind this step is the emergence of new AI technology, not only from Google but also from other industry players.

This development follows reports of OpenAI leveraging internet content for its ChatGPT service. It's worth noting that this announcement is not final. Google's statement emphasizes its commitment to engage in discussions and collaboration with the online community.

In a statement released on Twitter, Google said:

 

"Today, we're starting a public discussion, inviting members of the web and AI community to provide input on the approach to the complementary protocol. We want to engage a variety of voices from different web publishers, civil society, academics, and other fields worldwide to join this discussion, and we'll gather those interested to participate in the coming months."

That concludes the overview of GPTBot, its current operation, and how you can limit its interactions on your website.