Google New Crawler to Block AI: Google-Extended

In a recent development to give website owners greater control over their content, Google has introduced Google-Extended, a new addition to its web crawling family.

This specialized crawler allows website administrators to determine whether their content can be used to assist Bard and Vertex AI generative APIs or future Google AI products.

 

Functions of Google-Extended

In Google's documentation, here is the information we obtained:

 

 

1. Easier Content Control

With the introduction of Google Extended, website owners can now manage how their content is used in Google AI initiatives. This new crawler serves as a way to grant or deny permission to Google to use its content in specific AI projects.

 

2. Restricting Access for Bard

One of the key features of Google-Extended is its ability to restrict access to AI projects such as Bard. Website administrators who wish to prevent Bard from using their content can easily do so by specifying this preference in their robots.txt file, using the user agent "Google-Extended."

It's important to note that while Google-Extended facilitates content control, it does not affect regular Google crawling operations. Google will continue to crawl websites using the standard Googlebot and other bots. The use of Google Extended only serves as a signal to Google not to incorporate content into projects like Bard or other AI efforts.

 

How to Control Content Using Google-Extended

For its usage, webmasters can update their robots.txt file by adding:

 

User-agent: Google-Extended

Disallow: /

 

By adding these rules to robots.txt, you can start controlling your content to prevent it from being crawled by Bard and Vertex AI.

 

Conclusion

With the introduction of Google-Extended, website owners have an easier time controlling the use of their content by AI. However, regular content control can still be done using robots.txt as usual.

Google-Extended serves as a signal to search engines not to use the page's content for AI development purposes such as Google Bard.