Google Removes Crawl Rate Limiter Tool as of January 2024

On January 8, 2024, Google officially removed the Crawl Rate Limiter tool from Google Search Console. The announcement regarding this removal was made by Google in November 2023, and now the feature has been completely disabled. Here are the details.

What is the Crawl Rate Limiter Tool?

The Crawl Rate Limiter Tool was a feature previously available in Google Search Console and was used to limit the speed at which Googlebot crawled pages on websites.

Simply put, users could control how often Googlebot visited their site to gather information and index pages.

If your site has limited resources (such as low bandwidth or CPU capacity), limiting crawling speed can prevent potential server downtime.

However, Google does not recommend excessively limiting crawling speed unless there are serious issues.

Reasons for Removal

Google explained that the removal of this feature was due to several reasons:

• Improvement of Google's crawling system: Google stated that they have made various improvements to their crawling system, making manual settings by site owners no longer necessary.
• Availability of other tools: Google also provides various other tools to help site owners manage crawling.
• Low usage: Google noted that the Crawl Rate Limiter Tool was rarely used by site owners. This could be due to the effectiveness of the tool. Typically, this tool requires several days to implement a new crawl rate after changes, so users no longer use it.

Impact of Removal

The removal of the Crawl Rate Limiter Tool means that site owners can no longer manually adjust the crawling speed of Googlebot. However, Google ensures that they will continue to maintain optimal crawling speed for each site.

Google explained that they will use several factors to determine crawling speed, such as:

• Site popularity: More popular sites will be crawled more frequently.
• Site size: Larger sites will be crawled more slowly.
• Server resources: Google will consider a site's server resources when determining crawling speed.

Conclusion

The removal of the Crawl Rate Limiter Tool is a significant change in the Google Search Console. However, Google ensures that they will continue to maintain optimal crawling for each site.

If you encounter issues related to crawling being too fast, you can report them through the report form provided by Google.