Google Released Robot.txt Report in Google Search Console

Google has released a new robot.txt report in Google Search Console, replacing the previously existing Robot.txt Tester. This update allows website owners and developers to manage their robot.txt files. What information can be surfaced in this report? Let's delve into it.

The Robot.txt Report in Google Search Console

This latest robot.txt report not only provides basic information about the robot.txt files that can be found but also offers in-depth insights into the interaction between crawlers and these files.

To access it, the report is located in the Settings menu, and you can open the detailed report in the robot.txt column. The following is an overview of the new robot.txt report.

Reported Information

Google explains that in this report, users can access various details about robot.txt, such as:

• Which robot.txt files Google found for the top 20 hosts on your site. In other words, the report gives an overview of how Google interacts with the robot.txt files on the top 20 main hosts of the site.
• Users can now see when the Google crawler last visited their robot.txt files.
• Tracking any warnings or issues that may be encountered during the crawling process of robot.txt files.

In addition to the above information, there is a crucial feature of this report: the ability to "recrawl" the robot.txt file. If users encounter issues or need to quickly update the file, they can easily request a recrawl through the provided interface.

This allows site owners to manage and synchronize their robot.txt information more effectively. However, Google explains that requesting a recrawl is not necessary as the crawler regularly visits updated robot.txt files.

Here are some conditions that might require a recrawl request on the robot.txt file:

• If users have changed the robot.txt rules to unblock some important URLs and want to inform Google quickly.
• If users have successfully fixed fetch errors or other critical errors and want to notify Google for further action.