Controlling Googlebot Interaction on A Web Page

Googlebot is a bot used by the Google search engine to explore and index content across the web. By understanding how Googlebot interacts with different sections of a website page, we can ensure that our content is properly shown and indexed by the search engine.

Google has various crawlers specialized for different purposes. Examples of Googlebot agents for crawling pages include Googlebot Smartphone, Googlebot Desktop, Googlebot Image, Googlebot News, and many more.

Website owners have several ways to control Googlebot's interaction within their websites. Here are some controls that can be implemented:

 

1. Reducing Crawling Speed

Having well-crawled pages is an essential part of SEO. However, for large websites or those with high traffic, you may want to reduce the crawling speed to prevent Googlebot from overloading the server. There are two ways to reduce crawling speed or crawling rate:

• Use Search Console: You can adjust the crawling rate through the Crawler Rate Setting page in Search Console to set the maximum crawling rate limit for your website. This setting will take effect within a few days.
• Set error pages: If you need to quickly reduce the crawling speed, you can return error pages with status codes 500, 503, or 429. Googlebot will reduce the crawling rate when encountering a large number of such errors.

 

It's important to note that reducing the crawling speed may have some negative consequences, such as:

• New pages may take longer to be indexed.
• Existing pages may not be regularly updated in search results.
• Deleted pages may remain in the index for a longer period.

 

Therefore, reducing the crawling speed should only be done if necessary to maintain the server or improve website performance.

 

2. Blocking Googlebot from Crawling Specific Page Sections

Googlebot is responsible for indexing websites to make them appear in search results. However, there may be times when web owners want to limit Googlebot's interaction with their websites, such as restricting Googlebot from indexing specific pages or directories.

But can we block Googlebot from crawling specific sections of a page? John Mueller and Garry Illyes addressed this question on the Search of The Records podcast. Mueller stated that it is not possible to restrict Googlebot from crawling specific sections of a page.

However, there are alternative methods that web owners can use in this case. Nonetheless, these practices are not recommended as they can cause crawling and indexing issues. Here are three ways to limit Googlebot's interaction with specific sections of a page:

 

1. Using The data-nosnippet HTML Attribute:

The first approach mentioned by Mueller is using the "data-nosnippet" HTML attribute. This attribute can be added to specific HTML elements on a web page to instruct search engines not to display related text in search result snippets.

Search result snippets are brief descriptions that appear below the search result title on a search engine results page (SERP). By using the "data-nosnippet" attribute, website owners can control which parts of their content are displayed in search result snippets.

Here's an example of using the data-nosnippet attribute:

 

 

This approach can be useful for handling sensitive information or providing a different user experience in search results. However, it's important to note that this strategy cannot be considered an ideal solution due to potential limitations or issues associated with its use.

 

2. Using iframes

The second recommended strategy involves using iframes or JavaScript with the source blocked by robots.txt. An iframe is an HTML element used to embed another document within the current HTML document, while JavaScript is a commonly used programming language to enhance interactivity on web pages.

By blocking the source file of an iframe or JavaScript in the site's "robots.txt" file, website owners can prevent search engines from accessing and indexing that specific content. However, Mueller warns against using this strategy.

They caution that using iframe or JavaScript files can cause issues during the crawling and indexing process. These issues can be difficult to diagnose and resolve, thereby hindering the overall visibility of the website in search results.

 

3. Blocking Googlebot According to Guidelines

Website owners can also limit Googlebot's access to crawl the entire page by using the robots.txt file. Here's an example of creating a robots.txt file to block Googlebot from crawling:

1. Open a text editor or HTML editor application you use.
2. Create a new text file and name it "robots.txt".
3. Open the "robots.txt" file with a text editor and add the following rules:

 

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

 

4. Save the "robots.txt" file

5. Upload the "robots.txt" file to the root directory of the website. Usually, the root directory is the main directory of the website on the server.

6. Verify if the "robots.txt" file is accessible by opening the following URL in a browser: http://www.example.com/robots.txt

 

Overall, Mueller and Illyes offer two potential strategies to limit Googlebot's interaction with specific sections of a page. The first strategy involves using the data-nosnippet HTML attribute to control the content displayed in search result snippets, while the second strategy involves using iframes or JavaScript with blocked sources.

However, they emphasize that both strategies are not ideal due to potential limitations, compromises, or complications that may arise. Therefore, it's important to follow the recommended practices provided in Google's guidelines.

Not only Google, but website owners can also control the crawling speed of the Bingbot agent. You can instruct Bingbot to crawl the site faster or slower than the normal crawl rate.