Need help?

Tell us your SEO needs, our marketing team will help you find the best solution

Here is the officially recognized list of our team members. Please caution against scam activities and irresponsible individuals who falsely claim affiliation with PT cmlabs Indonesia Digital (cmlabs). Read more
Marketing Teams

Agita

Marketing

Ask Me
Marketing Teams

Irsa

Marketing

Ask Me
Marketing Teams

Yuliana

Business & Partnership

Ask Me
Marketing Teams

Thalia

Business Development Global

Ask Me
Marketing Teams

Robby

Business Development ID

Ask Me
Marketing Teams

Dwiyan

Partnership

Ask Me
Marketing Teams

Rochman

Product & Dev

Ask Me
Marketing Teams

Said

Career & Internship

Ask Me

We regret to inform you that the Mobile Friendly Test is currently unavailable due to system maintenance until further notice.

Check

Stay informed with our new tool, cmlabs Surge. Discover popular trends and events!

Check

Your Opinion Matters! Share your feedback in our Plagiarism Checker Survey?

Check

Discover your business trends effortlessly! The traffic projection calculator is the perfect tool to help you understand demand in your industry sector. Choose your sector and see its traffic projections now!

Check

There is no current notification..

Search Engine

Controlling Googlebot Interaction on A Web Page

news headline image
Googlebot performs its job 24/7 in discovering new or updated pages. However, the interactions of Googlebot can be managed so that they do not visit certain pages within a website. In addition to blocking access, webmasters can also slow down the crawling speed of Googlebot. For more details, please read this article.

Limiting page interactions with Googlebot is done under certain conditions. Managing Googlebot's interactions with different sections of a website page is a process of optimizing how Googlebot, Google's web crawler algorithm, interacts with the content and structure of a website page. Here's how to manage it.

Shared 0 times

Disclaimer: We offer ad-free and organic news content to our readers.


Key Takeaways
  • Website owners can limit Googlebot's interaction on a page using robots.txt.

  • Restricting Googlebot from crawling specific sections of a page is not possible.

  • Methods such as using the data-nosnippet HTML attribute and iframes can be used to limit Googlebot's interaction, but they are not recommended.


Googlebot is a bot used by the Google search engine to explore and index content across the web. By understanding how Googlebot interacts with different sections of a website page, we can ensure that our content is properly shown and indexed by the search engine.

Google has various crawlers specialized for different purposes. Examples of Googlebot agents for crawling pages include Googlebot Smartphone, Googlebot Desktop, Googlebot Image, Googlebot News, and many more.

Website owners have several ways to control Googlebot's interaction within their websites. Here are some controls that can be implemented:

 

1. Reducing Crawling Speed

Having well-crawled pages is an essential part of SEO. However, for large websites or those with high traffic, you may want to reduce the crawling speed to prevent Googlebot from overloading the server. There are two ways to reduce crawling speed or crawling rate:

  • Use Search Console: You can adjust the crawling rate through the Crawler Rate Setting page in Search Console to set the maximum crawling rate limit for your website. This setting will take effect within a few days.
  • Set error pages: If you need to quickly reduce the crawling speed, you can return error pages with status codes 500, 503, or 429. Googlebot will reduce the crawling rate when encountering a large number of such errors.

 

It's important to note that reducing the crawling speed may have some negative consequences, such as:

  • New pages may take longer to be indexed.
  • Existing pages may not be regularly updated in search results.
  • Deleted pages may remain in the index for a longer period.

 

Therefore, reducing the crawling speed should only be done if necessary to maintain the server or improve website performance.

 

2. Blocking Googlebot from Crawling Specific Page Sections

Googlebot is responsible for indexing websites to make them appear in search results. However, there may be times when web owners want to limit Googlebot's interaction with their websites, such as restricting Googlebot from indexing specific pages or directories.

But can we block Googlebot from crawling specific sections of a page? John Mueller and Garry Illyes addressed this question on the Search of The Records podcast. Mueller stated that it is not possible to restrict Googlebot from crawling specific sections of a page.

However, there are alternative methods that web owners can use in this case. Nonetheless, these practices are not recommended as they can cause crawling and indexing issues. Here are three ways to limit Googlebot's interaction with specific sections of a page:

 

1. Using The data-nosnippet HTML Attribute:

The first approach mentioned by Mueller is using the "data-nosnippet" HTML attribute. This attribute can be added to specific HTML elements on a web page to instruct search engines not to display related text in search result snippets.

Search result snippets are brief descriptions that appear below the search result title on a search engine results page (SERP). By using the "data-nosnippet" attribute, website owners can control which parts of their content are displayed in search result snippets.

Here's an example of using the data-nosnippet attribute:

Picture 1: Example of data-nosnippet code

 

 

This approach can be useful for handling sensitive information or providing a different user experience in search results. However, it's important to note that this strategy cannot be considered an ideal solution due to potential limitations or issues associated with its use.

 

2. Using iframes

The second recommended strategy involves using iframes or JavaScript with the source blocked by robots.txt. An iframe is an HTML element used to embed another document within the current HTML document, while JavaScript is a commonly used programming language to enhance interactivity on web pages.

By blocking the source file of an iframe or JavaScript in the site's "robots.txt" file, website owners can prevent search engines from accessing and indexing that specific content. However, Mueller warns against using this strategy.

They caution that using iframe or JavaScript files can cause issues during the crawling and indexing process. These issues can be difficult to diagnose and resolve, thereby hindering the overall visibility of the website in search results.

 

3. Blocking Googlebot According to Guidelines

Website owners can also limit Googlebot's access to crawl the entire page by using the robots.txt file. Here's an example of creating a robots.txt file to block Googlebot from crawling:

  1. Open a text editor or HTML editor application you use.
  2. Create a new text file and name it "robots.txt".
  3. Open the "robots.txt" file with a text editor and add the following rules:

 

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

 

4. Save the "robots.txt" file

5. Upload the "robots.txt" file to the root directory of the website. Usually, the root directory is the main directory of the website on the server.

6. Verify if the "robots.txt" file is accessible by opening the following URL in a browser: http://www.example.com/robots.txt

 

Overall, Mueller and Illyes offer two potential strategies to limit Googlebot's interaction with specific sections of a page. The first strategy involves using the data-nosnippet HTML attribute to control the content displayed in search result snippets, while the second strategy involves using iframes or JavaScript with blocked sources.

However, they emphasize that both strategies are not ideal due to potential limitations, compromises, or complications that may arise. Therefore, it's important to follow the recommended practices provided in Google's guidelines.

Not only Google, but website owners can also control the crawling speed of the Bingbot agent. You can instruct Bingbot to crawl the site faster or slower than the normal crawl rate.


Article Source

As a dedicated news provider, we are committed to accuracy and reliability. We go the extra mile by attaching credible sources to support the data and information we present.

  1. Mengubah Googlebot Crawl Rate
    https://support.google.com/webmasters/answer/48620?hl=en#:~:text=.com%2Ffolder%20.-,Limit%20the%20crawl%20rate,cannot%20increase%20the%20crawl%20rate
  2. Google Crawlers
    https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers 
  3. Mengurangi kecepatan Google dalam crawling
    https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate?hl=id 
  4. Search Engine Journal: “How To Control Googlebot’s Interaction With Your Website” https://www.searchenginejournal.com/how-to-control-googlebots-interaction-with-your-website/488863/ 
  5. Crawl control di Microsoft Bing https://www.bing.com/webmasters/help/crawl-control-55a30303 
  6. Google Developers: “Cara menulis dan mengirimkan file robots.txt” https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt?hl=id 

Disclaimer: All news published by cmlabs has undergone a strict verification and data processing process based on the cmlabs News Publication Guidelines. However, the data or core news we write may undergo changes, reductions, or additions. Consequently, cmlabs assumes no liability for any losses or damages that may arise from the use of this information. We encourage readers to conduct additional verification before making decisions based on the information written on this page.

Shared 0 times

Tati Khumairoh

Tati Khumairoh

An experienced content writer who is eager in creating engaging and impactful written pieces across various industries. Using SEO approach to deliver high-quality content that captivates readers.

Another post from Tati

Written in SEO Guidelines

7 Tips for Writing SEO Friendly Articles to Rank on Google

Wed 24 Apr 2024, 09:25am GMT + 7
Written in Blogs

None Can Guarantee Google Ranking, What Does SEO Agency Sell?

Wed 21 Feb 2024, 11:22am GMT + 7
Written in cmlabs News

Google Update: Circle to Search & AI-Powered Multisearch

Wed 24 Jan 2024, 08:24am GMT + 7
Written in cmlabs News

Structured Data Update for Products: suggestedAge Property

Fri 19 Jan 2024, 08:24am GMT + 7

More from cmlabs News your daily dose of SEO knowledge booster

In the development of its latest search engine, Bing has partnered with GPT-4 to deliver the most advanced search experience. Here are the details.

Bard, an experimental conversational AI service, combines information with language model intelligence. Check out the details here.

With the rapid advancement of AI technology, major search engines like Google and Bing are now equipped with their respective generative AI. Here is the detail.

Featured cmlabs News

Updated 2 months ago

featured cmlabs news

Tati Khumairoh

Previously, Bing allowed users to submit sitemaps without domain authentication or anonymously. However, due to certain reasons, this feature has been removed. Therefore, webmasters need to find alternatives to ensure their websites remain well-indexed.

Sequence Stats Updates

Compare and Discover Same Content Features Enablement

August 28, 2023

On this day, Sequence Stats has introduced two exciting features for the Most Popular Indonesian Online Publishers: "Compare" and "Discover Same Content."

Introducing Sequence Stats' Sentiment Analysis Tool

August 15, 2023

We're thrilled to unveil our latest innovation, the Sentiment Analysis tool which is designed to elevate your media insights and understanding the sentiment.

Introducing People and Brand Mention 2.0

August 09, 2023

As the demand is escalating, we are here to announce the launching of People and Brand Mention 2.0.

WRITE YOUR COMMENT

You must login to comment


All Comments (0)

Sort By