Need help?

Tell us your SEO needs, our marketing team will help you find the best solution

Here is the officially recognized list of our team members. Please caution against scam activities and irresponsible individuals who falsely claim affiliation with PT CMLABS INDONESIA DIGITAL (cmlabs). Read more
Marketing Teams

Agita

Marketing

Ask Me
Marketing Teams

Destri

Bizdev

Ask Me
Marketing Teams

Thalia

Bizdev Global

Ask Me
Marketing Teams

Irsa

Marketing

Ask Me
Marketing Teams

Yuliana

Business & Partnership

Ask Me
Marketing Teams

Dwiyan

Partnership

Ask Me
Marketing Teams

Rifqi

Product & Dev

Ask Me
Marketing Teams

Rochman

Product & Dev

Ask Me
Marketing Teams

Said

Career & Internship

Ask Me

There is no current notification..

Search Engine

Google Updates All Information on Crawler Documentation

news headline image

Google has just updated the information about its crawler and divided the discussion into various separate pages for more detail. Check it out here!

Shared 0 times

Disclaimer: We offer ad-free and organic news content to our readers.


Key Takeaways
  • Google has just updated the information about its crawler and divided the discussion into various separate pages.

  • The purpose of this update is to provide a more comprehensive explanation of the crawler bot and the products affected.

  • Additionally, Google has also provided the latest information on the user agent string from GoogleProducer, content encoding, and the migration of JavaScript guidelines from the blog to the documentation.


In a significant update, Google has enhanced its crawler documentation, now featuring additional details and reorganized content into three distinct sections, each focusing more specifically on the topic.

Previously, the documentation was consolidated into a single page that encompassed all aspects of crawler functionality. 

With this latest revision, Google has separated the information and introduced new sections that clarify the products impacted by each crawler and provide guidance on using the user agent token. This update aims to deepen the understanding of all aspects related to crawler pages.

 

What’s Changed in the Crawler Documentation?

Google has added two new sections across all crawlers: the impacted products and examples of robots.txt groups. Below is a screenshot showing the addition of these two new sections in Google’s crawler documentation:

Figure 1: Two new sections in Google’s crawler documentation.
Figure 1: Two new sections in Google’s crawler documentation.

Google stated that the reason for updating this information is the length of the discussion, which limited their ability to add new information about their crawlers.

Additionally, in the changelog, Google noted that they have added several pieces of information, including updates to the HTTP user agent string from GoogleProducer, new content encoding information, and the migration of JavaScript guidelines from the blog to the documentation, allowing users to study the guidelines more freely.

 

Google's General Crawler and Its Impact on Google Products

Google's documentation for the general crawler explains the user agent string in HTTP requests, the user agent token for the User-agent: line in robots.txt, and the Google products affected by crawling preferences.

Google also clarifies that some crawlers have more than one token, allowing users to match just one token to qualify.

Here are the details of the crawler list along with the affected products:

  • Googlebot: This crawling preference affects Google Search, including all its features, Discover, Google Images, Google Video, and Google News.
  • Googlebot Image: This crawling preference impacts Google Images, Discover, Google Video, and all Google Search features that include images, logos, and favicons.
  • Googlebot Video: This preference affects Google Search features related to video and other products that depend on video.
  • Googlebot News: This preference influences all Google News features, such as the News tab in Google Search and the Google News app.
  • Google Storebot: This preference affects all Google Shopping features, including the Shopping tab in Google Search and Google Shopping.
  • Google Inspection Tool: This preference impacts search testing tools like the Rich Result Test and URL inspection in Search Console. This crawler does not affect Google Search or other products.
  • GoogleOther: This preference does not impact specific products, as it is a general crawler used to retrieve publicly accessible content from websites.
  • GoogleOther-Image: Similar to GoogleOther, this crawler does not affect specific products. GoogleOther-Image is the version of GoogleOther used to fetch URLs of publicly uploaded images.
  • GoogleOther-Video: Like GoogleOther, this crawler does not impact specific products. Its function is the same as GoogleOther-Image, but it retrieves video URLs.
  • Google-CloudVertexBot: This crawler affects crawling requested by site owners to build Vertex AI Agents and does not impact Google Search or other products.
  • Google-Extended: This crawler is an independent product token that website owners can use to manage whether their sites contribute to enhancing Gemini Apps and the generative Vertex AI API, including the upcoming model generations driving these products. This crawler does not affect site inclusion or ranking in search results.

 

Crawlers for Special Cases by Google and Their Impact on Products

In addition to general crawlers, Google also explains crawlers for special cases and their impact on products, which can be summarized as follows:

  • APIs-Google: This crawler affects push notification messages from Google APIs.
  • AdsBot Mobile Web: This crawler influences Google Ads' ability to assess the quality of ads on a webpage.
  • AdsBot: This preference has the same impact as AdsBot Mobile Web.
  • AdSense: This preference affects Google AdSense, where its crawler visits participating sites to provide relevant ads.
  • Google-Safety: The user agent Google-Safety is responsible for searches specifically aimed at identifying abuse, such as discovering malware in links posted publicly on Google properties. Thus, its usage is not affected by search preferences.

 

List of User-Triggered Fetchers by Google

Another page added by Google as part of the crawler documentation update is dedicated to user-triggered fetchers. This page discusses bots activated by user requests to perform fetching functions within Google products, along with a list of these bots.

Here is the list of user-triggered fetchers and their associated products:

  • Feedfetcher: This bot is used to crawl RSS or Atom feeds for Google News and PubSubHubbub.
  • Google Publisher Center: This bot retrieves and processes feeds explicitly provided by publishers on the Google News landing page.
  • Google Read Aloud: This bot fetches and reads web pages using text-to-speech (TTS) based on user requests.
  • Google Site Verifier: This bot retrieves verification tokens from Search Console.

The changes and additional information made by Google for each of its crawlers have made the content more comprehensive. This allows you to understand how these crawlers impact various aspects of Google, even if not all of them have a direct influence.


News Source

As a dedicated news provider, we are committed to accuracy and reliability. We go the extra mile by attaching credible sources to support the data and information we present.

  1. Google Search Central (common crawlers) - https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
  2. Google Search Central (special-case crawlers) - https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers
  3. Google Search Central (user-triggered fetchers) - https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers
  4. Google Search Central (latest documentation updates) - https://developers.google.com/search/updates#reorganizing-the-crawler-documentation

Disclaimer: All news published by cmlabs has undergone a strict verification and data processing process based on the cmlabs News Publication Guidelines. However, the data or core news we write may undergo changes, reductions, or additions. Consequently, cmlabs assumes no liability for any losses or damages that may arise from the use of this information. We encourage readers to conduct additional verification before making decisions based on the information written on this page.

Shared 0 times

alivia-ariatna

Alivia Ariatna

As an experienced SEO content writer, I specialize in crafting compelling, keyword-optimized content with extensive research on it. I stay updated with the latest SEO trends and best practices to ensure my content meets both user intent and search engine requirements.

Another post from Alivia

Written in cmlabs News

Google Adds New Recommendations for Product Markup

Tue 08 Oct 2024, 11:24am GMT + 7
Written in cmlabs News

Bing Develops Generative AI Search for Quality Results

Tue 08 Oct 2024, 10:57am GMT + 7
Written in cmlabs News

Google Search Now Supports AVIF Image File Format

Tue 10 Sep 2024, 10:59am GMT + 7
Written in cmlabs News

xAI Released Grok-2 & Grok-2 Mini to Make AI-Generated Images

Tue 10 Sep 2024, 09:36am GMT + 7

More from cmlabs News your daily dose of SEO knowledge booster

In the development of its latest search engine, Bing has partnered with GPT-4 to deliver the most advanced search experience. Here are the details.

Bard, an experimental conversational AI service, combines information with language model intelligence. Check out the details here.

With the rapid advancement of AI technology, major search engines like Google and Bing are now equipped with their respective generative AI. Here is the detail.

WRITE YOUR COMMENT

You must login to comment


All Comments (0)

Sort By