CMLABS' SEO TERMS

WHAT IS WEB CRAWLER

CMLABS / SEO Terms / Dictionary / What Is Web Crawler?

Definition

Web crawlers, spiders, or search engine bots have the task to download and index content from all over the Internet. The purpose of such a bot is to learn about every page on the web, to display the information on SERP when someone searches for it.

It's called "web crawler" because "crawl" is a technical term for accessing websites and obtaining data through software programs automatically.

How Crawler Works

Web crawler bots start indexing from seed, or list of known URLs. They "crawl" web pages at the URL first. When the bot crawls the web page, it will find a hyperlink to another URL, and it will be added to the list of pages that will be explored next.

List of active Web Crawlers on the Internet

Crawler bots from several leading search engines, namely:

  • Google: Googlebot (there are actually two crawlers, Googlebot Desktop and Googlebot Mobile, for desktop and mobile search)
  • Bing: Bingbot
  • Yandex (Russian search engine): Yandex Bot
  • Baidu (Chinese search engine): Baidu Spider

There are also many less common web crawler bots, some of which are not related to any search engine.

Web Crawlers and SEO

SEO stands for search engine optimization, and is a strategy for preparing content for search engine indexing so that the website can appear on the main page in the SERP.

If spider bots do not crawl the website, the website or content page would not be indexed, and certainly will not appear in search results. Therefore, if the website owner wants to get organic traffic from search results, they should not block this bot.

Other Terms/Names

spider

Related Terms

Keyword

Keyword is a keyword typed by the user in a search engine to find an information.

Robot.txt

Robots.txt is a file that is used by crawlers on your web pages to find out which files are allowed and not allowed to be visited.

SERP

SERP is a place where keywords that user inputs are displayed on. SERP contains related information in response to the specific query by a user.

Find other important terms in the following SEO Terms:

EN
ID EN
Enable Notifications.    Ok No thanks