We use cookies

This site uses cookies from cmlabs to deliver and enhance the quality of its services and to analyze traffic..

Master SEO vocabulary with all essential terms and meanings here.

Robots.txt

Last updated: Aug 12, 2022

WHAT IS ROBOTS.TXT?

Robot.txt is a file used by search engines’ crawlers in your website to classify pages that people can visit. In certain cases, web developers provide a PUBLIC page for users, not search engines such as Google, Bing, and Yahoo.

The purpose of this file is a robot exclusion protocol. It is a de facto standard in the communication law and a border between websites and non-human users.

Robots exclusion protocol or robots txt allows web developers to decide in which part/file/folder of their website that can be accessed by bot or crawler.

How Robot.txt Works

Robot.txt provides instructions for bots. The web crawler will try to crawl robot.txt first before crawling all other pages in a domain. The instructions written on robot.txt will be carried out by the crawler such as no follow, do follow, or other instructions.

The crawler bot will follow the most specific set of instructions in the robots.txt file. If there are contradictory instructions in the file, the bot will follow more detailed instructions.

Robot.txt Function

The functions of robot.txt are as follows:

To Control Crawler/Useragent Activity

One of the main functions of robot.txt is to control the crawler activity on the website. Without using robot.txt, crawlers will crawl all pages including duplicate content. If you don't allow bots to crawl it, then you'll need to add instructions to robot.txt

Blocking Pages From Appearing in the SERP

There are times when you don't want a page on your website to appear in the SERP for several reasons. Case in point, you are not targeting product subcategory pages to appear on the SERP. Therefore, you can instruct the crawler not to display it.

Samples of Codes or Robots.txt Syntax

user-agent: Googlebot disallow: /login  user-agent: Googlebot-news disallow: /media  user-agent: Googlebot-image

Based on the syntax sample above, here is the explanation:

  • Googlebot user-agent is prohibited to crawl into the /loginfolder.
  • Googlebot-news user-agent is prohibited to crawl into the /media folder.
  • Googlebot-image user-agent is allowed to look over into all of the folders inside the www.cmlabs.co website without any limitations.

Sample and Implementation of robots.txt URL

In general cases, robots.txt implementation is NOT VALID for a subdomain, protocol, and port. However, it will be VALID for all files in all of the sub-directories on the host, protocol, and port.

Check the sample location of the robots.txt file in the directory of the website server:

Valid Example

http://robots.co/robots.txt http://robots.co/folder/file/robots.txt

cmlabs

Invalid Example 

http://other.cmlabs.co/robots.txt https://cmlabs.co/robots.txt 
http://cmlabs.co:8181/robots.txt

cmlabs

Important note

When this page is published (on May 21st, 2020), the definition and implementation of the robots.txt are only applicable to Google. In another word, other search engines such as Bing, Yahoo, Yandex, etc do not always use the same standard.

However, a global standardization has been a discussion among the international communities.

Misunderstanding

Robots.txt is not the right file to be used to hide a file or page from the crawler of search engines.

The right answer for: what should we do to hide files from Google? Is by inserting nonindex tag.

<meta name="robots" content="noindex"> <meta name="googlebot" content="noindex">

RESPONSE HEADER

HTTP/1.1 200 OK (…) X-Robots-Tag: noindex (…)

Changes in Protocol Standards

On July, 1st 2019, Google through its official blog announced that robots.txt protocol was prepared to be the Internet standard. It means that all of the search engines will be agreed to this provision.

Robots Exclusion Protocol Draft

Related Terms

User-agent / bot

User-agent is a robot that is used by search engines to crawl all websites on the internet.

Find other important terms in the following SEO Terms:

A:

Algorithm Update

Alt Tag

Anchor Text

AMP

B:

Backlink

Bounce Rate

Broken Link

Bug

Bandwidth

Breadcrumb

C:

Cache

CSS

Cost Per Click

Click

Cookies

Conversion Rate

CTR

Crawl

D:

Do-Follow

Dwell Time

Domain

Domain Authority

Disavow Link

E:

Exact Match Keyword

E-A-T

F:

Featured Snippet

G:

Google

Google Analytic

Github

H:

HTML

HTTP

I:

Impresi

Inbound Link

J:

Javascript

K:

Keyword

Keyword Density

KPI

L:

Long Tail Keyword

Landing Page

LSI Keyword

Linkbait

M:

Meta Description

Metadata

Marketing Mix

N:

No-Follow

Noopener

O:

Organic Search

P:

Paid Search

Permalink

PBN

PWA

Pagespeed Insight

Pixel

Q:

Query

R:

Robots.txt

Referral Traffic

S:

SERP

Semantic HTML

Site Speed

Social Signal

Snippet

Spam,

SSL Certificate

Server

T:

Trafic Web

Title Tag

U:

URL

User Interface (UI)

User Experience (UX)

 

V:

Visit

W:

WWW

Web Crawler

WP Plugin

Woocommerce

X:

XML Sitemap

Y:

YMYL Page

Yandex

Yoast SEO

cmlabs

cmlabs

WDYT, you like my article?

Latest Update
Last updated: Oct 04, 2024
Last updated: Oct 04, 2024
Last updated: Oct 04, 2024

Streamline your analysis with the SEO Tools installed directly in your browser. It's time to become a true SEO expert.

Free on all Chromium-based web browsers

Install it on your browser now? Explore Now cmlabs chrome extension pattern cmlabs chrome extension pattern

Need help?

Tell us your SEO needs, our marketing team will help you find the best solution

Here is the officially recognized list of our team members. Please caution against scam activities and irresponsible individuals who falsely claim affiliation with PT CMLABS INDONESIA DIGITAL (cmlabs). Read more
Marketing Teams

Agita

Marketing

Ask Me
Marketing Teams

Destri

Bizdev

Ask Me
Marketing Teams

Thalia

Bizdev Global

Ask Me
Marketing Teams

Irsa

Marketing

Ask Me
Marketing Teams

Yuliana

Business & Partnership

Ask Me
Marketing Teams

Dwiyan

Partnership

Ask Me
Marketing Teams

Rifqi

Product & Dev

Ask Me
Marketing Teams

Rochman

Product & Dev

Ask Me
Marketing Teams

Said

Career & Internship

Ask Me

There is no current notification..