robots.txt file

Get the Detail Overview on Robots.txt

What is Robots.txt?

Robots.txt is a file that guide the search engine crawler or spider not to crawl or index pages that part of it. Webmaster using this file for blocking unwanted pages from search engine indexing.

To implement robots.txt file you need to create a simple txt file. Integrate the code in the file and save it with the name “robots”. Upload this file over the server in the root folder.


Detail overview on Robots.txt file –

Case 1: If you want to block all the content of your website from all the crawler/spider of different search engines use the below mentioned code.

User-agent: *
Disallow: /

Case 2: Want to craw and index each and every thing.

User-agent: *
Allow: /

Case 3: To disallow indexing of specific folder.

User-agent: *
Disallow: /folder/

Case 4: Want to block specific folder but allowing indexing of specific file of that folder.
In the below example you can see we are allowing indexing of myfile.html but disallowing Googlebot to index “folder1”.

User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html


Based on your requirement you include any number of folder or pages in robots.txt file.

Let’s have a look on the robots.txt file which we have created for our website.

User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-admin
Disallow: /demo
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-includes/js
Disallow: /trackback
Disallow: */trackback
Disallow: /*?*
Disallow: /*?
Disallow: /*~*
Disallow: /*~
Disallow: /backup
Disallow: /ax
Disallow: /cgi-bin


If you want to block specific page from indexing, you can use robots Meta tag. This tag need to be integrated in the header part of the page.

The below is the syntax of Meta robots tag.

meta name=”robots” content=”noindex”

Description: This code will block the page from index over search engine. But the link will be followed.

meta name=”robots” content=”noindex,nofollow”

Description: This code will block the page from indexing and the link will not be followed.

0 0 0 0 0

Leave a Reply

Your email address will not be published. Required fields are marked *