Robots.txt Checklist

Before moving on the most essential topic of Robots.txt checklist we first need to understand the basic elements of Robots.txt.

  • 1. User-agent allow you to specify the robot name for Google, Bing, Yahoo, Yandex or any other.
  • 2. Disallow guide the robots not to crawl.
  • 3. Allow section allow the robots to crawl.
  • 4. Crawl-delay guide the robots to wait for a certain amount of time before starting the crawling.
  • 5. Noindex guide the search engine to remove page from indexing.
  • 6. Using # you can specify the comment. This help developer and other to better understand the code.
  • 7. Supporting strings like * and $ strings for text matching and URL.

Most important points relating to Robots.txt.

  • 1. Always have robots.txt file in the root. For example if I am having a website with a name then the path of the robots.txt will be
  • 2. Include all the files and directory which you don’t want to index.
  • 3. You can also specify the file or directory which you want to index. So, based on your requirement you can include the files.
  • 4. There is no need to block the java script and CSS. So, don’t disallow them.
  • 5. You can manage the crawl time by using crawl-delay. But the best way to manage the crawling time is to use Google Search Console.
  • 6. To check robots.txt file you can use Google Search Console. Using it you can validate it.
  • 7. Make sure that the size of Robots.txt should not be more than 500 kb.

Syntax overview –

Indexing of everything using robots.txt over search engine

User-agent: *


Blocking indexing of everything using robots.txt from search engine crawler

User-agent: *

Disallow: /

For detail overview on robots.txt please explore our blog – Meta Robots and Linking Attribute.

Get the Digital Marketing Updates and Insight with AllJobsHub

0 1 1 0 1

One thought on “Robots.txt Checklist”

  1. As a new blogger, i did not think that ROBOT.TXT file is so important. After reading it I understood. Thanks for sharing us this valuable content.

Leave a Reply

Your email address will not be published. Required fields are marked *