robots-txt-checklist

Robots.txt Checklist

Before moving on the most essential topic of Robots.txt checklist we first need to understand the basic elements of Robots.txt.


  • 1. User-agent allow you to specify the robot name for Google, Bing, Yahoo, Yandex or any other.
  • 2. Disallow guide the robots not to crawl.
  • 3. Allow section allow the robots to crawl.
  • 4. Crawl-delay guide the robots to wait for a certain amount of time before starting the crawling.
  • 5. Noindex guide the search engine to remove page from indexing.
  • 6. Using # you can specify the comment. This help developer and other to better understand the code.
  • 7. Supporting strings like * and $ strings for text matching and URL.


Most important points relating to Robots.txt.

  • 1. Always have robots.txt file in the root. For example if I am having a website with a name alljobshub.com then the path of the robots.txt will be alljobshub.com/robots.txt
  • 2. Include all the files and directory which you don’t want to index.
  • 3. You can also specify the file or directory which you want to index. So, based on your requirement you can include the files.
  • 4. There is no need to block the java script and CSS. So, don’t disallow them.
  • 5. You can manage the crawl time by using crawl-delay. But the best way to manage the crawling time is to use Google Search Console.
  • 6. To check robots.txt file you can use Google Search Console. Using it you can validate it.
  • 7. Make sure that the size of Robots.txt should not be more than 500 kb.


Syntax overview –

Indexing of everything using robots.txt over search engine

User-agent: *

Disallow:


Blocking indexing of everything using robots.txt from search engine crawler

User-agent: *

Disallow: /


For detail overview on robots.txt please explore our blog – Meta Robots and Linking Attribute.


Get the Digital Marketing Updates and Insight with AllJobsHub




0 1 1 0 1

One thought on “Robots.txt Checklist”

  1. As a new blogger, i did not think that ROBOT.TXT file is so important. After reading it I understood. Thanks for sharing us this valuable content.

Leave a Reply

Your email address will not be published. Required fields are marked *