Posted In: robots.txt directives

Robots Exclusion Protocol: robots.txt

What is Robots Exclusion Protocol?Robots Exclusion Protocol is a convention where directives are written with an objective to restrict or channel web crawlers to access parts of website. A part of website may be publicly visible and rest is private for all or some web crawlers.The standard was proposed by Martijn Koster.The robots.txt file need

This Article is TAGGED in robots HTTP header, robots meta tag, robots.txt, robots.txt directives, web crawler and robots.txt. Read more