Home > web crawler and robots.txt

Robots Exclusion Protocol: robots.txt

What is Robots Exclusion Protocol?Robots Exclusion Protocol is a convention where directives are written with an objective to restrict or channel web crawlers to access parts of website. A part of website may be publicly visible and rest is private for all or some web crawlers.The standard was proposed by Martijn Koster.The robots.txt file need

This Article is TAGGED in , , , , . Read more