Home > robots.txt

Robots Exclusion Protocol: robots.txt

What is Robots Exclusion Protocol?Robots Exclusion Protocol is a convention where directives are written with an objective to restrict or channel web crawlers to access parts of website. A part of website may be publicly visible and rest is private for all or some web crawlers.The standard was proposed by Martijn Koster.The robots.txt file need

This Article is TAGGED in , , , , . Read more

Robots.txt: User agents

Popular Search Index bots and their User agents that can be used in robots.txtCrawlerUser AgentsDescription   Googlebot(Web) Googlebot Generic Google bot crawler for web searchGooglebot News Googlebot-News For Google NewsGooglebot Images Googlebot-Image For Images SearchGooglebot videoGooglebot-Video  For Video SearchGooglebot mobile Googlebot-Mobile For Google Mobile web search resultsGoogle Mobile AdSense Mediapartners-Google Mediapartners Google AdSense Mediapartners-Google Mediapartners Google AdsBot landing page quality check AdsBot-Google Only visits landing pages used

This Article is TAGGED in , , , , , , . Read more