What is Robots Exclusion Protocol?Robots Exclusion Protocol is a convention where directives are written with an objective to restrict or channel web crawlers to access parts of website. A part of website may be publicly visible and rest is private for all or some web crawlers.The standard was proposed by Martijn Koster.The robots.txt file need
Read morePopular Search Index bots and their User agents that can be used in robots.txtCrawlerUser AgentsDescription Googlebot(Web) Googlebot Generic Google bot crawler for web searchGooglebot News Googlebot-News For Google NewsGooglebot Images Googlebot-Image For Images SearchGooglebot videoGooglebot-Video For Video SearchGooglebot mobile Googlebot-Mobile For Google Mobile web search resultsGoogle Mobile AdSense Mediapartners-Google Mediapartners Google AdSense Mediapartners-Google Mediapartners Google AdsBot landing page quality check AdsBot-Google Only visits landing pages used
Read more