Posted In: google bots

Robots.txt: User agents

Popular Search Index bots and their User agents that can be used in robots.txtCrawlerUser AgentsDescription   Googlebot(Web) Googlebot Generic Google bot crawler for web searchGooglebot News Googlebot-News For Google NewsGooglebot Images Googlebot-Image For Images SearchGooglebot videoGooglebot-Video  For Video SearchGooglebot mobile Googlebot-Mobile For Google Mobile web search resultsGoogle Mobile AdSense Mediapartners-Google Mediapartners Google AdSense Mediapartners-Google Mediapartners Google AdsBot landing page quality check AdsBot-Google Only visits landing pages used

This Article is TAGGED in google bots, robots.txt, SEO, user agents, web crawler, yahoo bots, yandex bots. Read more