This post was most recently updated on December 28th, 2016
Popular Search Index bots and their User agents that can be used in robots.txt
| Crawler | User Agents | Description |
| Googlebot (Web) |
Googlebot | Generic Google bot crawler for web search |
| Googlebot News | Googlebot-News | For Google News |
| Googlebot Images | Googlebot-Image | For Images Search |
| Googlebot video | Googlebot-Video | For Video Search |
| Googlebot mobile | Googlebot-Mobile | For Google Mobile web search results |
| Google Mobile AdSense | Mediapartners-Google Mediapartners | |
| Google AdSense | Mediapartners-Google Mediapartners | |
| Google AdsBot landing page quality check | AdsBot-Google | Only visits landing pages used in AdWords campaigns |
| YandexBot | YandexBot | main indexing bot |
| YandexMedia | YandexMedia | indexes multimedia data |
| YandexImages | YandexImages | Yandex images indexer |
| YandexCatalog | YandexCatalog | Yandex Catalog robot |
| YaDirectFetcher | YaDirectFetcher | contextual ad on search |
| YandexBlogs | YandexBlogs | the blog search robot that indexes comments to the posts |
| YandexNews | YandexNews | Yandex News |
| YandexPagechecker | YandexPagechecker | a robot that accesses the page when microformats are validated using the Microformats validator form |
| YandexMetrica | YandexMetrica | yandex Metrica Robot to measure site traffic |
| YandexMarket | YandexMarket | Yandex product store |
| YandexCalendar | YandexCalendar | Yandex calendar to manage ur dates |
| Yahoo Pipes 1.0 | Yahoo Pipes 1.0 | Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web. |
| Yahoo! Slurp | Yahoo! Slurp | Yahoo Web search |
