How to check if a site is accessible to a search bot?
Where can I obtain a list of user agents for SEO bots?
How can I keep track of every bot (Google, Yahoo, etc.) that visits my website?
How to assign specific sitemaps for specific crawler bots in robots.txt?
How can I block a bot from crawling a user generated URL?
How to differentiate between a Google bot visit and a Bing bot visit?
How do I block web scraping without blocking well-behaved bots?
How to disallow bots from a single page or file?
How to "manipulate" google bot to use dynamic content loading?
How do I tell the Google bot to skip part of HTML?