How can I keep track of every bot (Google, Yahoo, etc.) that visits my website?
To keep track of every bot that visits your website, you can use server logs. These logs will typically include information such as the IP address, user agent, and request method for every visit to your site, including those made by bots. You can analyze these logs to identify bot traffic and keep track of which bots are visiting your site, how frequently, and for what purposes. Additionally, you can use tools such as Google Analytics to get a better understanding of the types of bots that are visiting your site and the behavior of these bots.