How to assign specific sitemaps for specific crawler bots in robots.txt?

by jose_gulgowski , in category: SEO Tools , 2 years ago

How to assign specific sitemaps for specific crawler bots in robots.txt?

Facebook Twitter LinkedIn Telegram Whatsapp Pocket

2 answers

by genevieve_boehm , 2 years ago

@jose_gulgowski 

To assign specific sitemaps for specific crawler bots in robots.txt, you can use the "Sitemap" directive followed by the URL of the sitemap, and then specify the user-agent(s) that you want to apply the directive to. Here's an example:

1
2
3
4
5
User-Agent: Googlebot
Sitemap: https://example.com/google_sitemap.xml

User-Agent: Bingbot
Sitemap: https://example.com/bing_sitemap.xml


In the above example, we have specified two user-agents, Googlebot and Bingbot, and assigned separate sitemaps for each of them. The Googlebot will be directed to crawl the sitemap located at https://example.com/google_sitemap.xml, while Bingbot will crawl the sitemap located at https://example.com/bing_sitemap.xml.


Note that you can also use the "Disallow" directive to block certain bots from accessing certain parts of your website, and the "Allow" directive to allow specific bots to access certain parts of your website that you have blocked from others. Here's an example:

1
2
3
4
5
6
7
User-Agent: *
Disallow: /private/
Disallow: /admin/

User-Agent: Googlebot
Disallow: /private/
Allow: /admin/


In the above example, we have blocked all bots from accessing the /private/ and /admin/ directories, but allowed Googlebot to access the /admin/ directory.

by declan_ritchie , 10 months ago

@jose_gulgowski 

This is useful when you want to control the access of specific crawlers to certain parts of your website or when you want to provide different sitemaps to different bots. It's important to note that not all crawlers support the "Sitemap" directive in robots.txt, so it's always a good practice to submit your sitemaps to search engines directly through their respective webmaster tools.