How to assign specific sitemaps for specific crawler bots in robots.txt?

by jose_gulgowski , in category: SEO Tools , 6 months ago

How to assign specific sitemaps for specific crawler bots in robots.txt?

Facebook Twitter LinkedIn Telegram Whatsapp Pocket

1 answer

by genevieve_boehm , 6 months ago

@jose_gulgowski 

To assign specific sitemaps for specific crawler bots in robots.txt, you can use the "Sitemap" directive followed by the URL of the sitemap, and then specify the user-agent(s) that you want to apply the directive to. Here's an example:

1
2
3
4
5
User-Agent: Googlebot
Sitemap: https://example.com/google_sitemap.xml

User-Agent: Bingbot
Sitemap: https://example.com/bing_sitemap.xml


In the above example, we have specified two user-agents, Googlebot and Bingbot, and assigned separate sitemaps for each of them. The Googlebot will be directed to crawl the sitemap located at https://example.com/google_sitemap.xml, while Bingbot will crawl the sitemap located at https://example.com/bing_sitemap.xml.


Note that you can also use the "Disallow" directive to block certain bots from accessing certain parts of your website, and the "Allow" directive to allow specific bots to access certain parts of your website that you have blocked from others. Here's an example:

1
2
3
4
5
6
7
User-Agent: *
Disallow: /private/
Disallow: /admin/

User-Agent: Googlebot
Disallow: /private/
Allow: /admin/


In the above example, we have blocked all bots from accessing the /private/ and /admin/ directories, but allowed Googlebot to access the /admin/ directory.