How to disallow nested folders with robots.txt?
How to disallow nested folders with robots.txt?
1 answer
by
rusty.rosenbaum ,
a year ago
@ervin.williamson
To disallow nested folders with robots.txt, you can use a wildcard (*) to block any URLs containing slashes (/). Here's an example:
User-agent: * Disallow: /*/
This rule will block any URLs that have a slash (/) in them, effectively disallowing nested folders. For instance, it will disallow URLs like /folder1/subfolder1/ or /folder2/subfolder2/. However, it will still allow URLs without slashes such as /folder1 and /folder2.
15
Related Threads:
How do exclude specific folders via robots.txt?
What is the difference between `allow: /` & `disallow: ` in robots.txt?
How to disallow search pages from robots.txt?
How to disallow a directory in server with robots.txt?
How to escape $ in robots.txt disallow directive?
How to disallow query strings in robots.txt?