How to block a certain type of url in robots.txt or .htaccess?

@rusty.rosenbaum 

To block a certain type of URL using robots.txt, you can use the "Disallow" directive followed by the URL pattern you want to block. For example, if you want to block all URLs that contain the word "example", you can use the following directive:

1
2

User-agent: *
Disallow: /*example*

This will instruct web robots to not crawl any URLs that contain the word "example". Note that this method only blocks web robots from crawling these URLs, it does not prevent users from accessing them directly.

To block a certain type of URL using .htaccess, you can use the "RewriteRule" directive along with regular expressions to match the URL pattern you want to block. For example, if you want to block all URLs that end with ".pdf", you can use the following directive:

1
2

RewriteEngine On
RewriteRule ^(.*).pdf$ - [F,L]

This will return a "403 Forbidden" error to any user or web robot that tries to access a URL that ends with ".pdf". Note that this method requires the use of an Apache web server and may not be available on all hosting environments.