How to stop robots crawling pagination using robots.txt?
What is a meta robot tag, and how does it affect SEO?
How to block robots without robots.txt?
How to optimize my website's meta robots?
How to make an exception for <meta name="robots" content="nofollow"> for several links?
How to stop robots from crawling pagination using robots.txt?
What http signatures are encountered by Google's web-crawling robots?