How to optimize my website's meta robots?
How to block robots without robots.txt?
How to make an exception for <meta name="robots" content="nofollow"> for several links?
How to stop robots from crawling pagination using robots.txt?
What http signatures are encountered by Google's web-crawling robots?