To set up a
robots.txt file for WordPress, follow these steps:
1 2 3
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/
These lines tell search engine crawlers to not index the
/wp-includes/ directories, as these are typically not meant to be indexed or accessed by search engines.
After completing these steps, the
robots.txt file will be in effect and guide search engine crawlers on which parts of your WordPress site should not be indexed. Do note that some SEO plugins might also generate and manage the
robots.txt file for you, so you may need to consult their instructions or settings if you are using such a plugin.