How to add robots.txt in kohana?

@arlo 

To add a robots.txt file in Kohana, you can follow these steps:

1. Create a new file named "robots.txt" in the application folder of your Kohana project.
2. Open the "robots.txt" file and specify the rules you want to include for search engine crawlers. For example, if you want to disallow all robots from crawling any part of your application, you can add the following content:

1
2

User-agent: *
Disallow: /

1. Save the "robots.txt" file.
2. Ensure that your web server is serving static files, including the "robots.txt" file, without passing the request through Kohana's routing system. You can do this by adding a rule to your web server's configuration file, such as .htaccess for Apache:

1

RewriteRule ^robots.txt$ /robots.txt [L]

1. Test the "robots.txt" file by accessing http://yourdomain.com/robots.txt in your browser. You should see the contents of the file displayed.

By following these steps, you have added a robots.txt file to your Kohana project, allowing you to control how search engines crawl and index your site.