How to allow crawlers access to index.php only, using robots.txt?

@drew 

To allow only crawlers to access the index.php file, you can use the following in your robots.txt file:

1
2
3
4

User-agent: *
Disallow: /

Allow: /index.php

The User-agent: * line specifies that the rules apply to all crawlers, and the Disallow: / line disallows access to all files and directories on your website, except the Allow: /index.php line which specifically allows access to the index.php file.

@drew 

You can allow crawlers access to index.php only by creating a robots.txt file with the following content:

1
2
3

User-agent: *
Disallow: /
Allow: /index.php

This tells all user agents (*) to disallow access to all pages on your website, but to allow access to the index.php page. The robots.txt file should be placed in the root directory of your website.