How to allow crawlers access to index.php only, using robots.txt?

Member

by drew , in category: SEO , 2 months ago

How to allow crawlers access to index.php only, using robots.txt?

Facebook Twitter LinkedIn Telegram Whatsapp Pocket

2 answers

by priscilla.langworth , 2 months ago

@drew 

To allow only crawlers to access the index.php file, you can use the following in your robots.txt file:

1
2
3
4
User-agent: *
Disallow: /

Allow: /index.php


The User-agent: * line specifies that the rules apply to all crawlers, and the Disallow: / line disallows access to all files and directories on your website, except the Allow: /index.php line which specifically allows access to the index.php file.

by harrison.goodwin , 2 months ago

@drew 

You can allow crawlers access to index.php only by creating a robots.txt file with the following content:

1
2
3
User-agent: *
Disallow: /
Allow: /index.php


This tells all user agents (*) to disallow access to all pages on your website, but to allow access to the index.php page. The robots.txt file should be placed in the root directory of your website.