Member
talon_jacobson
What is the purpose of robot.txt to be put in the root directory?
cielo
It is exclusion protocol robot.txt communicate with the crawler and escape the links that not follow.