What is the use of "disallow: /sitepanel/ and disallow: /cgi-bin/"?

by rusty.rosenbaum , in category: SEO , a month ago

What is the use of "disallow: /sitepanel/ and disallow: /cgi-bin/"?

Facebook Twitter LinkedIn Telegram Whatsapp Pocket

1 answer

Member

by delpha , a month ago

@rusty.rosenbaum 

"Disallow: /sitepanel/" and "Disallow: /cgi-bin/" are used in the robots.txt file to instruct web robots (also known as crawlers or spiders) which parts of a website they are not allowed to crawl or index.


Specifically, "Disallow: /sitepanel/" tells robots not to crawl any page or directory that contains "/sitepanel/" in the URL, while "Disallow: /cgi-bin/" instructs them not to crawl any page or directory that contains "/cgi-bin/" in the URL.


The "Disallow" directive is used to restrict access to specific pages or directories that contain sensitive information or that you do not want to be indexed by search engines. By using the "Disallow" directive in the robots.txt file, you can prevent web robots from accessing certain pages or directories on your website.


In summary, "Disallow: /sitepanel/" and "Disallow: /cgi-bin/" are used to prevent web robots from crawling or indexing certain pages or directories on a website, which may contain sensitive information or that you do not want to be publicly accessible.