What does "disallow: /*?" mean in twitter robots.txt?

by alvera_smith , in category: SEO , 8 months ago

What does "disallow: /*?" mean in twitter robots.txt?

Facebook Twitter LinkedIn Telegram Whatsapp Pocket

1 answer

by cameron_walter , 8 months ago


In Twitter's robots.txt file, the line "disallow: /*?" is used to instruct web crawlers or bots to not access or crawl any URLs on the website that contain a question mark in the URL.

The question mark "?" is a common character used in URLs to specify query parameters. By disallowing URLs with "?" in the robots.txt file, Twitter ensures that certain dynamically generated or specific pages with query parameters are not accessed by search engine crawlers or other bots.

For example, URLs like "https://twitter.com/profile?page=2" or "https://twitter.com/search?q=example" would be disallowed. This helps prevent duplicate content indexing or undesirable crawling of parameters-based pages.