How to delay google crawling until the page is ready?

Member

by vinnie , in category: SEO , 2 years ago

How to delay google crawling until the page is ready?

Facebook Twitter LinkedIn Telegram Whatsapp Pocket

3 answers

Member

by delpha , 2 years ago

@vinnie 

You can delay Google crawling by using the "robots" meta tag in the HTML of your page. You can specify that the page should not be crawled using the "noindex" value, until the page is ready to be indexed. For example:

1
<meta name="robots" content="noindex">


When you are ready to have the page crawled and indexed, you can remove the "noindex" value or change it to "index":

1
<meta name="robots" content="index">


It's important to note that this only affects Google's crawling of the page and not necessarily the display of the page to users. To ensure that users do not see an unfinished version of the page, you should also implement proper redirects or staging environments to control the display of the page.

by jose_gulgowski , 2 years ago

@vinnie 

To delay Google crawling until the page is ready, you can use the "robots" meta tag in the HTML head section of the page and specify "noindex,nofollow". Once the page is ready, you can remove or change the "robots" meta tag to allow Google to crawl and index the page. You can also use a "noindex" robots file or HTTP header to block Google from crawling the page temporarily. Additionally, you can use the "Google Search Console" to temporarily block Google from crawling specific URLs on your website.

by clarabelle , 2 years ago

@vinnie 

There are a few ways to delay Google's crawling of a web page until it's ready:

  1. Use the "robots" meta tag: You can add the "robots" meta tag in the HTML head section of your page with the content "noindex, nofollow" to prevent Google from indexing the page until you're ready for it to be crawled.
  2. Block the page with robots.txt: You can use the robots.txt file to block Googlebot from accessing the page until it's ready.
  3. Use a "noindex" header: You can return a "noindex" header in the HTTP response when the page is not ready to be crawled.


It's important to keep in mind that even though you delay crawling, Google may still crawl the page at some point in the future, so be sure to remove the blocking mechanisms once the page is ready to be indexed.