It is possible to delay the crawling of a web page by Search Engines spiders. We can write robots.txt rules for delaying google and other Search Engine spiders to crawl and index our web page. The simple rule to delay all Search Engine spiders is :



Here you can write the time delay at the place of t.  The parameter t is counted in seconds and only after that amount of seconds only google and other spiders can crawl your web page. So t is the  number of seconds to wait between successive requests to the same web server.


So if you wish to delay Search Engine spiders for 30 seconds you can write the rule as:

How to delay spiders for specific Search Engine

If you want to delay the spiders of a specific Search Engine you can change the User-agent with specific Search Engine bot.

If you like to delay the Bing bots you need to replace User-agent with msnbot. To delay Yahoo spiders you should add Slurp.


The respective google bot is Googlebot.

Home