How do I rate limit google's crawl of my IP block?

Solution 1:

If you run BGP you could simply rate-limit AS15169 (AS-GOOGLE), but doing it by hand is likely to be far too error-prone.

Solution 2:

I found these notes interesting to pursue

1. Get yourself a smart robots.txt and other robots.txt posts there
2. A post on Google's Dirty Little Secret by someone troubled with google bots
3. Google web crawlers

Solution 3:

You can go to google and create an account with the webmaster tool and then you can control the crawl rate for each site. Go to Site Configuration::Settings::Crawl Rate. This won't let you schedule your sites in a certain order I don't believe, but you can at least slow it down for all of them.