Googlebot went nuts!

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
I have a mirror of wikipedia. I try to keep track of how many
googlebot visits were done to the mirror pages over time. Yesterday,
googlebot went NUTS and retrieved roughly 20 times the # of pages as

This gets pretty close to what I think is a sane limit of
queries. Does google recognize crawl-delay in robots.txt?

algebra/2005_07_04/count.txt:             Googlebot Wiki visits: 1773
algebra/2005_07_05/count.txt:             Googlebot Wiki visits: 1667
algebra/2005_07_06/count.txt:             Googlebot Wiki visits: 985
algebra/2005_07_07/count.txt:             Googlebot Wiki visits: 1065
algebra/2005_07_08/count.txt:             Googlebot Wiki visits: 1637
algebra/2005_07_09/count.txt:             Googlebot Wiki visits: 1528
algebra/2005_07_10/count.txt:             Googlebot Wiki visits: 947
algebra/2005_07_11/count.txt:             Googlebot Wiki visits: 1918
algebra/2005_07_12/count.txt:             Googlebot Wiki visits: 2633
algebra/2005_07_13/count.txt:             Googlebot Wiki visits: 50869
cooldictionary/2005_07_04/count.txt:             Googlebot Wiki visits: 197
cooldictionary/2005_07_05/count.txt:             Googlebot Wiki visits: 875
cooldictionary/2005_07_06/count.txt:             Googlebot Wiki visits: 311
cooldictionary/2005_07_07/count.txt:             Googlebot Wiki visits: 663
cooldictionary/2005_07_08/count.txt:             Googlebot Wiki visits: 985
cooldictionary/2005_07_09/count.txt:             Googlebot Wiki visits: 417
cooldictionary/2005_07_10/count.txt:             Googlebot Wiki visits: 582
cooldictionary/2005_07_11/count.txt:             Googlebot Wiki visits: 472
cooldictionary/2005_07_12/count.txt:             Googlebot Wiki visits: 636
cooldictionary/2005_07_13/count.txt:             Googlebot Wiki visits: 2878

Re: Googlebot went nuts!

On Thu, 14 Jul 2005 13:01:46 +0200, Ignoramus13314  

Quoted text here. Click to load it

G is experimenting all the time. It seems to me that the best we can all  
do right now is to take a break, and wait till Summer is over.

-- - chemical calculators for labs and education
BATE - program for pH calculations
CASC - Concentration and Solution Calculator
pH lectures - guide to hand pH calculation with examples

Re: Googlebot went nuts!

Ignoramus13314 wrote:
Quoted text here. Click to load it

I think this may be related what I was describing in the "2 different
google crawlers" thread in this group.

-- - Open source (GPL) security tools and how-tos.

Site Timeline