Googlebot - aggressive dowloading?

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

Yesterday we uploaded several hundred pages of new content.  Now, I monitor
Googlebot's activity on our site to try to learn its methods, and sure
enough it visited this morning and stumbled across the new pages.

I would have expected it to download a page every few seconds or so, but it
swallowed the lot in just a few minutes, peaking at 5 requests per second.

The site is capable of serving thousands of requests per second, so I have
no issue, but is this normal or is Googlebot out of control?

Brian Wakem

Re: Googlebot - aggressive dowloading?

Hi Brian -

Quoted text here. Click to load it

Not new pages in my case, but I've noticed Googlebot lately has been
downloading a whole lot of pages at one time rather than just a few at
a time.

Ken /

Re: Googlebot - aggressive dowloading?

Brian Wakem wrote:

Quoted text here. Click to load it

I have this happening on my sites every once in a while. It seems to be a
part of the normal behavior whereby Google seems to be checking if the
site is able to handle big load. It's hard to say why they do that, but in
some instances my sites did not hold up, and the ranks seem to be falling
down for those site pretty steadily ever since. It is difficult to see if
that has to do with the sites' lousy performance under Googlebot's
crossfire because the sites are also pretty new (couple months old), so
they could simply be suffering from the sandbox syndrome ;-) Also, if
seems to be the case that the site that did not hold well during one of
these heavy hit visits, is not probed anymore, and the total number of
visits tends to be low. As if Google does not want to overload a site that
it considers week on the server side. Pretty logical thinking, but it can
also mean that on a larger site with a lot of fresh content (like a forum
for example) Googlebot will always be behind when indexing new documents.

I take it as your site did hold pretty well, so congrats: you will have
more of your pages indexed in a very near future.


Free Search Engine Scripts

Article posted with Web
Developer's USENET Archive
Web and RSS gateway to
your favorite newsgroup -
alt.www.webmaster - 42985 messages and counting!

Site Timeline