Do you have a question? Post it now! No Registration Necessary. Now with pictures!
November 7, 2003, 1:51 am
rate this thread
their robot does not obviously visit my site every few seconds. So what
exactly does the following mean ?
"How often will Googlebot access my web pages?
For most sites, Googlebot should not access your site more than once every
few seconds on average. Since network delays are involved it is possible
over short periods the rate will appear to be slightly higher. If you find
that we are placing too high a load on your site, please let us know by
sending us e-mail at email@example.com. "
Re: Googlebot Questions
You may have only one or perhaps a few pages on your site, but some of
us have thousands. Gooblebot may be inclined to visit them all, if it
can find them and thinks your site is important enough. If Googlebot -
which is really a whole bunch of different servers - were to try to
slurp the thousands of pages on my site all up at once, my server
wouldn't be able to handle the load. So they intentionally space out
visits to the same site by a few seconds, to avoid overloading a
server. I'm getting about 5-10 hits per minute from Googlebot right
now, which is about right.
Googlebot will visit highly-ranked pages about once a day or
so. Sometimes it grabs the same page two or three times a day. I
have seen it re-visit the same page over and over again in the same
few hours, perhaps in a loop, probably something broken on their end
that isn't supposed to happen.