Google spidering & traffic

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
Google and other engines spider my website every single day and as
result generates QUITE A BIT of traffic.

I've tried:

<META NAME="revisit-after" CONTENT="7 days">

--but the engines still visit daily.

I dont want to use:


--as it will prevent spidering all together unless I remove the tag
every week or so for a few days?

I'm not sure about robot.txt as I want the entire site indexed (just not
every single day) and no pages excluded.

Any advice would be appreciated!


Re: Google spidering & traffic

Quoted text here. Click to load it

Personally, I don't bother with this one. Its a request for robots to
revisit after 7 days, but its only a request. It doesn't mean only revisit
after 7 days. Also, its a hint for the current page. So you are saying for
*this* page *please* visit again after 7 days.

Quoted text here. Click to load it

How many pages to you have? In my experience search engines don't tend to
pull down the same pages regularly (unless they've changed), but if you have
a large site then they can spend days (or more) working through all the
pages on your site.

Regarding google, whilst I don't know a way of saying only spider one day in
seven, you can change the crawl rate. Go to (which redirects you), sign up
for a Google Webmaster tools account. There is a minor hoop to jump through
so they they can validate that you are responsible for the domain you claim
to be responsible for, but thereafter you can change the crawl rate down
from "Normal" to "Slower".

Be aware that this will mean it takes longer for google to pick up changes
and new pages on your sit.e

Can't advise about other robots. Normally about all you can do is block
them, which you probably don't want.

Hope this helps.
Brian Cryer

Re: Google spidering & traffic

Scripsit Cinamon Thunder:

Quoted text here. Click to load it

Do you mean the _spider visits_? Even if a spider visits your all pages
every day, that's still just one HEAD request per page. If that's
considerably more than normal traffic to your site, I'd be more worried
about the low usage than the spiders.

(I'm assuming that your server sends adequate Last-Modified headers so that
spiders can just send a HEAD request and see that nothing has changed,
instead of retrieving the actual content.)

Quoted text here. Click to load it

Important search engines probably ignore that routinely, since there has
been too much abuse.

Quoted text here. Click to load it

Please convince me that daily visits by spiders are a _problem_. Many
webmasters would pay real money to achieve such a situation.

Jukka K. Korpela ("Yucca")

Re: Google spidering & traffic

Here are spider bandwidth stats for the 19th

Googlebot (Google)                    262.93 MB    19 Feb 2007 - 23:54
Inktomi Slurp                                74.83 MB    19 Feb 2007 - 23:58
MSNBot                                            270.99 MB    19 Feb 2007 -
Unknown robot (identified by 'robot')    597.13 KB    19 Feb 2007 - 05:23

Thats 1.2+ Gig for the day?

Yet when I look at my overall summary for the 19th:

Day    Number of visits    Pages    Hits    Bandwidth
19 Feb 2007    287    24266    45669    181.29 MB

I smell a rat...


Quoted text here. Click to load it

Re: Google spidering & traffic

On Feb 20, 9:20 pm, wrote:
Quoted text here. Click to load it

There's a mistake there. Either you typed 597.13 Kb when you meant
597.13 Mb or you read it the other way round,

Quoted text here. Click to load it

Depends on which flavour of smoke and mirrors your stats use.

Now as you full quoted and top posted you probably didn't read Jukka's
reply carefully, so I'll just draw you attention to this part:

Quoted text here. Click to load it


Site Timeline