searchbot hits vs. user hits

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
I'm trying to find out which bots generate more valid user hits
than their bots hit me. Is there a site somewhere that tells
how to recognise hits generated by search engines vs. those by
their bots.

I've figured out a few of them, but there are a lot of crawlers
out there I can't identify user hits from.

See for how far I've got so far.

Re: searchbot hits vs. user hits

Quoted text here. Click to load it

This comparison is meaningless.

You should compare the COST of serving search engines vs. the VALUE of
visitors. For example, if 10,000 page downloads from a particular
engine costs me $1, but 1,000 visitors referred by that engine bring
me $10, then serving that engine is worthwhile. This does not even get
into the possibility that those visitors may link to you and ehnance
your reputation.

Quoted text here. Click to load it

I would just try to watch logs for a while. Googlebot, slurp and
msnbot account for most bot traffic, not counting idiot downloaders.


Site Timeline