Looking for a web log analyzer

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

I'm looking at a couple of the web log analyzers and not finding them
what they're cracked up to be.

Web Log Storming claims it isn't finding any data in my (IIS 7.5) log
files, even though I can see that it's sequentially finding and
processing them.

WebLog Expert removes spider hits from the totals, which is good--except
that it discerns spiders only from the user agent string. In one report
I produced, which covers a three-week period, the third largest number
of visits comes from, with a total of 3,203 hits (more
than four times as many page hits as from any other source) in 30
visits. Every single one of these hits is to the same page, the index to
our news pages. These requests come every day, throughout the day. The
IP address belongs to Sungard Availability Services. In sixth place we
have 767 hits in 24 visits from gw1.mcgraw-hill.com, obviously another
spider: in one session, for example, it downloaded 36 pages in 13
seconds (and without requesting any of the CSS or image files contained
therein). Failure to exclude these spiders makes the results pretty useless.

Now that I think about it, probably any client IP address that is the
source of a request for robots.txt should be treated as a spider.

First: can anyone tell me what I might be missing with Web Log Storming?

Second: is Web Log Storming just going to have the same flaw as WebLog
Expert anyway?

Third: is there a product that usefully screens out the spiders?

As far as expense is concerned, this is a 96-page website belonging to a
rather small company. WebTrends isn't in the running.

Re: Looking for a web log analyzer

Quoted text here. Click to load it

Something custom and small? I do things like that for a living :-).

John Bokma                                                               j3b

Hacking & Hiking in Mexico -  http://johnbokma.com/
http://castleamber.com/ - Perl & Python Development

Site Timeline