Teaching a Crawler to Identify a Blog

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
Hello All,

I am currently trying to teach a web crawler how to identify blogs,
that is I am trying to determine a fairly inclusive set of criteria
that will help my crawler to identify them.

I have noticed that many Blogs include

div class=blogsomething (A format class conveniantly named blog)

xml tags

and/or php code.

I do know that cms(content management system) is used for several
blogs, does anyone else have any suggestions to help me determine

I am aware that any criteria is subjective, especially when
considering sites such as slashdot which has been around longer than


Re: Teaching a Crawler to Identify a Blog

Metropolis wrote:

Quoted text here. Click to load it

Maybe *some* blogs contain this tag, but I'm betting most don't.
Quoted text here. Click to load it

So do lots of other websites, and I'm betting other websites have 'em more
than blogs do.

Quoted text here. Click to load it

How can you tell it's a PHP document? You can't see any PHP code because
what you are served up is a static HTML page. The only hint you can have is
that the file extension ends with .php but not all PHP pages end in .php

In any case just because it's PHP doesn't make it a blog.

I think you'll need to do a lot more than your suggestions here to determine
if it's a blog or not.

A lot of them do have date boxes on the page somewhere so you can navigate
back to previous days postings. Things like this, and other elements that
are common to blogs, are what you should be looking for, and not stuff like
whether it contains XML style tags or PHP file extensions.

Chris Hope - The Electric Toolbox - http://www.electrictoolbox.com /

Re: Teaching a Crawler to Identify a Blog

Quoted text here. Click to load it

All true.

Start by thinking about how -you- identify a blog. That ain't easy,
if my attempts at explaining what a blog is to other people is any

Look for references to time and self. E.g. "yesterday, I"

What IS a blog, anyway?

Not duck soup, or a piece of cake, this problem.

Site Timeline