Need advice: what is the best for this ?

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
Hi all

I am looking for a free software that would help me to build a search
It is not something that has to be state of the art but I am looking for
My target is to index around 500 sites. I don't know yet the equivalent
of these in HTML pages (I mean the number of indexed pages).

I was looking at Nutch when I came across to this page:

There is a "Tool list" that impressed me and left me with no words. I am
new to this domain
and I have no idea what would be the best fit for my purposes.

I am planning to run it on Linux on a regular PC. It must be free, easy
to setup and maintain.

Thank you in advance

Re: Building a search engine

MiniME wrote:

Quoted text here. Click to load it

First of all, allow me to congratulate you on this initiative. Never forget
that Google started by creating an engine to index and search the Stanford
University Web site.

You made an excellent choice by going with Linux. The best tools will be
available to you and the price will be nill. As far as I can tell, even
Brin was using Python and other tools that were *NIX-related. You will
first need to look into 'wget' or equivalent. For example:

wget -r -l1 -H -t1 -nd -N -np -A.html -erobots=on

$ man wget

for more details. Make sure you honour the robots.txt file or else you can
hammer the server and get reported for abuse.

You then need to scan the HTML files (or whatever type suits your purpose)
for tokens (e.g. words). Use tools like lex or YACC to do that. You
probably want to create some token tables and perform some analysis upon
them. If you wish to index just 500 sites, consider starting with just 5
for testing purposes. Also make sure you have enough storage space, plenty
of RAM, high computational power, but abive all, _bandwidth_. You cannot do
much unless you have a Gigabit backbone or fast Ethernet.

In the future, please use a subject line that is meaningful. "check this
out", "help please" or "how do I do this" make it impossible for readers to
discern posts where they can help from those which are utter gibberish or
jargon to them.

Hope it helps,


Roy S. Schestowitz

Site Timeline