Search Engines you would ban

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
I've got crawling all over one of my sites at the moment. But
I've never had a single visitor come them. Looking at the search results
relevancy I can understand why we like Google so much (well some of us).

Personally I think is a waste of my bandwidth and I'm thinking
of adding them to my robots.txt. Are there other sites people in the
group would ban?

Re: Search Engines you would ban

On Sat, 21 Aug 2004 15:25:17 +0200, David Off wrote:

Quoted text here. Click to load it

Funender has made an excellent list to start from.


Re: Search Engines you would ban

Will Spencer wrote:
Quoted text here. Click to load it
hmmm interesting list, although I assume that email harvesters will
ignore robotx.txt.

Looking at my logs there seems to be a bit of a search engine feeding
frenzy on at the moment with Google, Yahoo, MSN and a number of others
crawling all over the place. Now it is great that they want to get their
   databases up to date but at the same time I'm dubious about letting
some of the no hopers waste server resouces when they are never going to
provide me with any traffic.

Site Timeline