Do you have a question? Post it now! No Registration Necessary. Now with pictures!
April 5, 2006, 6:50 am
rate this thread
I need to be advised by you.
I have this problem:
Inktomi spiders browse all my site all the day (in particular my web board):
every day I receive the visit of about 40-70 different spiders from Inktomi.
Now I can read my logs no more. Besides the most of my web server resources
are wasted by this kind of users and Inktomi send my only a few
users.... but ... but.. I have read in a web site that Yahoo has purchased
Inktomi and that it utilize his tecnology to index all web sites.
In another web site I have read that Inktomi works for other
engines indexing web site for them too.
...I would like to receive more visitors from Yahoo, but
I can't waste such level of resources from my web server.
What can I do?
Could I ban Inktomi using my htacces, or robots.txt file?
In such case what is the better way to do it?