Do you have a question? Post it now! No Registration Necessary. Now with pictures!
April 5, 2006, 6:45 am
rate this thread
I need to be advised by you.
I have this problem:
Inktomi spiders browse all my site all the day in particular my web board
every day receive the visit of about 40-70 different spides from Inktomi.
Now i can read my logs o more. besides the most of my web server resources
are wasted by this kind of users. Besides Inktomi send my almost a few
users.... but ... but I have read in a web site that yahoo has purchased
Inktomi and that utilize his tecnology to index
all web sites. In another web site I have read that inktomi works for other
engines indexing too.
I would like to receive more visitors from Yahoo, but
I can't waste such lebvel of resources from my web server. What can I do?
could I ban Inktomi in my htacces, or robots.txt file?
In such case what is the better way to do it?