Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- David Off
October 27, 2004, 9:58 am
rate this thread
the last few days. I decided to dig a bit further as the results may be
First of all, these guys, Convera, have literally been bombarding my
site over the last couple of days, asked for all the pages at least
twice. It seems they carry out corporate and govt snooping on websites.
Anyone known more about this crawler? It doesn't seem to benefit the site.
they do ask for robots.txt first.
There is also a fair bit of traffic from Exabot
http://www.exava.com/webmasters.html , this is apparently a new search
engine, nothing much on the homepage yet.