New Robots

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

I noticed a surge in traffic in my log files for a site I manage over
the last few days. I decided to dig a bit further as the results may be
of interest.

First of all, these guys, Convera, have literally been bombarding my
site over the last couple of days, asked for all the pages at least
twice. It seems they carry out corporate and govt snooping on websites.
Anyone known more about this crawler? It doesn't seem to benefit the site. ConveraCrawler/0.2 ConveraCrawler/0.2 ConveraCrawler/0.2 ConveraCrawler/0.2

they do ask for robots.txt first.

There is also a fair bit of traffic from Exabot Exabot , this is apparently a new search
engine, nothing much on the homepage yet.

Site Timeline