Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- BrandProtect.com Crawler using bandwidth
August 25, 2007, 10:38 pm
rate this thread
I recently discovered that one of my domains exceeded it's monthly
bandwidth. After reviewing the logs files, I noticed that a crawler from
BandProtect.com sucked 500MB of images from the website.
Is there a way to keep this crawler from visiting a website? I assume that
they would probably ignore a robots.txt file since they appear to be doing
Re: BrandProtect.com Crawler using bandwidth
Was this a one off, or are they still at it?
1. Its worth trying robots.txt. You are right, they might ignore it, but its
a good place to start.
2. Failing that, your control panel may provide you with a means to block
specific IP addresses. You could identify the ip address of the crawler and
block that ip address. It would mean that a genuine surfer from that IP
address couldn't see your site, but realistically that's not likely to be an
issue. Some crawlers use more than one ip address, but I suspect that's not
likely to be the case here.
3. This is really an alternative to the above - why not contact them and
ask? Their website seemed to be down (or part of the internet somewhere
between here and there) when I looked just now, but there are bound to be
contact details somewhere on their site.
Hope this helps.