Crawler using bandwidth

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

I recently discovered that one of my domains exceeded it's monthly
bandwidth. After reviewing the logs files, I noticed that a crawler from sucked 500MB of images from the website.

Is there a way to keep this crawler from visiting a website? I assume that
they would probably ignore a robots.txt file since they appear to be doing
investigative work.

thank you

Re: Crawler using bandwidth

Quoted text here. Click to load it

Crawler name is

Re: Crawler using bandwidth

Quoted text here. Click to load it

Was this a one off, or are they still at it?

Quoted text here. Click to load it

Three ideas:

1. Its worth trying robots.txt. You are right, they might ignore it, but its
a good place to start.

2. Failing that, your control panel may provide you with a means to block
specific IP addresses. You could identify the ip address of the crawler and
block that ip address. It would mean that a genuine surfer from that IP
address couldn't see your site, but realistically that's not likely to be an
issue. Some crawlers use more than one ip address, but I suspect that's not
likely to be the case here.

3. This is really an alternative to the above - why not contact them and
ask? Their website seemed to be down (or part of the internet somewhere
between here and there) when I looked just now, but there are bound to be
contact details somewhere on their site.

Hope this helps.
Brian Cryer

Site Timeline