Limiting Recursive Gets

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

There's a bot in the wild that seem to be causing numerous clients in
dynamic space to wget -r (or the java equivalent). User agents are
not reliable so the only way to identify this behavior is a sequence
of gets without a referrer.

I'm trying to find a way to limit this activity to a list of known
search engines. Any ideas?

Displayed Email Address is a SPAM TRAP
Our DNSRBL - Eliminate Spam:
Multi-RBL Check:
The Dirty Dozen Spammiest Ranges:

Re: Limiting Recursive Gets

Fleeing from the madness of the jungle
and said:

Quoted text here. Click to load it

only reliable way I can suggest is a honey-pot - assumes the bot is  
ignoring robots.txt

William Tasso

Site Timeline