LWP::Parallel concerns

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
Maybe I'm wrong, but if I were to use LWP::Parallel to parse a remote
site for a few hours,  then couldn't this be possibly interpreted as a
Denial of Service? And if could be interpreted as a possibly Denial of
Service attack, what could I do to possibly avoid it?

Re: LWP::Parallel concerns

Quoth chadda@lonemerchant.com:
Quoted text here. Click to load it

By default LWP::Parallel won't make more than 5 requests to any given
host at a time. If you are worried that even this many over a sustained
period would be considered abuse, you could reduce it with ->max_req; or
you could use ->on_connect to do some more sophisticated rate-limiting.


The Earth is degenerating these days. Bribery and corruption abound.
Children no longer mind their parents, every man wants to write a book,
and it is evident that the end of the world is fast approaching.
       Assyrian stone tablet, c.2800 BC                        ben@morrow.me.uk

Re: LWP::Parallel concerns

Quoted text here. Click to load it

Okay. Thanks.

Re: LWP::Parallel concerns

Quoted text here. Click to load it

On the other hand, this would not help at all if what you are doing is
against the terms of use the web site and use scraping to re-construct a
data set to which you do not have commercial use rights.

If you then go and set up a commercial operation of any sort using the
data you obtained that way, well, I am no lawyer, but I think a decent
case could be made against you.


(remove .invalid and reverse each component for email address)

comp.lang.perl.misc guidelines on the WWW:
http://www.rehabitation.com/clpmisc /

Site Timeline