Alternatives to LWP::Parallel

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

I'm in the process of writing a script that will take a whole bunch of
URL's and get their HTTP status.  I was going to use
LWP::Parallel::UserAgent to handle parallel HTTP requests.  However, I
found that this module does not work with later version of libwww and
it looks like this module is no longer maintained.

So I started looking around at other methods/modules I could use.
There are certainly quite a few to choose from and some are quite
complex for my needs.

Some of the modules I looked at were:

IO::Lambda::HTTP (this looked the most robust/well maintained, but has
a learning curve to it)
POE::Component::Client::HTTP (as above)
AnyEvent::HTTP (this looked promising, but I couldn't get any results
with it)
HTTP::Client::Parallel (doesn't appear to have any function to limit
number of connections)
HTTP::Async (a little buggy and didn't seem to produce expected

One essential feature I need is the ability to throttle/limit the
number of simultaneous connections or number of connections/requests
per minute.

I'd appreciate any advice as to what method/module might be best for
me.  The other governing factor is that I need it to run with
ActivePerl on Linux and Windows.

Re: Alternatives to LWP::Parallel

Peter wrote:
Quoted text here. Click to load it

Why not simply Parallel::ForkManager and LWP::UserAgent?


Re: Alternatives to LWP::Parallel

Quoted text here. Click to load it

Ah, now that looks to be just what I need.  I don't know how I managed
to miss that one.  Thanks.

Site Timeline