Maximize LWP https performance

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

I am attempting to load test an https web application using a
custom perl application. The order of requests made to the web
application is important so I can't just fire off all the requests
in parallel a la LWP::Parallel. So, my latest approach has been to
fork multiple times to achieve parallelism, with each fork
simulating a web application user. Via this technique I achieve a
sustained rate of about 12 requests / second which I feel, perhaps
incorrectly(?), that I should be able to improve upon.

I notice that some of the requests take an unusually long time to
complete. Seemingly unrelated to the web application server
responsiveness. Thus, I have two theories:

1) Contention over use of network. Requests are getting stacked up
at the mercy of the network scheduler.

I'm not entirely sure of a good way of confirming or disproving
this? It may be that splitting the execution across multiple hosts
is a way forward but I'd like to maximize performance on a single
host first before duplicating it across additional hosts...

2) SSL connection is biting.

Profiling a single process (no forking) suggests to me that a fair
bit of time is spent dealing with SSL connections.

%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 78.0   1.970  1.970    180   0.0109 0.0109  Crypt::SSLeay::Conn::connect
 8.16   0.206  0.206 114428   0.0000 0.0000  HTTP::Headers::_header
 6.81   0.172  0.358  15674   0.0000 0.0000  Net::SSL::read
 5.31   0.134  0.134  15674   0.0000 0.0000  Crypt::SSLeay::Conn::read
 4.32   0.109  0.143   1854   0.0001 0.0001  HTML::Parser::parse

I am using keep alives:

my $userAgent = LWP::UserAgent->new( keep_alive=>1 );

So... I was wondering if anyone has done heavy load testing on
https sites via perl previously and could give me a few pointers
or hints please!

In fact, any help would be much appreciated!

Best wishes,


Re: Maximize LWP https performance

Quoted text here. Click to load it

What is the "seemingly unrelated" conclusion based on?

Quoted text here. Click to load it

Make a dummy web application that does nothing but serve static files.
For example, go through the web app, saving the response for each request.
Then copy these responses to web server and make your client requests these
static files rather than doing actual dynamic requests.  If your network is
the problem (unlikely, I suspect), you will not see much improvement.  If
it isn't the problem, you should see a big improvement.

Quoted text here. Click to load it

Change your app to allow non-SSL connections, then run the benchmark again
that way and see if it speeds up.

Quoted text here. Click to load it

You seem to be profiling the client program, not the web-app server
programs, right?

Quoted text here. Click to load it

Was this generated with dprofpp, dprofpp -r, or dprofpp -u, or something

What was the header from the profile?, e.g.
               Total Elapsed Time =    1.67 Seconds
                        User Time =    0.61 Seconds


-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Site Timeline