Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Maximize LWP https performance
- Daniel Parry
May 28, 2008, 12:02 pm
rate this thread
I am attempting to load test an https web application using a
custom perl application. The order of requests made to the web
application is important so I can't just fire off all the requests
in parallel a la LWP::Parallel. So, my latest approach has been to
fork multiple times to achieve parallelism, with each fork
simulating a web application user. Via this technique I achieve a
sustained rate of about 12 requests / second which I feel, perhaps
incorrectly(?), that I should be able to improve upon.
I notice that some of the requests take an unusually long time to
complete. Seemingly unrelated to the web application server
responsiveness. Thus, I have two theories:
1) Contention over use of network. Requests are getting stacked up
at the mercy of the network scheduler.
I'm not entirely sure of a good way of confirming or disproving
this? It may be that splitting the execution across multiple hosts
is a way forward but I'd like to maximize performance on a single
host first before duplicating it across additional hosts...
2) SSL connection is biting.
Profiling a single process (no forking) suggests to me that a fair
bit of time is spent dealing with SSL connections.
%Time ExclSec CumulS #Calls sec/call Csec/c Name
78.0 1.970 1.970 180 0.0109 0.0109 Crypt::SSLeay::Conn::connect
8.16 0.206 0.206 114428 0.0000 0.0000 HTTP::Headers::_header
6.81 0.172 0.358 15674 0.0000 0.0000 Net::SSL::read
5.31 0.134 0.134 15674 0.0000 0.0000 Crypt::SSLeay::Conn::read
4.32 0.109 0.143 1854 0.0001 0.0001 HTML::Parser::parse
I am using keep alives:
my $userAgent = LWP::UserAgent->new( keep_alive=>1 );
So... I was wondering if anyone has done heavy load testing on
https sites via perl previously and could give me a few pointers
or hints please!
In fact, any help would be much appreciated!
Re: Maximize LWP https performance
What is the "seemingly unrelated" conclusion based on?
Make a dummy web application that does nothing but serve static files.
For example, go through the web app, saving the response for each request.
Then copy these responses to web server and make your client requests these
static files rather than doing actual dynamic requests. If your network is
the problem (unlikely, I suspect), you will not see much improvement. If
it isn't the problem, you should see a big improvement.
Change your app to allow non-SSL connections, then run the benchmark again
that way and see if it speeds up.
You seem to be profiling the client program, not the web-app server
Was this generated with dprofpp, dprofpp -r, or dprofpp -u, or something
What was the header from the profile?, e.g.
Total Elapsed Time = 1.67 Seconds
User Time = 0.61 Seconds
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate