case study: want to print redirects when LWP::UserAgent makes a request

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

I'm trying to figure out, if I have a LWP::UserAgent object and a
HTTP::Request object, how I can use the user-agent to request the
request object and print out a list of HTTP redirects that it
encounters along the way.

But, that's actually not the question.  What I want to know is: How do
more experienced Perl programmers find the answers to these questions
when they need to get something done?  I keep encountering problems
like this, reading the documentation, hitting a dead end, and having
to post in a forum or ask a friend.  It's hard to believe that real
Perl programmers would have to do this for every simple question like
this, or nothing would ever get done.

Suppose you're an experienced Perl programmer but you don't know the
answer to this particular question.  I look first in the documentation
page UserAgent.html.  I see it says "The difference from request() is
that simple_request() will not try to handle redirects or
authentication responses. The request() method will in fact invoke
this method for each simple request it sends.", so I know about the
difference between those two.

The next thing that stands out is:
"$ua->redirect_ok( $request )
This method is called by request() before it tries to follow a
redirection to the request in $request. This should return a TRUE
value if this redirection is permissible."
It's not clear whether LWP::UserAgent calls this for *every* redirect,
or just the first one.  (Taken literally, the documentation says it
will only do it for the first one -- "before it tries to follow a
redirection to the request in $request".)  But still, that would mean
I'd have to create a subclass of LWP::UserAgent to override behavior
of this function.  I'm hoping there's something easier.

So that's where I give up and ask someone.  What would you do?  Is
there something in the documentation that would make it obvious to you
what to try next?  What do I need to train myself to look for?

Re: case study: want to print redirects when LWP::UserAgent makes a request

Quoted text here. Click to load it

my $response = $ua->get( $url );

returns a HTTP::Response

HTTP::Response documentation shows a method redirects, which "Returns a
list of redirect responses that lead up to ... The list order is oldest
first ..."

Quoted text here. Click to load it

use YAML::Syck;
print Dump $response;

This shows a lot of info about $response
Knowing what the get method returns (to what it's blessed) and reading
the documentation.

Quoted text here. Click to load it

Now and then I look at the source to understand what some things do. The
documentation is not always complete. Now and then I contact authors
with a request to update the documentation. On CPAN there is also the
possibility to add annotations.

John Bokma                                                               j3b

Hacking & Hiking in Mexico - - Perl & Python Development

Re: case study: want to print redirects when LWP::UserAgent makes a request

Quoted text here. Click to load it

I agree that this isn't crystal-clear, but I don't think your "literal"
interpretation is reasonable: Note that it says "redirection TO the
request in $request". So if you have a redirect cascade

then it will first be called with $request "GET ",
and if that was allowed and succeeded with the new $request
"GET "

For current versions of LWP::UserAgent, redirect_ok takes two arguments,
btw - the new request and the response which caused the redirect.

Quoted text here. Click to load it

Yes. If the default redirect_ok isn't sufficient (AFAIK you can turn on
and off redirects for each request type (GET, POST, ...), but that's it)
then subclassing and overriding this method is the cleanest (and
probably simplest) way.

Quoted text here. Click to load it

It sounds like just turning off redirects (see requests_redirectable) or
using simple_request is sufficent.

Quoted text here. Click to load it

What I generally do when the documentation isn't sufficiently clear is
to write small test scripts which use the function I want to explore. I
may even use the debugger to step through this function and see what it

I'm rather impatient though (as a Perl programmer I can admit that) and
if I can't figure out how something works within a short time I write it
myself. So supposing I couldn't figure out how redirect_ok was supposed
to work, I'd just fall back to using simple_request and write an extra
loop to handle the redirects.


Site Timeline