LWP::UserAgent vs. WWW::Mechanize

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
Hi all,

I want to use WWW::Mechanize to pull some data of my corporate
intranet. LWP::UserAgent works fine, but if I use WWW::Mechanize, I
get a DNS error. Any ideas that I can use to debug further?

My script looks like this
use WWW::Mechanize;
use strict;

my $url = 'http://www.mysite.mycompany.com /'; # Just any URL on the
my $ua = LWP::UserAgent->new;
$ua->agent("Nightly/0.1 ");

my $req = HTTP::Request->new(GET => $url);
my $res = $ua->request($req);
print $res->content;

print "+++++++++++++++++++++++++++++++++++++++";

my $m = WWW::Mechanize->new();
$m->agent("Nightly/0.1 ");

$req = HTTP::Request->new(GET => $url);
$res = $m->request($req);

print $res->content;

print "+++++++++++++++++++++++++++++++++++++++";

my $c = $m->content;
print $c;

In the first case, I get the page I want. In the next two cases, I get
the following
<HEAD><TITLE>502 Bad Gateway</TITLE></HEAD>
<H1>Bad Gateway</H1>
The following error occurred:<P>
[code=DNS_NO_DATA] The IP address was not found during the DNS lookup.
Contact your system administrator.
Please contact the administrator.

When I do an nslookup the IP address is returned correctly.
c:\> perl -v

This is perl, v5.10.0 built for MSWin32-x86-multi-thread
(with 5 registered patches, see perl -V for more detail)

Copyright 1987-2007, Larry Wall

Binary build 1004 [287188] provided by ActiveState http://www.ActiveState.com
Built Sep  3 2008 13:16:37

Perl may be copied only under the terms of either the Artistic License
or the
GNU General Public License, which may be found in the Perl 5 source

Complete documentation for Perl, including FAQ lists, should be found
this system using "man perl" or "perldoc perl".  If you have access to
Internet, point your browser at http://www.perl.org /, the Perl Home


Appreciate any ideas/help.


Re: LWP::UserAgent vs. WWW::Mechanize

"Theo R." asks:

Quoted text here. Click to load it

Just a wild guess: Do you have the environment variable http_proxy
set, but no value for no_proxy?

If I recall correctly, then the user agent of WWW::Mechanize defaults
to using env_proxy, whereas LWP::UserAgent doesn't.  At least, this
would explain your observation: You try to pull an intranet page from
your proxy, but the proxy is using external name servers which don't
know about intranet servers.  And it would be easy to verify.

You could also simply try to create your object like this:

my $m = WWW::Mechanize->new(noproxy => 1);


Quoted text here. Click to load it

Re: LWP::UserAgent vs. WWW::Mechanize

Quoted text here. Click to load it
Yup. Bullseye on the wild guess.

Quoted text here. Click to load it
Interesting; I went back and read the module docs.
* noproxy => [0|1]
Turn off the automatic call to the LWP::UserAgent env_proxy function.

In short, passing noproxy or setting NO_PROXY works. Thanks a lot!


Site Timeline