Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- LWP::UserAgent and 404 page not found
June 22, 2005, 10:47 pm
rate this thread
I'm using LWP::UserAgent (Active Perl v188.8.131.528) in a web site
crawler, but there's a page I just can't read -
http://www.psychology.bangor.ac.uk/ gives '404 not found' It is
similarly inaccessible for many of the web checkers out there (like
http://validator.w3.org /) but is okay with 'real' browsers like Internet
Explorer and Netscape.
There's a redirection there somewhere behind the scenes to index.php
(which can be read), but then that is so for our main web page
http://www.bangor.ac.uk/ as well and that redirects okay.
I suppose the problem is not understanding how redirection takes place.
Is it a server issue? Do the regular browsers 'guess' at filenames if
none are given? Is there some browser/server negotiation which is not
An extract from the code which exhibits the symptoms is below (but note
the folding of the 'my $referer' line!)
I'd appreciate any help you can give - I've drawn blanks elsewhere!
#the page which refers to the culprit:
my $referer = http://www.bangor.ac.uk/corporate/informationabout/depts.php ';
#the inaccessible page
my $url='http://www.psychology.bangor.ac.uk /';
#but these are okay
# $url='http://www.informatics.bangor.ac.uk /';
# $url='http://www.psychology.bangor.ac.uk/index.php ';
# $url='http://www.bangor.ac.uk /';
#open the browser
my $browser = LWP::UserAgent->new;
#try to get the page
my $response = $browser->get($url, Referer => $referer);
print "Response $response\n";
my $status= $response->status_line;
($status) = split(' ',$status.' ');
print "Status_line $status\n";
Re: LWP::UserAgent and 404 page not found
They seem to be doing a redirect based upon the language that your broswer
declares itself to accept. As you aren't doing this you get an error page.
my $response = $browser->get($url, Referer => $referer, ACCEPT_LANGUAGE =>
- » In search of elegant code: Adding to a hash using (key => value) syntax
- — Previous thread in » PERL Discussions