Click here to get back home

Net::Google 502 Bad gateway errors

 HomeNewsGroups | Search | About
 comp.lang.perl.modules    Post an article   get this group's latest topics as an RSS feed add this group's latest topics to your My MSN content add this group's latest topics to your My Yahoo content
Subject Author Date
Net::Google 502 Bad gateway errors Ben Bullock 05-17-2006
Posted by Ben Bullock on May 17, 2006, 2:15 am
Please log in for more thread options


Does anyone here use the Net::Google module? It seems to run very slowly,
and I keep getting "502 Bad gateway" errors. In the end I just downloaded
the search page using libwww and parsed it myself, which is almost instant.

--
sci.lang.japan FAQ: http://www.sljfaq.org/afaq/afaq.html


Posted by Henry McGuinness on May 18, 2006, 4:41 am
Please log in for more thread options



> Does anyone here use the Net::Google module? It seems to run very slowly,
> and I keep getting "502 Bad gateway" errors. In the end I just downloaded
> the search page using libwww and parsed it myself, which is almost
> instant.

Hi,
I think the "Bad Gateway" stuff is actually a Google api/server problem. I
had this and it was nothing to do with Net::Google or my query. (though you
might have posted some code :)) - I assume you have a google api key, by
the way. I looked at a google api newsgroup and this was a fairly regular
problem

Believe it or not the solution is probably: leave it a couple of minutes and
try again a few times. It does work, but sometimes rather sporadically. I
found the Yahoo api stuff much more reliable. have a look at that:
http://developer.yahoo.com/

Others were having this problem in January and posted some code (though they
were using SOAP::Lite, adjust your code accordingly):

http://groups.google.co.uk/group/google.public.web-apis/browse_frm/thread/6f1ef9463b961feb/091e5c9505d5f81a?lnk=st&q=google+api+502+bad+gateway&rnum=1&hl=en#091e5c9505d5f81a

cheers
Henry



Posted by Ben Bullock on May 18, 2006, 11:18 pm
Please log in for more thread options


>
>> Does anyone here use the Net::Google module? It seems to run very slowly,
>> and I keep getting "502 Bad gateway" errors. In the end I just downloaded
>> the search page using libwww and parsed it myself, which is almost
>> instant.

> I think the "Bad Gateway" stuff is actually a Google api/server problem. I
> had this and it was nothing to do with Net::Google or my query. (though
> you might have posted some code :)) - I assume you have a google api key,
> by the way. I looked at a google api newsgroup and this was a fairly
> regular problem

Yes, I've got one of those.

> Believe it or not the solution is probably: leave it a couple of minutes
> and try again a few times. It does work, but sometimes rather
> sporadically. I found the Yahoo api stuff much more reliable. have a look
> at that: http://developer.yahoo.com/

OK, but what about just putting a search URL together then grabbing the page
via libwww and parsing it using regular expressions though? That works
almost every time, I never get any 502 errors, and the results are virtually
instant. I can't see why I would want to use this Net::Google thing in the
first place.

> Others were having this problem in January and posted some code (though
> they were using SOAP::Lite, adjust your code accordingly):

That is where the error comes in, it gives the line numbers from that code.
I'm not sure I'd want to spend hours fixing it though, because I'm far from
being an expert in Perl and it would take a lot of time for me to do so. I
suppose the nature of my question was to try to find out why anyone would
use Net::Google when it seems to endlessly choke and run very slowly, and
it's easy to get the stuff from Google via another method.




Posted by Henry McGuinness on May 19, 2006, 6:29 am
Please log in for more thread options




>> Believe it or not the solution is probably: leave it a couple of minutes
>> and try again a few times. It does work, but sometimes rather
>> sporadically. I found the Yahoo api stuff much more reliable. have a look
>> at that: http://developer.yahoo.com/
>
> OK, but what about just putting a search URL together then grabbing the
> page via libwww and parsing it using regular expressions though? That
> works almost every time, I never get any 502 errors, and the results are
> virtually instant. I can't see why I would want to use this Net::Google
> thing in the first place.

Maybe, but watch google's terms of service on this. They don't like
"automated queries" and are not specific about all the types of query this
includes: http://www.google.co.uk/intl/en/terms_of_service.html

>
>> Others were having this problem in January and posted some code (though
>> they were using SOAP::Lite, adjust your code accordingly):
>
> That is where the error comes in, it gives the line numbers from that
> code.

Yes. What I meant was you'll have to change their SOAP::Lite syntax to the
Net::Google query you want to do so you'll have:

...do {

#your Net::Google search here instead of the doGoogleSearch stuff...
if ( $@ ) { ......




cheers
Henry



Posted by Ben Bullock on May 21, 2006, 3:03 am
Please log in for more thread options


>
>
>>> Believe it or not the solution is probably: leave it a couple of minutes
>>> and try again a few times. It does work, but sometimes rather
>>> sporadically. I found the Yahoo api stuff much more reliable. have a
>>> look at that: http://developer.yahoo.com/
>>
>> OK, but what about just putting a search URL together then grabbing the
>> page via libwww and parsing it using regular expressions though? That
>> works almost every time, I never get any 502 errors, and the results are
>> virtually instant. I can't see why I would want to use this Net::Google
>> thing in the first place.
>
> Maybe, but watch google's terms of service on this. They don't like
> "automated queries" and are not specific about all the types of query this
> includes: http://www.google.co.uk/intl/en/terms_of_service.html

I'm guessing that would be more directed at people like scroogle
(http://www.scroogle.org/) than someone like me, who's unlikely to download
more than 100 pages a day.



Similar ThreadsPosted
I got a #1 on google in 24 hrs on Google - Free video January 17, 2008, 1:22 pm
Problem with LWP and Google API March 22, 2005, 3:45 pm
Google Spreadsheets November 7, 2006, 8:31 am
Perl interface to Google GDS SDK May 9, 2005, 2:22 am
ANNOUNCE: Google::Adwords v0.6 October 27, 2006, 11:24 pm
ANNOUNCE: Google::Adwords v0.6 October 27, 2006, 11:24 pm
[ANNOUNCE] Google::Adwords v1.0.1 February 23, 2007, 9:33 pm
ANNOUNCE: Google::Adwords v1.2.3 April 27, 2007, 9:17 am
Google API using SOAP Lite doesn't like '&' in query June 29, 2005, 2:06 pm
ANNOUNCE - Perl module Google::Adwords v0.4 October 12, 2006, 9:23 am

Our other projects:

Art Dolls, Fairies and Mermaids - Sunnyfaces.net

Roy's Linux, Programming and Search Engines messages

1-Script XML SitemapXML Sitemap