Click here to get back home

Using a buffer with HTTP::Response?
 Home | NewsGroups | Search | About
Login Password
Register Now!
 comp.lang.perl.misc    Post an article   get this group's latest topics as an RSS feed add this group's latest topics to your My MSN content add this group's latest topics to your My Yahoo content
Subject Author Date
Using a buffer with HTTP::Response? Leif Wessman 05-23-2008
Posted by Leif Wessman on May 23, 2008, 6:50 am
Please log in for more thread options
I'm parsing a text-file, line by line. But it is almost 100 MB. Is it
possible to use a buffer instead of loading the complete text file
into memory?

This is the code I'm using today:

my $url = "http://datafeed.com/somefile.txt";
my $req = new HTTP::Request GET => $url;
my $ua = new LWP::UserAgent;
my $res = $ua->request($req);
if ($res->is_success) {
my @lines = split /\n/, $res->content;
foreach $line (@lines) {
# do stuff
}
}

Thanks!

Posted by Bjoern Hoehrmann on May 23, 2008, 6:55 am
Please log in for more thread options
 * Leif Wessman wrote in comp.lang.perl.misc:
>I'm parsing a text-file, line by line. But it is almost 100 MB. Is it
>possible to use a buffer instead of loading the complete text file
>into memory?
>
>This is the code I'm using today:
>
>my $url = "http://datafeed.com/somefile.txt";
>my $req = new HTTP::Request GET => $url;
>my $ua = new LWP::UserAgent;
>my $res = $ua->request($req);

The second parameter to the request method takes a content callback,
you can also specify the special field :content_cb, see the notes in
`perldoc LWP::UserAgent` and see `perldoc lwpcook` for an example.
--
Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

Posted by Dr.Ruud on May 24, 2008, 10:10 am
Please log in for more thread options
Leif Wessman schreef:

> I'm parsing a text-file, line by line. But it is almost 100 MB.
> Is it possible to use a buffer instead of loading the complete
> text file into memory?

You don't want to buffer, you want to process line by line.


> my $req = new HTTP::Request GET => $url;
> my $ua = new LWP::UserAgent;

I would write those as:

my $req = HTTP::Request->new( GET => $url );
my $ua = LWP::UserAgent->new();


> my @lines = split /\n/, $res->content;
> foreach $line (@lines) {
> # do stuff
> }

while ( $res->content =~ m/ (.*) \n? /gx )
# do stuff with $1
}

--
Affijn, Ruud

"Gewoon is een tijger."

Posted by Dr.Ruud on May 24, 2008, 10:44 am
Please log in for more thread options
Dr.Ruud schreef:

> while ( $res->content =~ m/ (.*) \n? /gx )
> # do stuff with $1
> }

Rather:

while ( $res->content =~ m/ (.*) \n? /gx ) {
# do stuff with $1
}

--
Affijn, Ruud

"Gewoon is een tijger."

Similar ThreadsPosted
HTTP::Response and gzip July 25, 2006, 12:46 pm
HTTP::Request/GET response not expected May 6, 2005, 12:00 pm
Reading HTTP response body that is gzip'd *and* in UTF-8 August 4, 2006, 2:36 pm
To get the HTTP response headers such as content length from a web server February 23, 2007, 6:16 am
HTTP Proxy via HTTP Layer by Perl? October 19, 2004, 5:56 pm
Help testing post response July 13, 2004, 3:05 pm
XML Response customizing using PERL June 19, 2006, 1:41 pm
Does anybody know why mx1.hotmail.com doesn't response correctly? Thanks June 11, 2007, 4:09 pm
simple response not implemented August 1, 2007, 7:08 am
FAQ: What is the correct form of response from a CGI script? October 22, 2004, 11:10 am

Our other projects:

Art Dolls, Fairies and Mermaids - Sunnyfaces.net

Roy's Linux, Programming and Search Engines messages

1-Script XML SitemapXML Sitemap