Click here to get back home

Memory issues

 HomeNewsGroups | Search | About
 comp.lang.perl.misc    Post an article   get this group's latest topics as an RSS feed add this group's latest topics to your My MSN content add this group's latest topics to your My Yahoo content
Subject Author Date
Memory issues jm 03-29-2008
---> Re: Memory issues Joost Diepenmaa...03-29-2008
Posted by jm on March 29, 2008, 8:45 am
Please log in for more thread options
Based on the fact that perl contains many memory leaks,

A universal way to measure how many memory is malloced is required.

Is there standard way to measure how many memory a process has
allacated, which run with cygwin perl, active perl, and strawberry perl?

This should help to localize which code makes memory leaks.

Posted by Joost Diepenmaat on March 29, 2008, 9:15 am
Please log in for more thread options

> Based on the fact that perl contains many memory leaks,

It doesn't.

> A universal way to measure how many memory is malloced is required.

I don't understand what that means.

> Is there standard way to measure how many memory a process has
> allacated, which run with cygwin perl, active perl, and strawberry perl?

That's not nearly universal. See Win32::Process::Info

> This should help to localize which code makes memory leaks.

It hardly would.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

Posted by jm on March 29, 2008, 9:55 am
Please log in for more thread options
Joost Diepenmaat a écrit :
>
>> Based on the fact that perl contains many memory leaks,
>
> It doesn't.

I wrote a sample of code to illustrate the issue.

The code create a 10 mega characters string. this is the only big data
in this sample.

Then, the main part of the code just modify this data; that mean that
memory usage should (in my humble opinion) stay near of 10 or 20 (or 40)
mega bytes.

The main program does not manipulate directly the string, but makes
functions aa and ab to manipulate this string. Those two functions aa
and ab just make substitutions within the string.

After creating the first string, perl use (around) 20 Mbytes. It is okay.

Calling function aa (one or several times) makes a memory leak (or
memory empreint) of 150 Mbytes.
I mean that once I called this function I do not know how to free those
150 mega bytes, but if I call this same function again I will not loose
more memory.

When I call the function ab, which is quite similar to function aa,
I have the same memory issue, but with only 50 Mbytes more.



Hereafter the result of the script, and the script.
System is debian etch, with 512 Mbytes memory.

----- Result of script: ----------------------------------
/tmp$ perl essai.pl
10000001
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
20492 pts/1 R+ 0:00 0 1022 22977 20988 4.0 perl essai.pl

10000001
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
20492 pts/1 S+ 0:43 0 1022 159921 158132 30.5 perl essai.pl

10000001
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
20492 pts/1 R+ 0:57 0 1022 159921 158132 30.5 perl essai.pl
10000001
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
20492 pts/1 R+ 4:34 0 1022 218529 216740 41.9 perl essai.pl
10000001
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
20492 pts/1 R+ 8:10 0 1022 218529 216740 41.9 perl essai.pl


----- Script: ------------------------------------------


sub aa($)
{
my ($d) = @_;
$d =~ s/x(.....)/$1y/g ;
$d =~ s/x(.....)/$1z/g ;
$d =~ s/x(.....)/$1a/g ;
$d =~ s/x(.....)/$1b/g ;
$d =~ s/x(.....)/$1c/g ;
return $d;
}

sub ab($)
{
my ($d) = @_;
$d =~ s/a(.....)/$1y/g ;
$d =~ s/b(.....)/$1z/g ;
$d =~ s/c(.....)/$1a/g ;
$d =~ s/y(.....)/$1b/g ;
$d =~ s/z(.....)/$1c/g ;
return $d;
}


my $c= 'x' x (1000*1000*10) ;
$c .= "\x" ;
print length($c) ."\n" ;
my $v = qx( ps v $$ );
print "$v\n" ;
$c = aa($c);
print length($c) ."\n" ;
my $v = qx( ps v $$ );
print "$v\n" ;
$c = aa($c);
$c = aa($c);
$c = aa($c);
$c = aa($c);
$c = aa($c);
print length($c) ."\n" ;
my $v = qx( ps v $$ );
print $v;
$c = ab($c);
$c = ab($c);
$c = ab($c);
$c = ab($c);
$c = ab($c);
print length($c) ."\n" ;
my $v = qx( ps v $$ );
print $v;
$c = ab($c);
$c = ab($c);
$c = ab($c);
$c = ab($c);
$c = ab($c);
print length($c) ."\n" ;
my $v = qx( ps v $$ );
print $v;

Posted by Jürgen Exner on March 29, 2008, 11:23 am
Please log in for more thread options
>Joost Diepenmaat a écrit :
>>
>>> Based on the fact that perl contains many memory leaks,
>>
>> It doesn't.
>
>I wrote a sample of code to illustrate the issue.
>
>The code create a 10 mega characters string. this is the only big data
>in this sample.

Which subsequently you copy a few times.

>Then, the main part of the code just modify this data; that mean that
>memory usage should (in my humble opinion) stay near of 10 or 20 (or 40)
>mega bytes.
>
>The main program does not manipulate directly the string, but makes
>functions aa and ab to manipulate this string. Those two functions aa
>and ab just make substitutions within the string.
No, they don't modify the string at all, they modify a _copy_ of the
original string.

>----- Script: ------------------------------------------
>
>
>sub aa($)
>{
> my ($d) = @_;

And here you create a copy of the original string.

> $d =~ s/x(.....)/$1y/g ;
> $d =~ s/x(.....)/$1z/g ;
> $d =~ s/x(.....)/$1a/g ;
> $d =~ s/x(.....)/$1b/g ;
> $d =~ s/x(.....)/$1c/g ;
> return $d;

You return that copy ...

>}
>
>
>my $c= 'x' x (1000*1000*10) ;
>$c .= "\x" ;
>print length($c) ."\n" ;
>my $v = qx( ps v $$ );
>print "$v\n" ;
>$c = aa($c);

...and you save that copy in $c, such that the memory cannot be reused.

The rest of the code seems to duplicate that action several times using
successively updated versions of the string as function argument, such
that successivly new copies of the string are created.

jue

Posted by Peter J. Holzer on March 30, 2008, 8:37 am
Please log in for more thread options
> Joost Diepenmaat a écrit :
>>
>>> Based on the fact that perl contains many memory leaks,
>>
>> It doesn't.
>
> I wrote a sample of code to illustrate the issue.
[...]
> Calling function aa (one or several times) makes a memory leak (or
> memory empreint) of 150 Mbytes.
> I mean that once I called this function I do not know how to free those
> 150 mega bytes, but if I call this same function again I will not loose
> more memory.

Then it's not a memory leak. A memory leak is when memory which has been
allocated cannot be (re)used. But in your case it can be reused (and is,
if you call the function again).

Perl is certainly wasteful with memory - The data structures have a lot
of overhead, and it often doesn't free memory because it might need it
again later - but AFAIK perl itself doesn't leak. (Perl programs often
leak - the garbage collector cannot detect cycles, for example, so
the programmer has to remember to do that).

        hp

Similar ThreadsPosted
Huge Memory Load for reading into memory November 6, 2006, 7:10 pm
Get Win32 Total Physical Memory & Available Physical Memory stats October 26, 2004, 10:48 pm
2 issues with "tie" August 26, 2007, 9:32 pm
DBI Performance Issues August 25, 2006, 10:33 am
Perl script issues - Need help January 20, 2005, 9:54 pm
Known issues with Perl under Cygwin? August 26, 2005, 6:43 pm
Perldoc security issues September 12, 2005, 2:16 pm
having issues using awk and/or converting to perl January 9, 2006, 6:21 am
taint issues in utf8_heavy.pl August 23, 2006, 7:43 am
Inline replacement issues October 18, 2006, 3:53 pm

Our other projects:

Art Dolls, Fairies and Mermaids - Sunnyfaces.net

Roy's Linux, Programming and Search Engines messages

1-Script XML SitemapXML Sitemap