Click here to get back home

Memory issues

 HomeNewsGroups | Search | About
 comp.lang.perl.misc    Post an article   get this group's latest topics as an RSS feed add this group's latest topics to your My MSN content add this group's latest topics to your My Yahoo content
Subject Author Date
Memory issues jm 03-29-2008
---> Re: Memory issues Joost Diepenmaa...03-29-2008
Posted by jm on March 29, 2008, 7:40 pm
Please log in for more thread options
jm a écrit :
>
> Modifying a little bit again the script, and checking execution with
> ltrace, I observed malloc is called 1871 times when free is just called
> 922 times.
> Isn't it an issue?
>
> and did:

ltrace perl essai.pl 2>&1 | grep 'malloc\|free\|realloc' | perl
observe_malloc_free.pl > memory.log

with observe_malloc_free.pl in mail bottom.

Hereafter, this result of memory leaks:

1 NULL is freed, but thats not a memory leak!

the rest can be read like this:
7 10 Mbytes data are not freed.
1 13 Mbytes data is not freed
8272 4 bytes data are not freed
9843 4080 bytes data are not freed

but I still do not know why...

/tmp$ cat memory.log | sed 's/.*=>//g' | sort | uniq -c
7 10
10 100
7 10000004
22 11
4 112
10 116
1 1192
26 12
1 124
4 128
12 13
1 131716
1 13334528
13 14
3 140
6 15
28 16
9 17
14 18
8 19
19 2
14 20
1 2048
6 21
5 22
3 23
230 24
1 240
6 25
7 256
4 27
1 2712
143 28
4 3
2 30
1 31
125 32
2 33
2 34
1 36
8272 4
3 40
1 4048
1 4064
9843 4080
8 4096
2 4373
1 44
1 45
78 48
1 49156
2 496
72 5
1 50
1 512
65 52
50 56
1 58
6 6
1 628
1 635
13 64
18 7
1 76
45 8
1 80
1 8080
2 84
4 88
74 9
1 98
1 freeing not allocated memory : NULL :



>> --- scipt ---------------------------
>> sub aa($)
>> {
>> my ($d) = @_;
>> $$d =~ s/x(.....)/$1y/g ;
>> $$d =~ s/x(.....)/$1z/g ;
>> $$d =~ s/x(.....)/$1a/g ;
>> $$d =~ s/x(.....)/$1b/g ;
>> $$d =~ s/x(.....)/$1c/g ;
>> return $d;
>> }
>>
>> sub ab($)
>> {
>> my ($d) = @_;
>> $$d =~ s/a(.....)/$1y/g ;
>> $$d =~ s/b(.....)/$1z/g ;
>> $$d =~ s/c(.....)/$1a/g ;
>> $$d =~ s/y(.....)/$1b/g ;
>> $$d =~ s/z(.....)/$1c/g ;
>> return $d;
>> }
>>
>>
>> my $s= 'x' x (1000*1000*10) ;
>> $s .= "\x" ;
>> my $c = $s;
>> print length($$c) ."\n" ;
>> my $v = qx( ps v $$ );
>> print "$v\n" ;
>> $c = aa($c);
>> print length($$c) ."\n" ;
>> my $v = qx( ps v $$ );
>> print "$v\n" ;
>> $c = aa($c);
>> $c = aa($c);
>> $c = aa($c);
>> $c = aa($c);
>> $c = aa($c);
>> print length($$c) ."\n" ;
>> my $v = qx( ps v $$ );
>> print "$v\n" ;
>> $c = ab($c);
>> $c = ab($c);
>> $c = ab($c);
>> $c = ab($c);
>> $c = ab($c);
>> print length($$c) ."\n" ;
>> my $v = qx( ps v $$ );
>> print "$v\n" ;
>> $c = ab($c);
>> $c = ab($c);
>> $c = ab($c);
>> $c = ab($c);
>> $c = ab($c);
>> print length($$c) ."\n" ;
>> my $v = qx( ps v $$ );
>> print "$v\n" ;

-- observe...pl --------------------------

my %hash = ();


while (<>)
{
my $line = $_;
#print "jmg:" . $line;
if ( $line =~ m/malloc\(([0-9]*)\).*= *([0-9xa-fNUL]*)/ )
{
my $size = $1;
my $ad = $2;
#print "malloc : $1 : $2 : \n" ;
if ( defined ( $hash) )
{
print "redundant malloc : $1 : $2 : \n" ;
}
$hash = $size;
}
elsif ( $line =~ m/realloc\(([0-9xa-fNUL]*) *, *([0-9]*)\).*=
*([0-9xa-fNUL]*)/ )
{
my $adp = $1;
my $size = $2;
my $ad = $3;


if ( not defined ( $hash) )
{
print "realloc not allocated memory : $adp : \n" ;
}
delete $hash ;

#print "malloc : $1 : $2 : \n" ;
if ( defined ( $hash) )
{
print "redundant malloc : $size : $ad : \n" ;
}
$hash = $size;
}
elsif ( $line =~ m/free\(([0-9xa-fNUL]*)/ )
{
my $ad = $1;
#print "free : $1 : \n" ;
if ( not defined ( $hash) )
{
print "freeing not allocated memory : $1 : \n" ;
}
delete $hash ;
}
else
{
print "???:" . $line;
}

}

foreach my $key ( keys ( %hash) )
{
print $key . ' => ' . $hash . "\n" ;
}

Posted by smallpond on March 29, 2008, 7:52 pm
Please log in for more thread options
> jm a =E9crit :
>
>
>
> > Modifying a little bit again the script, and checking execution with
> > ltrace, I observed malloc is called 1871 times when free is just called
> > 922 times.
> > Isn't it an issue?
>
> > and did:
>
> ltrace perl essai.pl 2>&1 | grep 'malloc\|free\|realloc' | perl
> observe_malloc_free.pl > memory.log
>
> with observe_malloc_free.pl in mail bottom.
>
> Hereafter, this result of memory leaks:
>
> 1 NULL is freed, but thats not a memory leak!
>
> the rest can be read like this:
> 7 10 Mbytes data are not freed.
> 1 13 Mbytes data is not freed
> 8272 4 bytes data are not freed
> 9843 4080 bytes data are not freed
>
> but I still do not know why...
>


I don't know much about the perl garbage collector,
but memory is not freed immediately when the ref
count goes to 0. When I run your program and watch
with top, VM goes to 200 MB and stays there for the
whole run. That seems to be some upper bound where
the garbage collector is running. Memory use does
not continue to go up.



Posted by jm on March 29, 2008, 8:12 pm
Please log in for more thread options
smallpond a écrit :
>> jm a écrit :
>>
>>
>>
>>> Modifying a little bit again the script, and checking execution with
>>> ltrace, I observed malloc is called 1871 times when free is just called
>>> 922 times.
>>> Isn't it an issue?
>>> and did:
>> ltrace perl essai.pl 2>&1 | grep 'malloc\|free\|realloc' | perl
>> observe_malloc_free.pl > memory.log
>>
>> with observe_malloc_free.pl in mail bottom.
>>
>> Hereafter, this result of memory leaks:
>>
>> 1 NULL is freed, but thats not a memory leak!
>>
>> the rest can be read like this:
>> 7 10 Mbytes data are not freed.
>> 1 13 Mbytes data is not freed
>> 8272 4 bytes data are not freed
>> 9843 4080 bytes data are not freed
>>
>> but I still do not know why...
>>
>
>
> I don't know much about the perl garbage collector,
> but memory is not freed immediately when the ref
> count goes to 0. When I run your program and watch
> with top, VM goes to 200 MB and stays there for the
> whole run. That seems to be some upper bound where
> the garbage collector is running. Memory use does
> not continue to go up.

This is because I have only 500 Mbytes on my computer.
So I made a perl demo program which works within this limit.

Instead of a 10 MBytes string, you can (try to) use a 40 Mbytes string,
or a 100 Mbytes string.

And then, you will see if the garabage collector start at 200 Mbytes,
... or not.

What I only showed with ltrace and observe_malloc_free.pl is that when
the program stops, garbage collector did not collected all garbage.



Posted by Peter J. Holzer on March 30, 2008, 9:02 am
Please log in for more thread options
> I don't know much about the perl garbage collector,
> but memory is not freed immediately when the ref
> count goes to 0.

This is wrong. When the ref count goes to zero, perl immediately calls
free.

free may decide to keep the memory around for a subsequent malloc call,
but that doesn't have anything to do with perl - only with your system's
malloc/free implementation (if you use the system's malloc, which is the
default on most platforms, I think).

> When I run your program and watch with top, VM goes to 200 MB and
> stays there for the whole run. That seems to be some upper bound
> where the garbage collector is running.

Perl doesn't have a garbage collector which runs periodically.

        hp

Posted by Joost Diepenmaat on March 29, 2008, 8:13 pm
Please log in for more thread options

> Modifying a little bit again the script, and checking execution with
> ltrace, I observed malloc is called 1871 times when free is just called
> 922 times.
> Isn't it an issue?

Please keep in mind that perl's memory allocation strategy in general is
optimized for longer running programs, not for one-off scripts (which
makes sense, since one-off scripts don't usually need the performance
gains). This means that for instance subroutines will get memory
allocated on the assumption that they'll be called again, and will take
about as much memory the next time.

This is NOT a memory leak per se, but it does mean that if you have a
subroutine that takes 100Mb to complete, your program will take that
memory and probably not give it back until the program ends. IOW, if you
have a long-running program that only means you need 100Mb for it to
run, it does NOT mean it takes a 100Mb for each call.

In your test case, don't assume that just because the regular expression
replacements don't in theory *need* to use any additional RAM, they
won't. Especially not if you're using UTF-8 encoded strings (which you
are). Perl algorithms tend to exchange RAM for speed in most cases
anyway, and replacing a match with a new string of exactly the same
length in bytes in a unicode string is a pretty uncommon use-case, so
it's likely not optimized.

Anyway, I've not seen a serious memory leak in perl itself in ages, and
I run perl processes that use up to 8 Gb of RAM and run for months
without issues.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

Similar ThreadsPosted
Huge Memory Load for reading into memory November 6, 2006, 7:10 pm
Get Win32 Total Physical Memory & Available Physical Memory stats October 26, 2004, 10:48 pm
2 issues with "tie" August 26, 2007, 9:32 pm
DBI Performance Issues August 25, 2006, 10:33 am
Perl script issues - Need help January 20, 2005, 9:54 pm
Known issues with Perl under Cygwin? August 26, 2005, 6:43 pm
Perldoc security issues September 12, 2005, 2:16 pm
having issues using awk and/or converting to perl January 9, 2006, 6:21 am
taint issues in utf8_heavy.pl August 23, 2006, 7:43 am
Inline replacement issues October 18, 2006, 3:53 pm

Our other projects:

Art Dolls, Fairies and Mermaids - Sunnyfaces.net

Roy's Linux, Programming and Search Engines messages

1-Script XML SitemapXML Sitemap