Using loop labels and iterating.

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

This is a script that opens up a log file and searches for the word
'started' and if it finds the word, it prints it and iterates to the
next log file. But for the servers that didn't start, I want it to
print "didn't start". Please help, thanks.

use diagnostics;
use warnings;

open(SCRATCHPAD,"LOG.txt") or die "can't open $!";
my @uniq = <SCRATCHPAD>;

foreach my $stuff ( @uniq ) {

open(STARTUP, "/c$/$stuff/esp.log") or "die can't open $!";
my @log_file = <STARTUP>;

for my $line ( @log_file )  {

next LINE if $line !~ /ESP server (started)./;
my $capture = "$1\n";
print "$stuff =>  $capture";



Re: Using loop labels and iterating.

Am 17.02.2012 04:22, schrieb Nene:
Quoted text here. Click to load it

It's good that you have diagnostics and warnings enabled.
Especially with variables in the filename expression of
your open statements, you should use the three argument

Your example code has a number of issues: Your second die
statement is broken (the die's inside the string), you
close the outer file multiple times inside the loop, and
a little bit of indentation would go a long way in making
it more legible.

You don't say if you plan to do more processing of the
server names later on, which would merit storing them in
@uniq, or not. For sake of conserving memory I've omitted
doing that and just iterate over the file line by line.

I hope the code below is self-explanatory enough.

use diagnostics;
use warnings;

open(my $scratchpad, "<", "LOG.txt")
     or die "can't open LOG.txt: $!";

while( my $stuff = <$scratchpad> )
     my $status = "didn't start";

     chomp $stuff;
     open( my $startup, "<", "/c$/$stuff/esp.log" )
         or die "can't open $stuff/esp.log: $!";

     LINE: while( my $line = <$startup> )
         if( $line =~ /ESP server (started)/ )
             $status = $1;
             last LINE;

     close $startup;
     print "$stuff => $status$/";

close $scratchpad;


Re: Using loop labels and iterating.

Quoted text here. Click to load it

    use File::Slurp qw/slurp/;

    my $startup = slurp "/c$/$stuff/esp.log";
    $startup =~ /ESP server started/
        and print "$stuff => started\n";

Unless the logfile is *huge* (by which I mean several GB, these days),
it's going to be more efficient to read and match it all in one go.

$/ is the input line separator. It isn't usually right to use it as the
output separator. You can avoid having to print separators at all by
setting $\, or using 5.10's 'say'.

If you'd rather avoid File::Slurp (though there's no good reason you

    my $startup = do {
        open my $STARTUP, "<", "..." or die ...;
        local $/ = undef;

Oh, and my usual plug for 'autodie', which will add all those 'or die's
for you.


Re: Using loop labels and iterating.

Quoted text here. Click to load it

It is probably going to be faster which might be a good thing if the
assumption that the machine is exclusively dedicated to this
particular task holds because it will then hog CPU and memory most
effectively. OTOH, except if the logfile is *huge*, is the difference
large enough to matter, especially considering that the computer will
very likely have more important things to do than 'monitoring itself'?

OTOH, the source code of that was 'an interesting read'. I suggest
that 'Author: Uri Guttman' should be considered a sufficient reason to
avoid any code blindly, based on that.

Re: Using loop labels and iterating.


Quoted text here. Click to load it

For instance, if PERL_IMPLICIT_SYS is not set, sysread will simply do
a read system call and that may return less data than was requested
for any number of reasons. As far as I could determine, the module
does a single sysread call for 'small files' and returns the
results. The way 'atomic updates' are implemented is known to not work
with certain filesystems because it relies on rename having barrier
semantics wrt data writes and this might not be the case (out of my
head, this will break on 'older' version of ext4, on XFS and on any
filesystem which always performs metadata updates sychronously, IOW,
UFS and FFS). There's some hardcoded support for dealing with
'Windows' textfiles in an atrociously inefficient way (by doing
single-character deletions on the read buffer which is an O(n*n)
algorithm). There's no support for dealing with any other kind of


File::Slurp and reading files in general

Am 17.02.2012 16:46, schrieb Rainer Weikusat:
Quoted text here. Click to load it

Talking about this:

what is actually the advantage of sysread over the "Perl way" of reading
from a file with code like $result = <$fileHande> (or @result =
<$fileHandle>, depending on wantarray)? To slurp, I simply undef $/ and
everything seems fine... I have never used sysread in Perl - should I
consider using it?

I didn't expect such a long code for some "simple" thing like reading in
a file, especially not a comment telling me it is using "DEEP DARK MAGIC".

As I am usually using an "enterprise" Linux distribution (RHEL clone), I
still need to make scripts compatible with 5.8.8 where File::Slurp is
not a core module...

- Wolf

Re: File::Slurp and reading files in general

Quoted text here. Click to load it

In theory it can be faster. Perl's normal PerlIO layer puts a layer of
buffering between your program and the kernel; if you care a lot about
performance you probably want to avoid that and do the buffering
yourself more intelligently. Most of the time this will make no
difference and isn't worth worrying about.

Quoted text here. Click to load it

No, not unless you have a reason to. I use File::Slurp because (and only
because) it has a clean, simple interface for getting a file into a
string, without needing to mess about opening filehandles and setting

Quoted text here. Click to load it

The DEEP DARK MAGIC is specifically to do with slurping the DATA
filehandle. This is harder than you might think to do correctly, and
it's useful to know it will Just Work. (Using the untaint flag to
identify a DATA handle is a hack, hence the comment.)

Quoted text here. Click to load it

File::Slurp is never a core module (or, at least, not to date). However,
I'm sure RHEL has a package for it.


Re: File::Slurp and reading files in general


Quoted text here. Click to load it

Provided that's really just what you want, consider using this:

sub slurp
    my $fh;

    open($fh, '<', $_[0]) or die("open: $_[0]: $!");
    local $/ unless wantarray();
    return <$fh>;

That's less buggy (because it leaves the intricacies of dealing with
system specific I/O to perl) and possibly even faster (again because
it uses features perl already has).

Re: File::Slurp and reading files in general

Quoted text here. Click to load it

It bypasses the Perl I/O buffering mechanism, including any
translations layers etc which might be active as part of that. I
wouldn't use it for reading 'text files'. It is, however, useful when
more control about the actual I/O operations performed by a program is
required than the read-in-advance/ write-behind buffering mechanism
offers. This would usually either be the case if the I/O is actually
'real-time' IPC, eg, when the program is acting as a server on an
AF_UNIX datagram socket or when reliability is an important concern,
eg, when doing 'atomic updates' of files which must (to the degree the
application can guarantee this) remain in a consistent state even when
power suddenly goes away. Since "whining about the evil filesystem"
doesn't really help to solve the problem, this required a multi-step
procedure where one stop must have been completed before the next one

Re: File::Slurp and reading files in general

Quoted text here. Click to load it

I'll just quote from a posting I wrote about 2 years ago (incidentally
in reply to Uri Guttman's claim that sysread is faster):

| So I grabbed the server with the fastest disks I had access to (disk
| array of SSDs), created a file with 400 million lines of 80 characters
| (plus newline) each and ran some benchmarks:
| method                  time      speed (MB/s)
| ----------------------------------------------
| perlio  $/ = "\n"       2:35.12   209
| perlio  $/ = 96      1:35.36   340
| perlio  $/ = 48576   1:35.25   340
| sysread bs = 4096       1:35.28   340
| sysread bs = 1048576    1:35.18   340
| The times are the median of three runs. Times between the runs differed
| by about 1 second, so the difference between reading line by line and
| block by block is significant, but the difference between perlio and
| sysread or between different blocksizes isn't.
| I was a bit surprised that reading line by line was so much slower than
| blockwise reading. Was it because of the higher loop overhead (81 bytes
| read per loop instead of 4096 means 50 times more overhead) or because
| splitting a block into lines is so expensive?
| So I did another run of benchmarks with different block sizes:
| method                    block     user  system  cpu  total
| read_file_by_perlio_block 4096     0.64s  26.87s  31%  1:27.91
| read_file_by_perlio_block 2048     1.48s  28.65s  34%  1:28.56
| read_file_by_perlio_block 1024     5.14s  29.03s  37%  1:30.59
| read_file_by_perlio_block  512    11.98s  31.33s  47%  1:31.22
| read_file_by_perlio_block  256    26.84s  33.13s  61%  1:36.85
| read_file_by_perlio_block  128    43.53s  29.05s  71%  1:41.66
| read_file_by_perlio_block   64    77.26s  28.16s  88%  1:59.70
| read_file_by_line                104.68s  28.01s  93%  2:22.34
| (the times are a bit lower now because here the system was idle while it
| had a (relatively constant) load during the first batch)
| As expected elapsed time as well as CPU time increases with shrinking
| block size. However, even at 64 bytes, reading in blocks is still 20%
| faster than reading in lines, even though the loop is now executed 27%
| more often.
| Conclusions:
|  * The difference between sysread and blockwise <> isn't even measurable.
|  * Above 512 Bytes the block size matters very little (and above 4k, not
|    at all).
|  * Reading line by line is significantly slower than reading by blocks.

Quoted text here. Click to load it

I haven't benchmarked $/ = undef but based on the results above I would
expect it to be as fast as sysread.


   _  | Peter J. Holzer    | Deprecating human carelessness and
|_|_) | Sysadmin WSR       | ignorance has no successful track record.
| |   |         |
__/   | |  -- Bill Code on

Re: Using loop labels and iterating.

Nene wrote:
Quoted text here. Click to load it

This may work better (UNTESTED):

use warnings;
use strict;

open my $SCRATCHPAD, '<', 'LOG.txt' or die "can't open 'LOG.txt'
because: $!";

while ( my $stuff = <$SCRATCHPAD> ) {
     chomp $stuff;
     open my $STARTUP, '<', "/c$/$stuff/esp.log" or die "can't open
'/c$/$stuff/esp.log' because: $!";
     while ( my $line = <$STARTUP> )  {
         if ( $line =~ /ESP server started\./ ) {
             print "$stuff =>  started";
             next NODE;
     print "$stuff =>  didn't start";


Any intelligent fool can make things bigger and
more complex... It takes a touch of genius -
and a lot of courage to move in the opposite
direction.                   -- Albert Einstein

Site Timeline