Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Re: perl multithreading performance
- Leon Timmermans
August 28, 2008, 5:49 pm
rate this thread
On Wed, 27 Aug 2008 14:25:32 -0700, dniq00 wrote:
Is your computer 64 or 32 bits? In the former case mmap will work for
such large files, but the latter it won't. In that case it may not be a
bad idea to split the log files into chunks that do fit into your memory
space. An additional advantage of that would be that you may not need to
use threads at all.
Re: perl multithreading performance
Assuming <> is actually referring to a single file (if it doesn't, you
can just process several files in parallel), the same approach can be
used even without mmap:
Fork $num_cpu worker processes. Let each process seek to position
$i * $length / $num_cpu, and search for the start of the next line. Then
start processing lines until you get to position ($i+1) * $length / $num_cpu.
Finally report result to parent process and let it aggregate the