Re: perl multithreading performance

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

On Wed, 27 Aug 2008 14:25:32 -0700, dniq00 wrote:
Quoted text here. Click to load it

Is your computer 64 or 32 bits? In the former case mmap will work for
such large files, but the latter it won't. In that case it may not be a
bad idea to split the log files into chunks that do fit into your memory
space. An additional advantage of that would be that you may not need to
use threads at all.



Re: perl multithreading performance

Quoted text here. Click to load it

Assuming <> is actually referring to a single file (if it doesn't, you
can just process several files in parallel), the same approach can be
used even without mmap:

Fork $num_cpu worker processes. Let each process seek to position
$i * $length / $num_cpu, and search for the start of the next line. Then
start processing lines until you get to position ($i+1) * $length / $num_cpu.
Finally report result to parent process and let it aggregate the


Site Timeline