Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
Re: best way to make a few changes in a large data file
That's going to suffer from the same problem as the 'put the changes
into a hash' idea I posted earlier: Searching for a particular key for
a large number of times in a small hash, with most of these searches
being unsuccessful, is going to be slower than building a large hash
and (successfully) searching for a small number of keys in that. And
since there's no way to determine the key of a particular 'big file'
line except by reading this line (which implies reading everyting up to
this line) and parsing it and now way to generate the output stream
except by writing out all 'new' lines in the order they are supposed
to appear, it won't be possible to save any I/O in this way.
There are a number of possibilities here but without knowing more
about the problem, it is not really possible to make sensible
suggestion (Eg, what is supposed to be saves, memory or execution time?
Is it possible to change the process generating the 'big files'? If
not, how often is a file created and how often processed?).