Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
February 11, 2006, 4:49 pm
rate this thread
I read in a csv file with 60000 lines (20 fields per record), and store
the data to a local array. The file is read in and stored just fine and
Now, if I try to assign that array to a session variable it chokes.
e.g. create array and load each element with a row from the file (btw,
each row is an array as well, using fgetcsv()). When local array is
loaded, I assign to session var as so:
$_SESSION['mydata'] = localArray;
What happens is the server will sit there and churn for a long time. CPU
goes to 99% use and memory use explodes on the apache thread. It will
keep churning a looong time, so long that I had to stop the server to
make it quit. This happens on my windows server as well as fedoracore4
I've tried these variations all with the same results:
- store reference to array in session var (although since it's an array,
I think it makes no difference). e.g. $_SESSION['mydata'] =& myArray;
- read the file directly into a sesson var, instead of creating a local
array first. e.g. $_SESSION['mydata'] = $row; (grabbing one row at a
time from file)
Same thing always happens.
Anybody have any insight into storing large arrays as session vars? Is
it bad practice? Can it be done? What's the scoop?
Any help appreciated!
Re: Large array stored in session variable causes server to choke
Let's see... 60K records x 20 fields per record is 1.2M fields. That's
a lot of data. Then multiply by the average size of a field and you
have a lot of MB. I sure wouldn't try to store that much in a session!
Then PHP has to serialize the data to write it to the session file. And
unserialize when it reads the data in. A lot of work for that much data.
Doesn't look like a sound design to me... Maybe use a database instead?
Remove the "x" from my email address
JDS Computer Training Corp.
RETRACTION - Large array stored in session variable causes server to choke
sorry, no that was a mozilla bug not displaying lines with that many
non-space characters... I've tested sessions upto 2M here - that's about the
limit for this old hardware...
I hear (and see in my config file) that PHP gets (by default) 8M of ram
to play with.
60K x 20 is 1200K fields
there's only room for an about 8 bytes per field, four of which are
probably going to be a pointer of some sort there's probably another four
needed for memory allocation control, or a reference counte etc... looks
like you're doomed before you even start.
I did some testing:
print '<br>' .$c; flush();
I get about 5500 rows by 20 elements by 1 character each before I exceed
if you bump the limit up to say 400M you might have some success
(dependant on record sizes).
that means your server wants around 40G of ram to be able to handle 100
simultaneous requessts... (is 100 a reasonable figure??)
ISTM it might be time to re-evaluate the task and either do without the huge
array, or store and process it using some other language (like a databaee
using SQL or a custom app using C, or a combination oof the two)
Re: RETRACTION - Large array stored in session variable causes server to choke
so the sizeof(double)+sizeof(unsigned int)+sizeof(unsigned
I forgot my C!
I think it's 8+4+1+1.14 bytes to start with, + the sizeof(the HashTable)
when that's all done.
not much detail there. the details about the internal structure of a
hashtable weren't given. sorry. But I suspect this struct is only necessary
once, not repeated every array element, due to the refcount. a refcount is
only needed by the parser/interpreter for certain purposes (like language
"references" & mainly keeping track of pointer counts under the hood).