Intense CPU strain....suggestions

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
I am writing YAPSS (Yet Another Photo Sharing Site).

Currently, the user can upload up to six pictures at once.  When the
uploads are complete, I then have a script that will resize the pix to
different thumbnails for speed.  However, imagine resizing 6 pictures
of 2300x2200 pixels in 6 different sizes (maintaining the best quality
I can) on a Pentium 3 450Mhz running Ubuntu and 384 megs of ram.


Obviously, this is a test server and the real deal will be more
powerful but I don't want my users sitting around waiting on
resizing.  I would prefer to add the pictures to a queue for later
processing and allow the users to go do other things on the site.

My question is, how would you recommend I kick off this script and
allow the user to browse away?  I've heard people suggest using an
iframe but that sounds kludgey.  I've also considered using some type
of local XML-RPC server/client and have a script make a call.  But
would the client then have to sit and wait for a response?

As you can see, I am on the planning stages at this point.

I do like the idea of having a server running waiting to accept resize-
requests.  That way, he could easily be moved to dedicated hardware
later on.

Thoughts/opinions welcomed!



Re: Intense CPU strain....suggestions

cbmeeks contained the following:

Quoted text here. Click to load it

Why not just upload the pics and leave resizing til the pics are fetched
for the first time?  After that fetch from the filesystem.

Geoff Berrow (put thecat out to email)
It's only Usenet, no one dies.
My opinions, not the committee's, mine.
Simple RFDs

Re: Intense CPU strain....suggestions

I thought about that.  The problem is, that I want to allow very high
res uploads.  If the user uploaded 20 pictures, and they went to a
gallery page for the first time, it would take a while to resize.
That could leave a bad first impression.

Quoted text here. Click to load it

Re: Intense CPU strain....suggestions

cbmeeks wrote:
Quoted text here. Click to load it

Well, you could kick off a batch job to do your resizing, then return to  
the user.  That way it can keep processing the images.  The only thing  
you'll have to handle is when the user wants to display his gallery  
before it's ready.

Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.

Re: Intense CPU strain....suggestions

Quoted text here. Click to load it

That's exactly how I want to do it.  I just don't know the best way to
execute it.  I like the idea of having an XML-RPC server running but I
am concerned with security (as I should be).

So, let's say I have the following code (using CI):

$this->Upload->do_uploads();  // handle the actual uploading.

$this->Photos->do_resizing();  // handle the resizing


"Waiting" doesn't get kicked off until after do_resizing  (sorry, new
to PHP).  I'm sure there is a smarter way.  I'd like to avoid pop-ups,
iframes, javascript, etc if possible.

I also thought about having the "do_resizing" simple add some records
to a db and have a program running on the server full time that checks
the db frequently.  But wouldn't that be wasteful?  I like the idea of
programs laying dormant until something kicks it in the pants and says
"do something!"  lol



Re: Intense CPU strain....suggestions

cbmeeks wrote:
Quoted text here. Click to load it

First of all, the redirect won't happen until the code reaches there -  
which is after your resizing.  So that's normal operation.

You could have a cron job kicked off every few minutes to resize the  
pictures, but again, you'll run into the problem that after uploading  
the user won't be able to see the pictures he's just uploaded - at least  
in various sizes.

You could use shell_exec() to start off a background process, i.e.  
(assuming Unix/Linux):

   shell_exec('nohup myprog file1 file2 file3 2> /dev/null &');

nohup says to execute a command without hangups and no output, 2>  
/dev/null throws away any error messages, and the '&' at the end says  
run in the background.

You'll still run into a problem that the resizing might not occur as  
fast as the user wants to display it; you'll have to be able to handle  
this in your display routines.

Alternatively, you could do something like:

echo 'Resizing your images, please wait...<br>';
$this->Upload->do_uploads();  // handle the actual uploading.
$this->Photos->do_resizing();  // handle the resizing
echo 'Done!.  <a href="nextpage.php">Click here to continue</a>';

The flush() works with most browsers if you keep the html simple - it  
forces the output to the browser and the browser displays it.  But if  
you start to get fancy (i.e. tables, etc.), many browsers will wait for  
the end of the element to actually display.

The only problem with this is, once you've sent *any* output to the  
browser, you can't redirect them to another page via php (you could use  
an html redirect).  That's why the link.

Just some ideas.

Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.

Re: Intense CPU strain....suggestions

Thanks for the ideas!

I think I might have a somewhat complex system that might work.

First of all, when pictures are uploaded, I record the information in
the db.  I could add a column that says "is_ready" so that they
wouldn't show in the galleries and the resize could mark them ready.

I'm really leaning towards and XML-RPC system.  The XML-RPC server
could sit and wait for requests.  The client ("do_upload") could send
a request when uploading is finished.  That should take just a second
or so...hopefully.

The request could contain information on what to resize.

The XML-RPC server then could instruct the resize script to run and
then the script could mark the image ready.

I would need to make sure I knew how to run a robust XML-RPC server
capable of handling many requests.

I know this whole system seems complicated but I am trying to think
scalability.  The XML-RPC server could easily be moved to any machine.

Thanks for suggestions guys.


Quoted text here. Click to load it

Site Timeline