Output status during processing

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

I hate to come here with a question without any tried code but I really
do not even know where to start, I have never attempted this before
without a net.

What I would like to do is provide an output to the browser during a
long processing term.

Exactly what I am doing is this.

I am loading a csv file to update inventory.

During this process I am flushing the current Table and then loading the
new and then comparing the inventory table to a product table and would
like to provide status of where we are in the process what items matched
and changed and so forth in real time.

I have done some similar tasks in JS but with minimal steps in the
processing then what I am trying to accomplish.

I did some binging and found that it may have to do with output
buffering but nothing particular to my situation. (as far as I could find)

If this is possible could someone point me in the right direction and I
will do the leg work.


Re: Output status during processing

Am 04.11.2012 20:23, schrieb Scott Johnson:
Quoted text here. Click to load it

Are you starting your long running job in the browser? Not good, the whole setup
designed for request-response, and not long running jobs.

I would try to design the long running update-compare job as a separate process,
writing status or progress information to a file/a db table. Then you have a
corresponding script you call in your browser, showing you status and progress.
could be done with whole page refreshs or with ajax.


Re: Output status during processing

On 11/4/2012 1:28 PM, M. Strobel wrote:
Quoted text here. Click to load it

Ok that makes makes sense.

And yes it was started in the browser on a file upload from a form field.

I could use AJAX to load the file and then process it in the background
sending status points as it goes through AJAX.

I have done this is the past just not for so many steps to perform.

The more I read about the PHP buffer functions, I could not find a way
that would periodically send text without a header issue.

Guess why they have different tools for the different needs.

Thanks Str.


Re: Output status during processing

On 11/4/2012 4:52 PM, Scott Johnson wrote:
Quoted text here. Click to load it

Nope, but it has nothing to do with php.  HTTP is a request/response
protocol; the client sends a request and expects a response within the
time specified by the browser's timeout value.

But then this causes a problem also, because PHP has a buffer (for
efficiency), the server has a buffer (also for efficiency), routers
along the way may have buffers (again, for efficiency), and the client's
system may have a buffer (hmmm... efficiency?) and finally the browser
may have a buffer.

The bottom line is - the HTTP protocol is pretty much useless for
long-running tasks.  The time it takes to send data to the browser is
dependent on a lot of things, and the browser's timeout values are
independent of anything the server can control.

If you need such processing, I suggest you look into other methods of
communications.  Java applets, for instance, can work with non-http

Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.

Re: Output status during processing

On 11/4/2012 5:21 PM, Jerry Stuckle wrote:
Quoted text here. Click to load it

Yeah I can see there can be a bottle-neck/lag issue anywhere which would
be useless for anything real time (which I use loosely).

Thanks for the input.


Re: Output status during processing

On 11/04/2012 08:10 PM, Scott Johnson wrote:
Quoted text here. Click to load it

You can also have the browser request a reload every few seconds, it
isn't exactly what you're asking for but it can give more or less the
same benefits.

Re: Output status during processing

On Sun, 04 Nov 2012 13:52:35 -0800, Scott Johnson wrote:

Quoted text here. Click to load it

Back up a step.

The server can't "send" status updates to the browser using ajax in the
normal case, rather the browser would request "request" status updates
from the server, usually in relation to some browser event (which might
be a javascript timer timeout, or the user clicking a "get update" button
or link).

Ajax, which is built on the http request / response, which is in turn
built on a tcp/ip connection, is not a technology which generally allows
the server to push unsolicited data to the browser, rather it is a
technology that allows the browser to make requests to the server and
receive data back without reloading the page being displayed.

Specifically, when you make an http request to a server, a tcp connection
is opened to the server. The connection is usually closed after the
server has sent the response. Making a connection to the server is easy,
because tcp/ip knows that the server is listening to http on port 80. In
the other direction, there is no official port that a browser listens out
on for a server to make a connection to it, which makes it difficult for  
the server to open a connection to the browser to just send it something.

I know it may sound like a petty distinction to make, but making the
distinction helps you keep it clear in your mind how it all fits
together, and in the concept of php running behind a web-server, the
context is generally servicing and responding to requests from the
browser, not pushing data to the browser.


Denis McMahon

Re: Output status during processing

Denis McMahon wrote:

Quoted text here. Click to load it

_Usual_, not normal.

Quoted text here. Click to load it

There is no javascript.

Quoted text here. Click to load it

Quoted text here. Click to load it

… is a misnomer used only by people who do not know better.  It does not
need to be asynchronous, it does not have to be JavaScript (or any other
ECMAScript implementation), and it often does not use XML.

Quoted text here. Click to load it

_HTTP_ and _TCP/IP_

Quoted text here. Click to load it

_Usually_, not generally.

Quoted text here. Click to load it


Utter nonsense.  HTTP/1.1 uses persistent TCP connections by default.  
HTTP/1.1 (RFC 2616) is currently an IETF Draft Standard.  It was first
issued in 1999 and is widely implemented today: HTTP/1.1 is the default
protocol version in modern Web browsers.

Quoted text here. Click to load it

Or any other port.  80/tcp is the default port for HTTP, not the only one.

Quoted text here. Click to load it

An official port number is not required.  A privileged application in the
browser (a browser extension) can listen for push notifications from a
remote client on an unused TCP port > 1024.  Which is done.

However, it is also possible that an HTTP client keeps a listen-connection
(and therefore a local TCP port) open for a specific HTTP server, waiting
for push updates from that server through that connection.  That application
model is currently called Comet, among other names.

<http://en.wikipedia.org/wiki/Comet_ (programming)>

(The caveat there is that an HTTP client is supposed to limit the number of
connections to the same server or proxy, meaning that this "connection port"
of the browser's HTTP client cannot be used for accessing other resources as
Web browsers follow that recommendation in the RFCs.)

There are other approaches (like server-sent events and the HTML5 WebSocket
API) that accomplish the same.  They are referred to at the bottom of that

Quoted text here. Click to load it

If only you knew what you are talking about.

Anyone who slaps a 'this page is best viewed with Browser X' label on
a Web page appears to be yearning for the bad old days, before the Web,
when you had very little chance of reading a document written on another
computer, another word processor, or another network. -- Tim Berners-Lee

Re: Output status during processing

Scott Johnson wrote:

Quoted text here. Click to load it

Contrary to popular belief, it is possible with PHP alone to provide step-
by-step feedback to the client (as well as to a server-side log file) while
a server-side script is being executed.

About one year ago, I have done something very similar to what you want to
do: I have written a PHP script to import product information from a CSV
file into a Magento eCommerce database (in a way that Magento's built-in
import feature was not capable of at the time).

Here is basically what I have done:

1. Enable PHP's output buffering (ob_start()).

2. Process a record in the CSV (with fgetcsv() and the Magento API),
   writing status information to the server-side log file and (optionally)
   to the standard output (with file_put_contents() and `echo',

3. Flush and clear the output buffer:

      * Flushes all output buffers and restarts output buffering
      * @author sebastian@jcompare.com
      * @link http://php.net/manual/en/function.ob-flush.php
     function flush_buffers()

   That displays what is written in step 2 in the console or the browser
   window (depending on how the script is run).  In order to have the last
   row visible, client-side scripting is used to scroll down to it in
   regular intervals.

4. Continue with step 2 if there are more records or files;
   exit successfully otherwise.

That has worked beautifully both with text/plain and text/html.

The output-buffering approach does _not_ provide you with the ability to
update arbitrary parts of an (X)HTML document, though; it would create the
response document content on the fly instead.  If you need arbitrary display
updates, you should look into asynchronous XMLHttpRequest, with which you
would poll status information from the server accessing a (PHP-)generated
resource in regular intervals, and update your (X)HTML document accordingly.  
(The output-buffering approach was adequate and sufficient in my case; the
XHR approach was and is employed by the Magento UI but was not sufficient
for the used data format at the time.)

Also, you should be aware that no matter how you do it, neither a database
connection is likely to be stable indefinitely nor a PHP script is likely to
run forever.  In particular, on the server my script ran, the import was
supposed to run regularly (from CSV files that were uploaded by the customer
at arbitrary times, with the provision that the shop information needed to
be as up-to-date as possible).  IOW, a cron job was set up which started PHP
CLI to run the PHP script if it not already ran.

But because a CSV file could have had many (thousands of) records (read:
product variants), and there were several such files, it was not possible to
import all records from all files in one run.  Either the database
connection used by the Magento API was severed or the PHP script would be
terminated early (because of max_execution_time in php.ini, which for
security reasons could not be set to 0 for the Web version that was still
needed for "manual" import).  

I have solved this with storing the position of the last import in another
file, so that when the script was executed next time it would check if a
previous position was saved and continue from there, or it would continue
from the beginning of the file(s).


When all you know is jQuery, every problem looks $(olvable).

Re: Output status during processing

On 11/5/2012 5:22 AM, Thomas 'PointedEars' Lahn wrote:
Quoted text here. Click to load it

Ahh very interesting.

Thank you very much for the information.  This may be just the ticket
for me.  I will let you know how it ends up.

Again thanks

Re: Output status during processing

Scott Johnson wrote:

Quoted text here. Click to load it

You are welcome.

But *please* stop wasting resources (and trying your readers' nerves) and
*trim* *your* *quotes* to the relevant parts next time.  It is absolutely
unacceptable that you post 103 lines of quoted text for 9 lines of new (own)
text (of which one line is an attribution line even):



P.S.: You have e-mail.
var bugRiddenCrashPronePieceOfJunk = (
    navigator.userAgent.indexOf('MSIE 5') != -1
    && navigator.userAgent.indexOf('Mac') != -1
)  // Plone, register_function.js:16

Site Timeline