Slowing down an array / batch

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
The more I learn the more I realize I don't know -

I have an aray looping that's sending info each time it loops to a
remote server, then getting a response and storing the response in a

My problem is that I have a lot of data that I have ready to go
through this loop but more than the remote server can handle if I just
set it loose.

I have about 20,000 rows of data that will be looped through it so
that's 20,000 connections and responses to the remote serve and
storing of the responses locally.

I'm not even sure if arrays are built to handle this much? Do you know
if there's a limit on an array?

My main problam is that the remote server will only allow 250 requests
per minute and 30,000 requests per day.

I've pasted my loop example below, but I need advice on how to either
create a batch system to make sure I onl loop through the array 250
times a minute and no more than 30,000 times a day   -   or a simple
way to pause the loop.

If anyone has an idea please and can point me in the right direction
it would really help.

// Here's the general idea below, see part of the loop:

while($row = mysql_fetch_array($result, MYSQL_ASSOC))
   $items = " <br>";

   fputs($fp, "$itementer");

   $remote_server_response = fgets($fp, 128);

// the fgets response is then added to a mysql database each time the
array loops


Re: Slowing down an array / batch wrote:
Quoted text here. Click to load it

You'll need access to both servers.  Then you'll need to write a script
which can merge multiple requests into one and send them.  On the other
end, separate the requests, process them and combine the responses into
one string for returning to the caller.

Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.

Re: Slowing down an array / batch

Quoted text here. Click to load it

This is one of the problems Jerry - I will not be able to access both
The set up is that I can only control what I send to the ther server,
and collect a response for each item sent to the other server.

I'm thinking that either there's a way to run the whole array and
batch output (slowing it down) or maybe I have to try and make my
array's from only 250 rows of data at a time and then somehow not
start the next loop (of the next 250 rows) for another minute.

It may be a case that the latter is the only option - I'm not sure.
But I'm thinking that this would have to be done by putting a loop
inside a loop to stop at the end but keep checking for 250 rows at a
time and then some phpcode to wait for 60 seconds before running the
next 250

Does anyone have any idea if loops within loops work ok like this -
and any code to use to pause the process for 60 seconds in between the
inside loops?


Re: Slowing down an array / batch wrote:
Quoted text here. Click to load it

If you can't do anything on their server, you're stuck.  But eventually
you will run up against their 30K/day request limit.  You indicated
you're already running 20K/day.  You don't have much room left
(especially if you have a peak, their system is down for maintenance for
a while, etc.).

I suspect your best bet is to put all of the info in a work table
locally.  As you process a request, delete it from the work table.

Then use a cron job to start a task every minute or so.  Process a few
entries and terminate.

Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.

Site Timeline