Any way PHP can directly initiate/execute a HTTP connection+download?

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
I've searched the regular/tutorial documentation for PHP, and I
can't find any pure-PHP tool for, given a URL (with query string),
connect to that place and pass back the string of HTML or other
ordinary HTTP response. On a Unix/Linux host where you have full
account priviledges, of course you can use backtick to run *any*
Unix/Linux program from inside PHP. But I don't have access to
PHP/MySQL on my shell account, I'm using a free PHP/MySQL hosting
service, which of course doesn't allow backtick.

The only workaround I can think of is to issue a redirect to a CGI
script running Common Lisp on my Unix shell account, with GET query
string specifying what exactly the CGI/CL script is supposed to do,
have *that* application do all the Web spidering, collect the
results and write to file, then have GCI/CL connect back to the
main PHP server to POST the results, which are written to a local
file on the PHP server, and finally the CGI/CL script redirects
back to the PHP server, with GET query string to tell where to find
that data, and then the PHP script loads the local file and
processes it to the database. The GET query strings in both
directions would use public-key encryption/signature to prevent
mis-use. If the public-key signature from PHP to CGI/CL isn't
correct, or anything else is wrong with the request, the CGI/CL
script aborts immediately without consuming many CPU cycles on the
shell account. If the public-key signature from CGI/CL to PHP isn't
correct, etc., the PHP script ignores the request and doesn't even
try to load the local file much less make any changes to the

Before doing that messy workaround, I'd like to be sure there's no
way to program PHP to act like a "spider", browsing various Web
sites and collecting data from them. The specific application I
have in mind is that the user of the PHP application at specifies a keyword to search for job ads, the
spider browses and other archives of job ads, collects
job ads that match the keyword, reformats them for my purposes, and
passes the list of reformatted job ads back to the main PHP
application, to store in the database any that aren't already
there, and to index per my proposed system of filtering job ads to
eliminate any ads that require skills/experience the user doesn't
have. Whether I have PHP do many direct connections to job boards
and reformat the data itself, or PHP does single connection to
CGI/CL script that does all the spidering and reformatting to
simply pass back to the PHP script, in either case I'd like PHP to
be able to make that/those HTTP connection(s)/download(s) directly
so it's transparent to the user, rather than require the user to
deal with repeated redirects to switch back and forth between
direct connections to PHP and to CGI/CL. Another advantage of
direct connection is that URL of the CGI/CL application is *never*
seen by the user's Web client, so there's no way he/she can harvest
that URL and later make direct unauthorized connections to flood my
shell account.

Re: Any way PHP can directly initiate/execute a HTTP connection+download?

Robert Maas, wrote:

Quoted text here. Click to load it


$url = " ";



For fancier stuff, use CURL.

Iván Sánchez Ortega -ivan-sanchezortega-es-

The time is right to make new friends.

Re: Any way PHP can directly initiate/execute a HTTP connection+download?

On Jun 17, 4:13=A0pm, (Robert Maas, ) wrote:
Quoted text here. Click to load it


Re: Any way PHP can directly initiate/execute a HTTP connection+download?

Quoted text here. Click to load it

(I have no idea why you used a non-USASCII character there.)

I think I my have already used file_get_contents to get the
contents of a local file loaded into a PHP variable, but I had no
idea it could be used with a URL to do a HTTP GET request. The
various online tutorials I had been using to teach myself PHP
didn't seem to mention that capability. So when I saw your
suggestion, I did a Google search to try to confirm your claim, and
I found this very important info:

   First i [sic] need to warn you about something. You can only use fopen() and
   file_get_contents() when fopen wrappers is enabled. This parameter is
   specified in the php.ini file and cannot be changed at run time using
   ini_set(), To know whether you can use these two or not you can use
   the following code to check the value of fopen wrapper seting [sic].
   if (ini_get('allow_url_fopen') == '1') {
      // use fopen() or file_get_contents()
   } else {
      // use curl or your custom function

So I wrote a test script that does the ini_get and shows me the
return value. It came back NULL ('echo' prints it as nothing at
all). That could mean either that the free PHP/MySQL hosting
service I'm using doesn't allow opening HTTP connections, or
ini_get itself is broken on the hosting service and always returns
NULL regardless of the status of the configuration on the server.
Give that the hosting service disallows CREATE VIEW, I'm guessing
it also disallows HTTP connections to remote servers.

So it looks like I'll need to use my workaround of redirecting to a
CGI/CommonLisp application, with PK encrypted query string to tell
it what to do, have Common Lisp do all the spidering, and when done
connect back to the PHP hosting site and run an upload script that
writes the block of data to a local file there, and then finally
the CGI/CommonLisp script redirects back to the PHP hosting site to
process that local file to install the data into the MySQL
database. Now if allow_url_fopen is enabled on my Unix shell
account, then I can instead redirect to a PHP script that checks
for valid request before doing file_get_contents(urlOfCGICLScript)
to run the CommonLisp script under it, and then redirect from that
PHP script on shell account back to PHP script on free hosting
service. Or if allow_url_fopen isn't even allowed on my Unix shell
account, I could use backtick to execute a shell command that runs
lynx with command-line URL to the CGI/CommonLisp application. Let
me trim down that allow_url_fopen test script so it will work here
.. yeah, it returns 1 instead of NULL here, so I won't need
backtick. But let me actually try file_get_contents before I count
my chickens ...  yes it works, at least for a .txt file at that URL.
Now let me try one more, this time some CGI script at the URL ...
yup, it works for the test PHP script on my shell account to use
file_get_contents to invoke a CGI/CL script.

If you're curious, here's the final test script:
  $bool1 = ini_get('allow_url_fopen');
  echo "allow_url_fopen = $bool1<br>\n";
  $url = " ";
  $data = file_get_contents($url);
  $len = strlen($data);
  echo "Length of file-data is $len bytes.<br>";
  echo "<pre>

So why would I have PHP on hosting service redirect to PHP on shell
which uses file_get_contents to run the CGI/CL script, instead of
just having PHP on hosting service redirect to CGI/CL script?
Because some spammer who doesn't like me blocking his spambots with
Turing tests all over the place might harvest the redirection URL
and then wage a dDOS botnet flood attack against it. dDOS flood
against the CGI/CL script would likely crash the shell machine,
starting up thousands of simultaneous CGI/shTrampoline/CMUCL
triples, but dDOS flood against the PHP script, starting up
thousands of scheduled PHP responses within the Apache server,
never getting into the CGI/CL except for legitimate (public-key
signed) requests, wouldn't do much harm, and would be manageable.

So thanks for the clue giving me the keyword to do Google search to
find out how to tell if I can do what you suggested on each hosting
service and shell account. That was a key high-level design
decision I needed to do before starting to write any of the code
for the spider.

Re: Any way PHP can directly initiate/execute a HTTP connection+download?

On 18 Jun 2009, (Robert Maas, ) wrote:

Quoted text here. Click to load it

<article snipped>

Right, the fopen wrappers need to be enabled.  You might want to have
a read in's manual:


Quoted text here. Click to load it

Don't use `echo' to test for the type and value of a variable, use
`var_dump()' to get the explicit type and value.  You should RTM for
`ini_get()', it can return an empty string for failures and NULL

Quoted text here. Click to load it

They may have disabled various functions within PHP.  I know someone
who has GoDaddy's free hosting, and they disable a myriad of PHP
functions, including `ini_get()', `ini_set()', `fsockopen()', and who
knows what else.

Quoted text here. Click to load it

It's probably not broken, but they may have disabled `ini_get()'.  If
your host indeed disabled `ini_get()', it will always return NULL.

Quoted text here. Click to load it

Set up a test script on your hosting environment to briefly test for
various functions like `ini_get()' and `fsockopen()'.  If you have
`fsockopen()', you can use that to easily make a GET request, even if
fopen wrappers are disabled.  I usually use my own function wrapping
an `fsockopen()' call, which can make GET, HEAD, or POST requests.  
It doesn't require libcurl or fopen wrappers, which is why I still
sometimes use it.

If you decide to try a test script to see if your host disables
functions, you need to use `function_exists()' to check.  Try
something like:


/* add more functions to the array as needed */
$funcs = array('ini_get', 'fsockopen');

foreach ($funcs as $f) {
  echo "Checking for $f()... ",
    (function_exists($f) ? 'Yes' : 'No'),



Good luck!

Anonymous (1984 IOCCC winner):
int i;main()"];read('-'-'-',i+++"hell\
o, world!\n",'/'/'/'));}read(j,i,p)

Re: Any way PHP can directly initiate/execute a HTTP connection+download?

Robert Maas, wrote:
Quoted text here. Click to load it
Quoted text here. Click to load it

So, why don't you spend $5/mo. and get a real hosting account?  How much
time do you want to waste for $60/yr.?

Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.

Re: Any way PHP can directly initiate/execute a HTTP connection+download?


on 06/17/2009 06:13 PM Robert Maas, said the
Quoted text here. Click to load it

You may want to try this pure PHP HTTP client class. It can send any
type of PHP requests and give you access to any type of responses.

Now if you want something to help you spider sites, there is a recently
published class that makes it easier to browse sites like you would with
a regular browser, and can follow specific links and submit given forms
that may be found in the pages.


Manuel Lemos

Find and post PHP jobs /

PHP Classes - Free ready to use OOP components written in PHP /

Site Timeline