Persistent/keepalive HTTP with the PHP Curl library?

Solution 1:

cURL PHP documentation (curl_setopt) says:

CURLOPT_FORBID_REUSE - TRUE to force the connection to explicitly close when it has finished processing, and not be pooled for reuse.

So:

1. Yes, actually it should re-use connections by default, as long as you re-use the cURL handle.
2. by default, cURL handles persistent connections by itself; should you need some special headers, check CURLOPT_HTTPHEADER
3. the server may send a keep-alive timeout (with default Apache install, it is 15 seconds or 100 requests, whichever comes first) - but cURL will just open another connection when that happens.

Solution 2:

Curl sends the keep-alive header by default, but:

1. create a context using curl_init() without any parameters.
2. store the context in a scope where it will survive (not a local var)
3. use CURLOPT_URL option to pass the url to the context
4. execute the request using curl_exec()
5. don't close the connection with curl_close()

very basic example:

function get($url) {
    global $context;
    curl_setopt($context, CURLOPT_URL, $url);
    return curl_exec($context);
}

$context = curl_init();
//multiple calls to get() here
curl_close($context);

Solution 3:

1. On the server you are accessing keep-alive must be enabled and maximum keep-alive requests should be reasonable. In the case of Apache, refer to the apache docs.

2. You have to be re-using the same cURL context.

3. When configuring the cURL context, enable keep-alive with timeout in the header:

   curl_setopt($curlHandle, CURLOPT_HTTPHEADER, array(
       'Connection: Keep-Alive',
       'Keep-Alive: 300'
   ));
   

Solution 4:

If you don't care about the response from the request, you can do them asynchronously, but you run the risk of overloading your SOLR index. I doubt it though, SOLR is pretty damn quick.

Asynchronous PHP calls?