Using Internet Explorer to invoke PHP/CURL for long running data API causes Apache 2 server to freeze and require restart
I'm running a PHP program that works fine as long as it's not invoked by a Microsoft Internet Explorer browser, after which it spawns the below processes, locks up Apache 2 and requires a restart of the web server (on Ubuntu 12.04 LTS).
bob@drools:/etc/php5/apache2# ps auxwww | grep apache2
root 8737 0.1 2.5 369164 25800 ? Ssl 12:41 0:00 /usr/sbin/apache2 -k start
www-data 8743 0.0 3.2 393748 33268 ? Sl 12:41 0:00 /usr/sbin/apache2 -k start
www-data 8755 0.1 3.3 393856 33904 ? Sl 12:41 0:00 /usr/sbin/apache2 -k start
www-data 8779 0.1 3.2 393724 33252 ? Sl 12:45 0:00 /usr/sbin/apache2 -k start
www-data 8782 0.1 3.2 393716 33236 ? Sl 12:45 0:00 /usr/sbin/apache2 -k start
www-data 8785 0.1 3.2 393684 33204 ? Sl 12:45 0:00 /usr/sbin/apache2 -k start
www-data 8812 1.1 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8815 1.3 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8818 1.3 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8821 1.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8824 1.4 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8827 1.4 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8830 1.4 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8835 2.5 3.2 393684 33256 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8838 2.8 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8841 2.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8844 2.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8847 3.2 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8850 3.0 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8853 3.2 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8856 3.2 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8861 3.3 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8864 3.6 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8867 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8870 3.6 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8873 3.6 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8876 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8879 3.3 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8881 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8883 3.6 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8886 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8891 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8894 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8896 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8900 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8901 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8904 3.5 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8909 3.8 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8912 3.8 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8915 3.8 3.2 393684 33264 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
www-data 8918 3.6 3.2 393684 33260 ? Sl 12:47 0:00 /usr/sbin/apache2 -k start
root 8922 0.0 0.1 9396 2000 pts/0 S+ 12:47 0:00 grep --color=auto apache2
It used to lock up the entire server until I changed some of the "mpm_" module parameters to something more reasonable in /etc/spache2/apache2.conf.
Given the issues with Internet Explorer, I've even added this line:
**" SetEnvIf User-Agent ".*MSIE.*" nokeepalive "**
in the virtual hosts file located here: /etc/apache2/sites-available.
There are a number of articles written on the issue, but I've not had any success implementing any of them:
Apache Server 2 hangs after receiving requests from IE 10/11:
More R&D: Internet Explorer 10 (Windows 8) crashing Apache
The PHP program uses cURL to take a list of 25 items and perform a (GET) API call for each to an external server that returns JSON data for further processing. It's a classical long running data program.
What bakes my noodle is that it runs fine in every other browser except Internet Explorer - which causes the web server to misbehave.
I've interrogated the listed R&D and then some, implemented the suggested fixes, yet I still get the same predictable, recreatable, problematic server behavior.
I need to figure out how to protect the server from behaving badly when it encounters and the Internet Explorer browser making these particular requests of it. I'd like to understand why it happens in the first place.
Any guidance, perspectives, direction or solutions would be greatly appreciated...
Here's a snapshot of my cURL code:
<?php
// *** CURL Init, SetOps, and Execution Statements ****
$ch = curl_init();
// *** Execute the API call for each part number and store in the Associative Array ****
$index=0;
foreach ($partNumbersArray as $partNum) {
$MyValue = $partNum;
$MyUrl = $MyNiinjaBaseURL."/".$APICmd1."/".$MyDataSet."/".$MyValue."?key=".$MyKey."&$"."filter=substringof('".$MyValue."',PartNumbers)";
// *** cURL SetOpts, and Execution Statements ****
curl_setopt($ch, CURLOPT_URL, $MyUrl);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
// curl_setopt($ch, CURLOPT_TIMEOUT, 15); // <= THIS *never* worked with any reliability ....
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$server_output = curl_exec ($ch); // <= THIS executes the cURL call and stores the resulting JSON object in the variable '$server_output'
$niinjaResultsJsonArray[$MyValue] = $server_output; // Add the JSON object to the Array and index to PartNumber
$index++; // Increment the index
} // End Execution of NIINJA API Calls
// ** Close the CURL Object and release resources
curl_close ($ch);
?>
Here is the PHP information page: http://www.versaggi.net/phptest.phtml
A long time ago, I saw Apache lock-ups resulting from an Apache process making a call via HTTP to another URL serviced by an Apache process on the same server. I sometimes wound up with a bunch of processes waiting on such calls with no available Apache processes to service them. In my case I had a translation layer in front of some web pages, but calling an API on your own site is much the same thing.
The characteristics of the browser making the original call could make this more likely to occur. For example, keep-alive, timeout behaviour and so forth, but it's not fundamentally the browser at fault.
If it's anything like what I saw, then you want to be looking at the timeout behaviour in your use of curl. The code you've included suggests you're onto that, but you might need to be more fine-grained in your understanding of exactly what point in the request it's getting to. It might be interesting to look at it with tcpdump (or ngrep, Wireshark, or whatever). Also it'd be good to know what system call is in progress when the calling process hangs. That is, look at it with strace -p [PID].
You should probably also be thinking about whether you can remove the HTTP call from your use of the API. Can you keep things within the same Apache process by making a direct call to the appropriate code that handles the API request?
It's probably relevant to tell people how you're running PHP (E.g. mod_php, fpm, etc.). That may be part of understanding the mechanism by which the code locks up.