HTTPS is over 50 times slower then HTTP

https apache-2.2 ssl ubuntu

I have a website that uses https to transmit a javascript file to the client. The website is getsimpleapps.com.

It turns out that this file is loading 52 times slower with https (20.08s - 29.08s) that with http (380ms).

The homepage of the site shares the same slowness as the javacript file.

  • http://getsimpleapps.com
  • https://getsimpleapps.com

I've recently switched over from dreamhost to linode, and hacked at getting SSL to work on the new server until it did. I didn't do any crazy configuring.

The linode is running Ubuntu 12.04 and the site is on top of a (LAMP) stack.

My question to the stack overflow community is: How do I go about fixing SSL & HTTPS on my server? I know that stack overflow is littered with questions regarding the slowness of HTTPS but no real solutions are given. A ubuntu tutorial or configuration guide would be ideal.

file : /etc/apache2/sites-enabled/getsimpleapps.com

<VirtualHost *:80>
     ServerAdmin [email protected]
     ServerName getsimpleapps.com
     ServerAlias www.getsimpleapps.com
     DocumentRoot /srv/sites/getsimpleapps.com/public/
     ErrorLog /srv/sites/getsimpleapps.com/logs/error.log
     CustomLog /srv/sites/getsimpleapps.com/logs/access.log combined
</VirtualHost>

<VirtualHost 50.116.58.18:443>
     SSLEngine On
     #SSLCertificateFile /etc/apache2/ssl/www.getsimpleapps.com.crt
     #SSLCertificateKeyFile /etc/apache2/ssl/www.getsimpleapps.com.key
     #SSLCACertificateFile /etc/apache2/ssl/comodo.crt
     SSLCertificateFile /etc/apache2/ssl/dreamhost/dh.crt
     SSLCertificateKeyFile /etc/apache2/ssl/dreamhost/dh.key
     SSLCACertificateFile /etc/apache2/ssl/dreamhost/dh.cer

     ServerAdmin [email protected]
     ServerName getsimpleapps.com
     ServerAlias www.getsimpleapps.com
     DocumentRoot /srv/sites/getsimpleapps.com/public/
     ErrorLog /srv/sites/getsimpleapps.com/logs/error.log
     CustomLog /srv/sites/getsimpleapps.com/logs/access.log combined
</VirtualHost>

Curl from local workstation

thomas@workstation:~$ time curl -Iv https://getsimpleapps.com/
* About to connect() to getsimpleapps.com port 443 (#0)
*   Trying 50.116.58.18... connected
* Connected to getsimpleapps.com (50.116.58.18) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*    subject: OU=Domain Control Validated; OU=Provided by New Dream Network, LLC; OU=DreamHost Basic SSL; CN=getsimpleapps.com
*    start date: 2012-02-23 00:00:00 GMT
*    expire date: 2013-02-22 23:59:59 GMT
*    subjectAltName: getsimpleapps.com matched
*    issuer: C=GB; ST=Greater Manchester; L=Salford; O=Comodo CA Limited; CN=PositiveSSL CA
*    SSL certificate verify ok.
> HEAD / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: getsimpleapps.com
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 02 Aug 2012 20:31:39 GMT
Date: Thu, 02 Aug 2012 20:31:39 GMT
< Server: Apache/2.2.22 (Ubuntu)
Server: Apache/2.2.22 (Ubuntu)
< X-Powered-By: PHP/5.3.10-1ubuntu3.2
X-Powered-By: PHP/5.3.10-1ubuntu3.2
< Set-Cookie: ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2298c7e45da25e4aaf80f7a1e36ed4a006%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A13%3A%2250.75.209.154%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A81%3A%22curl%2F7.21.4+%28universal-apple-darwin11.0%29+libcurl%2F7.21.4+OpenSSL%2F0.9.8r+zlib%2F1.2.5%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1343939499%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D80bf8ae5040fc47780ccd59f1fb8b267; expires=Thu, 02-Aug-2012 22:31:39 GMT; path=/
Set-Cookie: ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2298c7e45da25e4aaf80f7a1e36ed4a006%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A13%3A%2250.75.209.154%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A81%3A%22curl%2F7.21.4+%28universal-apple-darwin11.0%29+libcurl%2F7.21.4+OpenSSL%2F0.9.8r+zlib%2F1.2.5%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1343939499%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D80bf8ae5040fc47780ccd59f1fb8b267; expires=Thu, 02-Aug-2012 22:31:39 GMT; path=/
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Content-Type: text/html
Content-Type: text/html

< 
* Connection #0 to host getsimpleapps.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

real    0m29.078s
user    0m0.018s
sys 0m0.005s

Curl from linode server (via ssh)

thomas@vannevar:~$ time curl -Iv https://getsimpleapps.com/happy-ending/api/script.js?shop=holstee.myshopify.com
* About to connect() to getsimpleapps.com port 443 (#0)
*   Trying 50.116.58.18... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*    subject: OU=Domain Control Validated; OU=Provided by New Dream Network, LLC; OU=DreamHost Basic SSL; CN=getsimpleapps.com
*    start date: 2012-02-23 00:00:00 GMT
*    expire date: 2013-02-22 23:59:59 GMT
*    subjectAltName: getsimpleapps.com matched
*    issuer: C=GB; ST=Greater Manchester; L=Salford; O=Comodo CA Limited; CN=PositiveSSL CA
*    SSL certificate verify ok.
> HEAD /happy-ending/api/script.js?shop=holstee.myshopify.com HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: getsimpleapps.com
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 02 Aug 2012 20:43:30 GMT
Date: Thu, 02 Aug 2012 20:43:30 GMT
< Server: Apache/2.2.22 (Ubuntu)
Server: Apache/2.2.22 (Ubuntu)
< X-Powered-By: PHP/5.3.10-1ubuntu3.2
X-Powered-By: PHP/5.3.10-1ubuntu3.2
< Set-Cookie: ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2204a54136cab08f9fdc5f082ebb8e739a%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%2250.116.58.18%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A97%3A%22curl%2F7.22.0+%28i686-pc-linux-gnu%29+libcurl%2F7.22.0+OpenSSL%2F1.0.1+zlib%2F1.2.3.4+libidn%2F1.23+librtmp%2F2.3%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1343940210%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7De7d7b8e2ca69b34c531ba7472b4b21b7; expires=Thu, 02-Aug-2012 22:43:30 GMT; path=/
Set-Cookie: ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2204a54136cab08f9fdc5f082ebb8e739a%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%2250.116.58.18%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A97%3A%22curl%2F7.22.0+%28i686-pc-linux-gnu%29+libcurl%2F7.22.0+OpenSSL%2F1.0.1+zlib%2F1.2.3.4+libidn%2F1.23+librtmp%2F2.3%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1343940210%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7De7d7b8e2ca69b34c531ba7472b4b21b7; expires=Thu, 02-Aug-2012 22:43:30 GMT; path=/
< Content-Type: text/javascript
Content-Type: text/javascript
* no chunk, no close, no size. Assume close to signal end

< 
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

real    0m25.991s
user    0m0.015s
sys 0m0.022s

I had the same issue, with nearly identical response time differences between HTTP and HTTPS. Turns out the issue was as in the answer by @htmltiger: Apache2 was simply running out of worker processes.

This causes new requests to be queued until a worker becomes free and can process the next one [source]. I suppose the reason why this only affects HTTPS and not also HTTPS is that nearly all your traffic is over HTTP and Apache gives HTTP and HTTPS requests the same priority, taking one request from each queue in turn. So when the HTTPS queue is much longer, requests wait much longer. Indeed there are two queues, as the queue is simply the Linux TCP connection queue mechanism, and Linux provides one queue per port.

Diagnostics

If this is your problem, the following symptoms will apply:

  • The best indicator: on your server, apachectl status shows that all allowable worker processes are running. This is the case when no dots . are shwon in the process scoreboard line, indicating no "Open slot with no current process" left. The line might look like this for example:

    KKKKKKRKKKRRCWKKKCCKWKKKKCRCKKKKKKKCKCKKKKWRKKKKWRWKKKKKKCWKKWKKK

  • You see messages like this in your main Apache2 error log (/var/log/apache2/error.log, not domain specific ones):

    [mpm_prefork:error] [pid 4715] AH00161: server reached MaxRequestWorkers 
    setting, consider raising the MaxRequestWorkers setting

  • There are many processes in your Apache backlog. According to this in-depth article, you can see this from the unacked: value in ss -lti '( sport = :https )' output. Depending on the version or configuration of ss, that value might be missing though.

  • Most of the delay (say, 17 of 20 s) is shown in the Firefox Network Console, in the "Timings" tab for the initial URL requested, as "Blocking".

Solution

This assumes you use the prefork MPM server module in Apache. It's similar for the "event" and "worker" MPM modules though – details.

  1. Edit /etc/apache2/mods-enabled/mpm_prefork.conf and increase the MaxRequestWorkers setting.

  2. If you increase it beyond the default of 256, you also have to set ServerLimit to the same value to make your change effective.

  3. Apply the changes: service apache2 reload

  4. Make sure in the scoreboard output of apachectl status that the new MaxRequestWorkers setting is effective. It has to be equivalent to the length of the scoreboard line in characters.

  5. If the setting is not effective yet, search in /etc/apache2 for old configuration directives (and their even older deprecated synonyms) that could overwrite your change:

    grep -R MaxRequestWorkers /etc/apache2/*
grep -R MaxClients /etc/apache2/*

Differential Diagnoses

In case you see HTTPS being much slower than HTTP but not every single time in a series of page reloads (just on average), then you might have a variant of this fancy problem, with two Apache2 servers running on SSL port 443.

Related

Celery Daemon receives unregistered tasks
Is it possible to get a list of running processes with a Cloudwatch Alarm?
Still getting aacraid: Host adapter abort request errors after following recommended steps
LXC, Port forwarding and iptables
Exim 4.71 does not add DKIM signature to outgoing emails
Lower TCP throughput from 1Gbps server than 100Mbps server over large RTT
strange ZFS disk space usage report for a ZVOL
zabbix agents with the same IP moving between proxies
Mysql Slave stuck in "System lock"
Win7 clients failing with cached credentials on samba4 RODC
strongSwan setup where both sides are behind NAT
Can I make systemd show service status after starting or stopping a service?