apache Proxy 4 seconds connection delay

apache-2.2 centos reverse-proxy

I have a big data site (wordpress secondry develop) one domain www.example.com. I have stored it into 3 server.

In DNS setteing 
IP address (IN A / AAAA) 46.192.22.01
IP address (IN A / AAAA) 46.192.22.02
IP address (IN A / AAAA) 46.192.22.03
  • server1 46.192.22.01 /var/www/html -> /music & /video
  • server2 46.192.22.02 /var/www/html -> /article & /photo
  • server3 46.192.22.03 /var/www/html -> /products & /showroom

httpd.conf file:

Timeout 45
KeepAlive Off
MaxKeepAliveRequests 256
KeepAliveTimeout 3
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      192
MaxClients       192
MaxRequestsPerChild  2500
</IfModule>

LoadModule auth_basic_module modules/mod_auth_basic.so
#LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
#LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
#LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
#LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
#LoadModule ldap_module modules/mod_ldap.so
#LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
#LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
#LoadModule ext_filter_module modules/mod_ext_filter.so
#LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
#LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
#LoadModule dav_module modules/mod_dav.so
#LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
#LoadModule info_module modules/mod_info.so
#LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
#LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
#LoadModule substitute_module modules/mod_substitute.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
#LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule cgi_module modules/mod_cgi.so
#LoadModule version_module modules/mod_version.so

ExpiresActive On
<FilesMatch "\.(jpg|jpeg)$">
 ExpiresDefault "access plus 1 year"
</FilesMatch>
<FilesMatch "\.(css|png|js|gif)$">
 ExpiresDefault "access plus 1 month"
</FilesMatch>

<ifmodule mod_deflate.c>
DeflateCompressionLevel 6
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/atom_xml
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/x-httpd-php
</ifmodule>

Server 1 

ServerName www.example.com:80
<VirtualHost *:80>
    ServerName www.example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/html
    <directory "/var/www/html">
        Options Indexes FollowSymLinks
        AllowOverride all
        Order Deny,Allow
        Deny from all
        Allow from all
    </directory>
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
    ProxyRequests Off
    ProxyPass /article http://46.192.22.02/article retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /article http://46.192.22.02/article
    ProxyPass /photo http://46.192.22.02/photo retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /photo http://46.192.22.02/photo
    ProxyPass /products http://46.192.22.03/products retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /products http://46.192.22.03/products
    ProxyPass /showroom http://46.192.22.03/showroom retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /showroom http://46.192.22.03/showroom
</VirtualHost>

server2 

ServerName www.example.com:80
<VirtualHost *:80>
    ServerName www.example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/html
    <directory "/var/www/html">
        Options Indexes FollowSymLinks
        AllowOverride all
        Order Deny,Allow
        Deny from all
        Allow from all
    </directory>
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
    ProxyRequests Off
    ProxyPass /music http://46.192.22.01/music retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /music http://46.192.22.01/music
    ProxyPass /video http://46.192.22.01/video retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /video http://46.192.22.01/video
    ProxyPass /products http://46.192.22.03/products retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /products http://46.192.22.03/products
    ProxyPass /showroom http://46.192.22.03/showroom retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /showroom http://46.192.22.03/showroom
</VirtualHost>

server3 

ServerName www.example.com:80
<VirtualHost *:80>
    ServerName www.example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/html
    <directory "/var/www/html">
        Options Indexes FollowSymLinks
        AllowOverride all
        Order Deny,Allow
        Deny from all
        Allow from all
    </directory>
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
    ProxyRequests Off
    ProxyPass /music http://46.192.22.01/music retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /music http://46.192.22.01/music
    ProxyPass /video http://46.192.22.01/video retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /video http://46.192.22.01/video
    ProxyPass /products http://46.192.22.03/products retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /products http://46.192.22.03/products
    ProxyPass /showroom http://46.192.22.03/showroom retry=1 acquire=3000 timeout=600 Keepalive=On
    ProxyPassReverse /showroom http://46.192.22.03/showroom
</VirtualHost>

all the /etc/hosts are

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Then .htaccess file

server 1

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

RewriteRule ^music/track/(\d+)?$ music/track.php?number=$1
RewriteRule ^video/introduce/(\d+)?$ video/introduce.php?number=$1

RewriteCond %{HTTP_HOST} ^example\.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

server 2

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

RewriteRule ^article/introduce/(\d+)?$ article/introduce.php?number=$1
RewriteRule ^photo/products/(\d+)?$ photo/products.php?number=$1

RewriteCond %{HTTP_HOST} ^example\.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

server 3

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

RewriteRule ^products/smartphone/(\d+)?$ products/smartphone.php?number=$1
RewriteRule ^showroom/smartphone/(\d+)?$ showroom/smartphone.php?number=$1

RewriteCond %{HTTP_HOST} ^example\.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

But when I open http://www.example.com/products/smartphone/102034 is very slow.(3-5 seconds delay), but http://46.192.22.03/products/smartphone/102034 is very fast (about 0.2 seconds)

tail -50 /var/log/messages return logs like 

Nov 18 11:46:20 dhclient[857]: DHCPACK from 46.192.22.01 (xid=0x6caa5cd9)
Nov 18 11:46:21 dhclient[857]: bound to 46.192.22.02 -- renewal in 1481 seconds.

I am not good at server part tech. Can any professor help me to find out and fix where is the problem cause DNS very slow? Thanks.

UPDATE 1 (2014/11/23)

During 2 days googling, still not find an anwser. every page loding need about 4 seconds connection which was view in chrome console. If typing ip instead domainn, only 0.2-0.4 seconds connection. I had disable my ipv6 in all 3 server, I had add retry=1 acquire=3000 timeout=600 Keepalive=On after ProxyPass...

#dig example.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63728
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.               IN      A

;; ANSWER SECTION:
example.com.        7199    IN      A       46.192.22.01
example.com.        7199    IN      A       46.192.22.02
example.com.        7199    IN      A       46.192.22.03

;; Query time: 121 msec
;; SERVER: 46.61.167.22#53(46.61.167.22)
;; WHEN: Sun Nov 23 20:51:03 2014
;; MSG SIZE  rcvd: 81

I doubt is the reverse proxy really what I need? Hane I found a wrong method to connection my server?

Indeed, I have 1 site with large data, now I am seprete it into 3 centos6.6 64bit server (with 3 ips). each server have unique folder

  • server1 46.192.22.01 /var/www/html -> /music & /video
  • server2 46.192.22.02 /var/www/html -> /article & /photo
  • server3 46.192.22.03 /var/www/html -> /products & /showroom

so it is not a cluster balance or proxy balance, but how to setting (or install other software) to let internet know if some one type "http://www.example.com/music" it will map into server 1; some one type "http://www.example.com/products" it will map into server 3... sorry my mother language is not english, so i have never find a right way, till today, only reverse proxy could meet my requires.

Sorry, I only have some php,mysql skill, little apache centos skill, Need a detailed solution. Many thanks.

UPDATE 2 (2014/11/24)

Another day working. it seems not DNS lookup problem. here is some screenshort when use domain chrome cosole with domain

then the same page with ip

chrome cosole with ip

To avoid browser cache effection. I have cleaned all the browser cache, then tried first loading ip then load domain. the biggest diference are in "stalled" and "initial connection".

I use top command moniter the memory and cpu usage, the 3 servers look normal when the speed is slowly.

s1 

top - 18:21:57 up 21:44,  1 user,  load average: 1.08, 1.28, 1.10
Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.3%us,  2.4%sy,  0.0%ni, 77.8%id, 12.2%wa,  0.0%hi,  0.0%si,  0.2%st
Mem:   1922200k total,  1847432k used,    74768k free,     2224k buffers
Swap:  1048572k total,    81224k used,   967348k free,    84816k cached

s2

top - 18:31:24 up 21:55,  1 user,  load average: 0.15, 0.09, 0.02
Tasks: 119 total,   1 running, 118 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.0%us,  1.0%sy,  0.0%ni, 95.8%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   1922200k total,  1669328k used,   252872k free,    25652k buffers
Swap:  1048572k total,    69256k used,   979316k free,   458528k cached

s3

top - 18:21:21 up 1 day,  5:57,  1 user,  load average: 1.03, 0.88, 0.90
Tasks: 105 total,   2 running, 103 sleeping,   0 stopped,   0 zombie
Cpu(s): 25.7%us, 20.0%sy,  0.0%ni, 53.9%id,  0.0%wa,  0.0%hi,  0.2%si,  0.2%st
Mem:   1922200k total,  1758088k used,   164112k free,    58712k buffers
Swap:  1048572k total,      128k used,  1048444k free,   226396k cached

please look the load average. 2 server are over value 1.

Indeed In DNS, I set server 1 at first A address position, but why server 2 become the main server?


Solution 1:

It still seems that the issue is DNS related, if you're having different results when connecting to the site via hostname and via IP.

Try adding all three machines to /etc/hosts on all of them, i.e.

46.192.22.01 server1
46.192.22.02 server2
46.192.22.03 server3

As a side comment, your DHCP lease time seems to be very short with about 1500 seconds. Typical DHCP leases last a day or longer. If the IPs are static, which seems to be the case, you might also consider removing them from DHCP pool and making IP configuration static on those three machines.

Solution 2:

It could be the problem is with the Apache backends having a problem resolving the proxy server. If this is the case you can fix it by either hardcoding the details for the proxy server in each of the /etc/hosts file, ie by adding a line like

proxy.ip.addr www.example.com example.com

Or by turning off reverse DNS lookups in Apache using the directive HostnameLookups Off

This does, of-course assume the problem to be DNS related and not related to the path traffic is taking across the network (although a DNS fault is a lot likely to be the issue).

You should also check your /etc/resolv.conf file on each system to make sure that they are all accessing a valid nameserver which is responding quickly - you can check this by looking at this file to ensure the nameserver is set to one "close" to you, then pinging a few places by domain name and seeing how long they take to respond. (If you implement the steps above, this bit becomes irrelevant, but its still worth trying to resolve)

Solution 3:

By the looks of your configuration nothing should be wrong. Perhaps the speed between the 3 servers is somehow limited. I would suggest trying to add the following parameters to ProxyPass as a test.

flushpackets=On keepalive=On

for example:

ProxyPass /products http://46.192.22.03/products flushpackets=On keepalive=On

Also make sure the receiving server is not trying to do a reverse dns lookup which has a timeout which is being reached. The ips you listed (Not sure if they are the real ip's or fake ip's) do not have a reverse dns name.

Related

Is there a single point-of-reference that lists the release dates for the Live Communications Server (and other related client/server applications) [closed]
Can SCCM or WSUS actively initiate a new TCP connection to the client?
No sound on Ubuntu 20.04 [duplicate]
Ubuntu 18.04: How to create a persistent dummy network interface
Reinstall software-center in Ubuntu 16.04
Ubuntu 12.04 touch-pad and keyboard problems asus q500a
pip upgrade not working
How can I load the `vsftpd` daemon at system-boot, but have it "start" in a "stopped" state?
How to remove duplicate imagemagick icons
PPTP VPN missing in "connection type dialogue" in Ubuntu 16.04
Ubuntu install can't find hard drives
How do i find all files in a folder recursively by the specific name and rename them?