[centos@staging03 ~]$ sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 127.0.0.1:80                0.0.0.0:*                   LISTEN      3600/httpd          
tcp        0      0 127.0.0.2:80                0.0.0.0:*                   LISTEN      1574/varnishd       
tcp        0      0 172.31.22.60:80             0.0.0.0:*                   LISTEN      1539/nginx          
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1251/sshd           
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1501/master         
tcp        0      0 127.0.0.1:443               0.0.0.0:*                   LISTEN      3600/httpd          
tcp        0      0 127.0.0.1:6082              0.0.0.0:*                   LISTEN      1573/varnishd       
tcp        0      0 127.0.0.1:9000              0.0.0.0:*                   LISTEN      3468/php-fpm        
tcp        0      0 127.0.0.1:11211             0.0.0.0:*                   LISTEN      1229/memcached      
tcp        0      0 127.0.0.1:6379              0.0.0.0:*                   LISTEN      1061/redis-server 1 
tcp        0      0 :::22                       :::*                        LISTEN      1251/sshd           
tcp        0      0 :::3306                     :::*                        LISTEN      1383/mysqld 

I checked to investigate what's the issue with my server, and when I did:

curl 127.0.0.1:80

I got:

Authorization Required

This server could not verify that you are authorized to access the document requested. Either you supplied the wrong credentials (e.g., bad password), or your browser doesn't understand how to supply the credentials required.

On a different server where everything is working, I get a blank response. So I am thinking this is why I am getting a 500 varnish error from Apache.

In the Apache log, I didn't really get anything when I curled, but before that I got:

[Wed Oct 27 17:02:25 2021] [notice] caught SIGTERM, shutting down
[Wed Oct 27 17:02:25 2021] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Oct 27 17:02:25 2021] [notice] Digest: generating secret for digest authentication ...
[Wed Oct 27 17:02:25 2021] [notice] Digest: done
[Wed Oct 27 17:02:25 2021] [notice] FastCGI: process manager initialized (pid 3602)
[Wed Oct 27 17:02:25 2021] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fastcgi/2.4.6 configured -- resuming normal operations

So it seems FastCGI is properly configured and the issue I am getting from Apache is an authentication issue strangely enough. Is there anything else I can do to pin point what the problem is?

Varnish gives the following:

   12 TxHeader     b X-Varnish: 1537309960
   12 RxProtocol   b HTTP/1.1
   12 RxStatus     b 500
   12 RxResponse   b Internal Server Error
   12 RxHeader     b Date: Wed, 27 Oct 2021 21:14:18 GMT
   12 RxHeader     b Server: Apache/2.2.15 (CentOS)
   12 RxHeader     b Expires: Wed, 11 Jan 1984 05:00:00 GMT
   12 RxHeader     b Cache-Control: no-cache, must-revalidate, max-age=0

However, I have no way of checking what the 500 Internal Server Error is, because the error logs for php seems to be empty.

Solution 1:

TODO Apache

1. Increase the log level in Apache
2. Test the difference between an HTTP call to a static file in Apache and a call to PHP
3. Monitor Apache's error log with the increased verbosity

The goal is to get an HTTP 200 out of Apache by running curl http://127.0.0.1, either on the homepage or some static file.

TODO Varnish

1. Upgrade Varnish to a supported & maintained version
2. Add a backend probe in your VCL
3. Monitor the backend health via VSL

Based on the VSL output you shared I can see that you're running an ancient version of Varnish. See https://www.varnish-software.com/developers/tutorials/installing-varnish-centos/ to learn how you can instal Varnish 6.0 LTS on CentOS.

Not only do you have a version of Varnish that is secure, your VSL tools (such varnishlog) are also far superior than in the version you're running.

Here's an example of a backend that has a health probe:

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .probe = {
        .url = "/";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

This will allow you to continuously monitor the health of your backend. You can do this with the following command:

sudo varnishlog -g raw -i Backend_health

The output will help you understand how the backend is behaving and which HTTP status code it returns to Varnish.

Combine this with your quest to get an HTTP 200 status code from Apache and you'll get pretty close to a final solution.