I have installed GitLab on a cloud VM and shortly after stood up a second VM as a postfix server with the same provider (Hetzner)

I first noticed a problem when I went back to the GitLab server to enable SMTP Email and nothing was being sent, not even any logs on the mail server to say a connection was attempted.

On the GitLab server

I tried to ping the mail server from gitlab and got back only a single response and no further responses not matter how long I wait.

root@gitlab:~# ping mail.simoncarr.co.uk
PING mail.simoncarr.co.uk(2a01:4f8:c2c:a992:: (2a01:4f8:c2c:a992::)) 56 data bytes

As you can see it is an IPV6 response. I don't remember doing anything other than enabling the ufw that would have impacted networking on gitlab.

I have turned off ufw on gitlab and the mail server, and get the same behaviour.

If I ping an external server I still get an IPV6 response, but I do at least get multiple responses.

root@gitlab:~# ping bbc.co.uk
PING bbc.co.uk(2a04:4e42:600::81 (2a04:4e42:600::81)) 56 data bytes
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=1 ttl=58 time=3.68 ms
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=2 ttl=58 time=3.47 ms
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=3 ttl=58 time=3.52 ms
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=4 ttl=58 time=3.50 ms
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=5 ttl=58 time=3.49 ms

On the mail server

If I ping the gitlab server, I get IPV4 response and they are repeated as you would expect.

PING gitlab.simoncarr.co.uk (168.119.124.76) 56(84) bytes of data.
64 bytes from static.76.124.119.168.clients.your-server.de (168.119.124.76): icmp_seq=1 ttl=58 time=0.878 ms
64 bytes from static.76.124.119.168.clients.your-server.de (168.119.124.76): icmp_seq=2 ttl=58 time=0.463 ms
64 bytes from static.76.124.119.168.clients.your-server.de (168.119.124.76): icmp_seq=3 ttl=58 time=0.353 ms
64 bytes from static.76.124.119.168.clients.your-server.de (168.119.124.76): icmp_seq=4 ttl=58 time=0.419 ms

If I ping an external server from the mail server, I again get an IPV6 response.

root@mail:~# ping bbc.co.uk
PING bbc.co.uk(2a04:4e42:600::81 (2a04:4e42:600::81)) 56 data bytes
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=1 ttl=58 time=6.68 ms
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=2 ttl=58 time=6.24 ms
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=3 ttl=58 time=6.23 ms
64 bytes from 2a04:4e42:600::81 (2a04:4e42:600::81): icmp_seq=4 ttl=58 time=6.16 ms

The only thing, that I have changed on the mail server, that is related to networking is to add reverse DNS addresses to both IPV4 and IPV6 addresses.

Getting back to my actual problem.

I want to be able to connect from GitLab server to my mail server to send SMTP email. Even with ufw disabled on both servers, GitLab is not communicating with the mail server. I can't help but think the issue is related to the behaviour I have described above.

HTTP web traffic to my GitLab server is working fine and my mail server is sending and receiving email fine, from mail clients and other SMTP servers.

Solution 1:

On suitable ways to debug your problem

When testing a service, it's always better to try and connect to the service instead of using ping. While ICMP echo might help in diagnosing some problems, it doesn't really tell much here, because there are too many reasons why it might respond or not, and it has less to do with the actual problem.

In your DNS, you have:

;; ANSWER SECTION:
simoncarr.co.uk.        14400   IN      MX      11 mail.simoncarr.co.uk.

;; ADDITIONAL SECTION:
mail.simoncarr.co.uk.   14400   IN      AAAA    2a01:4f8:c2c:a992::
mail.simoncarr.co.uk.   14400   IN      A       188.34.201.61

This means your mail server is advertised to have both IPv6 and IPv4 connectivity, but neither answers on the SMTP port 25:

$ nc 188.34.201.61 25 -vvv
nc: connect to 188.34.201.61 port 25 (tcp) failed: Connection timed out

$ nc 2a01:4f8:c2c:a992:: 25 -vvv
nc: connect to 2a01:4f8:c2c:a992:: port 25 (tcp) failed: Connection timed out

However, the IPv4 address answers on ports 465 (smtps, RFC 8314) and 587 (submission, RFC 6409), but the IPv6 address doesn't:

$ nc 188.34.201.61 465 -vvv
Connection to 188.34.201.61 465 port [tcp/submissions] succeeded!

$ nc 188.34.201.61 587 -vvv
Connection to 188.34.201.61 587 port [tcp/submission] succeeded!
220 mail.simoncarr.co.uk ESMTP Postfix (Ubuntu)

$ nc 2a01:4f8:c2c:a992:: 465 -vvv
nc: connect to 2a01:4f8:c2c:a992:: port 465 (tcp) failed: Connection timed out

$ nc 2a01:4f8:c2c:a992:: 587 -vvv
nc: connect to 2a01:4f8:c2c:a992:: port 587 (tcp) failed: Connection timed out

Knowing you have disabled the firewall, this suggests you either have wrong IPv6 address in your DNS or that the mail server doesn't have IPv6 connectivity altogether. Normally you could use ifconfig on the mail server to figure this out, but I got lucky as my first guess got right:

$ nc 2a01:4f8:c2c:a992::1 587 -vvv
Connection to 2a01:4f8:c2c:a992::1 587 port [tcp/submission] succeeded!
220 mail.simoncarr.co.uk ESMTP Postfix (Ubuntu)

How to fix your problems

1. Change the AAAA record in your DNS. It should probably be (notice the additional 1):

   mail.simoncarr.co.uk.   14400   IN      AAAA    2a01:4f8:c2c:a992::1
   

2. Fix your mail server. For incoming mail you need to have the SMTP server listening on TCP port 25. If you can connect to this port from the server itself, you might need to ask the service provider whether hosting mail servers are allowed or not. It's typical to block port 25 on connections not used for delivering mail as a way to fight SPAM from compromised computers.