Gitlab Container Registry timeout when connecting
Building an on-prem / self-hosted deployment of Gitlab for the first time. I've got the self-installation completed, single box and three dedicated hosts for runners (that I'm working to get them dockerized for docker-in-docker).
Did this from a package install two days ago, so nothing built from source, 99% of the things are still from out-of-the-box omnibus installer.
I'm trying to get the Container Registry up and running so that I can check-in containers and pull them for my CI/CD pipeline, but every time my local host tries to authenticate, it times out. When I enable the debug logs, it doesn't seem to generate anything?
PS C:\Users\my_user_name\docker-gl-runner> docker login gitlab.my_domain.tld:5050 -u my_user_name
Password:
Error response from daemon: Get https://gitlab.my_domain.tld:5050/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I do have a user account setup, I did generate a personal access token, and have tried both (username + password, username + token).
The service is listening on 5050:
[root@gitlab gitlab]# netstat -an | grep 50..
tcp 0 0 0.0.0.0:5050 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5001 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5001 127.0.0.1:46666 ESTABLISHED
tcp 0 0 127.0.0.1:5001 127.0.0.1:46560 TIME_WAIT
tcp 0 0 127.0.0.1:46666 127.0.0.1:5001 ESTABLISHED
And here are the relevant settings (I believe) in the gitlab.rb file:
[root@gitlab gitlab]# cat gitlab.rb | grep registry
# gitlab_rails['gitlab_default_projects_features_container_registry'] = true
registry_external_url 'https://gitlab.my_domain.tld'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.my_domain.tld"
gitlab_rails['registry_port'] = "5005"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
# gitlab_rails['registry_notification_secret'] = nil
# gitlab_rails['registry_api_url'] = "http://localhost:5000"
# gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key"
# gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer"
registry['enable'] = true
# registry['username'] = "registry"
# registry['group'] = "registry"
# registry['uid'] = nil
# registry['gid'] = nil
registry['dir'] = "/var/opt/gitlab/registry"
registry['registry_http_addr'] = "localhost:5000"
registry['debug_addr'] = "localhost:5001"
registry['log_directory'] = "/var/log/gitlab/registry"
registry['env_directory'] = "/opt/gitlab/etc/registry/env"
registry['log_level'] = "info"
registry['log_formatter'] = "text"
# registry['rootcertbundle'] = "/var/opt/gitlab/registry/certificate.crt"
# registry['health_storagedriver_enabled'] = true
# registry['storage_delete_enabled'] = true
# registry['validation_enabled'] = false
# registry['autoredirect'] = false
# registry['compatibility_schema1_enabled'] = false
Any thoughts on what's different, messed up, or should be looked into? Or maybe even a, "Here's what mine looks like, compare it to yours" (I'll show you mine if you show me yours?)
Okay, so I figured out the problems.
First of all, Centos8, firewalld, blocking the connection. Had to allow the docker0 zone to be trusted. Forgot that one.
Second, tcpdump showed that the connection was not being ACK'd. Turns out, the docker service wasn't running. You can config gitlab all you want, but you need Docker to be running. Who knew?
Third, after fixing both of those problems, I was hit with the self-signed certificate problem. If you're curious for the simple way to solve it:
openssl s_client -showcerts -connect gitlab.my_domain.tld:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >/etc/docker/certs.d/gitlab.my_domain.tld:5050/ca.crt
That'll pull down the cert from the remote host and trust it for Docker.