Ubuntu 18.04 - Temporary failure resolving 'us-east-1.ec2.archive.ubuntu.com'

amazon-web-services linux-networking ubuntu-18.04

We are running an Ubuntu 18.04 on an EC2 and have trouble installing updates.

This is what we get on doing a apt update

root@host-02:~# apt update
Err:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
  Temporary failure resolving 'us-east-1.ec2.archive.ubuntu.com'
Err:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
  Temporary failure resolving 'us-east-1.ec2.archive.ubuntu.com'
Err:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
  Temporary failure resolving 'us-east-1.ec2.archive.ubuntu.com'
Reading package lists... Done                      
Building dependency tree       
Reading state information... Done
22 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Temporary failure resolving 'us-east-1.ec2.archive.ubuntu.com'
W: Failed to fetch http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Temporary failure resolving 'us-east-1.ec2.archive.ubuntu.com'
W: Failed to fetch http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Temporary failure resolving 'us-east-1.ec2.archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

Our resolv.conf file looks good and is able to resolve sites,

root@host-02:~# cat /etc/resolv.conf
options timeout:2 attempts:5
; generated by /usr/sbin/dhclient-script
search domain.local ec2.internal
nameserver 10.12.x.x
nameserver 10.13.x.x
nameserver 10.1.0.2

root@host-02:~# curl -Iv cnn.com
* Rebuilt URL to: cnn.com/
*   Trying 151.101.193.67...
* TCP_NODELAY set
* Connected to cnn.com (151.101.193.67) port 80 (#0)
> HEAD / HTTP/1.1
> Host: cnn.com
> User-Agent: curl/7.58.0
> Accept: */*

For good measure, the curl to the affected URL also works

root@host-02:~# curl -I http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/bionic/InRelease
HTTP/1.1 200 OK
Date: Thu, 20 May 2021 02:17:06 GMT
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Thu, 26 Apr 2018 23:38:40 GMT
ETag: "3b180-56ac8e31ec000"
Accept-Ranges: bytes
Content-Length: 242048
Cache-Control: max-age=0, proxy-revalidate
Expires: Thu, 20 May 2021 02:17:06 GMT

I'm kind of stumped now. We are able to resolve outside apt, download using wget as shown below but apt refuse to resolve the URLs and download packages. Any ideas what is happening here?

root@host-02:~# wget https://downloads.tableau.com/esdalt/2020.4.0/tableau-tabcmd-2020-4-0.noarch.rpm
--2021-05-20 02:18:49--  https://downloads.tableau.com/esdalt/2020.4.0/tableau-tabcmd-2020-4-0.noarch.rpm
Resolving downloads.tableau.com (downloads.tableau.com)... 23.208.44.148
Connecting to downloads.tableau.com (downloads.tableau.com)|23.208.44.148|:443... connected.

Here are contents for /etc/apt/sources.list

root@jump-02:~$ cat /etc/apt/sources.list
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
## if you wish to make changes you can:
## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
##     or do the same in user-data
## b.) add sources in /etc/apt/sources.list.d
## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic main restricted
# deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
# deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic universe
# deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic universe
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-updates universe
# deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic multiverse
# deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic multiverse
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-updates multiverse
# deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu bionic partner
# deb-src http://archive.canonical.com/ubuntu bionic partner

deb http://security.ubuntu.com/ubuntu bionic-security main restricted
# deb-src http://security.ubuntu.com/ubuntu bionic-security main restricted
deb http://security.ubuntu.com/ubuntu bionic-security universe
# deb-src http://security.ubuntu.com/ubuntu bionic-security universe
deb http://security.ubuntu.com/ubuntu bionic-security multiverse
# deb-src http://security.ubuntu.com/ubuntu bionic-security multiverse

Good old nslookup to the rescue! So I finally found out that nslookup does not work but ping works.

root@host-02:~$ nslookup -v us-east-1.ec2.archive.ubuntu.com
;; Connection to 127.0.0.1#53(127.0.0.1) for us-east-1.ec2.archive.ubuntu.com failed: connection refused.
;; Connection to ::1#53(::1) for us-east-1.ec2.archive.ubuntu.com failed: connection refused.

It looked like the default resolver was broken.

root@host-02:~$ nslookup
> server
Default server: 127.0.0.1
Address: 127.0.0.1#53
Default server: ::1
Address: ::1#53
> us-east-1.ec2.archive.ubuntu.com
;; connection timed out; no servers could be reached

The resolver service was running but did not have permissions

root@host-02:~# systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-05-20 12:00:20 UTC; 14h ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 707 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 1110)
   CGroup: /system.slice/systemd-resolved.service
           └─707 /lib/systemd/systemd-resolved

May 21 00:00:24 host-02.ahrq.local systemd-resolved[707]: Failed to open /etc/resolv.conf: Permission denied
May 21 00:00:24 host-02.ahrq.local systemd-resolved[707]: Failed to open /etc/resolv.conf: Permission denied
May 21 00:30:25 host-02.ahrq.local systemd-resolved[707]: Failed to open /etc/resolv.conf: Permission denied
May 21 00:30:25 host-02.ahrq.local systemd-resolved[707]: Failed to open /etc/resolv.conf: Permission denied

Restored the default resolver and restarted the service and we are back in business!

root@host-02:~# ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
root@host-02:~# systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-05-21 02:22:28 UTC; 5s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 9074 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 1110)
   CGroup: /system.slice/systemd-resolved.service
           └─9074 /lib/systemd/systemd-resolved

May 21 02:22:28 host-02.ahrq.local systemd[1]: Starting Network Name Resolution...
May 21 02:22:28 host-02.ahrq.local systemd-resolved[9074]: Positive Trust Anchors:
May 21 02:22:28 host-02.ahrq.local systemd-resolved[9074]: . IN DS 19036 8 2 49aac11d7b6f644670254a1607371607a1a41855200fd2ce1cdde32f24e8fb5
May 21 02:22:28 host-02.ahrq.local systemd-resolved[9074]: . IN DS 20326 8 2 e06d44b80b8f1d39a95b0d7c65d08458e880409bbc683457104237c7f8ec8d
May 21 02:22:28 host-02.ahrq.local systemd-resolved[9074]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.ar
May 21 02:22:28 host-02.ahrq.local systemd-resolved[9074]: Using system hostname 'host-02.ahrq.local'.
May 21 02:22:28 host-02.ahrq.local systemd[1]: Started Network Name Resolution.

Finally

root@host-02:~# nslookup -v us-east-1.ec2.archive.ubuntu.com
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   us-east-1.ec2.archive.ubuntu.com
Address: 54.172.25.22

Related

Wouldn't setting the DNS server to 127.0.0.1 prevent my system from actually resolving remote hosts? I found this answer
Twilight Zelda Guardian Puzzle : Shortest Path (UPDATE: ADDED RULES)
Exotic maps $S_5\to S_6$
group of units in a topological ring
Euler characteristic of covering space of CW complex
Does this tricky series converge?
Geometric interpretation of different types of field extensions?
How to express $f(n\alpha)$ in terms of $f(\alpha)$
Is there an error in this AP Calculus quiz on function transformations?
Injecting a group into itself
Zero joint subsums of integers
Mathematical disciplines with high thresholds