apt update seems broken

On one of our servers, apt update gives weird errors:

me@mimer:/etc/apt$ sudo apt update
Ign:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Ign:2 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Ign:3 http://security.ubuntu.com/ubuntu xenial-security InRelease
Err:4 http://archive.ubuntu.com/ubuntu xenial Release
  503  Cache storage error - No such file or directory
Err:5 http://archive.ubuntu.com/ubuntu xenial-updates Release
  503  Cache storage error - No such file or directory
Ign:6 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial InRelease
Err:7 http://security.ubuntu.com/ubuntu xenial-security Release
  503  Cache storage error - No such file or directory
Ign:8 http://ppa.launchpad.net/costamagnagianfranco/borgbackup/ubuntu xenial InRelease
Err:9 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial Release
  503  Cache storage error - No such file or directory
Err:10 http://ppa.launchpad.net/costamagnagianfranco/borgbackup/ubuntu xenial Release
  503  Cache storage error - No such file or directory
Reading package lists... Done
E: The repository 'http://archive.ubuntu.com/ubuntu xenial Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://archive.ubuntu.com/ubuntu xenial-updates Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://security.ubuntu.com/ubuntu xenial-security Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://ppa.launchpad.net/certbot/certbot/ubuntu xenial Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://ppa.launchpad.net/costamagnagianfranco/borgbackup/ubuntu xenial Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

One difference to the machine below that for some reason mimer is also looking for Release, but I can't figure out why or where that's configured.

me@mimer:/etc/apt$ grep -R Release .
me@mimer:/etc/apt$

Nothing?

In the meantime, on another machine it works just fine:

me@freja:/etc/apt$ sudo apt update
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Hit:2 http://ppa.launchpad.net/ansible/ansible/ubuntu xenial InRelease                                                                   
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease                                                                                                               
Hit:4 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial InRelease                                                                                                
Get:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]                                                                                 
Hit:6 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu xenial InRelease                                                                                                                  
Ign:7 http://pkg.jenkins.io/debian-stable binary/ InRelease                                                                        
Hit:8 http://pkg.jenkins.io/debian-stable binary/ Release
Hit:9 https://download.docker.com/linux/ubuntu xenial InRelease
Hit:10 https://deb.nodesource.com/node_8.x xenial InRelease
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1120 kB]
Fetched 1338 kB in 1s (789 kB/s)                              
Reading package lists... Done
Building dependency tree       
Reading state information... Done
...

Some random greps and stuff I've tried:

mimer does have network and the files seem to be there?

me@mimer:/etc/apt$ curl -s http://archive.ubuntu.com/ubuntu/dists/xenial/Release | head
Origin: Ubuntu
Label: Ubuntu
Suite: xenial
Version: 16.04
Codename: xenial
Date: Thu, 21 Apr 2016 23:23:46 UTC
Architectures: amd64 arm64 armhf i386 powerpc ppc64el s390x
Components: main restricted universe multiverse
Description: Ubuntu Xenial 16.04
MD5Sum:

sources.list from mimer:

me@mimer:/etc/apt$ cat sources.list
#
#  /etc/apt/sources.list
#


#
# trusty
#
deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted universe
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe
deb http://security.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu xenial-security main restricted universe

Both machines are running 16.04.

me@mimer:/etc/apt$ lsb_release -id
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS

Now to something weird, if I change the sources.list to add [trusted=yes] it seems to work. Is there some key or certificate or something that can expire?

I've tried apt clean, removing the apt lock file etc.

Please help, I've been bashing my head against this all day and would be really happy to move on to what I was planning on doing today... :) If you want output from anything just let me know.


Solution 1:

So it turned out that there was an apt proxy configured on that server (small company, I wear many, many hats etc...) and apparently that proxy wasn't working anymore.

me@mimer:/etc/apt/apt.conf.d$ cat 00proxy 
Acquire::http::Proxy "http://1.2.3.4:3142";

I moved that file to another directory and then apt update now works again. Woo!