Perform (Ubuntu) do-release-upgrade using a local Aptly mirror (missing meta-release files)

Background

We currently have a local deb mirror setup using Aptly snapshots due to regulatory reasons (for example: servers are not allowed to fetch packages over the internet). All of this works just fine for normal upgrades. When we need to do a release upgrade (from Ubuntu 18.04 to 20.04) we can normally just replace the VM, but some servers require us to do an in-place release upgrade using our mirror.

Current state

We have configured /etc/update-manager/meta-release on our nodes to point to our mirror but of-course there is no meta-release and meta-release-lts files in our Aptly. If we then copy the official meta-release files to be served from our Aptly it still won't work because it refers to a bunch of different external URLs, installers, documentations, etc. and even if we would change the URLs to point to our Aptly all those files would not be there. So we would need to be copy and serve those too.

Solution?

Do we need to parse the meta-release files, download all the installers, etc. serve them and generate our own meta-release files from that? There must be a simpler way of doing this! Do we have to build this from scratch or is there tooling available?

https://www.aptly.info/

PS. We are thinking of switching to Pulp 3 since Aptly seems to be partially abandoned, not that we know if that would help with the release upgrade issue.

What we did is syncing our local mirror, downloading the meta-release & meta-release-lts files as well as the required updater including signature and release infos (bionic.tar.gz bionic.tar.gz.gpg etc) from default Ubuntu repositories.

archive.ubuntu.com/ubuntu/dists/bionic/main/dist-upgrader-all/current/

Then we made these available from our mirror and modified the contents of meta-release files matching our mirror URL.

[meta-release snippet]

[METARELEASE]
URI = http://$YOUR_MIRROR_HERE/ubuntu_1604/meta-release
URI_LTS = http://$YOUR_MIRROR_HERE/ubuntu_1604/meta-release-lts

[meta-release-lts snippet]

[...]
Release-File: http://$YOUR_MIRROR_HERE/ubuntu/dists/bionic/Release
ReleaseNotes: http://$YOUR_MIRROR_HERE/ubuntu/dists/bionic-updates/main/dist-upgrader-all/current/ReleaseAnnouncement
ReleaseNotesHtml: http://$YOUR_MIRROR_HERE/ubuntu/dists/bionic-updates/main/dist-upgrader-all/current/ReleaseAnnouncement.html
UpgradeTool: http://$YOUR_MIRROR_HERE/ubuntu/dists/bionic-updates/main/dist-upgrader-all/current/bionic.tar.gz
UpgradeToolSignature: http://$YOUR_MIRROR_HERE/ubuntu/dists/bionic-updates/main/dist-upgrader-all/current/bionic.tar.gz.gpg

We used "apt dist-upgrade -dy" to download/fetch all packages to local PACKAGECACHE first. You can override PACKAGECACHE and custom sources.list files using apt/dpkg flags, this can be skipped if full offline upgrade is not required.

/etc/apt/apt.conf.d/10upgrade

dir::etc::sourcelist /srv/apt/custom_sources.list;
dir::cache::archives /srv/apt/packages;"

With this setup you can fully upgrade using only local packages/configs and prevent communication with ubuntu.com repos during upgrade.

One caveat was using systemwide http proxy servers in /etc/environment - just override this by calling upgrader with custom env:

$ export HTTP_PROXY="" 
$ export http_proxy="" 
$ export DEBUG_UPDATE_MANAGER=true

The updater doesn't read /etc/apt/apt.conf.d/10proxy as this is only for transport.

Without these steps the upgrader still tries to reach archive.ubuntu.com using your configured proxy which results in upgrade failure.

And finally calling it using a subshell to catch all log output besides infos in /var/log/dist-upgrade/main.log

$ (do-release-upgrade -f DistUpgradeViewNonInteractive) &>> $somelog

Hope this helps (we did build it from scratch)