the size of apt-get update lists is too big

This question contains circumventions for an bug in Ubuntu:

• Bug #1001780 “apt-get update size is too big” : Bugs : Launchpad itself,

as well as some general advice on saving bandwidth when doing apt-get update.

That bug was closed on 8 Aug 2012, so you may only need the information related to this question should the bug reappear. If that happens please file another bug report on Launchpad.

I ran a clean install to Ubuntu 12.04 and so far everything has been working well. I especially commend the Ubuntu team for this release.

I only noticed that the size of repository update is now about ~13MB. Normally, it is about this size for the first time you run apt-get update after a clean install and then ~ 23kb - 1300kb for subsequent updates.

The output from apt-get update is the same I get for previous versions of Ubuntu (its pretty normal). Its a bit too long but look at an example output I got from running apt-get update.

Ign http://archive.canonical.com precise InRelease
Ign http://dl.google.com stable InRelease
Ign http://dl.google.com stable InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Hit http://download.virtualbox.org precise InRelease
Ign http://security.ubuntu.com precise-security InRelease
Ign http://linux.dropbox.com precise InRelease
Ign http://extras.ubuntu.com precise InRelease
Ign http://download.skype.com stable InRelease
Hit http://archive.canonical.com precise Release.gpg
Get:1 http://dl.google.com stable Release.gpg [198 B]
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net oneiric InRelease
Ign http://ppa.launchpad.net precise InRelease
Get:2 http://security.ubuntu.com precise-security Release.gpg [198 B]
Get:3 http://extras.ubuntu.com precise Release.gpg [72 B]
Hit http://download.virtualbox.org precise/contrib i386 Packages
Ign http://download.skype.com stable Release.gpg
Hit http://linux.dropbox.com precise Release.gpg
Ign http://us.archive.ubuntu.com precise InRelease
Ign http://us.archive.ubuntu.com precise-updates InRelease
Ign http://us.archive.ubuntu.com precise-backports InRelease
Hit http://archive.canonical.com precise Release
Get:4 http://dl.google.com stable Release.gpg [198 B]
Ign http://ppa.launchpad.net oneiric InRelease
Ign http://ppa.launchpad.net oneiric InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Ign http://ppa.launchpad.net precise InRelease
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net precise Release.gpg
Get:5 http://security.ubuntu.com precise-security Release [49.6 kB]
Hit http://extras.ubuntu.com precise Release
Ign http://download.skype.com stable Release
Ign http://download.virtualbox.org precise/contrib TranslationIndex
Get:6 http://us.archive.ubuntu.com precise Release.gpg [198 B]
Hit http://archive.canonical.com precise/partner i386 Packages
Hit http://linux.dropbox.com precise Release
Get:7 http://ppa.launchpad.net precise Release.gpg [316 B]
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://extras.ubuntu.com precise/main Sources
Get:8 http://ppa.launchpad.net precise Release.gpg [316 B]
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net precise Release.gpg
Get:9 http://us.archive.ubuntu.com precise-updates Release.gpg [198 B]
Ign http://archive.canonical.com precise/partner TranslationIndex
Ign http://download.skype.com stable/non-free i386 Packages/DiffIndex
Get:10 http://dl.google.com stable Release [1,347 B]
Hit http://linux.dropbox.com precise/main i386 Packages
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net oneiric Release.gpg
Hit http://extras.ubuntu.com precise/main i386 Packages
Ign http://extras.ubuntu.com precise/main TranslationIndex
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net oneiric Release.gpg
Hit http://ppa.launchpad.net oneiric Release.gpg
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net precise Release.gpg
Get:11 http://us.archive.ubuntu.com precise-backports Release.gpg [198 B]
Ign http://download.skype.com stable/non-free TranslationIndex
Get:12 http://dl.google.com stable Release [1,347 B]
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Ign http://linux.dropbox.com precise/main TranslationIndex
Hit http://ppa.launchpad.net precise Release
Ign http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Get:13 http://ppa.launchpad.net precise Release [11.9 kB]
Get:14 http://us.archive.ubuntu.com precise Release [49.6 kB]
Hit http://download.skype.com stable/non-free i386 Packages
Get:15 http://dl.google.com stable/main i386 Packages [1,268 B]
Ign http://dl.google.com stable/main TranslationIndex
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net oneiric Release
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net oneiric Release
Get:16 http://security.ubuntu.com precise-security/main Sources [7,089 B]
Hit http://ppa.launchpad.net oneiric Release
Get:17 http://dl.google.com stable/main i386 Packages [769 B]
Ign http://dl.google.com stable/main TranslationIndex
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise Release
Hit http://ppa.launchpad.net precise/main Sources
Hit http://ppa.launchpad.net precise/main i386 Packages
Get:18 http://security.ubuntu.com precise-security/restricted Sources [14 B]
Get:19 http://security.ubuntu.com precise-security/universe Sources [3,653 B]
Get:20 http://security.ubuntu.com precise-security/multiverse Sources [696 B]
Get:21 http://security.ubuntu.com precise-security/main i386 Packages [32.9 kB]
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main Sources
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Get:22 http://us.archive.ubuntu.com precise-updates Release [49.6 kB]
Ign http://ppa.launchpad.net precise/main Sources/DiffIndex
Ign http://ppa.launchpad.net precise/main i386 Packages/DiffIndex
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main Sources
Hit http://ppa.launchpad.net precise/main i386 Packages
Get:23 http://security.ubuntu.com precise-security/restricted i386 Packages [14 B]
Get:24 http://security.ubuntu.com precise-security/universe i386 Packages [8,594 B]
Get:25 http://security.ubuntu.com precise-security/multiverse i386 Packages [1,393 B]
Hit http://security.ubuntu.com precise-security/main TranslationIndex
Hit http://security.ubuntu.com precise-security/multiverse TranslationIndex
Hit http://security.ubuntu.com precise-security/restricted TranslationIndex
Hit http://security.ubuntu.com precise-security/universe TranslationIndex
Ign http://ppa.launchpad.net precise/main TranslationIndex
Get:26 http://us.archive.ubuntu.com precise-backports Release [49.6 kB]
Hit http://ppa.launchpad.net precise/main Sources
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Get:27 http://ppa.launchpad.net precise/main i386 Packages [1,276 B]
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main Sources
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main Sources
Get:28 http://us.archive.ubuntu.com precise/main Sources [934 kB]
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main Sources
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main i386 Packages
Hit http://security.ubuntu.com precise-security/main Translation-en
Hit http://security.ubuntu.com precise-security/multiverse Translation-en
Hit http://security.ubuntu.com precise-security/restricted Translation-en
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net oneiric/main Sources
Hit http://ppa.launchpad.net oneiric/main i386 Packages
Ign http://ppa.launchpad.net oneiric/main TranslationIndex
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net oneiric/main Sources
Hit http://security.ubuntu.com precise-security/universe Translation-en
Ign http://archive.canonical.com precise/partner Translation-en_US
Hit http://ppa.launchpad.net oneiric/main i386 Packages
Ign http://ppa.launchpad.net oneiric/main TranslationIndex
Hit http://ppa.launchpad.net oneiric/main Sources
Hit http://ppa.launchpad.net oneiric/main i386 Packages
Ign http://ppa.launchpad.net oneiric/main TranslationIndex
Hit http://ppa.launchpad.net precise/main Sources
Ign http://extras.ubuntu.com precise/main Translation-en_US
Ign http://download.virtualbox.org precise/contrib Translation-en_US
Ign http://archive.canonical.com precise/partner Translation-en
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main Sources
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Ign http://extras.ubuntu.com precise/main Translation-en
Ign http://download.virtualbox.org precise/contrib Translation-en
Hit http://ppa.launchpad.net precise/main Sources
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main Sources
Ign http://linux.dropbox.com precise/main Translation-en_US
Hit http://ppa.launchpad.net precise/main i386 Packages
Ign http://download.skype.com stable/non-free Translation-en_US
Ign http://linux.dropbox.com precise/main Translation-en
Ign http://download.skype.com stable/non-free Translation-en
Ign http://dl.google.com stable/main Translation-en_US
Ign http://dl.google.com stable/main Translation-en
Ign http://dl.google.com stable/main Translation-en_US
Get:29 http://us.archive.ubuntu.com precise/restricted Sources [5,470 B]
Get:30 http://us.archive.ubuntu.com precise/universe Sources [5,019 kB]
Ign http://dl.google.com stable/main Translation-en
Get:31 http://us.archive.ubuntu.com precise/multiverse Sources [155 kB]
Get:32 http://us.archive.ubuntu.com precise/main i386 Packages [1,274 kB]
Get:33 http://us.archive.ubuntu.com precise/restricted i386 Packages [8,431 B]
Get:34 http://us.archive.ubuntu.com precise/universe i386 Packages [4,796 kB]
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net oneiric/main Translation-en_US
Ign http://ppa.launchpad.net oneiric/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net oneiric/main Translation-en_US
Ign http://ppa.launchpad.net oneiric/main Translation-en
Ign http://ppa.launchpad.net oneiric/main Translation-en_US
Ign http://ppa.launchpad.net oneiric/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Get:35 http://us.archive.ubuntu.com precise/multiverse i386 Packages [121 kB]
Hit http://us.archive.ubuntu.com precise/main TranslationIndex
Hit http://us.archive.ubuntu.com precise/multiverse TranslationIndex
Hit http://us.archive.ubuntu.com precise/restricted TranslationIndex
Hit http://us.archive.ubuntu.com precise/universe TranslationIndex
Get:36 http://us.archive.ubuntu.com precise-updates/main Sources [31.2 kB]
Get:37 http://us.archive.ubuntu.com precise-updates/restricted Sources [765 B]
Get:38 http://us.archive.ubuntu.com precise-updates/universe Sources [10.1 kB]
Get:39 http://us.archive.ubuntu.com precise-updates/multiverse Sources [696 B]
Get:40 http://us.archive.ubuntu.com precise-updates/main i386 Packages [96.5 kB]
Get:41 http://us.archive.ubuntu.com precise-updates/restricted i386 Packages [770 B]
Get:42 http://us.archive.ubuntu.com precise-updates/universe i386 Packages [27.7 kB]
Get:43 http://us.archive.ubuntu.com precise-updates/multiverse i386 Packages [1,393 B]
Hit http://us.archive.ubuntu.com precise-updates/main TranslationIndex
Hit http://us.archive.ubuntu.com precise-updates/multiverse TranslationIndex
Hit http://us.archive.ubuntu.com precise-updates/restricted TranslationIndex
Hit http://us.archive.ubuntu.com precise-updates/universe TranslationIndex
Get:44 http://us.archive.ubuntu.com precise-backports/main Sources [700 B]
Get:45 http://us.archive.ubuntu.com precise-backports/restricted Sources [14 B]
Get:46 http://us.archive.ubuntu.com precise-backports/universe Sources [1,680 B]
Get:47 http://us.archive.ubuntu.com precise-backports/multiverse Sources [14 B]
Get:48 http://us.archive.ubuntu.com precise-backports/main i386 Packages [559 B]
Get:49 http://us.archive.ubuntu.com precise-backports/restricted i386 Packages [14 B]
Get:50 http://us.archive.ubuntu.com precise-backports/universe i386 Packages [1,391 B]
Get:51 http://us.archive.ubuntu.com precise-backports/multiverse i386 Packages [14 B]
Hit http://us.archive.ubuntu.com precise-backports/main TranslationIndex
Hit http://us.archive.ubuntu.com precise-backports/multiverse TranslationIndex
Hit http://us.archive.ubuntu.com precise-backports/restricted TranslationIndex
Hit http://us.archive.ubuntu.com precise-backports/universe TranslationIndex
Hit http://us.archive.ubuntu.com precise/main Translation-en
Hit http://us.archive.ubuntu.com precise/multiverse Translation-en
Hit http://us.archive.ubuntu.com precise/restricted Translation-en
Hit http://us.archive.ubuntu.com precise/universe Translation-en
Hit http://us.archive.ubuntu.com precise-updates/main Translation-en
Hit http://us.archive.ubuntu.com precise-updates/multiverse Translation-en
Hit http://us.archive.ubuntu.com precise-updates/restricted Translation-en
Hit http://us.archive.ubuntu.com precise-updates/universe Translation-en
Hit http://us.archive.ubuntu.com precise-backports/main Translation-en
Hit http://us.archive.ubuntu.com precise-backports/multiverse Translation-en
Hit http://us.archive.ubuntu.com precise-backports/restricted Translation-en
Hit http://us.archive.ubuntu.com precise-backports/universe Translation-en
Fetched 12.8 MB in 1min 33s (137 kB/s)

Is this a new feature in 12.04? Or, if it is unintended, is there a way I can fix this? Thanks.

I think there are a couple of things worth noting here.

Looking at the same us.archive.ubuntu.com or archive.ubuntu.com archives, You can see that the Main and Universe distribution Package.bz2 files are being marked as modified twice an hour, even though they contain the same content. These repositories are actually frozen. As they hold entries for each package in main and universe this is causing the average download to be very large. This causes the server to send apt-get update huge package file again even though nothing has actually changed. The package file contains the version number and description of the most current version of each package in the repository. The actual release files contain the date and time of the last real change.

chilicuil has kindly filed a bug report for this and at this writing this is thought to be a problem with Ubuntu archive mirroring mechanisms. Thank you, chilicuil. The bug is at http://bugs.launchpad.net/launchpad/+bug/1001780

Note that this bug has been closed, so the following circumvention should no longer be needed or used:

Since the affected repositories are the unchanging distribution repositories, one approach to circumventing this problem is to run the following commands immediately before running sudo apt-get update. Be sure to adjust them for the name of the archive site you use as well as the name of the release you are using. Not everyone will need all four commands--see the comments at the end of the commands. Don't use them for the development release.

• sudo touch /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_precise_main_binary-i386_Packages
• sudo touch /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_precise_universe_binary-i386_Packages # If you have enabled Ubuntu Universe
• sudo touch /var/lib/apt/list/us.archive.ubuntu.com_ubuntu_dists_precise_main_source_Sources # If you have enabled sources
• sudo touch /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_precise_universe_source_Sources # If you have enabled both Universe and sources

These commands tell the system that these files are up-to-date by updating the time on the archive files. Don't run these commands on other repositories, or after the bug is fixed.

Now that the problem has been fixed, it may make sense to consider how to further reduce bandwidth when running apt-get if you run it regularly and have either a slow or an expensive Internet connection.

1. It may make sense to run the update less often or, equivalently, choose a repository that is updated less often the main ones.

2. Many include the "source" repositories in their list of software sources. This is useful for doing apt-get source packagename, but that may be sort of old-fashioned.

   Unless you get source every day, you might consider removing the source setting, and enable it again and do an update only when you want to download some source. The source of various packages is always changing so the index to be downloaded is always changing, too.

   You can also consider using Ubuntu Distributed Development for smaller packages, using bazaar. In other words:

   bzr branch lp:ubuntu/sourcename

   This downloads both the package source and its history, in compressed form. This history for bigger packages may have many megabytes, so you may want to restrict this strategy to small packages. If you can use this instead of "apt-get source packagename" you can keep the source repositories out of you repo list altogether.

I tried switching to a mirror instead of using us.archive.ubuntu.com and it has fixed the problem! With the usual Ubuntu server, I found that just checking a mere half hour later I would get another ~13MB fetch from apt-get update; however, with the mirror, it's gone down to < 1MB.

Basically what I did was open up /etc/apt/sources.list and replace:

deb http://us.archive.ubuntu.com/ubuntu precise main restricted universe multiverse

with

deb http://us-west-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu precise main restricted universe multiverse

I don't know if there's something special about the Amazon S3 mirrors but you can locate a mirror near you and see if that helps. The first fetch was large (~16MB) but subsequent ones were < 1MB.

A workaround to update only the interested Index file:

Most of the time, we update package list to add a PPA. so this is a workaround for that

1. Create a file as /etc/apt/apt.conf.d/99mysettings and
   put this line in the file. Apt::Get::List-Cleanup 'false';.

   How ? : Type these in a terminal sudo gedit /etc/apt/apt.conf.d/99mysettings.

   Explanation: It causes apt-get not to delete the index files which are already downloaded.

2. Disable the repositories which you do not like to update now. You can go here to see how you can disable repositories .

3. Then run apt-get update command in a terminal. wait till it ends.

4. Finally again enable all repositories which you have disabled a few minutes ago. You have now updated PPA index with all previous indexes

Hope this will help.