Reading package list takes forever

I've been trying to update my stuff on my machine, and it seems like it can't read my package list. It seems like every time i do the sudo apt-get install *something* && sudo apt-get update it gets stuck at reading package list, this have not been a problem before. Here is my specs and whatnot :

Memory : 15.8 gb
Processor : AMD Phenom(tm) II x4 965 Processor x 4
Graphics : Gallium 0.4 on AMD BARTS
OS type : 32-bit
Netspeed :

Solution 1:

I've seen that as well.

I don't have a solution, but I have a workaround (echo 3 | sudo tee /proc/sys/vm/drop_caches) and potentially more information so someone can bring the investigation any further.

It's not a network issue because at "Reading package list...", it is just reading files in /var/lib/apt/lists/. A:

strace -tt -T -fo strace.log apt-get update

gives:

16394 14:43:03.921130 open("/var/lib/apt/lists/gb.archive.ubuntu.com_ubuntu_dists_precise_main_binary-i386_Packages", O_RDONLY|O_LARGEFILE) = 7 <0.000012>
[...]
16394 14:43:03.995238 read(6, "-3.1ubuntu2)\nConflicts: linux86\n"..., 32444) = 32444 <0.000111>
16394 14:43:05.787187 read(6, "c (<< 1:14.b.4-dfsg), erlang-exa"..., 32239) = 32239 <0.000069>
16394 14:43:05.788025 read(6, ".deb\nSize: 42130\nMD5sum: c7de671"..., 31695) = 31695 <0.000068>
16394 14:43:05.870734 read(6, "5: 29c4b395a92bdc12932f151c3643a"..., 31607) = 31607 <0.000071>
16394 14:43:05.890862 read(6, "e-pack-af-base\nFilename: pool/ma"..., 32538) = 32538 <0.000070>
16394 14:43:05.891425 read(6, "buntu-usb-live, ubuntu-dvd-live,"..., 32090) = 32090 <0.000066>
16394 14:43:05.891960 read(6, "cd9755b03ac2c9b8251125c7b6618\nDe"..., 32195) = 32195 <0.000034>
16394 14:43:06.043001 read(6, "rg>\nArchitecture: all\nVersion: 2"..., 32535) = 32535 <0.000072>

See how those 8 read system calls took over 2 seconds even though each individual call takes less than 1 ms. Running time apt-get update or looking at top, that process is not busy between those two calls. So why the delay?

Then I did:

echo t > /proc/sysrq-trigger

a few times and looked at the outcome in kern.log:

 apt-get         D 00000000     0 16790  12706 0x00000000
  e8695d30 00000086 f7bd5e6c 00000000 f7bd5e44 f74a6580 c1990e00 c1990e00
  efe46efe 000042cb f7b9de00 e71a7230 f74a6580 c107e116 00000000 00000000
  044aa200 00000000 00000000 00000000 00000000 e8695d0c e8695d0c c1038de8
 Call Trace:
  [<c107e116>] ? enqueue_entity+0x186/0x220
  [<c1038de8>] ? default_spin_lock_flags+0x8/0x10
  [<c15e13bd>] ? _raw_spin_lock_irqsave+0x2d/0x40
  [<c15e0533>] schedule+0x23/0x60
  [<c15deecf>] schedule_timeout+0x12f/0x290
  [<c1075c38>] ? ttwu_do_activate.constprop.86+0x58/0x70
  [<c1055190>] ? usleep_range+0x40/0x40
  [<c15e0846>] io_schedule_timeout+0x86/0xd0
  [<c15cef7d>] balance_dirty_pages.isra.17+0x3f5/0x4b4
  [<c15e118d>] ? _raw_spin_lock+0xd/0x10
  [<c1180781>] ? __set_page_dirty_buffers+0x81/0xb0
  [<c110deb5>] ? set_page_dirty+0x55/0x60
  [<c11812c9>] ? __block_page_mkwrite+0xe9/0x170
  [<c110f3ae>] balance_dirty_pages_ratelimited_nr+0xde/0x100
  [<c1126f53>] do_wp_page+0x503/0x830
  [<c1128ef7>] handle_pte_fault+0x267/0x2c0
  [<c1129c62>] handle_mm_fault+0x1e2/0x280
  [<c15e4988>] do_page_fault+0x158/0x4c0
  [<c104e4dc>] ? irq_exit+0x5c/0xa0
  [<c15e22d0>] ? do_debug+0x180/0x180
  [<c15e4830>] ? vmalloc_fault+0x195/0x195
  [<c15e1c53>] error_code+0x67/0x6c

So, not sure what that means but that looks about the handling of page faults, so points at a potential memory management issue.

I then tried a:

echo 3 >/proc/sys/vm/drop_caches

And that did make the problem go away.

Now, it very much looks like a kernel issue. So, I've updated to the latest kernel (3.8 backport from raring) and that's where I'm at. Will update if the problem persists with the newer kernel.

Edit

The problem does persist with the new kernel, though not as bad. And same thing,

echo 3 | sudo tee /proc/sys/vm/drop_caches

clears the problem for a while. I've only seen that happen on MSI laptops (Product Name: CR61 2M/CX61 2OC/CX61 2OD).

Edit December 2015

As confirmed by btrace aptitude/apt-get does appear to do some disk I/O at the time. It's got a temporary file (/var/cache/apt/pkgcache.bin.<random-chars>) mmapped in memory which is why it doesn't show in the strace output.

Still can't explain why it only happens on some machines only, why dropping caches helps, why switching to 64bit helps.

If somebody can reproduce it, an interesting test could be to see if that also happens when running under eatmydata or if moving /var/cache/apt onto tmpfs or a ramdisk helps.

Solution 2:

The advice at http://antti-juhani.kaijanaho.fi/newblog/archives/521 has sped it up for me several times on various computers:

sudo dpkg --clear-avail
sudo sync-available

(The blog also recommended sudo dpkg --forget-old-unavail between the 2 steps but apparently it's deprecated and no longer needed.)

Solution 3:

Follow the steps:

Clean out cache:
```
sudo apt-get clean
```

Move the sources.list so apt cannot use it:

mv /etc/apt/sources.list /etc/apt/sources.list1 && sudo apt-get update

Move it back then update:

mv /etc/apt/sources.list1 /etc/apt/sources.list && sudo apt-get update

Also check and remove any PPAs and source lines you don't need.

Solution 4:

On my system, the cause was an incorrect value in the LANGUAGE= environment variable. It should hold values such as en:fr:de, and not en_US.UTF-8,sl_SI.UTF-8:

root@fik:~
# locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8,sl_SI.UTF-8
LC_CTYPE=sl_SI.UTF-8
LC_NUMERIC=sl_SI.UTF-8
LC_TIME=sl_SI.UTF-8
LC_COLLATE=sl_SI.UTF-8
LC_MONETARY=sl_SI.UTF-8
LC_MESSAGES=en_US.UTF-8
LC_PAPER=sl_SI.UTF-8
LC_NAME=sl_SI.UTF-8
LC_ADDRESS=sl_SI.UTF-8
LC_TELEPHONE=sl_SI.UTF-8
LC_MEASUREMENT=sl_SI.UTF-8
LC_IDENTIFICATION=sl_SI.UTF-8
LC_ALL=

When ran (via strace), the apt-get update command clonks on the read() call. It takes ages to execute, and eats all the available cycles of one CPU core:

root@fik:~
# strace apt-get update
[snip]
read(5, "form, hardware::opengl, implemen"..., 32146) = 32146
read(5, " Maintainers <pkg-bluetooth-main"..., 32658) = 32658
read(5, ": 17569748\nMD5sum: 9c20d52f9a0d5"..., 32200) = 32200
brk(0x55ac79212000)                     = 0x55ac79212000
read(5, "scription-md5: ca1156b27bec24d4c"..., 32469) = 32469
read(5, " Boost.Math Library\nMulti-Arch: "..., 32477) = 32477
read(5, "epends: libc6 (>= 2.4), lsb-base"..., 32648) = 32648
^C--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
strace: Process 18452 detached

If I set LANGUAGE= to a correct value (such as en), everything goes back to normal again:

root@fik:~
# export LANGUAGE=en

root@fik:~
# locale
LANG=en_US.UTF-8
LANGUAGE=en
LC_CTYPE=sl_SI.UTF-8
LC_NUMERIC=sl_SI.UTF-8
LC_TIME=sl_SI.UTF-8
LC_COLLATE=sl_SI.UTF-8
LC_MONETARY=sl_SI.UTF-8
LC_MESSAGES=en_US.UTF-8
LC_PAPER=sl_SI.UTF-8
LC_NAME=sl_SI.UTF-8
LC_ADDRESS=sl_SI.UTF-8
LC_TELEPHONE=sl_SI.UTF-8
LC_MEASUREMENT=sl_SI.UTF-8
LC_IDENTIFICATION=sl_SI.UTF-8
LC_ALL=

root@fik:~
# apt-get update
Hit:1 http://ftp.at.debian.org/debian experimental InRelease
Ign:3 http://ftp.at.debian.org/debian jessie InRelease                                                      
Hit:4 http://ftp.at.debian.org/debian jessie-updates InRelease  
Hit:5 http://ftp.at.debian.org/debian jessie-backports InRelease                                                                             
Hit:6 http://ftp.at.debian.org/debian sid InRelease                                                                    
Hit:7 http://ftp.at.debian.org/debian stretch InRelease                               
Hit:8 http://ftp.at.debian.org/debian stretch-updates InRelease                                             
Hit:9 http://ftp.at.debian.org/debian jessie Release                                  
Hit:2 http://screenshots.getdeb.net xenial-getdeb InRelease                           
Hit:10 http://security.debian.org jessie/updates InRelease      
Hit:11 http://security.debian.org stretch/updates InRelease
Reading package lists... Done