How to check "apt-get upgrade" status after losing ssh connection?
The following logs are related to apt upgrades:
/var/log/apt/history.log
/var/log/apt/term.log
/var/log/dpkg.log
If the command was dist-upgrade
, there are additional logs in:
/var/log/dist-upgrade
FYI, it is usually safe to just re-run the upgrade and apt will continue where it left off when the process died due to disconnection. However...
A GNU Screen Primer:
When ssh'ing into a remote server and starting a long-running process in the foreground, it is best practice to use GNU Screen. Screen provides a virtual terminal that continues running even if your ssh connection is lost.
Install screen:
sudo apt-get install screen
Run screen:
screen
After running screen you will get a command line prompt as with a normal terminal. You can then run the upgrade from inside screen:
sudo apt-get upgrade
To understand how this works, "detach" screen by pressing Ctrl+a, d. This will return you to the non-screen terminal. You can see the list of running screens with
screen -list
If you only have one screen running, you can reattach it with:
screen -raAd
(This detaches screen in case it is attached elsewhere, and reattaches it to the terminal you are currently running.)
Typically you cannot scroll 'normally' from within screen without some extra setup. To scroll within screen, press Ctrl-Esc to enter cursor mode. You can then scroll down and up with j and k. Press Esc again to exit cursor mode.
There are many more resources on the net available for additional screen functions. It is an invaluable standard tool for system administration.
See also:
- How to keep processes running after ending ssh session?
To see real-time output from background apt
job, use:
sudo tail -f /var/log/apt/term.log
In addition to doublerebel's answer, I noticed an alternative today.
I went to bed last night after starting an upgrade over SSH. I stupidly forgot to start it in screen
and lost my SSH session overnight.
I was just about to start researching retty
when I noticed that root
had started a screen
session.
me@GAMMA:~$ ps aux | grep -E 'release|upgrade|apt'
root 6208 0.0 0.0 29140 1628 ? Ss 01:57 0:05 SCREEN -e \0\0 -L -c screenrc -S ubuntu-release-upgrade-screen-window /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
root 6209 0.2 5.6 287428 93144 pts/2 Ss+ 01:57 3:13 /usr/bin/python /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
root 6239 0.0 0.0 50052 1184 ? Ss 01:58 0:00 /usr/sbin/sshd -o PidFile=/var/run/release-upgrader-sshd.pid -p 1022
root 7306 0.0 4.6 287432 77284 pts/2 S+ 02:43 0:08 /usr/bin/python /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
me 26829 0.0 0.0 9440 956 pts/5 S+ 22:18 0:00 grep --color=auto -E release|upgrade|apt
So I listed root
's screens and attached to it:
me@GAMMA:~$ sudo screen -list
There is a screen on:
6208.ubuntu-release-upgrade-screen-window (12/11/2013 01:57:58 AM) (Detached)
1 Socket in /var/run/screen/S-root.
me@GAMMA:~$ sudo screen -x -r
And Bam! I was back in the game.
I had an upgrade in progress (asking me whether to keep or replace a config file; this information I took from /var/log/apt/term.log
) when my terminal was closed by another person. Restarting apt upgrade
complained about a lock file being held.
My fix was to do the following:
sudo killall apt # to release the lock
sudo dpkg --configure -a # to fix the interrupted upgrade
sudo apt upgrade # to make sure no packages are left not upgraded