Shutting an EC2 instance down locally (on the instance)?
edit: this seems to be a problem specifically with the AMI that I am trying to shut down. Other AMIs politely shut down with the "sudo poweroff" command.
The goal is for an EC2 instance to shut itself down. All my instances are Ubuntu 10.10.
I used to be able to execute the command "sudo poweroff -n -f" and the instance would shut down. However, now the instances stop responding and peg their CPUs at 50%.
I've tried every possible combination of shutdown/halt/poweroff and they all either:
- Do nothing, leaving the instance still running
- Put the instance into this halted/buys but still "on" state
Additional details / exact scenarios:
Run
sudo poweroff: I get the normal "The system is going down for power off NOW!" message. Then am returned to the shell. The system does not power off. If I execute "sudo poweroff" again, the system seems to shut down (ssh connections terminated), however the AMI does not shut down and remains in a "running" state with constant 50% CPU usage (as monitored from the AWS Management console.Run
sudo shutdown now: I get the normal "The system is going down for maintenance NOW!" message but nothing happens. The system keeps running normally. Tailing the syslog I see that the following messages:
-
Dec 21 16:24:36 ip-10-212-177-30 init: ssh main process (599) terminated with status 255
Dec 21 16:24:36 ip-10-212-177-30 init: rc main process (677) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty4 main process (678) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty5 main process (682) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty2 main process (685) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty6 main process (688) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: cron main process (692) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: irqbalance main process (701) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty3 main process (686) killed by TERM signal
All log messages after calling "sudo poweroff"
sudo poweroff
==> auth.log <==
Dec 21 21:39:37 ip-10-244-74-146 sudo: ubuntu : TTY=pts/1 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/sbin/poweroff
==> wtmp <==
02~~~runlevel2.6.35-28-virtualR?N??
Broadcast message from ubuntu@ip-10-244-74-146
(/dev/pts/1) at 21:39 ...
The system is going down for power off NOW!
==> boot.log <==
init: Disconnected from system bus
init: dbus main process (663) killed by TERM signal
hostname: the specified hostname is invalid
system is still running, no change
sudo poweroff
==> wtmp <==
~~~shutdown2.6.35-28-virtual)R?N^?
no more log files, system hangs, can't SSH in, CPU spins at 50%
sudo halt -p --verbose
ubuntu@ip-10-2-22-35:~$ /sbin/runlevel
N 2
ubuntu@ip-10-2-22-35:~$ sudo halt -p --verbose
Calling shutdown
Broadcast message from ubuntu@ip-10-2-22-35
(/dev/pts/1) at 17:31 ...
The system is going down for power off NOW!
ubuntu@ip-10-2-22-35:~$ /sbin/runlevel
2 0
ubuntu@ip-10-2-22-35:~$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.1 2892 1732 ? Ss 17:21 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S 17:21 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 17:21 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S 17:21 0:00 [migration/0]
root 5 0.0 0.0 0 0 ? S 17:21 0:00 [watchdog/0]
root 6 0.0 0.0 0 0 ? S 17:21 0:00 [events/0]
root 7 0.0 0.0 0 0 ? S 17:21 0:00 [cpuset]
root 8 0.0 0.0 0 0 ? S 17:21 0:00 [khelper]
root 9 0.0 0.0 0 0 ? S 17:21 0:00 [netns]
root 10 0.0 0.0 0 0 ? S 17:21 0:00 [async/mgr]
root 11 0.0 0.0 0 0 ? S 17:21 0:00 [pm]
root 12 0.0 0.0 0 0 ? S 17:21 0:00 [xenwatch]
root 13 0.0 0.0 0 0 ? S 17:21 0:00 [xenbus]
root 14 0.0 0.0 0 0 ? S 17:21 0:00 [sync_supers]
root 15 0.0 0.0 0 0 ? S 17:21 0:00 [bdi-default]
root 16 0.0 0.0 0 0 ? S 17:21 0:00 [kintegrityd/0]
root 17 0.0 0.0 0 0 ? S 17:21 0:00 [kblockd/0]
root 18 0.0 0.0 0 0 ? S 17:21 0:00 [ata_aux]
root 19 0.0 0.0 0 0 ? S 17:21 0:00 [ata_sff/0]
root 20 0.0 0.0 0 0 ? S 17:21 0:00 [khubd]
root 21 0.0 0.0 0 0 ? S 17:21 0:00 [kseriod]
root 22 0.0 0.0 0 0 ? S 17:21 0:00 [kmmcd]
root 24 0.0 0.0 0 0 ? S 17:21 0:00 [khungtaskd]
root 25 0.0 0.0 0 0 ? S 17:21 0:00 [kswapd0]
root 26 0.0 0.0 0 0 ? SN 17:21 0:00 [ksmd]
root 27 0.0 0.0 0 0 ? S 17:21 0:00 [aio/0]
root 28 0.0 0.0 0 0 ? S 17:21 0:00 [ecryptfs-kthrea]
root 29 0.0 0.0 0 0 ? S 17:21 0:00 [crypto/0]
root 35 0.0 0.0 0 0 ? S 17:21 0:00 [khvcd]
root 36 0.0 0.0 0 0 ? S 17:21 0:00 [kstriped]
root 37 0.0 0.0 0 0 ? S 17:21 0:00 [kmpathd/0]
root 38 0.0 0.0 0 0 ? S 17:21 0:00 [kmpath_handlerd]
root 39 0.0 0.0 0 0 ? S 17:21 0:00 [ksnapd]
root 40 0.0 0.0 0 0 ? S 17:21 0:00 [kondemand/0]
root 41 0.0 0.0 0 0 ? S 17:21 0:00 [kconservative/0]
root 143 0.0 0.0 0 0 ? S 17:21 0:00 [xfs_mru_cache]
root 144 0.0 0.0 0 0 ? S 17:21 0:00 [xfslogd/0]
root 145 0.0 0.0 0 0 ? S 17:21 0:00 [xfsdatad/0]
root 146 0.0 0.0 0 0 ? S 17:21 0:00 [xfsconvertd/0]
root 148 0.0 0.0 0 0 ? S 17:21 0:00 [xfsbufd/sda1]
root 149 0.0 0.0 0 0 ? S 17:21 0:00 [xfsaild/sda1]
root 150 0.0 0.0 0 0 ? S 17:21 0:00 [xfssyncd/sda1]
root 177 0.0 0.0 0 0 ? S 17:21 0:00 [flush-202:1]
root 183 0.0 0.0 2560 988 ? S 17:21 0:00 /sbin/plymouthd --mode=boot --attach-to-session
root 187 0.0 0.0 4484 1260 ? S 17:21 0:00 mountall --daemon
daemon 382 0.0 0.0 1968 536 ? Ss 17:22 0:00 portmap
root 413 0.0 0.0 2304 252 ? Ss 17:22 0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.leases eth0
statd 593 0.0 0.0 2104 824 ? Ss 17:22 0:00 rpc.statd -L
root 607 0.0 0.0 0 0 ? S 17:22 0:00 [kjournald]
root 624 0.0 0.0 0 0 ? S 17:22 0:00 [rpciod/0]
root 627 0.0 0.0 0 0 ? S< 17:22 0:00 [kslowd000]
root 628 0.0 0.0 0 0 ? S< 17:22 0:00 [kslowd001]
root 639 0.0 0.0 0 0 ? S 17:22 0:00 [nfsiod]
ganglia 684 0.0 0.0 4260 1644 ? Ss 17:22 0:00 /usr/sbin/gmond
root 723 0.0 0.1 52988 3168 ? Ss 17:22 0:00 php-fpm: master process (/etc/php5/fpm/main.conf)
www-data 724 0.0 0.1 52972 2536 ? S 17:22 0:00 php-fpm: pool www
www-data 725 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www
www-data 726 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www
www-data 727 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www
www-data 728 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www
www-data 729 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www
www-data 730 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www
www-data 731 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www
www-data 732 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www
www-data 733 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 734 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 735 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 736 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 737 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 738 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 739 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 740 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 741 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 742 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 743 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 744 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 746 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 747 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 748 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 750 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 751 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 753 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 754 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 755 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www
www-data 756 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 757 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 758 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 759 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 760 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 761 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 762 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 763 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 764 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 765 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
www-data 766 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www
root 1240 0.0 0.1 8848 3116 ? Ss 17:31 0:00 sshd: ubuntu [priv]
ubuntu 1334 0.0 0.0 8848 1368 ? S 17:31 0:00 sshd: ubuntu@pts/1
ubuntu 1335 0.5 0.3 9016 6048 pts/1 Ss 17:31 0:01 -bash
root 1373 0.0 0.0 1904 512 ? Ss 17:31 0:00 /bin/sh /etc/init.d/rc 0
root 1375 0.0 0.0 3000 1304 ? S 17:31 0:00 /bin/bash /etc/rc0.d/K01unregister stop
root 1394 0.0 0.1 5144 1952 ? S 17:31 0:00 ssh -i /XXXX [email protected] sudo sh -c 'sed -e "/10.2.22.35 fb?"-//W3C//DTD??lang="e
ubuntu 1400 0.0 0.0 4280 1088 pts/1 R+ 17:34 0:00 ps aux
last VMS console output after two calls to halt -p (in wierd 50% cpu hung state)
init: tty2 main process (679) killed by TERM signal
init: tty3 main process (680) killed by TERM signal
init: tty6 main process (682) killed by TERM signal
init: hwclock-save main process (881) terminated with status 1
init: Disconnected from system bus
init: dbus main process (618) killed by TERM signal
hostname: the specified hostname is invalid
[ 52.346590] System halted.
Thanks!
Solution 1:
By default, when you shutdown/halt an EBS boot instance it is simply moved to the "stopped" state. This leave the instance available for restarting. You are no longer paying for instance hours, but you will continue paying a minimal charge for the EBS volume(s).
You can change this behavior so that a shutdown/halt will terminate the instance by setting the instance-initiated-shutdown-behavior to "terminate".
I've written a bit about this here:
Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data
http://alestic.com/2010/01/ec2-instance-locking
You'll want to run the instance with the option:
--instance-initiated-shutdown-behavior terminate
With this flag set, either of the following commands will terminate the EC2 instance:
sudo shutdown -h now
# or
sudo halt
You can also "Change Shutdown Behavior" through the AWS console for running instances (right mouse click).
Solution 2:
Look into ec2stop / ec2-stop-instances commands that are a part of the AWS tools