Server Suddenly became Unresponsive to SSH
Server suddenly became unresponsive to SSH, I reported the previous time it happened here on another server I have about 2 months ago. The server setup is CentOS6 64bit, it runs a very basic apache2 server that serves up static files and should be under very small loads, as it behind a heavily cached Amazon CloudFront.
I was told to look through /var/log/messages but when I open it for the time of the problem ( around 18:00 on 2/27/2012 ) there a lot of messages there that I have no idea what they mean, and Googling is not helping much either so I was wondering if someone more experienced could possible take a look at this log and tell me if they see anything weird.
Feb 26 03:06:02 173 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Feb 26 03:06:02 173 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1439" x-info="http://www.rsyslog.com"] (re)start
Feb 26 04:20:01 173 auditd[1414]: Audit daemon rotating log files
Feb 26 13:37:09 173 auditd[1414]: Audit daemon rotating log files
Feb 27 08:03:03 173 auditd[1414]: Audit daemon rotating log files
Feb 27 18:35:13 173 init: tty (/dev/tty2) main process (1958) killed by TERM signal
Feb 27 18:35:13 173 init: tty (/dev/tty3) main process (1960) killed by TERM signal
Feb 27 18:35:13 173 init: tty (/dev/tty4) main process (1962) killed by TERM signal
Feb 27 18:35:13 173 init: tty (/dev/tty5) main process (1964) killed by TERM signal
Feb 27 18:35:13 173 init: tty (/dev/tty6) main process (1966) killed by TERM signal
Feb 27 18:35:15 173 qpidd[1893]: 2012-02-27 18:35:15 notice Shut down
Feb 27 18:35:16 173 abrtd: Got signal 15, exiting
Feb 27 18:35:20 173 acpid: exiting
Feb 27 18:35:20 173 init: Disconnected from system bus
Feb 27 18:35:20 173 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Feb 27 18:35:20 173 console-kit-daemon[1982]: WARNING: no sender#012
Feb 27 18:35:20 173 auditd[1414]: The audit daemon is exiting.
Feb 27 18:35:20 173 kernel: type=1305 audit(1330389320.836:995029): audit_pid=0 old=1414 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
Feb 27 18:35:20 173 kernel: type=1305 audit(1330389320.936:995030): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditctl_t:s0 res=1
Feb 27 18:35:20 173 kernel: Kernel logging (proc) stopped.
Feb 27 18:35:20 173 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1439" x-info="http://www.rsyslog.com"] exiting on signal 15.
Feb 27 18:38:07 173 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Feb 27 18:38:07 173 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1437" x-info="http://www.rsyslog.com"] (re)start
Feb 27 18:38:07 173 kernel: Initializing cgroup subsys cpuset
Feb 27 18:38:07 173 kernel: Initializing cgroup subsys cpu
Feb 27 18:38:07 173 kernel: Linux version 2.6.32-220.2.1.el6.x86_64 ([email protected]) (gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) ) #1 SMP Fri Dec 23 02:21:33 CST 2011
Feb 27 18:38:07 173 kernel: Command line: ro root=/dev/mapper/vg_173-lv_root rd_LVM_LV=vg_173/lv_swap rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_173/lv_root rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
Feb 27 18:38:07 173 kernel: KERNEL supported cpus:
Feb 27 18:38:07 173 kernel: Intel GenuineIntel
Feb 27 18:38:07 173 kernel: AMD AuthenticAMD
Feb 27 18:38:07 173 kernel: Centaur CentaurHauls
Feb 27 18:38:07 173 kernel: BIOS-provided physical RAM map:
Feb 27 18:38:07 173 kernel: BIOS-e820: 0000000000000000 - 000000000009b800 (usable)
Feb 27 18:38:07 173 kernel: BIOS-e820: 000000000009b800 - 00000000000a0000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 0000000000100000 - 00000000be7a8000 (usable)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000be7a8000 - 00000000be7f4000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000be7f4000 - 00000000be7fc000 (ACPI data)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000be7fc000 - 00000000bf47a000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf47a000 - 00000000bf47b000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf47b000 - 00000000bf48c000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf48c000 - 00000000bf48f000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf48f000 - 00000000bf4b0000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf4b0000 - 00000000bf4b2000 (usable)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf4b2000 - 00000000bf503000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf503000 - 00000000bf50d000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf50d000 - 00000000bf533000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf533000 - 00000000bf576000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf576000 - 00000000bf800000 (usable)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed40000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
Feb 27 18:38:07 173 kernel: DMI 2.7 present.
Feb 27 18:38:07 173 kernel: SMBIOS version 2.7 @ 0xF0480
Feb 27 18:38:07 173 kernel: AMI BIOS detected: BIOS may corrupt low RAM, working around it.
Feb 27 18:38:07 173 kernel: last_pfn = 0x240000 max_arch_pfn = 0x400000000
Feb 27 18:38:07 173 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Feb 27 18:38:07 173 kernel: total RAM covered: 8192M
Feb 27 18:38:07 173 kernel: Found optimal setting for mtrr clean up
Feb 27 18:38:07 173 kernel: gran_size: 64K chunk_size: 64K num_reg: 4 lose cover RAM: 0G
Feb 27 18:38:07 173 kernel: last_pfn = 0xbf800 max_arch_pfn = 0x400000000
Feb 27 18:38:07 173 kernel: init_memory_mapping: 0000000000000000-00000000bf800000
Feb 27 18:38:07 173 kernel: init_memory_mapping: 0000000100000000-0000000240000000
Feb 27 18:38:07 173 kernel: RAMDISK: 3711a000 - 37fef46f
Feb 27 18:38:07 173 kernel: ACPI: RSDP 00000000000f0450 00024 (v02 SUPERM)
Feb 27 18:38:07 173 kernel: ACPI: XSDT 00000000be7f4078 00074 (v01 SUPERM SMCI--MB 00000001 AMI 00010013)
Feb 27 18:38:07 173 kernel: ACPI: FACP 00000000be7faf40 000F4 (v04 SUPERM SMCI--MB 00000001 AMI 00010013)
Feb 27 18:38:07 173 kernel: ACPI: DSDT 00000000be7f4180 06DC0 (v02 SUPERM SMCI--MB 00000000 INTL 20051117)
Feb 27 18:38:07 173 kernel: ACPI: FACS 00000000bf50af80 00040
Feb 27 18:38:07 173 kernel: ACPI: APIC 00000000be7fb038 00092 (v03 SUPERM SMCI--MB 00000001 AMI 00010013)
Feb 27 18:38:07 173 kernel: ACPI: SSDT 00000000be7fb0d0 001D6 (v01 AMICPU PROC 00000001 MSFT 03000001)
Feb 27 18:38:07 173 kernel: ACPI: MCFG 00000000be7fb2a8 0003C (v01 SUPERM SMCI--MB 00000001 MSFT 00000097)
Feb 27 18:38:07 173 kernel: ACPI: HPET 00000000be7fb2e8 00038 (v01 SUPERM SMCI--MB 00000001 AMI. 00000004)
Feb 27 18:38:07 173 kernel: ACPI: SPMI 00000000be7fb320 00040 (v05 A M I OEMSPMI 00000000 AMI. 00000000)
Feb 27 18:38:07 173 kernel: ACPI: EINJ 00000000be7fb360 00130 (v01 AMI AMI EINJ 00000000 00000000)
Feb 27 18:38:07 173 kernel: ACPI: ERST 00000000be7fb490 00210 (v01 AMIER AMI ERST 00000000 00000000)
Feb 27 18:38:07 173 kernel: ACPI: HEST 00000000be7fb6a0 000A8 (v01 AMI AMI HEST 00000000 00000000)
Feb 27 18:38:07 173 kernel: ACPI: BERT 00000000be7fb748 00030 (v01 AMI AMI BERT 00000000 00000000)
Feb 27 18:38:07 173 kernel: No NUMA configuration found
Feb 27 18:38:07 173 kernel: Faking a node at 0000000000000000-0000000240000000
Feb 27 18:38:07 173 kernel: Bootmem setup node 0 0000000000000000-0000000240000000
Feb 27 18:38:07 173 kernel: NODE_DATA [0000000000017000 - 000000000004afff]
Feb 27 18:38:07 173 kernel: bootmap [000000000004b000 - 0000000000092fff] pages 48
Feb 27 18:38:07 173 kernel: (8 early reservations) ==> bootmem [0000000000 - 0240000000]
Feb 27 18:38:07 173 kernel: #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
Feb 27 18:38:07 173 kernel: #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
Feb 27 18:38:07 173 kernel: #2 [0001000000 - 000200c7e4] TEXT DATA BSS ==> [0001000000 - 000200c7e4]
Feb 27 18:38:07 173 kernel: #3 [003711a000 - 0037fef46f] RAMDISK ==> [003711a000 - 0037fef46f]
Feb 27 18:38:07 173 kernel: #4 [000009b800 - 0000100000] BIOS reserved ==> [000009b800 - 0000100000]
Feb 27 18:38:07 173 kernel: #5 [000200d000 - 000200d3a8] BRK ==> [000200d000 - 000200d3a8]
Feb 27 18:38:07 173 kernel: #6 [0000010000 - 0000012000] PGTABLE ==> [0000010000 - 0000012000]
Feb 27 18:38:07 173 kernel: #7 [0000012000 - 0000017000] PGTABLE ==> [0000012000 - 0000017000]
Feb 27 18:38:07 173 kernel: found SMP MP-table at [ffff8800000fcdc0] fcdc0
Feb 27 18:38:07 173 kernel: Reserving 129MB of memory at 48MB for crashkernel (System RAM: 9216MB)
Feb 27 18:38:07 173 kernel: Zone PFN ranges:
Feb 27 18:38:07 173 kernel: DMA 0x00000010 -> 0x00001000
Feb 27 18:38:07 173 kernel: DMA32 0x00001000 -> 0x00100000
Feb 27 18:38:07 173 kernel: Normal 0x00100000 -> 0x00240000
Feb 27 18:38:07 173 kernel: Movable zone start PFN for each node
Feb 27 18:38:07 173 kernel: early_node_map[5] active PFN ranges
Feb 27 18:38:07 173 kernel: 0: 0x00000010 -> 0x0000009b
Feb 27 18:38:07 173 kernel: 0: 0x00000100 -> 0x000be7a8
Feb 27 18:38:07 173 kernel: 0: 0x000bf4b0 -> 0x000bf4b2
Feb 27 18:38:07 173 kernel: 0: 0x000bf576 -> 0x000bf800
Feb 27 18:38:07 173 kernel: 0: 0x00100000 -> 0x00240000
Feb 27 18:38:07 173 kernel: ACPI: PM-Timer IO Port: 0x408
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
Feb 27 18:38:07 173 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Feb 27 18:38:07 173 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Feb 27 18:38:07 173 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Feb 27 18:38:07 173 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Feb 27 18:38:07 173 kernel: Using ACPI (MADT) for SMP configuration information
Feb 27 18:38:07 173 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Feb 27 18:38:07 173 kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 000000000009b000 - 000000000009c000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 000000000009c000 - 00000000000a0000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000be7a8000 - 00000000be7f4000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000be7f4000 - 00000000be7fc000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000be7fc000 - 00000000bf47a000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf47a000 - 00000000bf47b000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf47b000 - 00000000bf48c000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf48c000 - 00000000bf48f000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf48f000 - 00000000bf4b0000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf4b2000 - 00000000bf503000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf503000 - 00000000bf50d000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf50d000 - 00000000bf533000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf533000 - 00000000bf576000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf800000 - 00000000fed1c000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed40000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000fed40000 - 00000000ff000000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000ff000000 - 0000000100000000
Feb 27 18:38:07 173 kernel: Allocating PCI resources starting at bf800000 (gap: bf800000:3f51c000)
There's more but I can not share it here because there is a 30,000 character limit for a post.
EDIT : LIKE LAST TIME, my multipackets received number spiked at around the time of the incident.
04:20:01 PM eth1 82.02 84.18 13.40 93.86 0.00 0.00 0.00
04:30:01 PM lo 0.03 0.03 0.00 0.00 0.00 0.00 0.00
04:30:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
04:30:01 PM eth1 72.94 74.44 7.29 86.59 0.00 0.00 0.00
04:40:01 PM lo 0.01 0.01 0.00 0.00 0.00 0.00 0.00
04:40:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
04:40:01 PM eth1 77.25 83.17 6.75 99.82 0.00 0.00 0.00
04:50:01 PM lo 0.01 0.01 0.00 0.00 0.00 0.00 0.00
04:50:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
04:50:01 PM eth1 71.76 77.19 6.24 91.93 0.00 0.00 0.00
05:00:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:00:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:00:01 PM eth1 64.39 63.84 8.28 71.20 0.00 0.00 0.00
05:10:01 PM lo 0.03 0.03 0.00 0.00 0.00 0.00 0.00
05:10:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:10:01 PM eth1 65.18 66.56 5.77 76.39 0.00 0.00 0.00
05:20:01 PM lo 0.03 0.03 0.00 0.00 0.00 0.00 0.00
05:20:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:01 PM eth1 92.23 89.68 33.78 88.57 0.00 0.00 0.00
05:30:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:30:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:30:01 PM eth1 73.42 74.93 8.05 87.59 0.00 0.00 0.00
05:40:01 PM lo 0.03 0.03 0.00 0.00 0.00 0.00 0.00
05:40:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:40:01 PM eth1 60.54 58.69 19.79 59.61 0.00 0.00 507865361.98
05:50:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:50:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:50:01 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 2146553474.33
06:00:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:00:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:00:01 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 2150279010.21
06:00:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
06:10:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:10:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:10:01 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 2145874241.82
06:20:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:20:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:20:01 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 2150063723.97
06:30:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:30:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:30:01 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 2146303180.75
Average: lo 0.01 0.01 0.00 0.00 0.00 0.00 0.00
Average: eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth1 122.77 119.70 24.14 84.60 0.00 0.00 101305280.31
06:38:02 PM LINUX RESTART
06:40:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
06:50:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
EDIT: /var/log/secure from around the time of the incident:
Feb 27 18:35:15 173 runuser: pam_unix(runuser:session): session opened for user qpidd by (uid=0)
Feb 27 18:35:15 173 runuser: pam_unix(runuser:session): session closed for user qpidd
Feb 27 18:35:16 173 sshd[1774]: Received signal 15; terminating.
Feb 27 18:35:16 173 sshd[5490]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[5500]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[5500]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[5490]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[5510]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[5520]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[5510]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[5520]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[23046]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[23056]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[23066]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[23046]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[23056]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[23066]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[23076]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[23076]: pam_unix(sshd:session): session closed for user root
Feb 27 18:38:19 173 sshd[5112]: Server listening on 0.0.0.0 port 22.
Feb 27 18:38:19 173 sshd[5112]: Server listening on :: port 22.
Feb 27 18:38:20 173 runuser: pam_unix(runuser:session): session opened for user qpidd by (uid=0)
Feb 27 18:38:20 173 runuser: pam_unix(runuser:session): session closed for user qpidd
Feb 27 18:38:20 173 runuser: pam_unix(runuser:session): session opened for user qpidd by (uid=0)
Feb 27 18:38:20 173 runuser: pam_unix(runuser:session): session closed for user qpidd
Feb 27 18:38:20 173 runuser: pam_unix(runuser-l:session): session opened for user qpidd by (uid=0)
Feb 27 18:38:20 173 runuser: pam_unix(runuser-l:session): session closed for user qpidd
Feb 27 18:38:34 173 sshd[5349]: Accepted publickey for root from MY IP port 40702 ssh2
Feb 27 18:38:34 173 sshd[5349]: pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 27 18:40:57 173 sshd[5349]: Received disconnect from MY IP: 11: disconnected by user
Feb 27 18:40:57 173 sshd[5349]: pam_unix(sshd:session): session closed for user root
So when I first try to connect after the incident it seems that it would accept my publickey but it would not respond.
Check /var/log/secure - if there are many Failed attempts, the sshd might be getting hung
If this is the case you probably want to move to key based auth or install something like Deny Hosts