How to fix “eth0: Detected Hardware Unit Hang” in Debian 9?
I have my own home server which acts as a gateway to internet (don't know if it is best name for it). Basically it has two ethernet ports, one connected to my ISP and other to LAN switch. Routing and NAT successfully configured and a bunch of other services.
Recently I migrated from Ubuntu 14.04 to Debian 9 (new, clean install) and now slowly restoring previous configuration. I'm stuck quite early as I made just basic network configuration to allow other computers/phones/TVs/etc access internet, but noticed that there are a lot of packet losses and connection seems to hang for a few seconds. Inspecting logs gave me this:
[ 212.088208] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
TDH <69>
TDT <aa>
next_to_use <aa>
next_to_clean <69>
buffer_info[next_to_clean]:
time_stamp <ffffa7f6>
next_to_watch <69>
jiffies <ffffa9e8>
next_to_watch.status <0>
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <3800>
PHY Extended Status <3000>
PCI Status <10>
[ 214.072275] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
TDH <69>
TDT <aa>
next_to_use <aa>
next_to_clean <69>
buffer_info[next_to_clean]:
time_stamp <ffffa7f6>
next_to_watch <69>
jiffies <ffffabd8>
next_to_watch.status <0>
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <3800>
PHY Extended Status <3000>
PCI Status <10>
[ 216.088094] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
TDH <69>
TDT <aa>
next_to_use <aa>
next_to_clean <69>
buffer_info[next_to_clean]:
time_stamp <ffffa7f6>
next_to_watch <69>
jiffies <ffffadd0>
next_to_watch.status <0>
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <3800>
PHY Extended Status <3000>
PCI Status <10>
[ 218.071082] ------------[ cut here ]------------
[ 218.072129] WARNING: CPU: 0 PID: 0 at /build/linux-EAZfyE/linux-4.9.51/net/sched/sch_generic.c:316 dev_watchdog+0x22d/0x230
[ 218.073249] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
[ 218.074368] Modules linked in: xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul iTCO_wdt iTCO_vendor_support ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf pcspkr i915 sg drm_kms_helper lpc_ich mei_me mfd_core drm ie31200_edac joydev evdev mei edac_core shpchp i2c_algo_bit battery video button ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb glue_helper lrw gf128mul ablk_helper cryptd aes_x86_64 mbcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid0 multipath linear hid_generic usbhid hid
[ 218.078853] raid1 md_mod sd_mod crc32c_intel i2c_i801 ahci i2c_smbus libahci libata scsi_mod ehci_pci ehci_hcd xhci_pci xhci_hcd e1000e ptp usbcore pps_core usb_common fan thermal
[ 218.082049] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.0-4-amd64 #1 Debian 4.9.51-1
[ 218.083772] Hardware name: /DQ77KB, BIOS KBQ7710H.86A.0051.2013.0329.1350 03/29/2013
[ 218.085468] 0000000000000000 ffffffffa7729974 ffff98909e203e20 0000000000000000
[ 218.087205] ffffffffa7476eae 0000000000000000 ffff98909e203e78 ffff989094e28000
[ 218.088980] 0000000000000000 ffff989094fb9c80 0000000000000001 ffffffffa7476f2f
[ 218.090766] Call Trace:
[ 218.092579] <IRQ>
[ 218.092597] [<ffffffffa7729974>] ? dump_stack+0x5c/0x78
[ 218.094413] [<ffffffffa7476eae>] ? __warn+0xbe/0xe0
[ 218.096268] [<ffffffffa7476f2f>] ? warn_slowpath_fmt+0x5f/0x80
[ 218.098133] [<ffffffffa74aed52>] ? enqueue_task_fair+0x82/0x940
[ 218.100024] [<ffffffffa792cb2d>] ? dev_watchdog+0x22d/0x230
[ 218.101909] [<ffffffffa792c900>] ? qdisc_rcu_free+0x40/0x40
[ 218.103860] [<ffffffffa74e4020>] ? call_timer_fn+0x30/0x110
[ 218.105766] [<ffffffffa74e4524>] ? run_timer_softirq+0x1d4/0x430
[ 218.107709] [<ffffffffa74f4ca0>] ? tick_sched_handle.isra.12+0x20/0x50
[ 218.109654] [<ffffffffa74f4d08>] ? tick_sched_timer+0x38/0x70
[ 218.111630] [<ffffffffa7a0b0d5>] ? __do_softirq+0x105/0x290
[ 218.113594] [<ffffffffa747cf8e>] ? irq_exit+0xae/0xb0
[ 218.115567] [<ffffffffa7a0aeee>] ? smp_apic_timer_interrupt+0x3e/0x50
[ 218.117536] [<ffffffffa7a0a202>] ? apic_timer_interrupt+0x82/0x90
[ 218.119509] <EOI>
[ 218.119527] [<ffffffffa78cd31a>] ? cpuidle_enter_state+0x11a/0x2b0
[ 218.121505] [<ffffffffa74b9634>] ? cpu_startup_entry+0x154/0x240
[ 218.123486] [<ffffffffa8138f57>] ? start_kernel+0x443/0x463
[ 218.125426] [<ffffffffa8138120>] ? early_idt_handler_array+0x120/0x120
[ 218.127400] [<ffffffffa8138408>] ? x86_64_start_kernel+0x14c/0x170
[ 218.129384] ---[ end trace 6cd1142bfcc66b87 ]---
[ 218.131367] e1000e 0000:00:19.0 eth0: Reset adapter unexpectedly
[ 222.052843] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
I have found this question: e1000e Reset adapter unexpectedly / Detected Hardware Unit Hang which seems to be same problem, but none of found fixes worked.
I tried:
- Booting kernel with
pcie_aspm=off - Turning off strange options:
ethtool -K eth0 gso off gro off tso off - Disabled ASPM in bios
- Disabled any power saving features in bios
- Used script
fixeep-82573-dspd.shwhich said that my hardware is not compatible with that fix or something like that - Compiled newest driver from Intel website
What else I can try? I already lost whole day on this, internet connection is unusable, everybody needs to use own LTE/3G internet on phones to access web.
Is Debian a bad choice for such server?
There's a question in ServerFault that has two more potential fixes:
https://serverfault.com/questions/616485/e1000e-reset-adapter-unexpectedly-detected-hardware-unit-hang
As mentioned in there, you might want to try:
- disabling enhanced C1 power saving state (C1E) in the BIOS settings, or
- disabling TCP checksum offloading with
ethtool -K eth0 tx off rx off
If you can extract the e1000e.ko kernel module from your backups, you can use the modinfo command on it to list the driver version.
To identify the possibility of a NIC firmware update, a precise identification of the exact NIC model is necessary. From your log output, I can see your NIC is PCI device 00:19.0 and the name of the network interface is eth0. Please run these commands as root:
# lspci -nn -s 00:19.0 -v
# ethtool -i eth0
The first command reveals the PCI ID numbers of the NIC, and the second command has the NIC firmware version number in its output.
If the NIC happens to be Intel 82579V (PCI ID 8086:1503), there's a known bug in it related to power saving states, that has also caused problems in Windows 8 and newer (i.e. with operating systems that are new enough to use the more advanced power saving states of modern hardware). An update from Ubuntu 14.04 to Debian 9 might have brought an equivalent update in power saving code of Linux, causing the bug to trigger.
Intel has a specific firmware update tool for the 82579V chip that can even be used with NICs integrated on motherboards. Unfortunately, I think the update tool must be run in Windows.
Problem solved by... switching cables and configuration between NIC's. So eth0 was LAN side device and eth1 was Internet side device. After switch eth0 became Internet device and eth1 LAN one. Dont know why and how, but simply it is working. Even under heavy load after a couple of hours everything is ok. When i switch it back to initial config, driver crashes after no more than 2 minutes.
I'm completely unable to find any reason why this happens, but well... now is good.