My server crash. This is the log. What's likely happen?
The problem is as it states itself. Line 261 of net/sched/sch_generic.c
which is generic packet scheduler routine.
The panic itself is here
Dec 21 19:47:45 localhost kernel: [<ffffffff8144a54d>] ? dev_watchdog+0x26d/0x280
So, the network device timed out. As the source code says, some queue was blocked and the timer expired. It was supposed to hold the device for certain particular time but the counter got over. Here is the relevant portion of the code.
if (!mod_timer(&dev->watchdog_timer,
258 round_jiffies(jiffies +
259 dev->watchdog_timeo)))
260 dev_hold(dev);
You see there is a watchdog timer and the counter is measured in jiffies. When this timer is over, it throws warning.
This has something to do with your network card or driver. I will straight away reject the theory of kernel bug unless they can prove it. And there is no way to tell it, unless someone has reported the exact call trace or Intel knows about this trace and it happens on the same hardware, same driver, same firmware. In a nutshell, without checking the kernel dump or vmcore, no experienced person will tell this is kernel bug. The part of kernel that deals with timer, is carefully crafted out and e1000 is not an obscure driver to deal with.
I don't want to diss your server guys but this is my take. It would be worthwhile to check your ethtool -S ethX
outputs to see whether there are any drops, overruns, timeouts etc.