Firmware Best Practices and Update Schedules

Firmware receives very little attention when it comes to updates. Out of sight, out of mind.

Many devices: RAID controllers, NICs, chipsets, and even hard drives, get some benefit from being updated. Better features, security/bug fixes, etc.

Most SA say, "Whenever it breaks, update the firmware." But this can lead to difficulties down the road; Several times, when contacting Dell about a failed hard drive, I've been asked if my hard drive firmware is up to date. All of my servers use some type of RAID configuration. If I already have a single drive failure, should I even consider trying to upgrading the firmware on the rest of the drives or the RAID controller? I would say no. But Dell seems to have a different view.

• What's a realistic update schedule for system firmware?
• Do you have any best practices to share?

(I am aware that Dell has a nice utility called Server Update Utility, which checks for all new firmware on any Dell server.)

Solution 1:

I update firmware in two key instances.

1. When staging up a server.
   • When I just get the server, I'll check the the HP web-site for the date of their latest "Firmware Update CD". If it's new enough, I'll run it against the server before bringing it up to production.
   • When I repurpose a server. Typically, this server is 2-5 years old and probably hasn't had a firmware update in that entire time. Since I'm reformatting it anyway, I'll update all the various firmware on the server.
2. When there is a vendor identified need to do so.
   • Sometimes there are major stability problems identified, like an inability to rebuild a RAID5 array after the wrong kind of failure, or a major performance bug in the TCP-offload engine on the NIC.
   • Sometimes when calling in for support, the support tech will request I update the firmware. I will do so then.

There is a third instance that I didn't list above, because it hasn't happened yet:

• When putting a much newer component into an older server. Sometimes the system BIOS will need updating to handle it.

Solution 2:

We use HP SIM (System's Insight Manager) to roll out firmware, we do it by platform - test first, then development, then integration, then reference then finally production - usually about a week or so per platform so we have a 5/6 week release to production window. Seems to work but one thing we NEVER do is roll out firmware at the same time as other updates like drivers/code etc. - saves a lot of finger-pointing.