Firmware update failures due to filesystem corruption on iLO3
I have a few dozen HP servers with iLO3 cards where the firmware update fails. When I try to update by uploading the .bin file via the web interface, I get a nondescript message saying "The last firmware update attempt was not successful. Ready for the next update.". The iLO event log only shows "Firmware upgrade via web page failed." for this attempt.
Upgrading via the iLO ssh interface also doesn't work (useless output snipped):
</>hpiLO-> cd /map1
</map1>hpiLO-> cd firmware1
</map1/firmware1>hpiLO-> show version
/map1/firmware1
Properties
version=1.15
</map1/firmware1>hpiLO-> load -source http://buildserver/ilo3_170.bin
status=1
status_tag=COMMAND SPAWNED
Attempting to load iLO3 firmware image.
status=0
If I try to use the XML api, I get the following non-xml returned after uploading the file:
<html>
<head>
<title>Server Error</title>
</head>
<body>
<h1>Server Error</h1>
A file system error was detected on the HP iLO3.<p>
Return to <a href="">last page</a><p>
</body>
</html>
At this point the iLO is reset, a new SSH key is generated, but the firmware is not updated with no indication as to why. Nothing appears in the iLO event log.
A reset of the iLO does not make this problem go away, a quick google also returned nothing useful. What can I do to make this problem go the way of the dodo?
HP had a pretty awful initial set of firmware releases for the ILO3 management interface. See the massive changelog documenting the evolution of the product. I was working for a firm that deployed a large number of ProLiant DL380 G7 servers and had to deal with the pain of managing the buggy feature set.
In your case, you're trying to move from a very old ILO revision (1.15, circa 2010) to the current version (1.70).
In order to accomplish this, you have to perform the upgrade in two steps:
First, upgrade to an ILO version between 1.20 and 1.28. I'd recommend 1.28. HP placed a firmware prerequisite on ILO 1.50 and newer in 2012, so this intermediate update is absolutely necessary. You may need to do this by hand using the shell interface.
From there, you should be able to bring the ILO version to the current 1.70.
I covered this in: Updating ILO on HP DL380 G7 fails due to firmware version gap when I encountered the issue.
When I had to do this en masse, I kept both firmware files handy in case I encountered super old or neglected servers.
