What happens when the USB key or SD card I've installed VMware ESXi on fails?

Here's the process I used to resolve this:

VMware ESXi can be installed in an embedded mode or an installable mode. As outlined here, the installation mode is determined by the destination media and the size of the volume available to the ESXi installer.

• USB, SDHC or any device less than 5GB in size: Embedded
• Hard drives/volumes greater than or equal to 5GB in size: Installable

One of the unique attributes of running ESXi in embedded mode is that the OS is loaded into RAM and only touches the USB/SD device hourly during normal operation. In my situation, the system continued to operate, even with a failed SDHC device.

The error message I received in the vCenter interface indicated that configuration changes would not be saved, but the cluster was still usable. I left the system in this state for several days until I could get to the datacenter to replace the SD card.

With regard to steps to take following a failure of a USB or SD device, it is important to extract and save a copy of your host's settings!!

This is easily accomplished via PowerCLI or the vSphere CLI. I used PowerCLI running from the vCenter server:

Get-VMHostFirmware -VMHost 10.10.8.22 -BackupConfiguration -DestinationPath C:\Users\ewwhite\Downloads

• Following that, I evacuated all virtual machines from the affected host and placed it in maintenance mode.
• The host was then shut down, the SDHC card replaced with a new device, and I installed ESXi again.
• Once the host was up again with a fresh ESXi install, I made the bare minimum configuration changes needed to make the host visible on the network; set IP information, vlan info and password.
• I reloaded the saved configuration to the host via PowerCLI...

For this step, I used:

Set-VMHostFirmware -VMHost 10.10.8.22 -Restore -SourcePath c:\Users\ewwhite\configBundle-10.10.8.22.tgz -HostUser root -HostPassword YoMama!!

Restoring the configuration forces a host reboot. Once up again, I was able to issue a reconnect to rejoin the host in vCenter and exit maintenance mode.

If PowerCLI not available, the ESXi shell commands look like:

vim-cmd hostsvc/firmware/backup_config 

This produces a web link that you'll be able to browse to and download a tarball of the host's configuration.

You can SCP a configuration file to a host and use the following to restore the settings.

vim-cmd hostsvc/firmware/restore_config /tmp/configBundle.tgz

That's all!

KB article: http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=true&languageId=&externalID=2004784

Note: In the event that your original SD flash card or USB drive fails, these steps can be used in a disaster recovery capacity to get a fully functional ESXi host running as quickly as possible on a replacement USB drive or SD flash card.

Disaster Recovery Scenario

If you have a local datastore containing production information and have completed the installation on a replacement USB flash drive or SD flash card in a disaster recovery scenario, you may be required perform these additional steps:

1. Connect to the ESXi host using the vSphere Client or add the ESXi host to an existing vCenter Server implementation.
2. Click the host.
3. Click the Configuration tab.
4. Click Storage.
5. Click Rescan to ensure you can see all datastores.
6. Browse the datastore and register any virtual machines that are located on the datastores.

I'd say just reinstall ESXi fresh on a new card, it's not worth your time or effort to try to preserve the potentially corrupt data on the current card.