Can't boot Windows Server 2012 from iSCSI

I'm working out how to install various OSes onto diskless servers to be run via iSCSI. The servers have no special hardware for this (no iSCSI HBAs etc.)

I briefly played with Windows Deployment Services but it seems to be about the opposite of what I want: I could be wrong but it seems designed to do a network-based install of Windows onto a system that will then boot from a local disk, whereas I want to install Windows such that it will run from an iSCSI disk, and I don't care if install media has to be inserted locally. Therefore my present approach doesn't use WDS.

This test was of Windows Server 2012 but likely applies to prior versions also.

In short, I am setting up a blank iSCSI target (I have tried several s/w vendors and it doesn't seem to make much difference although presently I'm using StarWind), and using PXE to chain to gPXE or iPXE to do an iSCSI boot (I've tried both, with similar result although I found gPXE a little easier to use.)

Of course, without an OS, the boot fails, but it leaves the LU hooked in through the BIOS disk interface such that when the local DVD-ROM is then booted, the installer sees the volume to install to.

I accomplish the boot by dropping to a gPXE prompt and typing the following commands (same in iPXE except for the interface #):

dhcp net1
set keep-san 1
sanboot iscsi:#.#.#.#::::iqn.xxxxxxxxxxxxxxxxxxx
exit

Windows Server 2012 seems to install and at some point reboots. iPXE kicks in to boot from iSCSI, and the 'Metro' logo appears - so at least something did get installed to the LU. However, the screen then turns light blue and a message briefly flashes in large text, something about the "PC" (it's actually a rack server but anyway) encountering a problem but I can't read it all before it vanishes and the computer is rebooting.

Eventually I get a text screen titled Windows Boot Manager which reads:

Windows failed to start. A recent hardware or software change might be the
cause. To fix the problem:

  1. Insert your Windows installation disc and restart your computer.
  2. Choose your language settings, and then click "Next."
  3. Click "Repair your computer."

If you do not have this disc, contact your system administrator or computer
manufacturer for assistance.

    File: \Windows\system32\ntoskrnl.exe

    Status: 0xc00000e9

    Info: The operating system couldn't be loaded because the kernel is
          missing or contains errors

Those paying close attention will realize these instructions would be very difficult to follow effectively, especially on physical hardware. (By the way, I have tried installs both to physical and virtual machines, with similar results.)

I am guessing what is happening here is that Windows 2012 gets as far as it does because iPXE has hooked Interrupt 13h (BIOS disk services), but once the system has booted far enough to switch to protected mode drivers, the boot is rudely interrupted because there is no hard disk to read from and/or Windows' iSCSI initiator fails.

The iPXE I'm trying is 1.0.0+ (3fcb) and the gPXE I tried was 1.0.1.

Is it possible to have Server 2012 run off iSCSI without a hardware HBA, and if so, what might I be doing wrong?

Solution 1:

(My initial answer was premature. As promised, I've rewritten it after having gotten everything working.)

First of all, I've found that in general iSCSI-boot-enabling software is half-baked, and the disparate systems involved interoperate very poorly. For this reason I recommend instead going with a hardware-based solution such as iSCSI HBAs if possible. With that said I'll relate my experiences here in case it helps anyone.

To summarize what I found (I'm assuming you've set up DHCP and TFTP for PXE and an iSCSI target and have gotten as far as chaining to gPXE or iPXE):

• gPXE and iPXE never write multiple NICs into the iBFT (iSCSI Boot Firmware Table), and this can impact Windows Server. I have discussed this problem in detail in a separate question here.

• In addition to the above design limitation, gPXE has an actual bug that likewise affects systems with multiple network ports. I'll explain below. In order to avoid this bug, I used the "UNDI only" build of gPXE. This prevents gPXE from accessing the NICs directly and makes it instead use an API provided by the NIC's PXE loader. This makes gPXE think there is only one network port (the one it was loaded on), and this evades the bug. I am not sure if this bug is present in newer iPXE versions.

• I was initially confused about the keep-san option in gPXE/iPXE. The keep-san flag affects gPXE's behaviour only if the boot fails. Therefore, this option is needed only on the very first boot when the installation is started.

• Windows Server (at least 2012 and probably others) apparently does not tolerate moving the iSCSI initiator that provides its system disk from one network port to another. If Windows is booted from an initiator on a different network port than the one to which it was installed, Windows will crash (BSOD and/or reboot) during boot, at the handoff to the MS initiator.

• There is an acknowledged feature/issue in Windows Server (2003 and up) where it will use the gateway, if one is specified, to access the target, even if the target is on the local subnet. If the gateway is unavailable or doesn't route back onto the same port, the boot will fail at the handoff to the MS initiator. Make sure no gateway setting is given out by the DHCP if one isn't needed.

The gPXE bug I mention above involves the iBFT (iSCSI Boot Firmware Table). This is an object which is placed into memory by the pre-boot system which contains information about the NICs, iSCSI initiator, and the iSCSI target to use as the system disk. The OS uses this information to continue booting once it switches to protected mode. The format is specified here.

Suspecting a problem in the information gPXE was placing in the iBFT, I programmed a boot sector which dumps the contents of the iBFT to the screen. Using this I found that the data written by gPXE is under certain circumstances erroneous.

As mentioned, gPXE only writes one NIC record into the iBFT, but in some situations, the information written to that one NIC record is jumbled up. The MAC address and PCI address will correspond to one NIC, but the local IP and gateway addresses will correspond to another. This is most likely to happen if the SAN is not on the first NIC.

To add to the confusion, this incorrect iBFT information is written if gPXE boots automatically, but when booting from gPXE's command prompt, depending on the exact sequence of commands entered, the correct information may be written. Throw in the fact that Windows will manifest symptoms identical to those caused by this bug if its NIC has been changed (even given a correct iBFT), and you can see why I tore my hair out.

By the way, in my original question I had thought that it was working for Server 2008 R2 but not Server 2012. (I'm editing that out as it's misleading.) I suspect there's actually no difference in their underlying behaviour and that the different outcomes owed to the subtleties of the above problems and minor variations in my tests.