Second NIC "not working properly" on diskless Server 2012

Solution 1:

I finally got to the bottom of this and did manage to get it working. In the process, though, I've come to the conclusion that iSCSI boot functionality in Windows, gPXE, and iPXE are all half-baked. I'll share the approach that worked for me in case it helps anyone else, but please be aware of some caveats:

• This is a poor solution. A hardware-based solution, such as an iSCSI HBA, will offer better performance, robustness, and will be far, far easier to set up.

• This solution does not scale well to large deployments, mainly because it requires too much manual labour per diskless server to set up.

• This solution isn't that simple. There may be a simpler solution (other than the obvious, use an iSCSI HBA.) If you know of one, please add it and I'll mark yours the answer if I can duplicate it.

• This solution is an ugly, ugly hack. Use at your own risk!!

Before I go on I want to clarify that any time I say "NIC", I am referring to what Windows considers a single "device", but which might actually only be one of several ports on an actual NIC. This terminology is consistent with both the iBFT standard itself and iPXE/gPXE.

Windows, when booted on its iSCSI initiator, has some very finicky requirements with regard to the iBFT (the table the 'iSCSI boot solution' writes into memory before calling the Windows boot loader, which tells it how to access the iSCSI LU). I've been able to piece together some "gotcha" rules (which may or may not hold in your particular situation):

• If a NIC is not in the iBFT, Windows will not work with it. It will manifest the symptom given in the question.

• The list of NICs in the iBFT has to be sorted in a particular order. I don't have full details as I only had two NIC ports in the test server, on the same NIC. One was PCI 08:04.0 and the other was PCI 08:04.1. If the iBFT listed the NIC at 08:04.1 before the one at 08:04.0, Windows got mad. (Note that there is nothing in the standards requiring a given order.)

• The iSCSI target must be reachable from the first NIC listed in the iBFT. This might require you to switch your SAN and LAN ports, due to the above rule.

• If the first NIC in the iBFT is not the same as it was when Windows was first installed, it will crash and reboot. This may require you to reinstall Windows if your setup wasn't right initially. (I'm not positive what constitutes "same", but a different port on the same NIC definitely wasn't "same".)

• The NIC sections must occur in memory in the same order in which they are listed in the control section, or Windows gets mad. (Note that the standards do not stipulate that the ordering must match - again, this is just Windows being lazy.)

The first rule is the rub. Neither gPXE 1.0.0 nor the Jan. 31, 2013 commit of its successor, iPXE, ever write multiple NICs to the iBFT, even where there are multiple NICs they know about. I've verified this by examination of their source code.

My hacky solution was to get the iPXE source tree, and modify the program such that it writes a second NIC section to the iBFT, corresponding to the other NIC in my server (the NIC I was not booting from.) I just hardwired the MAC address and PCI address. I found it was not necessary to place the IP stuff into the NIC's section - just leave it all zeroed and Windows will assign it later during boot. (Please note that the IP stuff does need to be written for the SAN NIC, but iPXE is already coded to do that.)

By using #define, the actual addresses can be entered into some convenient place rather than having to dig through source code every time you want to change them.

If you make this change, please be aware that the NIC sections have an index byte in their header. The iPXE code doesn't touch these (although it is given in the struct) because it never, ever writes more than one NIC, but if you write a second NIC, you will need to set its index byte to 1 or else Windows will not be happy.

The obvious large downside to this solution is that you have to recompile iPXE for every server, keep these separate versions of iPXE on the TFTP server, and configure the PXE server to issue the different boot programs to each of the servers.

Some knowledge of C programming is needed to make the initial change, along with a Linux distro and the GNU dev tools. The iBFT format is specified here.

I wish I could just post my changes here but I actually ended up changing a very old version which the ipxe.org website tricked me into downloading. (Apparently, they never tag stable releases; I've since learned all releases on the master branch are stable.) I'd rather not encourage anyone to use such an old version.

The newest version still has the same limitation. I'll forward this to their development list so hopefully it gets fixed.