Configuring SuperMicro IPMI to use one of the LAN interfaces instead of the IPMI port?
The SuperMicro X8SIE-F board has two dedicated LAN interfaces for the operating system (LAN1/2) and one dedicated LAN interface for IPMI.
Is it possible to configure IPMI to use one of the LAN1/2 interfaces, instead of the IPMI port? If so, what is the procedure?
Jiri's on the right track with the three options (Dedicated, Share, Failover) for the IPMI interface. The short answer is that yes, you can use LAN1 instead of the dedicated IPMI port, and it generally works that way with the default BIOS settings. It's not possible to run the IPMI on the LAN2 interface.
Here's a more detailed description of the three options:
Dedicated: Always use the dedicated IPMI interface. This is the option you want if you're trying to have the simplest setup, at the expense of additional cabling.
Shared: Always use the LAN1 interface. This is the option you want if you're trying to reduce your cabling to each server, and understand the tradeoffs. Under the covers, there's a virtual switch in hardware that's splitting out traffic to the IPMI card from traffic to the rest of the system; the IPMI card has a separate MAC address to differentiate the traffic. On modern Supermicro boards, you can also set the IPMI traffic to run on a different VLAN from the rest of the system, so you can tag the IPMI traffic. There are some definite security implication to this design; it's not difficult for the main system to access the IPMI network, if you were trying to keep them separated. A failure of the LAN1 interface often means that you lose primary and out-of-band connectivity at the same time.
Failover (factory default): On boot, detect if the dedicated IPMI interface is connected. If so, use the dedicated interface, otherwise fall back to the shared LAN1. I've never found a good use for this option. As best I can tell, this setup is fundamentally flawed - I haven't tested it extensively, but I've heard reports it'll fail to detect the dedicated interface in many circumstances because the upstream switch isn't passing traffic - for example, after a power outage if the switch and system come up simultaneously, or if the switch is still blocking during the spanning tree detection. Combine this with the fact that the check only happens at boot, and it's just generally hard to control what interface you end up using.
The following raw commands will surely prove useful to the next person who reads this topic. I got these directly from Supermicro support. Cheers:
To get LAN mode: ipmitool raw 0x30 0x70 0x0c 0.
To set LAN mode dedicated: ipmitool raw 0x30 0x70 0x0c 1 0.
To set LAN mode onboard/shared: ipmitool raw 0x30 0x70 0x0c 1 1.
To set LAN mode failover: ipmitool raw 0x30 0x70 0x0c 1 2.
These raw values will also work with ipmicfg (just remember to use -raw instead of raw).
After setting the required LAN mode, do not forget to do a hard reboot.
In these boards, the ipmi interface can only be configured on the 1rst LAN or in the dedicated one, and you can choose it by issuing some raw commands. My steps to enable the dedicated LAN on a X8DTU-F board are these:
#install ipmitool (this is for debian)
apt-get install ipmitool
#insert the kernel modules needed for ipmi
modprobe ipmi_devintf
modprobe ipmi_si
modprobe ipmi_msghandler
#get the current mode (01 00 is dedicated mode)
ipmitool raw 0x30 0x70 0x0c 0
#send the raw command to enable dedicated lan
ipmitool raw 0x30 0x70 0xc 1 1 0
You can now use ipmitool to assign an ipaddress/netmask to the interface, and start working with it normally.
If you want to know all the comands in raw mode for the supermicro ipmi, there's a list here
I figured it out. The way it works is pretty hokey, but here it is:
It depends on whether there is a network cable plugged into the dedicated IPMI port, before the server receives power. If a cable is present, then IPMI is automatically run on that port. If there is no cable in that port, then IPMI is automatically assigned to the LAN1 port (shared with the operating system).
These settings are not persistent through power cycles unless you make them permanent using IPMI View tool > BMC Settings. If you do not do this, then every time the server gets power cycled, it will test the cable again and reassign IPMI accordingly.
You can also use ipmitool to set the lanport:
sudo ipmiutil smcoem lanport dedicated
Then check that your config is right:
sudo ipmiutil lan -e
You should see the line:
SuperMicro Lan Interface : 00 : Dedicated
Cheers!