Do PCI / PCIe buses and devices always enumerate in the same order?

I am looking into a situation where there are multiple, identical (National Instruments) PCIe DAQ modules in a PCI/PCIe chassis, connected via either a single or dual chassis controller. In the worst case scenario there can be 8 identical modules on the bus.

I have been led to believe that PCI enumeration is non-deterministic - i.e. that there is no guarantee that, boot to boot, the PCI bus will be enumerated in the same order. Is this true?

Does this also imply that PCI(e) bus/device/function values can change, boot to boot, even if the physical hardware layout does not?

In short, I understand that initial PCI enumeration is done by the BIOS, and that the boot OS (RHEL 6, in my case) uses this enumerated list to service identified hardware. Does this list always contain the found devices in the same order or not?

This is a hint in Red Hat Bugzilla that implies that BIOS ordering is always the same.

Please note that I am not (yet) talking about the order in which the OS handles the devices (e.g. for, in Linux, determining the /dev/ name order).

Thanks!

Solution 1:

The short answer is "Yes", if the BIOS (which performs initial enumeration) adheres to PCI specification.

According to: "PCI Express System Architecture" R. Budruk, D. Anderson, T. Shanley, ADDISON-WESLEY DEVELOPER´S PRESS, 2003. ISBN: 0-321-15630-7, page 743:

The specification states that the enumeration software must perform a depth-first search, so before proceeding to discover additional functions/ devices on bus 0, it must proceed to search bus 1.

And for RHEL 6:

9. Devices and Device Drivers

PCI Device Ordering

In Red Hat Enterprise Linux 6, the PCI device ordering is based on the PCI device enumeration. PCI device enumeration is based on the PCI enumeration algorithm (depth first then breadth) and is constant per system type. Additionally, once the devices are discovered, the module loading process is sequentialized, providing persistent naming of the interfaces.

However, (some versions of?) Linux support a kernel parameter "pci=bfsort" which changes the Linux enumeration algorithm to "breadth first". This will change the order that the Linux kernel loads and initialises PCI devices wrt how BIOS does.

In any case, from boot-to-boot, enumerated lists will still contain detected items in the same order.