Dell PowerEdge T-620 - Choosing Memory Modes in the BIOS

I'm trying to configure the Memory Mode, in the BIOS, of a Dell PowerEdge T620 with dual processors: Owner's Manual | Technical Guide

The system has 384GB of DDR3 LRDIMM Memory and I'm able to choose between any of the following modes:

1. Optimizer Mode
2. Spare Mode
3. Mirror Mode
4. Advanced ECC Mode
5. Spare with Advanced ECC
6. Dell Fault Resilient Mode

In addition to this, since all the DIMMs are identical, the memory configuration allows me to set Node Interleaving to "enabled" or "disabled", with the following description of node interleaving:

If the system is configured with matching memory, this field enables node interleaving. If set to Disabled (the default), the system supports Non-Uniform Memory architecture (NUMA) (asymetric) memory configurations. This field is active only if the memory configuration can support interleaving.

This system will be running Proxmox to host virtual machines.

Which mode would you choose and why? Would you enable node interleaving and why?

from mellanox

Node Interleaving - When node interleaving is disabled, NUMA mode is enabled. Conversely, enabling Node Interleaving means that memory is interleaved between memory nodes, and there is no NUMA presentation to the operating system. For performance reasons, we wish to disable interleaving (and enable NUMA), thus ensuring that memory is always allocated to the local NUMA node for any given logical processor.

An attempt at describing what's happening:

• simplest case you have one CPU and two (or four) DIMM slots. The memory controller is now within the CPU so ideally when a core within a given CPU needs data from RAM you want it to fetch it from the closest RAM- from it's memory controller to its nearest DIMM slot. Nothing to worry about in this one cpu scenario.
• you have two (or more) CPUs, two (or more) different memory controllers, but they can access RAM through the other CPU memory controller to get to its DIMM and access 500+ GB of total ram or whatever you may have. So what do you do...
  • there are two ways to do this (I think). The simple way, just interleave and always access every other CPU's DIMMS but at a penalty: memory timings come into play and so you just always use the slowest (longest) access time as the same access time for everything. Otherwise you would be restricted to total ram = just what's available to one cpu, not from 2, 4, or 8 cpu's. So when accessing local ram via local dimms to the memory controller of the cpu where a process is, it could have been fast but now is the (long) time based on accessing the farthest DIMM, so that all RAM can show up as shared memory. That I believe is why interleaved = performance hit. NUMA = non-uniform-memory-access. Because memory access can be however far away the timings would inherently be different and non uniform. But the BIOS/EFI and CPU memory controllers can handle all that via the NUMA model. Which is why everything is now NUMA. I'm pretty sure you don't want interleave; don't use interleave unless you get instruction from a credible source to use it for a specific reason that calls for it. I would be interested in hearing of such a reason if anyone knows of one.

It is a fact that ECC memory is slower than non-ecc regular desktop ram, because it has to do error checking. but don't take this out of context. if you value reliability and dependability then probably use advanced ECC mode. But if you're goal is all out performance then advanced (or any) ECC mode may not be ideal and the best choice I would suspect would be that #1 optimizer mode for your specific box... I would have to read Dell's definition of there menu offerings.

I prefer Advanced ECC. Optimized mode supports Single Device Data Correction (SDDC) only for memory modules that use x4 device width. It does not impose any specific slot population requirements. Advanced Error Correction Code (ECC) mode extends SDDC from x4 DRAM based DIMMs to both x4 and x8 DRAMs. This protects against single DRAM chip failures during normal operation. The restriction is that memory modules must be identical in size, speed, and technology. Identical DIMMs are installed in matched pair. Since you can order a server from start with this type of module compliance, there's no problem using AECC instead of optimized.

Note that both optimized and AECC modes will let the full amount of memory be usable.

If power is fine which it should be on a server, there is no actual reason for RAM modules to have problems, the probability of that is low.

If you want to make sure you don't have to open up the server in the low chance case of a memory failure, you may choose spare mode. In this mode, one rank per channel is reserved as a spare. If persistent correctable errors are detected on a rank, the data from this rank is copied to the spare rank, and the failed rank is disabled. The memory available to the operating system is reduced by one rank per channel.

For example, in a dual-processor configuration with 16x 4 GB single-rank memory modules, the available system memory is: 3/4 (ranks/channel) × 16 (memory modules) × 4 GB = 48 GB, and not 16 1x (memory modules) × 4 GB = 64 GB.

Mirroring is like a RAID 1 for memory. The bad part is that in a mirrored configuration, the total available system memory is one half of the total installed physical memory, exactly like with HDDs in RAID 1. So if you really need maximum protection you an choose this one but in your case you will have only 192 GB (half) to work with.

Fault Resilient Memory (FRM) is an operating mode introduced in newer PowerEdge servers. The mode establishes an area of memory that is fault resilient and can be used by hypervisors such as VMware ESXi to load vmkernel, critical applications or services to maximize system availability. The operating systems uses the resilient region exposed from the platform and map the process’s address ranges to enforce resiliency. This is good to be used for ESXi installations. FRM creates a highly resilient memory zone for the hypervisor, protecting it from severe memory errors. With the VMware Reliable Memory feature, vSphere 5.5 and later revisions can take advantage of this zone, providing the hypervisor with a strong protection from memory faults that would bring down the entire system.