I updated my CentOS 7 system. Why is Meltdown/Spectre only partially mitigated?
Like many of us, I spent yesterday updating a whole lot of systems to mitigate the Meltdown and Spectre attacks. As I understand it, it is necessary to install two packages and reboot:
kernel-3.10.0-693.11.6.el7.x86_64
microcode_ctl-2.1-22.2.el7.x86_64
I have two CentOS 7 systems on which I've installed these packages and rebooted.
According to Red Hat, I can check the status of mitigation by checking these sysctls and ensuring that they are all 1. However, on these systems, they are not all 1:
# cat /sys/kernel/debug/x86/pti_enabled
1
# cat /sys/kernel/debug/x86/ibpb_enabled
0
# cat /sys/kernel/debug/x86/ibrs_enabled
0
And I can't set them to 1, either:
# echo 1 > /sys/kernel/debug/x86/ibpb_enabled
-bash: echo: write error: No such device
# echo 1 > /sys/kernel/debug/x86/ibrs_enabled
-bash: echo: write error: No such device
I confirmed that Intel microcode appears to have loaded on boot:
# systemctl status microcode -l
● microcode.service - Load CPU microcode update
Loaded: loaded (/usr/lib/systemd/system/microcode.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2018-01-05 16:42:25 UTC; 9min ago
Process: 30383 ExecStart=/usr/bin/bash -c grep -l GenuineIntel /proc/cpuinfo | xargs grep -l -E "model[[:space:]]*: 79$" > /dev/null || echo 1 > /sys/devices/system/cpu/microcode/reload (code=exited, status=0/SUCCESS)
Main PID: 30383 (code=exited, status=0/SUCCESS)
Jan 05 16:42:25 makrura systemd[1]: Starting Load CPU microcode update...
Jan 05 16:42:25 makrura systemd[1]: Started Load CPU microcode update.
Even dmesg seems to have confirmed it:
[ 3.245580] microcode: CPU0 sig=0x50662, pf=0x10, revision=0xf
[ 3.245627] microcode: CPU1 sig=0x50662, pf=0x10, revision=0xf
[ 3.245674] microcode: CPU2 sig=0x50662, pf=0x10, revision=0xf
[ 3.245722] microcode: CPU3 sig=0x50662, pf=0x10, revision=0xf
[ 3.245768] microcode: CPU4 sig=0x50662, pf=0x10, revision=0xf
[ 3.245816] microcode: CPU5 sig=0x50662, pf=0x10, revision=0xf
[ 3.245869] microcode: CPU6 sig=0x50662, pf=0x10, revision=0xf
[ 3.245880] microcode: CPU7 sig=0x50662, pf=0x10, revision=0xf
[ 3.245924] microcode: CPU8 sig=0x50662, pf=0x10, revision=0xf
[ 3.245972] microcode: CPU9 sig=0x50662, pf=0x10, revision=0xf
[ 3.245989] microcode: CPU10 sig=0x50662, pf=0x10, revision=0xf
[ 3.246036] microcode: CPU11 sig=0x50662, pf=0x10, revision=0xf
[ 3.246083] microcode: CPU12 sig=0x50662, pf=0x10, revision=0xf
[ 3.246131] microcode: CPU13 sig=0x50662, pf=0x10, revision=0xf
[ 3.246179] microcode: CPU14 sig=0x50662, pf=0x10, revision=0xf
[ 3.246194] microcode: CPU15 sig=0x50662, pf=0x10, revision=0xf
[ 3.246273] microcode: Microcode Update Driver: v2.01 <[email protected]>, Peter Oruba
I have an Intel CPU formerly code named Broadwell:
processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 86
model name : Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
stepping : 2
microcode : 0xf
cpu MHz : 2499.921
cache size : 12288 KB
physical id : 0
siblings : 16
core id : 7
cpu cores : 8
apicid : 15
initial apicid : 15
fpu : yes
fpu_exception : yes
cpuid level : 20
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
bogomips : 3999.90
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
The cpuid utility reports:
# cpuid -1
Disclaimer: cpuid may not support decoding of all cpuid registers.
CPU:
vendor_id = "GenuineIntel"
version information (1/eax):
processor type = primary processor (0)
family = Intel Pentium Pro/II/III/Celeron/Core/Core 2/Atom, AMD Athlon/Duron, Cyrix M2, VIA C3 (6)
model = 0x6 (6)
stepping id = 0x2 (2)
extended family = 0x0 (0)
extended model = 0x5 (5)
(simple synth) = Intel Xeon D-1500 (Broadwell-DE V1), 14nm
miscellaneous (1/ebx):
process local APIC physical ID = 0x9 (9)
cpu count = 0x10 (16)
CLFLUSH line size = 0x8 (8)
brand index = 0x0 (0)
brand id = 0x00 (0): unknown
feature information (1/edx):
x87 FPU on chip = true
virtual-8086 mode enhancement = true
debugging extensions = true
page size extensions = true
time stamp counter = true
RDMSR and WRMSR support = true
physical address extensions = true
machine check exception = true
CMPXCHG8B inst. = true
APIC on chip = true
SYSENTER and SYSEXIT = true
memory type range registers = true
PTE global bit = true
machine check architecture = true
conditional move/compare instruction = true
page attribute table = true
page size extension = true
processor serial number = false
CLFLUSH instruction = true
debug store = true
thermal monitor and clock ctrl = true
MMX Technology = true
FXSAVE/FXRSTOR = true
SSE extensions = true
SSE2 extensions = true
self snoop = true
hyper-threading / multi-core supported = true
therm. monitor = true
IA64 = false
pending break event = true
feature information (1/ecx):
PNI/SSE3: Prescott New Instructions = true
PCLMULDQ instruction = true
64-bit debug store = true
MONITOR/MWAIT = true
CPL-qualified debug store = true
VMX: virtual machine extensions = true
SMX: safer mode extensions = true
Enhanced Intel SpeedStep Technology = true
thermal monitor 2 = true
SSSE3 extensions = true
context ID: adaptive or shared L1 data = false
FMA instruction = true
CMPXCHG16B instruction = true
xTPR disable = true
perfmon and debug = true
process context identifiers = true
direct cache access = true
SSE4.1 extensions = true
SSE4.2 extensions = true
extended xAPIC support = true
MOVBE instruction = true
POPCNT instruction = true
time stamp counter deadline = true
AES instruction = true
XSAVE/XSTOR states = true
OS-enabled XSAVE/XSTOR = true
AVX: advanced vector extensions = true
F16C half-precision convert instruction = true
RDRAND instruction = true
hypervisor guest status = false
cache and TLB information (2):
0x63: data TLB: 1G pages, 4-way, 4 entries
0x03: data TLB: 4K pages, 4-way, 64 entries
0x76: instruction TLB: 2M/4M pages, fully, 8 entries
0xff: cache data is in CPUID 4
0xb5: instruction TLB: 4K, 8-way, 64 entries
0xf0: 64 byte prefetching
0xc3: L2 TLB: 4K/2M pages, 6-way, 1536 entries
processor serial number: 0005-0662-0000-0000-0000-0000
deterministic cache parameters (4):
--- cache 0 ---
cache type = data cache (1)
cache level = 0x1 (1)
self-initializing cache level = true
fully associative cache = false
extra threads sharing this cache = 0x1 (1)
extra processor cores on this die = 0x7 (7)
system coherency line size = 0x3f (63)
physical line partitions = 0x0 (0)
ways of associativity = 0x7 (7)
ways of associativity = 0x0 (0)
WBINVD/INVD behavior on lower caches = false
inclusive to lower caches = false
complex cache indexing = false
number of sets - 1 (s) = 63
--- cache 1 ---
cache type = instruction cache (2)
cache level = 0x1 (1)
self-initializing cache level = true
fully associative cache = false
extra threads sharing this cache = 0x1 (1)
extra processor cores on this die = 0x7 (7)
system coherency line size = 0x3f (63)
physical line partitions = 0x0 (0)
ways of associativity = 0x7 (7)
ways of associativity = 0x0 (0)
WBINVD/INVD behavior on lower caches = false
inclusive to lower caches = false
complex cache indexing = false
number of sets - 1 (s) = 63
--- cache 2 ---
cache type = unified cache (3)
cache level = 0x2 (2)
self-initializing cache level = true
fully associative cache = false
extra threads sharing this cache = 0x1 (1)
extra processor cores on this die = 0x7 (7)
system coherency line size = 0x3f (63)
physical line partitions = 0x0 (0)
ways of associativity = 0x7 (7)
ways of associativity = 0x0 (0)
WBINVD/INVD behavior on lower caches = false
inclusive to lower caches = false
complex cache indexing = false
number of sets - 1 (s) = 511
--- cache 3 ---
cache type = unified cache (3)
cache level = 0x3 (3)
self-initializing cache level = true
fully associative cache = false
extra threads sharing this cache = 0xf (15)
extra processor cores on this die = 0x7 (7)
system coherency line size = 0x3f (63)
physical line partitions = 0x0 (0)
ways of associativity = 0xb (11)
ways of associativity = 0x6 (6)
WBINVD/INVD behavior on lower caches = false
inclusive to lower caches = true
complex cache indexing = true
number of sets - 1 (s) = 16383
MONITOR/MWAIT (5):
smallest monitor-line size (bytes) = 0x40 (64)
largest monitor-line size (bytes) = 0x40 (64)
enum of Monitor-MWAIT exts supported = true
supports intrs as break-event for MWAIT = true
number of C0 sub C-states using MWAIT = 0x0 (0)
number of C1 sub C-states using MWAIT = 0x2 (2)
number of C2 sub C-states using MWAIT = 0x1 (1)
number of C3 sub C-states using MWAIT = 0x2 (2)
number of C4 sub C-states using MWAIT = 0x0 (0)
number of C5 sub C-states using MWAIT = 0x0 (0)
number of C6 sub C-states using MWAIT = 0x0 (0)
number of C7 sub C-states using MWAIT = 0x0 (0)
Thermal and Power Management Features (6):
digital thermometer = true
Intel Turbo Boost Technology = true
ARAT always running APIC timer = true
PLN power limit notification = true
ECMD extended clock modulation duty = true
PTM package thermal management = true
HWP base registers = false
HWP notification = false
HWP activity window = false
HWP energy performance preference = false
HWP package level request = false
HDC base registers = false
digital thermometer thresholds = 0x2 (2)
ACNT/MCNT supported performance measure = true
ACNT2 available = false
performance-energy bias capability = true
extended feature flags (7):
FSGSBASE instructions = true
IA32_TSC_ADJUST MSR supported = true
SGX: Software Guard Extensions supported = false
BMI instruction = true
HLE hardware lock elision = true
AVX2: advanced vector extensions 2 = true
FDP_EXCPTN_ONLY = false
SMEP supervisor mode exec protection = true
BMI2 instructions = true
enhanced REP MOVSB/STOSB = true
INVPCID instruction = true
RTM: restricted transactional memory = true
QM: quality of service monitoring = true
deprecated FPU CS/DS = true
intel memory protection extensions = false
PQE: platform quality of service enforce = true
AVX512F: AVX-512 foundation instructions = false
AVX512DQ: double & quadword instructions = false
RDSEED instruction = true
ADX instructions = true
SMAP: supervisor mode access prevention = true
AVX512IFMA: fused multiply add = false
CLFLUSHOPT instruction = false
CLWB instruction = false
Intel processor trace = true
AVX512PF: prefetch instructions = false
AVX512ER: exponent & reciprocal instrs = false
AVX512CD: conflict detection instrs = false
SHA instructions = false
AVX512BW: byte & word instructions = false
AVX512VL: vector length = false
PREFETCHWT1 = false
AVX512VBMI: vector byte manipulation = false
UMIP: user-mode instruction prevention = false
PKU protection keys for user-mode = false
OSPKE CR4.PKE and RDPKRU/WRPKRU = false
BNDLDX/BNDSTX MAWAU value in 64-bit mode = 0x0 (0)
RDPID: read processor D supported = false
SGX_LC: SGX launch config supported = false
AVX512_4VNNIW: neural network instrs = false
AVX512_4FMAPS: multiply acc single prec = false
Direct Cache Access Parameters (9):
PLATFORM_DCA_CAP MSR bits = 1
Architecture Performance Monitoring Features (0xa/eax):
version ID = 0x3 (3)
number of counters per logical processor = 0x4 (4)
bit width of counter = 0x30 (48)
length of EBX bit vector = 0x7 (7)
Architecture Performance Monitoring Features (0xa/ebx):
core cycle event not available = false
instruction retired event not available = false
reference cycles event not available = false
last-level cache ref event not available = false
last-level cache miss event not avail = false
branch inst retired event not available = false
branch mispred retired event not avail = false
Architecture Performance Monitoring Features (0xa/edx):
number of fixed counters = 0x3 (3)
bit width of fixed counters = 0x30 (48)
x2APIC features / processor topology (0xb):
--- level 0 (thread) ---
bits to shift APIC ID to get next = 0x1 (1)
logical processors at this level = 0x2 (2)
level number = 0x0 (0)
level type = thread (1)
extended APIC ID = 9
--- level 1 (core) ---
bits to shift APIC ID to get next = 0x4 (4)
logical processors at this level = 0x10 (16)
level number = 0x1 (1)
level type = core (2)
extended APIC ID = 9
XSAVE features (0xd/0):
XCR0 lower 32 bits valid bit field mask = 0x00000007
XCR0 upper 32 bits valid bit field mask = 0x00000000
XCR0 supported: x87 state = true
XCR0 supported: SSE state = true
XCR0 supported: AVX state = true
XCR0 supported: MPX BNDREGS = false
XCR0 supported: MPX BNDCSR = false
XCR0 supported: AVX-512 opmask = false
XCR0 supported: AVX-512 ZMM_Hi256 = false
XCR0 supported: AVX-512 Hi16_ZMM = false
IA32_XSS supported: PT state = false
XCR0 supported: PKRU state = false
bytes required by fields in XCR0 = 0x00000340 (832)
bytes required by XSAVE/XRSTOR area = 0x00000340 (832)
XSAVE features (0xd/1):
XSAVEOPT instruction = true
XSAVEC instruction = false
XGETBV instruction = false
XSAVES/XRSTORS instructions = false
SAVE area size in bytes = 0x00000000 (0)
IA32_XSS lower 32 bits valid bit field mask = 0x00000000
IA32_XSS upper 32 bits valid bit field mask = 0x00000000
AVX/YMM features (0xd/2):
AVX/YMM save state byte size = 0x00000100 (256)
AVX/YMM save state byte offset = 0x00000240 (576)
supported in IA32_XSS or XCR0 = XCR0 (user state)
64-byte alignment in compacted XSAVE = false
Quality of Service Monitoring Resource Type (0xf/0):
Maximum range of RMID = 63
supports L3 cache QoS monitoring = false
L3 Cache Quality of Service Monitoring (0xf/1):
Conversion factor from IA32_QM_CTR to bytes = 32768
Maximum range of RMID = 63
supports L3 occupancy monitoring = true
supports L3 total bandwidth monitoring = true
supports L3 local bandwidth monitoring = true
Resource Director Technology allocation (0x10/0):
L3 cache allocation technology supported = true
L2 cache allocation technology supported = false
L3 Cache Allocation Technology (0x10/1):
length of capacity bit mask - 1 = 0xb (11)
Bit-granular map of isolation/contention = 0x00000c00
infrequent updates of COS = true
code and data prioritization supported = false
highest COS number supported = 0xb (11)
0x00000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
SGX capability (0x12/0):
SGX1 supported = false
SGX2 supported = false
MISCSELECT.EXINFO supported: #PF & #GP = false
MaxEnclaveSize_Not64 (log2) = 0x0 (0)
MaxEnclaveSize_64 (log2) = 0x0 (0)
0x00000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
Intel Processor Trace (0x14):
IA32_RTIT_CR3_MATCH is accessible = true
configurable PSB & cycle-accurate = false
IP & TraceStop filtering; PT preserve = false
MTC timing packet; suppress COFI-based = false
PTWRITE support = false
power event trace support = false
IA32_RTIT_CTL can enable tracing = true
ToPA can hold many output entries = false
single-range output scheme = false
output to trace transport = false
IP payloads have LIP values & CS = false
extended feature flags (0x80000001/edx):
SYSCALL and SYSRET instructions = true
execution disable = true
1-GB large page support = true
RDTSCP = true
64-bit extensions technology available = true
Intel feature flags (0x80000001/ecx):
LAHF/SAHF supported in 64-bit mode = true
LZCNT advanced bit manipulation = true
3DNow! PREFETCH/PREFETCHW instructions = true
brand = "Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz"
L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax):
instruction # entries = 0x0 (0)
instruction associativity = 0x0 (0)
data # entries = 0x0 (0)
data associativity = 0x0 (0)
L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx):
instruction # entries = 0x0 (0)
instruction associativity = 0x0 (0)
data # entries = 0x0 (0)
data associativity = 0x0 (0)
L1 data cache information (0x80000005/ecx):
line size (bytes) = 0x0 (0)
lines per tag = 0x0 (0)
associativity = 0x0 (0)
size (KB) = 0x0 (0)
L1 instruction cache information (0x80000005/edx):
line size (bytes) = 0x0 (0)
lines per tag = 0x0 (0)
associativity = 0x0 (0)
size (KB) = 0x0 (0)
L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax):
instruction # entries = 0x0 (0)
instruction associativity = L2 off (0)
data # entries = 0x0 (0)
data associativity = L2 off (0)
L2 TLB/cache information: 4K pages & L2 TLB (0x80000006/ebx):
instruction # entries = 0x0 (0)
instruction associativity = L2 off (0)
data # entries = 0x0 (0)
data associativity = L2 off (0)
L2 unified cache information (0x80000006/ecx):
line size (bytes) = 0x40 (64)
lines per tag = 0x0 (0)
associativity = 8-way (6)
size (KB) = 0x100 (256)
L3 cache information (0x80000006/edx):
line size (bytes) = 0x0 (0)
lines per tag = 0x0 (0)
associativity = L2 off (0)
size (in 512KB units) = 0x0 (0)
Advanced Power Management Features (0x80000007/edx):
temperature sensing diode = false
frequency ID (FID) control = false
voltage ID (VID) control = false
thermal trip (TTP) = false
thermal monitor (TM) = false
software thermal control (STC) = false
100 MHz multiplier control = false
hardware P-State control = false
TscInvariant = true
Physical Address and Linear Address Size (0x80000008/eax):
maximum physical address bits = 0x2e (46)
maximum linear (virtual) address bits = 0x30 (48)
maximum guest physical address bits = 0x0 (0)
Logical CPU cores (0x80000008/ecx):
number of CPU cores - 1 = 0x0 (0)
ApicIdCoreIdSize = 0x0 (0)
(multi-processing synth): multi-core (c=8), hyper-threaded (t=2)
(multi-processing method): Intel leaf 0xb
(APIC widths synth): CORE_width=4 SMT_width=1
(APIC synth): PKG_ID=0 CORE_ID=4 SMT_ID=1
(synth) = Intel Xeon D-1500 (Broadwell-DE V1), 14nm
The system is fully up to date:
# yum upgrade
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: centos.mirror.colo-serv.net
* epel: mirror.steadfast.net
* extras: centos.mirror.colo-serv.net
* updates: centos.mirror.colo-serv.net
No packages marked for update
I feel like I've missed something important, but at this point I really don't know what it could be. What's going on here? How do I get the system fully mitigated?
I am also seeing the same behavior on Fedora 27 workstations, a desktop with a Core i7-3770 CPU and a laptop with a Core i7-7500U.
Solution 1:
As noted in https://access.redhat.com/articles/3311301
CVE-2017-5715 (variant #2/Spectre) is an indirect branching poisoning attack that can lead to data leakage. This attack allows for a virtualized guest to read memory from the host system. This issue is corrected with microcode, along with kernel and virtualization updates to both guest and host virtualization software. This vulnerability requires both updated microcode and kernel patches. Variant #2 behavior is controlled by the ibrs and ibpb tunables (noibrs/ibrs_enabled and noibpb/ibpb_enabled), which work in conjunction with the microcode
...
As noted, installing the microcode update for your hardware, if provided by the hardware vendor, is necessary to protect against variant 2. Please contact your hardware vendor for microcode updates.
It seems that you also need a BIOS update to enable the mitigations for CVE-2017-5715.
I read this elsewhere too earlier but can't find the reference right now.
Solution 2:
UPDATE: it seems the microcode update released with OS updates are binary patches to the running microcode, rather than full-blown microcode replacement. In other words, they require a specific, base ucode version in the BIOS/processor to be able to patch it. For this reason, on all machines I had at hand a BIOS/firmware update is required, and this very BIOS update already bring the patched microcode embedded. For reference, DELL just released the required firmware update for 14/13/12th generation PowerEdge servers, and fix for 11th generation server will be released by the end of the month.
TL;DR: the microcode updates seem very rushed. I think we must wait some days/weeks to fully understand/discover which processors/steppings/models/SKU will receive the correct update.
Long version: of three updated systems, a single one (the older) correctly loaded the new microcode and enabled the relative mitigations:
- on a Ryzen box the loading failed with a "patch mismatch" error;
- on a Clarkdale (Core i5) laptop, no new microcode was loaded, yet the correct files are installed on
/lib/firmware/ - a very old PhenomII box correctly loaded the new microcode and enabled all appropriate mitigations.