Why is 1 of my 24 CPUs Pegged at 100%?

As others have already pointed out, we can see from that screenshot that the CPU that's working so hard is spending all its time in kernel mode. (The red color.)

Running Powershell as administrator, type:

Get-Process | Select Name, PrivilegedProcessorTime | `
Sort-Object PrivilegedProcessorTime -Descending

The process at the top of the list is the process currently using the most kernel mode CPU time right now. If that process is not "System," then you've just figured out what user mode process is causing this CPU usage. If the process with the highest Privileged Processor Time is System, which I suspect it is, then it's a little more complicated.

Open Process Explorer. Optionally, set up your symbol server. Make sure you are running with full UAC elevation. Right click the System "process" and go to Properties. Then go to the Threads tab. Sort the threads by CPU usage. The thread that's causing all this kernel mode work should be here. If you look at the module listed under Start Address, it should give you a clue as to what the work is related to. If it's NDIS.sys, for instance, that's a network interface driver. If you set up the symbol server, you should see the name of a function within a module (unless the module is non-Microsoft,) else you'll just see a numerical offset from the module's start address.

Alternatively, use Xperf from the Windows Performance Toolkit to profile interrupts, DPCs, etc.

xperf -on PROC_THREAD+LOADER+DPC+INTERRUPT

and stop recording with xperf -d logfile.etl

Xperf replaces the old Kernrate tool, and can net you some extremely detailed data.

When a CPU is doing work in kernel mode, it's mostly running interrupt service routines. (ISRs) When an interrupt occurs, user mode work is suspended on that processor, and the CPU runs the ISR registered to that interrupt. If you find your CPU spending an inordinate amount of time on these interrupts, that usually indicates a faulty device driver that needs to be updated.

What bugs me (no pun intended) about this scenario though is that it appears as though whatever kernel thread that is doing this seems to be affinitized to that one core. I wonder why the dispatcher seems to be only scheduling the thread to run on that one seemingly arbitrary core. So I have a feeling that we need to find whoever wrote this device driver and show them how to do threaded DPCs, and not to explicitly set an affinity on kernel threads, etc.

Show the "CPU Time" column on the "Details" tab in "Task Manager" and look for a process with a CPU time count that's steadily increasing. That's your wedged process. It should be using around 4.17% CPU constantly.