Does Mac OS X throttle the RATE of socket creation?
So it turns out the Mac OS X ephemeral port range is fairly low.
Wikipedia informs me that IANA suggests 49152 to 65535 as "dynamic and/or private ports" while many Linux kernels use 32768 to 61000. OS X uses the IANA range. This means Linux has almost twice the available ephemeral ports. Since each closed socket goes through a TIME_WAIT state (that I didn't know about) the rate is just overwhelming my system.
How to fix?
sudo sysctl -w net.inet.ip.portrange.first=32768
sudo sysctl -w net.inet.ip.portrange.hifirst=32768
This will give about double the range.
(Thanks to Spiff who answered in more detail here: https://superuser.com/questions/145989/does-mac-os-x-throttle-the-rate-of-socket-creation)