How to enable/disable auto AirPod switching on phone call?

Finally figured it out. It was an accessibility option.

  1. Navigate to Settings > Accessibility > Touch > Call Audio Routing
  2. Select "Automatic". This will only use the AirPods when they are connected to the phone. I had it on "Bluetooth Headset".