USB audio ducks when Notifications happen

I'm using a USB DAC (FiiO BTRK3). When my Mac gets a notification, it ducks the rest of the audio for about 2 seconds. It's a big duck too; All the way to 0 it seems.

• It doesn't happen when I'm using DAC as a BlueTooth device.
• Disabling audio ducking in VoiceOver Utility doesn't help.

What can I do to stop the ducking?

I still want the notifications. I still want to use it as a USB device.

I'm on Mojave 10.14.2 right now

The bluetooth receiver was also paired with my phone at the same time as the mac.

I've unpaired the phone and it stops the ducking.

I figured this out when a call came in and the ring tone played full blast and started trying to figure out how to prevent that from ever happening again.