Bluetooth lagging when Magic Mouse and a bluetooth speaker connected

I don't know that there's anything you can do about this—speakers just use up a lot of Bluetooth's available bandwidth. While I've never had trouble with JUST a speaker and mouse, I've found that if I connect a Bluetooth mouse, keyboard, and speaker at once, the speaker becomes horribly stuttery until either the mouse or keyboard is disconnected.

You say you've already tried different speakers; I might try different mice as well. You could try a USB Bluetooth adapter, but there's no guarantee that it will be any better. Also, if you have any additional Bluetooth devices you didn't tell us about, definitely disconnect those as well.

(At one point, I set up Eventscripts to automatically disconnect my keyboard whenever a Bluetooth speaker was connected. It worked, but was somewhat fragile, and I don't recommend it.)

You could also try lowering macOS's Bluetooth audio quality in the terminal so it uses less bandwidth. I have never been able to get acceptable results this way, but it could be worth a try. https://lifehacker.com/fix-your-bluetooth-audio-in-yosemite-with-this-terminal-1670380974

You're not going to like this solution, but I would recommend replacing either the mouse or speaker with a non-Bluetooth alternative, such as a wireless mouse with a USB dongle or a wifi speaker that works over airplay.

Sorry this isn't the answer you wanted.

Check the battery level in your keyboard or mouse. I've found this happens consistently when the peripheral battery is running low and the device seems to start spamming the Bluetooth connection with "help I'm running out of juice" requests. Swapping out your batteries for full ones will reliably fix the issue.

I had a related problem which was driving me nuts, & it took forever to track down. Seems wifi was stopping my Apple Magic Mouse from working - turns out the macBook was sharing my connection on channel 11, & as soon as I stopped wifi sharing the mouse worked great. I've got it sharing on channel 48 (5GHz) now, & so far everything works perfectly.

As bluetooth & wifi can share the same frequency band, it makes sense to use the 5GHz band for wifi, gives your bluetooth stuff more space.