Problem with XBox One headset when switching controllers

I've had an issue with my XBox One controller and headset since day one and although it's gotten better with all the system and controller updates, there's still an outstanding problem.

When playing a game with friends in a party, my headset works perfectly. I can hear people, they can hear me and the quality is great. My controller's battery is almost dead so I turn the controller off to swap to my backup. The headset is plugged into the controller when I turn it off. I plug the headset into the new controller and turn that controller on.

As soon as I turn the new controller on, the headset no longer functions at all. I start hearing people through my TV and no one can hear me (because I have kinect mic turned off). I try turning the controller off and on again, I try unplugging the headset and plugging it back in. Nothing works except for turning off the console and turning it back on with the controller and connected headset.

Has anyone else had this issue and if so, is there a fix, am I doing something wrong?

Your problem is most likely due to the controller not being assigned to your person.

The other controller, even though it has a headset plugged in is treated as belonging to a guest (or other profile if it is signed in) and, hence, will not have access to the party.

You can fix the issue by pairing the controller with your person manually. However, reading more on the topic, it seems that if you have set up Kinect Sign-in, the console will automatically recognise that you picked up a different controller and will assign it to you as you would expect.

Source: How to assign a profile to an Xbox One Wireless Controller

I hated this problem and couldn't figure out out for the longest time but I think I just found an easy fix. What worked for me was simply pressing the center xbox button (home) going to the top left of the screen to your profile (as if you were trying to switch profiles) selecting your profile and selecting "choose this person" (even though you are already signed in) hope it helps, it was driving me insane.