Why is my headset reducing sound volume/quality when recording my voice?

TL;DR: Bluetooth is limited in bandwidth.

To be able to let both incoming (microphone) and outgoing (speakers) audio data, your headset can use multiple audio codecs.

Depending on your usage, it automatically switch from a codec set to another. The output quality gets worse (which use less bandwith) to allow input data from you microphone to get to your computer.

Now you might ask, what if we wouldn’t use 2 separate codecs but keep up with one ?

You would either have have a poor audio quality all the time (no matter if the microphone is used or not), or have a good quality with recurring audio interruptions because it would be impossible for the headset and the computer to exchange all the required data in time.

Some headsets even support more than 2 codecs. In such case they can have a greater control of the input and output quality balance. They would ask to your computer/phone/... which codecs they do understand, and make the best choice (depending on if you use the microphone or not) to give you the best audio quality.

For a non-exhaustive list of codecs used by Bluetooth devices: Apt-X, SBC, AAC (also used in Apple M4A audio files), LDAC, ... Other audio codecs might include: AC3, Opus, MP3, FLAC, Ogg, ... but they are not used in Bluetooth.

Is this a setting ? How can I fix this ?

If your headphone does not explicitly allows you to switch the codec in use — in most cases — you can’t. The headphone already uses what it thinks as the best setting. Some headphones allow that, but it is mostly « high-end » (understand « expensive ») headphones.

It’s like in plumbing: a pipe can handle a given quantity of water pressure. But at some point, if you try to put more pressure, pipes might break/explode. You would either need to use a bigger pipe (an USB-C cable for example) or reduce the pressure (in your case, the headset reduce the audio quality).