How do I silence the message ringtone while talking?

Switch the Ring/Silent switch to Silent.

Here is how things work. The alert volume can be controlled from the sound preference pane. If you have change with buttons on, you cannot use the physical buttons during a call since that changes the call volume and not the alert volume. (the two are distinct sliders).

You can use the slider to change things (requires the phone to move away from your ear to disengage the proximity sensor) or simply flip the silent switch when you are talking to avoid hearing any alerts except for the call in progress.

My read of your description is things are working as designed / normally and you can use the silent toggle rather than the volume buttons.