Howto to set AltGr key with xmodmap
Solution 1:
According to the Xmodmap Manual, when you want to swap or change modifier keys you need to remove & clear their ties to their original keycodes and modifier groups. Then after assigning them to a new keycode you must also assign them to a new, unique group.
It looks like the problem is that you've assigned two different modifiers to the same key: [ISO_Level3_shift] & [Mode_switch]. And you've also assigned them to the same modifier group. The effect of assigning them to the same group is that: pressing one of them, activates/sends them both to the system.
[ISO_Level3_shift] & [Mode_switch] are not synonymous. Each of these keys is meant to provide access to a unique layer/level of the keyboard. And when pressed together they can access even higher levels if your keyboard supports them.
Another potential complication is that on virtually all systems Xmodmap is running on top of xkb (which defines certain aspects of the underlying virtual keyboard).
[Mode_switch] is synonymous with [AltGr] to Xmodmap providing access to level 2- that would be the second set of [q] & [Q] in your example. Whereas [ISO_Level3_shift] provides access to level three- the [ä] & [Ä] in your example.
But since both keys share the same modifier group in your example they are both being pressed which is probably trying to access your highest level; but since there seems to be an error* you are probably either not getting any output, or some unexpected output coming from the underlying xkb.
Basically you can fix this by changing your .xmodmap file. Just assign [Level3_shift] & [Mode_switch] to separate keys and separate modifier groups. Or, actually I would just recommend that you just remove the reference to Mode_switch altogether. I have found it to behave rather oddly; the 3rd level shift key seems to behave more consistently and reliably.
*-your second "adiaeresi" (sic.) is missing the final [s] so it wouldn't produce any character.
I know this rather confusing and convoluted for no apparent reason, plus this thread is old. But hopefully someone will find my explanation useful at some point!