Setting Hyper and Super modifiers for certain keys with setxkbmap or xmodmap

Problem: I want to modify/redefine my keyboard layout (pc105 se) so that when I press certain keys with mod3 (super) and mod4 (hyper), a different character is returned than usual.

E.g.

Keycode 61 produces 'minus' (-) when pressed normally and 'underscore' (_) when pressed with shift. When pressed with the first modifier group (AltGr) 'dead_belowdot' and 'dead_abovedot' is produced (without/with shift).

What I want is a left paranthesis or right paranthesis (without/with shift) when I press the key with the Super (win) key also pressed, and slash/backslash for the Hyper key (which I have configured to Caps Lock).

I have tried to add a .Xmodmap to my home directory and run it with xmodmap ~/.Xmodmap using the following syntax

keycode 61 = minus underscore a b c d e f g h i j k l m n o p

However, that will cause AltGr to result in c/d, while win/caps lock still only results in -/_

I have also tried modifying /usr/share/X11/xkb/symbols/se directly and tried the following changes:

key <BKSL>  { [apostrophe, asterisk, acute, multiply], [a,b,c,d], [e,f,g,h], [i,j,k,l]};

But all that happens is ' * ´ × ' * ' * (for normal, altgr, win and caps lock respectively). I have also tried

key <BKSL>  { [apostrophe, asterisk, acute, multiply, a,b,c,d, e,f,g,h, i,j,k,l]};

But as soon as there are more than four specified fields, it seems to break the key totaly, and the key reverts to default pc105 (us?) behaviour.

As I have failed to find anything on stackoverflow, superuser or google on how to deal with my scenario, I ask here.

This is my xmodmap -pm

xmodmap:  up to 3 keys per modifier, (keycodes in parentheses):

shift       Shift_L (0x32),  Shift_R (0x3e)
lock      
control     Control_L (0x25),  Control_R (0x69)
mod1        Alt_L (0x40),  Meta_L (0xcd)
mod2        Num_Lock (0x4d)
mod3        Hyper_L (0x42),  Hyper_L (0xcf)
mod4        Super_L (0x85),  Super_R (0x86),  Super_L (0xce)
mod5        ISO_Level3_Shift (0x5c),  Mode_switch (0xcb)

Two years later, I am finally proud to answer my own question.

I have found the easiest, most reliable and most extendable method is to solely rely on xmodmap.

1. Get keycodes of special keys

Run xev to determine the key code of keys you wish to use as mode switchers. Then press the keys you are interested in and note down the keycode

$ xev
KeyRelease event, serial 40, synthetic NO, window 0x2600001,
    root 0x2a6, subw 0x0, time 1221887800, (885,743), root:(3447,805),
    state 0x2010, keycode 66 (keysym 0xff7e, Mode_switch), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

The keycode is on the third line, keycode 66.

For me, these were the keys I was particularly interested in, and their corresponding key codes for me:

• Caps Lock, keycode 66
• Pause|Break, keycode 128
• Left Windows key, keycode 133
• Num Lock, key code 77

2. Set switcher keys

This is the setup I have opted for, but you can choose your own setup. I have saved this to a file $HOME/.xmodmap.

~/.xmodmap
keycode  66 = Mode_switch
keycode 127 = Multi_key
keycode 133 = Super_L
...

This way I have bound Caps Lock to Mode_switch, Pause Break to Compose and the left Win key to Super_L. The Compose key is particularly useful as it allows two keys to be combined to ligature. E.g. Compose followed by T and M results in the trademark sign ™.

3. Get current keymap bindings of xmodmap

In order to not destroy my keyboard layout, I wanted to keep my keys as close to my original layout as possible. In order to do that I printed the current keymap and looked for the keycodes I wanted to change.

$ xmodmap -pke | ag 'keycode  51' 

Off course, you could just as easily use grep or ack, depending on the tool you prefer to use.

$ xmodmap -pke | grep 'keycode  41' 
$ xmodmap -pke | ack 'keycode  31' 

The output of this is in the same format as the one you specify in your ~/.xmodmap like this:

keycode  31 = i I i I rightarrow idotless rightarrow idotless i I rightarrow idotless i I rightarrow idotless

4. Backup your existing xmodmap

Just in case something goes wrong, it's never a bad Idea to have your current xmodmap available to fall back to easily.

$ xmodmap -pke > ~/.xmodmap.bak

5. Add characters to 2nd, 3rd, 4th level of your keys

As far as I have understood, the format of the .xmodmap file is like this:

keycode <keycode> = <1st level> <Shift+1st> <2nd level> <shift+2nd> <3rd level> <shift+3rd level> <4th level> <shift+4th> ...

Now, I don't understand all those levels but Mode_Switch (Caps Lock in my case) will give you level 2 and Alt Gr gives me level 3, and that is pretty much all I am interested in so I left the rest of that line as it was in my ~/.xmodmap.bak and just changed the 3rd, 4th, 5th and 6th columns (level 2 and 3, plus shift) to valid characters (See here for a list of valid characters).

~/.xmodmap
...
keycode  51 = apostrophe asterisk dollar ampersand acute multiply acute multiply apostrophe asterisk acute multiply apostrophe asterisk acute multiply
keycode  59 = comma semicolon parenleft parenright dead_cedilla dead_ogonek dead_cedilla dead_ogonek comma semicolon dead_cedilla dead_ogonek comma semicolon dead_cedilla dead_ogonek
keycode  60 = period colon bracketleft bracketright periodcentered dead_abovedot periodcentered dead_abovedot period colon periodcentered dead_abovedot period colon periodcentered dead_abovedot
keycode  61 = minus underscore braceleft braceright slash backslash braceleft braceright
keycode  65 = space space space space space underscore space space
...

6. Test your .xmodmap

Now let xmodmap execute your .xmodmap and test your brand new keyboard layout.

$ xmodmap /home/user/.xmodmap

7. Make sure your keyboard layout is loaded on x init

I added this to my $HOME/.xinitrc:

~/.xinitrc
...
[-f $HOME/.xmodmap] && xmodmap $HOME/.xmodmap
...

I hope this was helpful!