What file is the setxkbmap option -rules meant to take, and how can I add keyboard variants to that file?

XKB's system keyboard configuration database is stored in /usr/share/X11/xkb. XKB layouts are defined in a RMLVO model: Rules, Model, Layout, Variant, Options. The main rules file in use these days is actually evdev; you can see this with setxkbmap:

$ setxkbmap -query -verbose 10
...
Trying to load rules file /usr/share/X11/xkb/rules/evdev...
Success.
Applied rules from evdev:
rules:      evdev
model:      pc105
layout:     us
variant:    altgr-intl
options:    caps:hyper,compose:menu
Trying to build keymap using the following components:
keycodes:   evdev+aliases(qwerty)
types:      complete
compat:     complete
symbols:    pc+us(altgr-intl)+inet(evdev)+capslock(hyper)+compose(menu)
geometry:   pc(pc105)

Each of the components mentioned in the last few lines above is a subdirectory of the system database in /usr/share/X11/xkb, and the values shown are filenames in that subdirectory (separated by +) that will be consulted to build this particular keymap. Parentheses mark a particular clause in the specified file (usually variants and options).

XKB tools (setxkbmap, xkbcomp, etc) can take an argument to search another location for the files, but this custom location must be in the same format as the system database. Here's what the system database looks like. See the README in each directory for details, and

/usr/share/X11/xkb/
├── compat       # ??? dark magic here, avoid
├── geometry     # as in physical, eg for generating layout maps
├── keycodes     # helpful for translating keycodes (from xev) to <FOO>
├── rules        # "evdev" is the important one; *.lst & *.xml are descriptions
├── symbols      # main layouts, variants, optional overrides
└── types        # ??? dark magic here, avoid

If you want to override these files or provide your own layouts, without merging them into the system database, what you'll do is create a similar directory structure for your own files. You probably won't need to do anything with geometry or keycodes unless you're building your own keyboards from scratch.

For a per-user configuration, $HOME/.xkb/ or $HOME/.config/xkb/ are ideal:

$HOME/.config/xkb/
...
├── rules
│   ├── evdev-local
│   ├── evdev-local.lst
│   └── evdev-local.xml
├── symbols
│   ├── my-fun-capslock-options
│   ├── my-US-Dvorak-layout
│   └── my-ZWERTY-layout
...

Once your directory structure is in place, you can load your customizations with the -I /path/to/local/xkb parameter:

setxkbmap -I $HOME/.config/xkb \
  -rules evdev-local           \
  -layout my-ZWERTY-layout     \
  -option myZWERTY:option1,compose:menu,fun:caps_is_insert

You can mix and match local options like fun:caps_is_insert with system options like compose:menu as long as the specified rules file defines both of them. (Symbol files can include other symbol files but I haven't found an include syntax for rules files. You'll probably have to copy the entire system evdev rules to your local version and add your modifications.)

Using custom locations may not be usable with other XKB configuration systems like localectl, GNOME's settings daemon, or Wayland compositors that aren't configurable with setxkbmap.

Resources:

• http://madduck.net/docs/extending-xkb/
• http://apps.jcns.fz-juelich.de/doku/sc/xkbmap
• https://wiki.archlinux.org/index.php/Keyboard_configuration_in_Xorg
• https://unix.stackexchange.com/questions/66624/where-is-xkb-getting-its-configuration
• Permanently set keyboard layout options with setxkbmap in gnome/unity