where does RPM install custom GPG keys?

Solution 1:

They are stored in the RPM database, which is in /var/lib/rpm. From the manpage

Digital signatures cannot be verified without a public key.  An ASCII 
armored public  key  can  be added  to  the rpm database using --import. An 
imported public key is carried in a header, and keyring management is 
performed exactly like package management. For example, all currently  
imported public keys can be displayed by:

rpm -qa gpg-pubkey*

Details about a specific public key, when imported, can be displayed by 
querying.  Here’s information about the Red Hat GPG/DSA key:

rpm -qi gpg-pubkey-db42a60e

Finally, public keys can be erased after importing just like packages. 
Here’s how  to  remove  the Red Hat GPG/DSA key

rpm -e gpg-pubkey-db42a60e

/etc/pki/rpm-gpg is the standard place for packages with repository configuration (like epel-release) to put keys they want to be imported. The yum configuration in the package will have the path to the key in the gpgkey directive. The first time you try to install a package from a repository yum prompts you to import the key.