Encrypted offsite backup using GPG with private key never on backup server?

This is definitely possible with the --show-session-key and --override-session-key options.

First you need the beginning of your encrypted file. This is where the encrypted session key is stored.

root@qwerty:~/gpg# head -c 1024k bigfile.gpg > head.gpg

Then copy it to your workstation and retrieve the session key

PS C:\Users\redacted\Downloads> gpg --show-session-key .\head.gpg
gpg: encrypted with 2048-bit RSA key, ID DC21D645, created 2016-02-01
  "admin <[email protected]>"
gpg: session key: '9:926EC16DF1248A1C4401F5AD5D86C63C1BD4BF351ECEFB121C57EC209DE3933D'

Now you can decrypt the file using your session key

root@qwerty:~/gpg# gpg -d -o bigfile --override-session-key 9:926EC16DF1248A1C4401F5AD5D86C63C1BD4BF351ECEFB121C57EC209DE3933D bigfile.gpg
gpg: encrypted with 2048-bit RSA key, ID DC21D645, created 2016-02-01
  "admin <[email protected]>"

It looks as though most of your question has been answered, however, if you're administrator team is wary of private keys ending up out of their local control ya might consider sshfs to mount the remote backups over a ssh session.

Install via apt on each remote administrator's system

sudo apt-get install sshfs

Assuming admins' ssh configuration looks something like below

# configuration for ssh login to remote server
Host Remote
    Hostname Remote.web.domain
    User admin
    IdentityFile ~/.ssh/private.key

Then your admins can use something like below for mounting

# make a mount point
mkdir -p /mnt/remote
# mount remote directory to local file system
sshfs Remote:/path/to/encrypted/dir /mnt/remote

To unmount after inspection the remote administrator can use the following

fusermount -u /mnt/remote

The sweet bit about using sshfs is that only public keys for GnuPG and ssh are needed on the remote server, the related private keys stay on the systems that own'em. Second nice bit is that until read or accessed most of the file info stays on its related file system.

If you're still looking for tools to facilitate auto encryption of logs or directories ya might want to check the prof of concept tool I've pushed to GitHub (specifically Scenario Four written for sshsf usage) which with a little customization will happily encrypt almost any data via GnuPG. But be warned that it is experimental and some of it's features may cause corruption of data if misused. Source code is less then ~1600~ lines so it's very possible to audit in less then a weekend.

Additional security can be had by setting up the remote server's ssh configuration to chroot users to only allow access to the encrypted directory and disable interactive shell for admins keys that are used in this fashion.

If you want the secret key kept off the hard disks, you could create a ramdisk (remember those?) and load the secret keys there from your secure not-on-server location as needed. Use it for decrypting and when done overwrite it with /dev/random. The secret has to go into RAM to be used by GPG anyhow, so why not twice?

If you can't let a secret key ever be on the server, even in RAM, then you have a technical impossibility. GPG must have the secret key somewhere in order to decrypt anything.

Ramdisk info: https://unix.stackexchange.com/questions/66329/creating-a-ram-disk-on-linux