SSH Agent forwarding using different usernames and different keys

Solution 1:

Step 1

Make sure your local agent is ready

Just because you can ssh into your bastion server without specifying your key path or getting prompted for the password, doesn't mean that your ssh agent is running and holding your key. Some modern OSes (eg: OSX) handle this for you.

On your local machine

$ ssh-add -L
ssh-rsa ObahfCbvagGbLbhSbeHfvatEBG13== ~/.ssh/mykey.pem
ssh-rsa LbhNerWhfgJnnlGbbPyrireEBG13== ~/.ssh/sharedkey.pem

^fig.1

That means your agent is running and has your key.

$ ssh-add -L
The agent has no identities.

^fig.2

That means you haven't added any keys to your agent. Fix that with:

ssh-add ~/.ssh/mykey.pem ~/.ssh/sharedkey.pem

^fig.3

Step 2

Make sure your remote agent is ready

SSH into your bastion server and repeat the check from fig.1 & fig.2. However, the error you are more likely to get is this:

$ ssh-add -L
Could not open a connection to your authentication agent.

^fig.4

That most likely means that your SSH client is not forwarding your authentication agent connection.

You can force this with the -A flag (as long as the sshd config on the server allows it, which is the default).

$ ssh -A bastion.ec2

^fig.5

Step 3

Make sure you're using the right keys

If you have added keys to your agent, your agent is forwarding, and your remote agent lists your local keys. There is only two likely reasons you are not getting a connection. Either you are not using the right key or you are not using the right username.

Output the public counterpart to your private key:

$ cd
$ cd .ssh
$ ssh-keygen -y -f mykey.pem
ssh-rsa ObahfCbvagGbLbhSbeHfvatEBG13
$ ssh-keygen -y -f sharedkey.pem
ssh-rsa LbhNerWhfgJnnlGbbPyrireEBG13

^fig.6

These should be the same as what you were seeing from ssh-add -L up to the == near the end.

Now one way or another you have to get into the box you are failing to connect to and look at the content of the $HOME/.ssh/authorized_keys file for the user you are trying to connect as. You need make sure the public key you output with the command above is in that file on a line by itself. You can't trust that the sharedkey.pub that the Bro 2 cubes emailed you is right. Verify! This may require getting someone else who can SSH in as that user to get you the authorized_keys file or getting root access. If you've come this far and it still isn't working, you are beyond taking shortcuts.

Step 4

Make it easy

Hopefully the steps above have gotten you in. Now let's make this headache go away for as long as you are using this workstation.

Configure your local ssh-client

Host *
    # A lot of people put an IdentityFile line in this Host * section.
    # Don't do that unless you will use only 1 key everywhere forever.
    #IdentityFile id_rsa

Host bastion.ec2
    # You want to make sure you always forward your agent to this host.
    # But don't forward to untrusted hosts. So don't put it in Host *
    ForwardAgent yes
    # Go a head and put the IP here in case DNS ever fails you.
    # Comment it out if you want. Having it recorded is a good backup.
    HostName 172.31.0.1
    # You don't want to create a proxy loop later, so be explicit here.
    ProxyCommand none
    # SSH should try using all keys in your .ssh folder, but if you
    # know you want this key, being explicit speeds authentication.
    IdentityFile ~/.ssh/mykey.pem

# Connect effortlessly by hostname or IP address
# This assumes that your internal DNS uses the fake TLD ec2
# This assumes that 172.31.0.0 is your C-Class subnet
Host *.ec2 172.31.*
    # This command says proxy all ssh connections through bastion as if
    # you had done an ssh -A
    ProxyCommand ssh -W %h:%p bastion.ec2
    ForwardAgent yes
    # These next lines are documentation you leave as a love letter to
    # your future self when all else fails or you have to help a
    # coworker and decide to look at your own config.
    # ssh-add ~/.ssh/*.pem
    # ssh -At bastion.ecs ssh [email protected]

^fig.7

If you take nothing else away from fig.7 it should be the proper use of ProxyCommand & ForwardAgent.

Auto populate your .bash_profile

You don't want to have to do ssh-add manually every time you log into your machine. ~/.bash_profile is a script that runs each time you login**. Put the line from fig. 3 in there and you should always have your agent ready.

** Don't confuse this with .bashrc which runs for each new [interactive] terminal. Your agent continues running even if you close all your terminal sessions. No Need to reload your keys

Alternative to using .bash_profile

I have also created a gist that add an OSX/macOS Launch Agent. You can use that method for starting your ssh-agent on boot. It is very easy to install:

curl -sSL https://gist.github.com/RichardBronosky/429a8fff2687a16959294bcee336dd2a/raw/install.sh | bash

Solution 2:

This is where you come in. How can I forward a different key for the 2nd connection?

You do not forward the keys. You forward agent and it can have completely independent keys than you are using for authentication in the first jump. Check the keys in your agent using ssh-add -L.

But even better, run the connection over ProxyCommand ssh -W %h:%p [email protected], which will avoid the need to forward the agent, the heap or command-line options and a need to authenticate from the immediate host.

You can simply put that in your config (see man ssh_config).