Solution 1:

OK, I got it!

the host names of the each node have to line up inside the container.

On the host machine (q2) I checked what hosts it knew of in the hosts file:

# This file was generated by OpsWorks
# any manual changes will be removed on the next update.

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

# OpsWorks Layer State
127.0.0.1 localhost.localdomain localhost
127.0.1.1 q2.localdomain q2

10.0.3.56 q1.localdomain q1
10.0.3.101 q2.localdomain q2


root@q2:/# ping q1
PING q1.local (10.0.3.56): 56 data bytes
^C--- q1.local ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

Then,it occurred to me that it doesn't matter at all what the host machine knows, it's what the docker container knows about. So, I shelled into the container and did the same thing:

root@q2:/# cat /etc/hosts
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2  q2.local

Now we are on to something! So, I added an entry for the node master:

root@q2:/# echo "10.0.3.56    q1.local q1" >> /etc/hosts
root@q2:/# which ping
/bin/ping
root@q2:/# ping q1
PING q1.local (10.0.3.56): 56 data bytes

And, took another swing inside the container

root@q2:/# rabbitmqctl stop_app
Stopping node rabbit@q2 ...
root@q2:/# rabbitmqctl join_cluster rabbit@q1                                     
Clustering node rabbit@q2 with rabbit@q1 ...
root@q2:/#

And now, each node recognizes that it's clustered! Woot!

I think for clustering with docker I am going to modify the docker command to mount the hosts /etc/hosts file into the docker image with -v /etc/hosts:/etc/hosts:ro and then this should just work magically

Another step I forgot to mention: The local Ubuntu box had an ancient version of erlang running that I had to remove (and it had rabbit as well).