How to setup password-less ssh between two vagrant machines?
The following answer suggested that I have to look at the permission all the way from root to the ssh folder.
https://superuser.com/a/255411/210224
In order to solve the problem I had to add the sticky bit to the home folder.
sudo chmod +t /home
Read the linked answer above for more details.