ssh key exists on remote, but still prompts for password?

Why is this happening?

$ ssh-copy-id me@server
/usr/local/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/local/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.

$ ssh me@server
me@server's password:

ssh is still prompting for a password, even though it says the key exists already on the remote server (which is Debian).

UPDATE: Contents of .ssh folder:

$ ls -al ~/.ssh
total 72
drwx------   9 me  staff   306 22 Apr 13:20 .
drwxr-xr-x+ 65 me  staff  2210 22 Apr 13:18 ..
-rwxr-xr-x   1 me  staff  1397 22 Apr 11:49 config
-rw-------   1 me  staff  1679 25 Feb 16:54 id_boot2docker
-rw-r--r--   1 me  staff   410 25 Feb 16:54 id_boot2docker.pub
-rw-------   1 me  staff  1675  6 Jan 14:00 id_rsa
-rwxr-xr-x   1 me  staff   381  6 Jan 14:00 id_rsa.pub
-rw-------   1 me  staff  4884 13 Mar 17:29 known_hosts
-rw-r--r--   1 me  staff  4492  9 Jan 11:33 known_hosts.old

Please check the sshd configuration file if Public Key Authentication is enabled

search in /etc/ssh/sshd_config for

#PubkeyAuthentication yes

If it is commented out remove the leading '#'. if it is set to 'no' change it to 'yes'

As @Darius mentioned please also check the setting AuthorizedKeysFile It should be commented out for default behaviour. Then create a authorized_keys file in ~/.ssh/ and add your public key string to this file. How to do this is described here in the Debian documentation

Take care of the permissions set on the ~/.ssh/ folder in your home directory. Only the owner must have read/write permission.

chmod -R go-xwr ~/.ssh

can fix the permissions