'confusion matrix' for English phonemes
Solution 1:
After googling for a while for "confusion matrix", found this resource:
http://people.cs.uchicago.edu/~dinoj/research/confmat.html
Perhaps is interesting to learn how to read a confusion matrix here:
http://en.wikipedia.org/wiki/Confusion_matrix
Edit: Thanks to J Trana for also pointing out the following papers:
Consonant And Vowel Confusion Patterns By American English Listeners (
Patterns of English phoneme confusions by native and non-native listeners
Solution 2:
The confusion matrix is highly dependent on the native language of the listener.
For example, a German speaker once asked me why we pronounce the German composer Bach's name with a /k/, when [bɑf] is clearly closer to [bɑx] than [bɑk]. It's not to modern English speakers. But it might have been closer in Middle English: the word laugh, which was originally pronounced with /x/, is now pronounced with an /f/ and not a /k/.
Similarly, this question and its answers show that Spanish speakers perceive /ɛ/ as the closest English phoneme to /e/, while American English speakers think it's /eɪ/.
And I suspect that /z/, /d/, and /v/ are the closest phonemes to /ð/ in various languages, because these are all substituted for /ð/ by foreigners learning English.
And even among native English speakers, people from California can't tell the difference between Don and Dawn, while people from the U.K. think they sound nothing alike.