"The man was either mad or both"
I realize this question is quite old, but I disagree with an earlier answer. While Stephen Fry's delivery is brilliant comedy in itself, there's also some word-play going on here.
The term mad can mean either angry or crazy. So here he's saying the man was either [meaning of] mad, or both; i.e. either angry or crazy or both.
The 'funny' comes from the fact that he should have said 'The man was either mad or X or both'. You can see that he looks at the book for the 'or X' that he may have missed.