If you want my sympathy, look it up in the dictionary

What it means is "I am unwilling to give you any sympathy."

The point is that the only place you will find sympathy from me is in a list of words, a dictionary, since I have none in my head or heart.

This is a pretty ugly way to speak, and is further emphasized by the second phrase, using the ugliest bracketing words they can think of (shit is before sympathy in the dictionary, and syphilis is after it.)

Moreover, it doesn't even really work as a dismissal, since the "sympathy" in the dictionary doesn't actually "belong" to the speaker.

Ugly and incorrect isn't a combination that reflects well on the speaker.

It's an expression not only of total lack of sympathy, but contempt - either for the person you're talking to and their problems, or for the concept of sympathy itself. Obviously this is extremely rude, and you would only either say it to an enemy or to a very good friend who knows you're joking.

A more common variation: "Where can you find sympathy these days? In the dictionary, between shit and syphilis" - is a cynical, off-color complaint about the human condition. In this variant, the speaker is essentially saying that s/he has experienced personal tragedy, but can't get anyone to sympathize. The speaker is complaining, but half-jokingly; s/he doesn't expect to be taken seriously.

When my father used the term he meant it this way:

If you are looking for sympathy, you can find it in the dictionary between "shit" and "syphilis", two other equally undesirable things to have.