What is the usage difference between dilemma and quandary

A dilemma is a choice between several options that are equally unfavourable.

A quandary is the state of being uncertain as to what you should do.

But dilemma is also used of the state of having a dilemma to solve, while quandary is also used of the cause of the quandary, so there is certainly a large overlap between these two.

Dilemma is still used by some to refer only two options (the original meaning) or at least where there are known options (as opposed to just "I don't know what to do about this"). Some people though use it more loosely to be more fully synonymous with quandary.

If you're at the mall choosing between red, green or blue socks, it is dilemma or quandary.

It would be an example of hyperbole to describe it as either. If you aren't a stickler for the dilemma = 2 things rule, then either is equally good (if over the top), otherwise it's only a quandary.

While they can be used interchangeably (hence the dictionaries listing them as synonyms as you say), dilemma does more often refer to the choice and quandary the mental and moral state of having to make it, which gives dilemma a slightly greater nuance of the difficulty faced, quandary a slightly greater nuance of the emotional response to it.

Per the definitions you link to, a quandary is:

a situation in which you are confused about what to do

This could be any sort of situation where the "best" choice is not obvious.

A dilemma is:

a situation in which you have to make a difficult choice

The intent here is that every option available to you is undesirable, but you nevertheless have to choose.

Often, when you have a dilemma, it is also a quandary, because which of the bad choices is "least bad" is not at all obvious and it is difficult to decide what to do. (This is not always the case, though; sometimes you might easily choose a "known to be about X bad" over an "I have no idea how bad it could get" option.)

On the other hand, when you have a quandary, it is entirely possible to be picking from a set of choices that are entirely good in different ways, or that balance good and bad aspects, or that are a relatively meaningless set of choices that you just can't decide between.

Picking socks may constitute a quandary if you simply can't determine which set of socks will best match your outfit, but it is definitely NOT a dilemma, where no matter what socks you pick your life will be difficult and the only question is how it will become difficult.