Difference between vex and annoy

Vex connotes being frustrated more than annoy does. Annoy connotes being bothered.

I might be annoyed by a mosquito buzzing around my ear; I might be vexed by a difficult math problem.

Vex is also much better for pangrams.

Contrary to @Manishearth's comment, I suggest on average vexation is stronger than annoyance. Definitions of vexed are likely to include annoyed, and vice-versa, but distressed is much more likely to only turn up with the former. It's also important to note the long-term usage shift...

...which means that in modern parlance, using vex is at least a slightly unusual choice. Sometimes it's slightly self-conscious/facetious, sometimes it's in order to make a subtle distinction. As J.R. points out, by choosing vex you may be able to convey that you're being bothered by something you can't easily resolve (i.e. - it's truly frustrating), whereas oftentimes if something simply annoys you, you can just deal with it.

In general my advice to a non-native speaker would be to avoid vex in most contexts - you need to have a good "ear" for the way other people use it, otherwise your choice of vocabulary will just end up sounding quaint or dated, if not actually archaic.