What's the difference between "formal" and "literary" language?

As I see it, literary means that it is more often found in books, novels, and such. You wouldn't hear it very often in spoken conversation.

Formal means that it is used in a context of formality -politicians, speeches, research studies...- , AND it could be both literary or spoken conversation.

I am not sure where you can find a list like that, but if you have a background in any other romance language, as a general rule, words that come from latin and greek tend to be more formal / literary, and words that come from germanic roots are usually more colloquial.

Looking at my Oxford Dictionary and what it has to say about such labels, I see that it makes a distinction between Register and Style;
Formal denotes Register and is 'more common in formal (esp. written English), while Literary denotes Style and is 'found mainly in literature'.
These are descriptive, telling you where they are normally found, rather that prescriptive which would be telling you where they must be used.