What is meant by saying "X, not to say Y"?

Solution 1:

You are right: literally not to say means not quite Y, but it is mostly used to mean even Y.

[Edited:] In this case, either the author uses this phrase in an unusual way, or he considers misogynistic an even stronger qualification than misanthropic. Perhaps his line of though was thus: they disliked people in general, and, what is even more salient, they specifically disliked women. I will assume the latter, since they use not to say in a conventional manner in your other example as well, dismissive, not to say derisive. Dismissing something is not as strong as ridiculing it.

It literally means they were misanthropic, and one might almost go so far as to say that they were misogynistic (but one doesn't). The latter is supposed to be an even stronger word than the former; the writer says he will not fully commit himself to "misogynistic" but offers it as a suggestion.

In practice, this figure of speech is mostly used when the writer is quite committed; it then simply introduces a phrase that is even stronger than what came before, without any serious holding back. Not to say is often more or less equivalent to even:

In somewhat different ways, Orwell and Larkin were phlegmatically pessimistic and at times almost misanthropic, even misogynistic.

If he did not intend it to be read this way, which I think he did, that means he would have used it in an unusual manner that might be misleading for some readers.

Solution 2:

I am more familiar with this as "not to mention", though the sense is, I think, the same:

"X, not to say/mention Y" is often used to strengthen the first claim/concept. Example:

Sarah is awful at her job! She never arrives on time, not to say/mention how little effort she makes when she is here.

The fact that 'she never arrives on time' gives good reason for my claim that 'Sarah is awful for her job', even without adding that she makes no effort. By using the 'not to say' clause, we are giving further reason to accept the original claim.

As a point of interest, another similar phrase exists with quite a different meaning:

If I say "X. That is not to say Y", it means that even though my saying X might imply Y, that is not the case.

An example should help:

Sarah is terrible at her job. That is not to say that I think she should be fired; perhaps further training would be a good idea, though.

Here, my first statement (X) might lead you to think that I would advocate Sarah being fired. I want to clarify that I do not feel that way, so I use the 'not to say Y' statement to ensure that the listener doesn't get the wrong idea.

Hope that helps.

Solution 3:

• My dictionary says that "Not to say..." is "used to introduce a stronger alternative or addition to something already said."

It is easy to become sensitive, not to say paranoid.
He was large, not to say fat.

• The OALD gives only one choice: "Used to introduce a stronger way of describing something"

A difficult, not to say impossible, task

So, in this case:

In somewhat different ways, Orwell and Larkin were phlegmatically pessimistic and at times almost misanthropic, not to say misogynistic.

I think that the matter is not which one between misanthropic and misogynistic is stronger (in general), but which one Orwell and Larkin were, more exactly, between those two. This is also what I get from the comment written by @Hal.