Solution 1:

"The "obvious" answer

I'm going to attempt a rare answer with no research.

I don't like the phrase non-comparable and would argue that it's a poor term for what is meant to describe the distinction of latter.

I prefer his latter book over his first

If this means a preference for the second book, it only works if there are only two books. It does not seem correct or clear to say:

Out of his first, second, and third books, I prefer the latter.

Maybe "latter" in its purest form is a binary term, functioning as the opposite of "former." Neither implies an immediate relationship as with preceding or following.

Perhaps over time it evolved to mean, simply, "later," and led to words like "lattermost," as a recent development, in meaning and character...

So what if I had done research?

It turns out, according the OED, that the opposite is true. Latter in its earliest uses in English simply meant "later than," or, even better, often "slower," or even just "worse" or "less than [in any sense]." Hardly a "non-comparable" term, in that it seemed to imply only that anything would be better, faster, or sooner in comparison.

In fact, the distinction between "latter" and "later" seems to have happened a little latter than I thought.

All examples of later earlier than 1500 have been regarded as showing latter adj., adv., and n., although it is conceivable that some (especially in senses A. 2a or B. 1) may rather belong as antedatings at later adv., adj., and int.)

Only first in c1225, long before the OED has, in the aforementioned quote, determined a distinction between antedatings of "later" and "latter," do we find a citation that fits the assumptions I laid out at the top of my answer.

That has been mentioned second of two, last of a group of more than two, or at or near the end of a preceding clause or sentence. Frequently opposed to former

Solution 2:

Latter usually has the distinction of being last on a list; "Dick, Jane and Spot played outside, the latter being Jane's pet dog." However, the idiom currently used with it is, latter-day: modern or contemporary, especially when mirroring some person or thing of the past. Curiously, I could have written that lead in as; "However, the popular latter-day usage is in the idiom, latter-day." and been correct.

Solution 3:

Job 19:25-27, in the King James Version:

For I know that my redeemer liveth, and that he shall stand at the latter day upon the earth: And though after my skin worms destroy this body, yet in my flesh shall I see God: Whom I shall see for myself, and mine eyes shall behold, and not another;

This "latter day," and other KJV references to latter days, possibly refer the last days of earth, and are echoed in the formal title of the Mormon church, The Church of Jesus Christ of Latter-Day Saints (established 1830).

And it is latter-day saints that provide most of the instances where latter is used with the indefinite article (a latter), e.g.:

A Latter Day Saint: Being the Story of the Conversion of Ethel Jones (1884) (this one is fiction, although most are memoirs) and the more intriguing Confessions of a Latter-day Virgin: A Memoir (2013).

There are also a number of legal references, such as "a latter law," which seem to be alternative spellings of later. "... a former will may be revoked by a latter one," A Treatise of Equity, 1805. Although the use of the indefinite article is infrequent, it seems to follow the usual conventions.

Former/latter are often used in formal writing to refer to the first-mentioned and second-mentioned of a pair, as in your fame-fortune duo; but if it's reversed to fortune-fame, then there are more ways to achieve the latter than the former.

late-later-last: The boy arrive late; his sister was even later; and the mother was the last of the family to arrive.

But the latest guest to arrive is the most recent to arrive, as in "the latest and greatest" or the latest fashion. late, OLD

An incomplete answer, unquestionably, but too long for a comment.