What does the phrase "ungodly hour" really mean?

Ungodly is being used in the sense of outrageous; shocking; dreadful; insufferable, rather than invoking any of its religious meanings. So "an ungodly hour" just means "an outrageous time of day".

But how did the phrase come to have a connotation of earliness?

When people complain about an "ungodly hour", it is usually because they are forced to be awake when they would prefer to be asleep.

It's not unusual to be compelled to get up earlier than you wish. Some people must get up early every weekday just to get to work on time. And on occasion they might have to get up very early - to catch an early flight, for example.

It is far less common to be required to stay up late.

So, in most people's experience, when they find themselves talking about an "ungodly hour", it is far more likely to be an early one than a late one.

It's the meaning of the word ungodly. "Ungodly" here means:

Informal unseemly; outrageous (esp in the phrase an ungodly hour)

So, it doesn't really matter what time it might be, it's just outrageous.

It can also be used for other things, as can be seen in this dictionary

outrageous; shocking; dreadful; insufferable: an ungodly roar; an ungodly hour to drop in.

The idea that an hour is ungodly is slightly hyperbolic; it implies that even God wouldn't be awake at this time, or that the hour is so improper for being awake and alert that it's morally obscene and thus abhorrent to God.

Plainly spoken, godly is pious or devoutly religious.

An ungodly hour for any given activity would therefore be one where the pious are commonly indisposed to participate in said activity.

I don't think any history would be needed to trace the derivation of this use, but a good story is always welcome illustrating when a use of a certain phrase explodes in popularity.