Origin of 'wee hours'

Solution 1:

I think you're strongly overestimating how old that phrase is, if you think it is explained by ancient time-keeping standards.

The real answer is quite simple, and I'll quote from the OED:

"the wee (small) hours = small hours":
"The early hours after midnight, denoted by the small numbers, one, two, etc."

And as you said, 'wee' is used as 'small'. Both idioms have recorded use in the 18th century, according the OED.

Solution 2:

According to Etymonline the expression is from the Scottish expression "wee sma' hours" probably referring to the small number that are used to refer to them (1/2/3 o'clock) as in small hours.

• Wee hours is attested by 1891, from Scot. wee sma' hours (1787, Burns)

Note also the Scottish usage of "wee" in suggesting the idea of a "short time":

• (mainly Scot) a short time (esp in the phrase bide a wee.)

From: Dictionary.com