What was going on with "quha", "quhat" and the like in Scots and English?

Subject to correction by those more knowledgeable than I about 16th century dialects: the author, who speaks Scots, is justifying the traditional Scots spelling quh for what he sounds with a strong initial 'guttural' - probably velar - fricative, followed by the labial : [xw].

He appeals to the ear: Is not the first sound at bottom [ex imo]a guttural? Since the only difference between Latin quo (pronounced with [kw]) and Scots quho (prounced with [xw]) is that the latter is 'aspirated' (fricative), the spelling with qu is superior to that with wh. This, he says, "No man can deny who has sucked the paps [suckled at the breasts] of reason."

The Doctour, who as an Englishman uses in the same words either a simple labial [w] or possibly a much lighter initial velar [hw], dismisses this as a "Scottish assumption".

One of the peculiarities of a 'Scottish accent' (insofar as such a thing exists, as opposed to Highland, Glasgow, Embro, etc.) is a slight aspiration on w, so that whether sounds as if it were spelt hwether. It looks as if the author here, being Scottish, championed spelling reform on the grounds that written English should reflect spoken, and so this aspiration should be reflected in an initial qu. It never caught on among writers in England, probably because they (we?) do not use the sound in the first place. I have seen quilk for whilk (which) in Renaissance English, but of course there were many Scottish divines writing authoritatively at the time.

Why qu rather than hw or cw? At this distance there's no knowing for sure, but a reasonable guess is possible. It is well-known (picking up your second point) that the Anglo-Saxon for 'woman', for example, was cwen, and that the Normans, finding it impossible either to pronounce or to spell, substituted the nearest sound in the Norman French vocabulary, qu for cw. (The Scots language, pronouncing wheen closer to queen in any case, might not have seen any need to change.) A Scot would obviously not want to bring in additional Saxon (Sassenach) overtones with cw, and might well prefer the Latinate qu to the entirely new hw.

And, by the way, suked the paepes = sucked the paps = imbibed the milk.

Some possibilities are shown in the Wikipedia article on the phonological history of wh.

The original proto-Indo-European has probably survived better in the Latin question words quis-quid-quo-quare-quando question words and had a kw-like pronunciation, which may have drifted through xw, hw, xʍ, ʍ, w, v and f in various Germanic languages and dialects.

The Scots quh- spellings are probably a combination of historical usage and a desire to emphasise that this is not the same sound as produced by the English. And there may be local differences; in Northern Scots the sound seems to have changed even more, under Gaelic influence.

As for "quho" and "quo" differing only in aspiration, this may be more explicitly related to the wine-whine merger, or lack of it in Scotland.

The text quoted here is written by Alexander Hume in about 1617, and it is also quoted in David Chrystal's The Stories of English on page 299 if anyone's interested.