Why are over-the-counter drugs so called, when it is the prescription drugs that are actually dispensed over the counter?

To summarise your question, you are asking about the phrase “over the counter” and you use as an example the experience at chemists/druggists in which “over the counter” items do not necessarily and literally go “over the counter”, whereas prescription only medicines do go “over the counter.”

You seem to assume that there is a literal passing of the purchase over the counter. There would have been at one time, but this is not the essence of the phrase.

Language and phrases develop: We say we dial a telephone number - but modern telephones have no dial.

The earliest mention (that I can easily find) of “over the counter” in a retail sense is from a publication by “The Business Historical Society Inc. 1823” entitled "Past, Present & Probably the Future State of the Wine Trade" By James Warre:

Mr. Barker, a licensed victualler in Holborn, “sells retail over the counter, in glasses, a pipe and a half of Port wine in a week. Some drink at the counter, others take it away in small bottles. The principal customer are small tradesmen, ..."

Here, “over the counter” has already taken on a figurative meaning: “directly to the consumer and without formality and restriction” and this is in contrast to sales of wine to a restricted group, i.e. members of the trade.

The figurative sense of “over the counter” is thus “without restriction/formalities” and it has been for over 200 years.

So he took me on a tour of the multiple gun stores and pawn Shops not far from the famous Vegas Strip. Our mission - to see if I could buy a gun over the counter. SBS News (2017) https://www.sbs.com.au/news/i-tried-to-buy-a-gun-at-a-las-vegas-department-store

This source [1] suggests comparing the expression to under the counter (and its American variant 'under the table'), the meaning of this expression being secret and often illegal transactions.

If we examine the etymology for under the counter, there are two origin stories found here:

a [2] "In Britain, during the Second World War, shopkeepers sometimes kept articles that were in great demand under the shop counter. They only sold them to special customers, often charging very high prices for them."

b [3] "the counter [is] the flat-surfaced furnishing or table over which legal business is conducted."

This background suggests to me the following suspicion. Perhaps non-prescription drugs are referred to as over the counter because there isn't much chance of their being involved in secret or illegal transactions. Such a loss of opportunity for underhandedness entails a loss of opportunity for under the counter business dealings. Moreover, we may also surmise, with inference from the first origin story, that non-prescription drugs are in less demand than their counterparts and hence provide less incentive for being kept under the counter for those special customers.

[1]: McGraw-Hill Dictionary of American Idioms and Phrasal Verbs. S.v. "over the counter."

[2]: Collins COBUILD Idioms Dictionary, 3rd ed.. S.v. "under the counter."

[3]: The American Heritage Dictionary of Idioms by Christine Ammer. S.v. "under the counter."