Why were Scottish & Irish names once rendered with apostrophes instead of "Mac" or "Mc"

While you may have seen M’ with an apostrophe, look carefully: you might have instead seen M‘ with the character used for an opening quotation mark.

The difference is small but significant. According to M‘CULLOCH AND THE TURNED COMMA, by Michael G. Collins (The Green Bag Second Series, pages 265-275, 2009),

the upside down and backwards apostrophe turns out to have been a routine way for eighteenth and early nineteenth century printers to recreate a lower case, superscript “c” after the letter “M”.

(page 266)

Collins quotes P. Luckombe, The History and Art of Printing 266 (London, J. Johnson 1771), as saying that the “inverted comma . . . serves as a superior c in the nominal appellation of Mac or M^c” (footnote 10, page 268, ellipsis in Collins).

Collins indicates that there are attested uses of the apostrophe in the same context (page 270). So as I said at the start of this answer, you would have to look carefully to identify the exact usage in the texts that you're reading.

As far as I can tell, this was purely a graphical abbreviation, like the superscript o in №: this would mean that M‘ was not meant to represent any difference in pronunciation from Mac or Mc, either in the vowel or in the consonant. (There is some variation in the pronunciation of names starting with this element: I just don't think this variation is connected in most cases with the variation in spelling.)

It's a good question and I didn't have the answer so I asked it on r/AskLinguistics on Reddit and got a good answer. I'm going to quote it (with little changes):

• The Victorians abbreviated things a lot in formal documents and this probably had an influence elsewhere. Older English language (and Latin) wills, records and legal documents are often full of abbreviations. Wm for "William". Answ for "answer" etc.
  
  Older stuff had a whole other set of scribal abbreviations with other characters and things. Wikipedia says they were used in Old English, but you absolutely see them in wills up to the 18th century (and likely beyond). I can't think of any examples but imagine &rew for "Andrew" and you're along the right lines.
  
  Why they stopped being used is another matter, that I don't quite have an answer for. A guess would be a gradual move over to typed or typeset documents, which lacked a lot of the scribal abbreviations that were used in older documents done by hand causing most of them just to quietly vanish. Wikipedia says that typewriters took off commercially in around the 1880s so that sort of fits chronologically.
  
  Here's is a guide to reading Victorian era handwriting, with a mention of abbreviations. The Bodleian has another list of examples in secretary hand - a kind of standardised handwriting and set of abbreviations that are used in recording information. The National Archives have quite a good set of tutorials on paleography generally.

• It's widely found in Scottish 19th-century sources. Here's an example featuring "Donald M'Donald" and "Patrick M'Caffery":
  
  

-- The Dundee Courier Newspaper, Friday, November 22, 1895

• As a lawyer, the example of this usage that I’m most familiar with is M'Naghten (the leading case from English common law that forms the basis of the insanity defense)

According to TribuneStar:

Mac- and Mc- come from the Gaelic word “meic,” meaning “son of.” Meic was contracted or abbreviated in various other ways, including mic, Mhic M’c, M’, Mcc, and Mc with two dots under the c. Such a large number of Irish names carried the mic prefix that it became an ethnic slur for the Irish people in general: “micks.” Some names beginning with Mc- or Mac- are not strictly patronymic, but are the professional or descriptive names of the father. Thus MacMaster denotes the “son of a master or cleric”, Macpherson means “son of the parson,” MacWard is the “son of a bard,” MacKinzie signifies “son of the fair one,” MacDuff means “son of the dark one,” and McDowell indicates a “son of the dark stranger.”

M’ appears to have been a common, now obsolete, abbreviation of Mc and Mac. A check with Google Books suggests that usage of M’ as Mac died out around the end of the 19th century.

From Britannica.com

Another fallacy has proven much more resilient, however: that the Mc form is Scottish and the Mac form Irish (sometimes the reverse is asserted). This is entirely without foundation. Both forms are used indiscriminately, Mc being, of course, simply an abbreviation of Mac, as is M’ (now nearly obsolete but once widely used).

From scottishhistory.com

Mac vs Mc:

They are both EXACTLY the same word, the Mc is actually the abbreviated form of Mac (and sometimes meic) and was usually written M'c (sometimes even M') with the apostrophe indicating that the name has been abbreviated (there are many other characters indicating abbreviation including two dots under the c).

I suggest a look at the entry for "mac" in a Scots Gaelic dictionary e.g. Faclair Gàidhlig is Beurla le Dwelly 'na bhroinn and listening to the pronunciation of mac /maxɡ/ at https://www.faclair.com/Listen/mac.mp3.

In Victorian times, Scotland - then often referred to as "Northern Britain" - was very popular as Queen Victoria often went to her home in Balmoral.

The pronunciation thus reflected the educated English Victorian's attempt at an approximation of the Gaelic pronunciation /M^h/

The emphasis is on the [m] and the velar fricative [x] is not easily pronounceable by the English who reduced it to an "h". This seems to have been reproduced in print as an apostrophe.