Button up that frog, will you?
The origin of the term frog in relation to the braid on military uniforms (and almost uniquely cavalry uniforms) might come from the old term for frock. The OED has:
Frock 2. An upper garment worn chiefly by men... 1375. Barbour Bruce x. 375 With blak froggis all helit thai The Armouris at thai on thame had. c 1425 Wyntoun Cron. viii. xxxviii. 57 Ilkane a gud Burdowne in hand, And royd Frogis on þare Armyng. c 1460 Towneley Myst. (Surtees) 241, I wold be fayn of this frog [Christ's coat] myght it fall vnto me. 1500–20 Dunbar Poems li. 3 To giff a doublett he is als doure, As it war off ane futt syd frog
A 14th Century 'frock' or 'frog'. Note the belt and cord carrying gear. The term frog-loop and frog-belt have been used to describe these items up to the 19th Century (see below) and even today a frog-loop is still a term for a retainer for a tool in a tradesman's belt.
There are also references in the Dictionary of the Scots Language:
Frog, Froig, n.1 [ME. frog, frogge (15th c.), of uncertain origin.] A frock; a cloak or coat. Barb. x. 375. With blak froggis all helyt thai The armouris at thai on thame had; Wynt. viii. 5702. [They had] royd frogis on thare armyng, To covyre thame for persaywyng; Crying of Play 39. Five thousand ellis ȝeid in his frog Of hieland pladdis of haire; Dunb. li. 3. To giff a doublett he is als doure, As it war off ane futt syd frog; Doug. vi. v. 132. In hevy wayt frog stad and chargyt soyr; 1550–1 Treas. Acc. IX. 467. Spanȝe freis … to be ane froig to my Lady;
So what we might be looking at is frog(frock)-button, and frog(frock)-belt, as below:
1719 De Foe Crusoe ii. iv. (1840) II. 68 He drew a hatchet out of a frog-belt. 1827 Hone Every-day Bk. II. 190 A coat with frog-buttons. 1867 Smyth Sailor's Word-bk., Frog-belt, a baldrick.
Curiously (and perhaps destructively to this argument) Defoe 1719 also has:
1719 De Foe Crusoe i. xv, A belt with a frog hanging to it, such as..we wear hangers in.
Tailoring of simple jackets, frocks or 'frogs' only became usual from the 1500's onwards. Prior to that they were held in shape with belts, cords, lacing and over the shoulder straps which served the dual function of carrying tools and possessions on hooks and loops, and in bags hung from them. Pockets are a relatively late invention in clothing. (See: http://www.theinnerbailey.com/baileybasics-sets-townsman-1.jpg)
My suggestion (tending towards an answer) is that this early fashion of belting, lacing and buttoning was contemporary with the use of the ME word 'frog' for jacket, and the the technical expressions for items associated with them such as frog-button, frog-belt, frog-loop and frogging were all developed during that time. As fashion evolved in the 1500-1600 period and clothes were more tailored to the shape of the body and pockets came into use, not only did the term 'frog' evolve into 'frock', but those belts and cords became less important for shaping the material and for carrying items. Frog as a word for a frock evolved, but the use of frog in association with those accoutrements became fossilized.
To see how these fossilized terms for buttoning and braiding (frog-buttons, frogging) came to be associated with a military fashion of the early 1700's (in England) we need to look at Eastern Europe in the late 1400's.
Frogging (as we call it) is most closely, and most extravagantly, associated with lightly armoured mounted troops known as Hussars or Dragoons. This link to an image from c.1550 clearly shows the horizontal braid on a mounted Hungarian soldier's uniform:
Such troops, and their dress styles, evolved in Eastern Europe before spreading to Western Europe in the late 1600's. None of the languages or cultures (including that of the Ottoman Empire) appear to have contributed the word 'frog' or 'frogging' along the way.
The actual fashion which became quickly extravagant and an actual impediment to military action (1746 Berkeley Let. Wks. 1871 IV. 306 Laces, frogs, cockades..are so many..obstacles to a soldier's exerting his strength.) may have originated in loops in the jacket to hold additional weapons such as daggers and short swords, or in the lacing used to tighten loose plates of armour or a baggy cloth jacket around the body.
When the new style reached England in the mid 1700's there was no current word to describe the braiding (that had become increasingly ornate as it travelled west across Europe). It was recognized, however, as similar to the ornamentation formerly (and no longer) used on the item of clothing once known as a frog. The names for that ornamentation (including 'frogging') had never evolved because the items had been largely disused for about a hundred years, but were revived in their original form to describe the ornamentation on this new (to English eyes) military fashion.
This speculative essay into the evolution of the use of the word 'frog', and the suggestion that it's original association with the article of clothing, and the accoutrements to that clothing went down different paths, and that the word for the accoutrements was frozen for some one or two hundred years while the word for the clothing evolved, and how the frozen word was revived to describe a new fashion element, also explains how we arrived at DeFoes (otherwise improbable) description of a 'frock with frog-buttons'. Essentially he is saying, 'a frock with frock-buttons', but the language available to him has him say otherwise.
An extract from The Life and Extraordinary History of the Chevalier John Taylor, etc. The biography was written by his son in 1761, and was printed in Dublin, Ireland.
The story begins when the Norwich-born John Taylor is 19 years old in 1722. He is trained as an eye surgeon in St. Thomas's hospital in London, and during his career as an oculist he worked in England, Holland, Ireland and France; on page 78, the doctor flees to Portugal. During the three years he lives there, the ‘Chevalier’ becomes incredibly wealthy “… he restored to Sight the Viceroy of Brazil; for which Feat he got, to use his own Words, a Hat full of Gold, ...” Later, he escapes back to England where his father lives.
Having lately landed from Portugal, he writes to his father, who sends him immediately two Suits of Cloaths of blue Damsk lined with black Silk, and embossed with Frogs of the same colour. The Air and Cut of these Cloaths expressed something foreign, which the Mob, in their great Wisdom, were pleased to call Frenchified.
CHAP. II
THUS equipped, the young Chevalier travels to the Borough-Fair; where he seems an human Exotic in the Eyes of that polite Assembly. They soon saw the Ridiculousness of his Dress, and judged the Singularity of it to be a public Crime, which, by the Law of Nations, the Populace have ever presumed to punish.
[…]
A handsome genteel young Man, with a modest and inoffensive Deportment, is made the Butt of Insolence, Brutality and Abuse, in the most merciless and savage Manner, because he wore Frogs upon his coat, his Hat under his Arm, and his Sword by his Side.
[…]
The prescribed Criminal, with more Frogs on his Coat than he set out with, escapes to the Water-Side half frighted to Death; takes Boat, and in a Day or two recovers his Reason, and his Cloathing in some Degree.
The above extract appears to confirm that frog ‘buttons’ were used to adorn men's coats, hats, and belts, as described in @John Mack's answer.
Frog waistcoat buttons before OED's 1746 ...
The earliest match I found for these ornamental loops and buttons is dated 1735 from The Account of the Behaviour, Confessions, and Dying Words, of the Malfactors, who were executed at Kennington-Common (source)
…when he went into the Tavern he was dressed in a lightcolour'd grey Coat and Breeches, and a white Waistcoat with Frogs; but that whilst this Deponent was getting Assistance to apprehend him he had chang'd his Cloaths, to a brown Coat and Buck-skin Breeches…
and after 1746
In The Fortunate Country Maid Vol II, 1748
“… Is he not a tall young Man with black Eyes, and wears a Plaister or large Patch on his Forehead, his Hair curled, and had on when he came a light brown Surtout with gold Frogs?”. You are right, (replied the Stranger) ‘it is certainly he; I am very lucky in meeting you! And, tell me, pray is he here still?
and in the same year: A catalogue of English heads: or, An account of about two thousand prints
- Edward Lord Hinchingbrooke. Cap, Collar unbutton'd frogs, dated 1701
(The print shows the eight-year-old Viscount, the eldest son of the 3rd Earl of sandwich)- The Right Honourable Stephen Poyntz, Esq; ... Wig, Neckcloth, Frogs to his Coat