Who named "Quotient groups"?

Read Julia Nicholson's paper "The development and understanding of the concept of quotient group", Historia Mathematica 20 (1993), 68--88. (If you have suitable permissions, you can read it online at http://www.sciencedirect.com/science/article/pii/S0315086083710074.) I will summarize the basic story.

The idea of a quotient group arose in work of Galois, Betti, Jordan, Dedekind, Frobenius, von Dyck, and Hölder. Jordan came close to the concept, but was missing the abstract idea of thinking about it as a set whose elements are cosets and he did not use the name you are asking about. It was finally defined and named by Hölder, who wrote $G|H$, rather than $G/H$, and called it the quotient of $G$ and $H$. So the answer to your question in the title is "Hölder".

Jordan, in the 1870s, almost had the full idea; he introduced the congruence relation $g_1 \equiv g_2 \bmod H$ when $H$ is a normal subgroup of $G$ and he showed multiplication on congruence classes mod $H$ is well-defined, but when it came to making a new group from this congruence relation Jordan defined the group operation on a fixed set of representatives mod $H$. To me this sounds very similar to the way addition mod $m$ can be described in an elementary way as "clock arithmetic", using a wrap-around addition directly on the set of numbers $0, 1, \dots, m$ instead of on the set of cosets mod $m$. Jordan wrote the resulting group as $\frac{G}{H}$ and called it le groupe suivant le module $H$.

Hölder, nearly twenty (!) years later in 1889, took the final step of viewing this new group as a set whose elements are equivalence classes (the $H$-cosets) in $G$. Jordan could not take that step because for him the elements of a group were always permutations. He did not have the language to speak about a group whose elements are cosets (of permutations). Here is the excerpt from Hölder's paper where he used the term "quotient": Man erhält so neue Operationen, welche gleichfalls eine Gruppe bilden. Diese vollständig bestimmte Gruppe ist es, welche in die Betrachtung eingeführt werden soll. Man könnte sie den Quotienten der Gruppen $G$ und $H$ nennen, dieselbe soll im Folgenden mit $G|H$ bezeichnet werden. There is a link to the paper in Bill's answer and Hagen von Eitzen provides a translation of this excerpt into English in a comment to that answer.

This paper of Hölder's is where the Jordan-Hölder theorem assumed its modern form, as a theorem comparing the quotient groups in two composition series of a finite group. Jordan's earlier version of the theorem only compared the quotients of the orders of successive subgroups in any two composition series. While Nicholson does not come out directly to say why the name "quotient group" was chosen, it seems to me that it was because quotients of orders of subgroups were, in the evolving Jordan-Hölder theorem, the precursor of quotient groups. It would also explain why such groups were written by Jordan using the notation of fractions.

Mathematicians writing in the years soon after Hölder's paper used the word he had introduced. In a paper in 1893, Cayley wrote $G/H$ and called it a quotient. The term quotient group was used by Burnside in his book Theory of Groups of Finite Order (1897), which was the first systematic textbook treatment of group theory in English.

In some other languages (e.g., Russian and German) they use the term "factor group", although this term was introduced by Hölder to mean something more precise: he used Factorgruppe (see p. 33 of his paper) to refer to the simple quotient groups that arise from a composition series. Why call them factor groups? Probably because these simple quotient groups "decompose" the original group, hence they are like prime factors. Later Burnside, in his book, treated the terms "quotient group" and "factor group" as synonyms for what we call quotient groups. Perhaps that is how the terms factor group and quotient group wound up getting used to mean the same thing.

You might also like to read section 1.1 of Leo Corry's book Modern Algebra and the Rise of Mathematical Structures, which discusses more generally the genesis of group theory.

According to Young, Amer. Jnl. Math, 1893, p.130 the terminology was introduced by Hölder when he proved the uniqueness of the factor groups in composition series (Jordan-Hölder theorem) in his 1889 paper in Mathematische Annalen titled Zurückführung einer beliebigen algebraischen Gleichung auf eine Kette von Gleichungen. This is often cited as the first clear formulation of the concept of a quotient group. Perhaps someone who reads German can tell us what the German term was and, further, tell us if Hölder gives any reason for the choice of the terminology.

Update: skimming Hölder's article confirms Young's claim: Holder uses the terminology "Quotienten der Gruppen", see the following excerpt from p.31

Julia Nicholson wrote in her 1993 Historia Math. paper on the history of quotient groups

In 1893 Cayley wrote a "Note on the so-called quotient G/H in the theory of groups," in which he cited Hölder's paper of 1889 and referred to a paper of Young's [1893], who, in his turn, attributed the definition of quotient group and the term 'quotient' to Hölder. In a paper of his own [1893, 317], Hölder cited Jordan, saying that quotient groups had already appeared in Jordan's work and that he (Hölder) drew up their theory anew in 1889.

If the group $G$ has $24$ members and a normal subgroup $H$ has $6$ members, then $G/H$ has $24/6$ members and $24/6$ is clearly a quotient. Division is an operation on numbers defined initially by thinking about finite sets: split a set of $24$ members into parts of size $6$; then how many such parts are there? Clearly that is a quotient, in the sense of a number that results from division, and clearly that's how quotient groups are formed: a quotient group is a group of cosets.

(This is "quotition division" as opposed to "partition division".)