How did early mathematicians make it without Set theory?

Some hints...

You can start reading Cauchy's elucidation of function (1823) :

On nomme quantité variable celle que l'on considère comme devant recevoir successivement plusieurs valeurs différentes les unes des autres. On appelle au contraire quantité constante toute quantité qui reçoit une valeur fixe et déterminée.

[We name variable a quantity that receives successively many different values. We name constant a quantity that receives a fixed and determined value.]

Lorsque des quantités variables sont tellement liées entre elles, que, la valeur de l'une d'elles étant donnée, on puisse en conclure les valeurs de toutes les autres, on conçoit d'ordinaire ces diverses quantités exprimées au moyen de l'une d'entre elles, qui prend alors le nom de variable indépendante; et les autres quantités, exprimées au moyen de la variable indépendante, sont ce qu'on appelle des fonctions de cette variable .

[When some variable quantities are linked together in a way that, having fixed the value of one of them, all others quantities can be determined, on conceive these different quantities as expressed by way of one of them, named independent variable. The remaining quantities, expessed by way of the independent variable, are named functions of that variable.]

Thus, in a nutshell, the concept of "function" was a primitive one, like today for set. A function is a correspondence (a relation) between two "variable quantities".

It is worth noting that Cauchy's definition of "variable quantity" was already present into de L'Hôpital's textbook : Analyse des infiniment petits pour l'intelligence des lignes courbes (1st ed, 1696), the first calculus' textbook. See :

• Robert Bradley & Salvatore Petrilli & CEdward Sandifer (editors), L’Hôpital's Analyse des infiniments petits (2015).

An early occurrence of "function" is in Leibniz, in De linea ex lineis numero infinitis ordinatim ductis inter se concurrentibus formata, easque omnes tangente, ac de novo in ea re Analysis infinitorum usu (1692), but a "reasonable" definition of function is available only with Johann Bernoulli, Remarques sur ce qu'on a donne jusqu'ici de solutions des problemes sur les isopdrimitres (1718) and Leonhard Euler, Introductio in analysin infinitorum (1748).

Regarding group we may see e.g. Arthur Cayley : he uses the name "set" in his definition of group (1854) :

A set of symbols : $1,α,β,\ldots$ all of them different, and such that the product of any two of them (no matter in what order), or the product of any one of them into itself, belongs to the set, is said to be a group.

Set here is not a mathematical object : no specific properties of sets are assumed.

I recently read a scholarly book relevant to this question.

Jeremy Gray, Plato's Ghost: The Modernist Transformation of Mathematics

The transformation in question goes from 1880 to 1910 (roughly speaking). Gray discusses how mathematics was done before, the turmoil surrounding the change, and how it was done after.

Recommended. But not for the faint of heart.

We can get a hint to an answer by considering some of today's undergraduate textbooks which do not assume a knowledge of set theory, and do not include a formal definition of a function. There are still a lot of such books.

An entire calculus course can be taught without dwelling on set theory. In fact, I know that if I have to teach calculus to a big class, the worst thing I can do is to spend half an hour on set theoretical issues; I can see the glaze settle over their eyes (except for the small minority of students who instead get a glint of excitement in their eyes, and who I am constantly on the lookout for).

A student can ingest what they need to know about derivatives and their applications, integrals and their applications, differential equations, et cetera, and that student can go on to apply calculus in many different branches of inquiry, without me ever spending that half hour on set theoretical issues. And that's a good thing.

The term "set theory" as used in the phrase "set theory was invented in the late nineteenth century" doesn't just refer to "the concept of a collection of objects". Of course people have always understood the notion of taking a collection of things and referring to them by a single name. What was new in the nineteenth century was the creation of a formal framework for talking about much more complex problems involving infinite sets - cardinality, for example, or theorems like Zorn's lemma. The Continuum Hypothesis. I don't need ZFC to talk about the set of fingers on my left hand, to conclude that that set contains five elements, or that its union with the set of fingers on my right hand contains ten elements. So this is a bit like asking "How did ancient man know that objects were solid without knowing about electrostatic force?".

Mathematicians in Cauchy's time thought of functions as "rules" that assigned one output number to each input number. I'm not sure why you think modern set theory is needed in order to be able to talk about such a concept.

As for Galois and Abel, the notion of a group as a set with axioms imposed on it didn't exist until much later, although I'm sure they wouldn't have had any issue with such a definition, other than that they might not have seen any motivation for it. They thought of groups as permutation groups on a (finite) set of solutions to an equation - because again, not having a precise notion of the Axiom Of Extensionality or the undecidability of CH doesn't prevent anyone from talking about swapping around elements in a finite set of objects.