What did mathematicians study as an undergraduate/graduate before modern mathematics such as modern algebra and analysis?

To give one example: Cauchy. I choose to discuss him because he straddled the threshold of "modern analysis."

He entered the lycée ("high school") École Centrale du Panthéon in 1802, studying humanities. Then, in 1805 he entered the École Polytechnique at age 15.

The "core" mathematics curriculum at the École Polytechnique was:

Analysis instruction: the Cours d'Analyse Algébrique by Garnier and the Traité Élémentaire de Calcul Différentiel et Intégral by Lacroix;

Mechanics instruction: the Traité de Mécanique, using the methods of Prony and edited by Francoeur and the Plan Raisonne du Cours de Prony;

Descriptive geometry instruction: the Géométrie Descriptive by Monge;

Applied analysis instruction: the Feuilles d'Analyse Appliquée a la Géométrie by Monge and the Application de l’Algèbre a la Géométrie by Monge and Hachette.

(Belhoste, Bruno. Augustin-Louis Cauchy: A Biography. New York: Springer-Verlag, 1991. pp. 7-10. Appendix II is Cauchy's outlines of his analysis courses he taught at the École Polytechnique from 1816-1819.
See the distribution of courses at the École Polytechnique when Cauchy was a student there.)

Also, according to the Oxford English Dictionary, "analysis" originally meant

the proving of a proposition by resolution into simpler propositions already proved or admitted (obs.); (later) algebra (now hist.). Now: the branch of mathematics concerned with the rigorous treatment of functions and the use of limits, continuity, and the operations of calculus.