Purpose of the Peano Axioms
Solution 1:
Peano axioms come to model the natural numbers, and their most important property: the fact we can use induction on the natural numbers. This has nothing to do with set theory. Equally one can talk about the axioms of a real-closed field, or a vector space.
Axioms are given to give a definition for a mathematical object. It is a basic setting from which we can prove certain propositions.
As it turns out, however, it is possible to use the natural numbers as a basis for some of our mathematics, and we can use Peano axioms to model first-order logic (its syntax and the inference rules), and the notion of a proof.
This can be seen as a basis for some of the mathematics we do, however it is often a syntactical basis only: we only use the integers to manipulate strings in our language and sequences of these strings. We do not have the notion of a structure, of a model.
But it seems that you are mainly confused by the use of the term "axioms". These are just basic properties for a mathematical object. In this case "the natural numbers". Much like there are axioms in geometry, but geometry doesn't usually serve as a basis for many parts in mathematics.
Solution 2:
You must separate two approaches, of course interconnected but conceptually independent from each other : "rigorization" and "foundation".
With "rigorization" I mean the work of most of XIX century mathematician on calculus, aimed at removing the obscurity inherited from Newton's and Leibniz's discoveries. The work of Cauchy , Weierstrass , Bolzano, Cantor, etc. gave us the modern definition of limit, etc. based on real numbers.
This effort of rigorization was completed at the turn of the century by Dedekind, Frege, Peano and Hilbert, who achieved several big results :
Dedekind (1872 - Stetigkeit und irrationale Zahlen (Continuity and irrational numbers) : analysis of irrational numbers and construction of real numbers as Dedekind cuts , i.e. as non-geometrical entities;
Frege (1879 : Begriffsschrift and 1884 : Die Grundlagen der Arithmetik) : modern mathematical logic and philosophical anlysis on the nature of numbers;
Dedekind (1888 - Was sind und was sollen die Zahlen? (What are numbers and what should they be?) and Peano (1889 - The principles of arithmetic presented by a new method ) : axiomatisation of natural number, i.e. characterization of the mathematical structure that we refer to as "natural numbers" in terms of some basic properties (as you said : "agreed to be basic and reasonable") form which all known properties of natural numbers (their "behaviour") can be deduced in a rigorous way (see also Frege) .
Hilbert ( 1899 - Grundlagen der Geometrie): modern axiomatisation of geometry ;
Hilbert ( 1928 - with Wilhelm Ackermann, Grundzüge der theoretischen Logik) : foundations of mathematical logic (based on works of Peano, Frege and Russell)
Hilbert ( 1920s-30s - with Paul Bernays, Grundlagen der Mathematik, vol. 1 - 1934 and vol.2 - 1939) : mathematical investigations on formal systems.
These mathematicians (with Russell, Brouwer, Weyl) tried also to develop research programs (e.g. logicism, intuitionism, formalism) aimed at answering basic philosphical questions about the existence of mathematical objects (i.e.numbers), the way we can have knowledge of them, etc.
Those research programs was greatly propelled by the discovery of Paradoxes (Cantor's, Russell's), so they become known as "foundational" programs, aimed at find the basic principles that can "secure" our mathematical knowledge.
One of the most important result of this movement was Zermelo's axiomatisation of Set Theory (ZFC) : with this, mathematicians was able to find some basic axioms (but this time NOT all "agreed and reasonable") capable of "generating" (up to now without contradictions) all known properties of set AND capable also of building a proxy for other mathematical structures, like natural numbers. This means that Peano Axioms for numbers are now theorems of Set Theory.
Can we say that we have reduced numbers to set ?
From one point of view : YES. The language of sets is so basic that quite all mathematical concepts can be "described" with it and sets' axioms are so powerful that all properties of the defined mathematical concept can be proved starting from them.
Form another point of view (more philosophical ) : NO. In what sense we can say that our basic insight into existence of natural number and their properties are less "clear" or "certain" than our insight into the existence of sets (the cumulative hierarchy) and their properties (i.e.Axiom of Choice) ?
Solution 3:
The axioms are to give us a statement of our knowledge of arithmetic that we can study through mathematical means. The idea is that to study what can be logically deduced about arithmetic, you first need as complete a statement as possible of your knowledge of how arithmetic works, in a form that makes logical consequence as unambiguous as possible. This is why Peano arithmetic. It also gives us a measure of, say, how well our set theory formalizes arithmetic; if our construction of the natural numbers yields the theorems of PA, we can with some confidence consider it to be $\mathbb{N}$.
PA would be a somewhat poor foundational system. There are lots of areas of mathematical study that aren't about numbers. Of course PA is fairly strong and can intepret a number of other formal systems, but to give an idea of how far that would get you, PA is equiconsistent with Zermelo set theory with the negation of infinity and foundation replaced by an $\in$-induction scheme, which is pretty weak.
Solution 4:
In his 1965 article
Benacerraf, Paul, What numbers could not be. Philos. Rev. 74 (1965) 47–73,
Benacerraf pointed out that if observer A learned that the natural numbers "are" the Zermelo ordinals $\emptyset$, $\{\emptyset\}$, $\{\{\emptyset\}\}$, etc., while observer B learned that they are the von Neumann ordinals $\emptyset$, $\{\emptyset\}$, $\{ \emptyset, \{\emptyset\}\}$, then, strictly speaking, they are dealing with different things. Nevertheless, observer A's actual mathematical practice is practically the same as observer B's. Hence, different ontologies may support one and the same practice.
Benacerraf's observation helps us place the set theoretic framework in a proper perspective: it can be the basis of formal mathematics if you choose it to be. As you have observed, in the case of Peano arithmetic it doesn't have to be.
Note also that many leading mathematicians no longer consider set theory to be the foundation of formal mathematics. This is because for various reasons set theory is too inflexible. Cutting edge research today often relies on category theory, rather than set theory, as the foundation.