Is $\mathbb{N}$ impossible to pin down?
Solution 1:
If you are a Platonist, or in any other sense believe that there is a "true" set of natural numbers, then you have pinned them down. If you believe that there is one concrete universe of sets, and suppose it even satisfies the axioms of $\sf ZF$, then that's universe $\omega$ can be thought of as the one true set of natural numbers.
But if you are not a Platonist. Whether you are a formalist, or supporting a multiverse approach, or maybe you just don't care enough to believe that one thing is true or another, then there is indeed a slight problem because we can switch between models of $\sf PA$ and models of $\sf ZF$, and thus get different "true" natural numbers.
But the point is this, in my opinion, that when we are set to work and we take $\sf ZF$ to be our foundational theory, then we fix one universe of set theory that we work in. And being foundational this universe cannot disagree with the notion of natural numbers coming from the logic outside of it (so in particular it is going to have other models of $\sf ZF$ within it). Then the natural numbers are the $\omega$ of that universe.
When you are done working with this universe you throw it in the bin, and get another when you need to. Or you can save that universe in a scrapbook if you like.
The reason we can do this, or better yet, the reason that people don't care about foundational problems in many places throughout mathematics, is that we use properties which are internal to that set, and therefore are true in any set with those properties, regardless it being the "true" or "false" set of natural numbers. If you write the sentence $\lnot\rm Con(\sf PA)$, and shown it to be independent of $\sf PA$ then you have shown that first-order logic is insufficient to determine all the true properties of the natural numbers. But those that it does determine are enough for a lot of things, and most people don't mind switching to second-order in some of the cases. In which case the above argument fails to hold.
Solution 2:
There are three closely related concepts:
The collection of natural numbers
The collection of (correct) formal proofs
The collection of finite sequences from a fixed finite alphabet (for example, the collection of finite sequences of 0s and 1s).
Forgetting about formal systems for a moment, if we can pin down just one of these three concepts, the other two are also going to be pinned down - but it is very difficult to define any of these three concepts without referring to the others. That difficulty shows up in particular when we try to define the natural numbers in formal systems, or try to formalize proofs within a formal system such as ZFC.
There is no reason that we must interpret this as saying that the categoricity of $\mathbb{N}$ is "illusory" - we could also view it as saying that effective formal systems are simply never strong enough to prove all number-theoretic truths. This deficiency of formal systems only affects $\mathbb{N}$ in settings where $\mathbb{N}$ is defined using a formal system. Many mathematicians feel they already understand what $\mathbb{N}$ is before they learn anything about formal systems, however.
Solution 3:
Yes, you are correct, at least in some sense.
But rather than thinking of "the natural numbers", I prefer to think more along the lines of, for each (adequate) set-theoretic universe, there is a particular model $\mathbb{N}$ having a specific, useful relationship to the universe containing it.
Solution 4:
When one speaks of "natural numbers" today, one realizes that these are infinite in number, but one also typically has in mind concrete counting numbers as examples. A quick thought experiment would indicate that the natural numbers are perhaps more speculative/ideal than is generally thought: consider a computer the size of the universe, computing the duration of time allotted to our universe, and allowed to use rapidly growing functions as rapid as the computer can design them. Let $N$ be the largest integer that can be expressed with all the allotted space and time, assuming the most efficient computer design and software. Now take the number $N+1$. This number cannot even in principle be expressed in any conceivable meaning of the term. For all practical purposes, this number is "infinite" or at any rate "inaccessible". What I am trying to suggest is that the properties of such numbers belong in the realm of the ideal, perhaps offering a consolation for being unable to pin them down as concretely as typical "counting numbers".
In fact, this type of observation is the motivational starting point of Edward Nelson's theory (see http://www.ams.org/journals/bull/1977-83-06/S0002-9904-1977-14398-X/home.html) where some integers are "standard" (i.e., "accessible"), and some are not. The discussion above is interpreted as suggesting that the integers are not as "uniform" as they are thought to be when one has ordinary "counting numbers" in mind. The stratification introduced by Nelson in terms of a new predicate "standard" results in a syntactic enrichment of the traditional ZFC where the usual reals contain infinitesimals and one can do infinitesimal calculus Leibniz's way. So the "non-uniqueness" of the natural numbers turns out to be an asset rather than otherwise.
Solution 5:
There's actually two distinct ways that the structure $\mathbf{N}=(\mathbb{N},S,0,+,\times)$ cannot be pinned down.
The first-order theory of $\mathbf{N}$ cannot be enumerated by a Turing machine. This can be deduced as a corollary of Godel's first incompleteness theorem.
The first-order theory of $\mathbf{N}$ isn't categorical, meaning that it doesn't pin down the structure $\mathbf{N}$ up to isomorphism. This can be viewed as a corollary of the upward Lowenheim-Skolem theorem, but it's best viewed as a corollary of the much stronger statement that the first-order theory of $\mathbf{N}$ fails even to be $\omega$-categorical.
As a general rule, whenever a non-expert mentions Godel's incompleteness theorems, I always begin by explaining this distinction; it's an empirical fact that doing so vastly increases the probability that the ensuing discussion will actually be enjoyable and thought-provoking :)