What in Mathematics cannot be described within set theory? [duplicate]

A brief digression before I attempt to answer your question: This question really begins to move out of mathematics proper and into the realms of philosophy of mathematics. (Which shouldn't be too surprising, since Suppes in fact was a well-respected philosopher of mathematics at Stanford! )But it really shows how it's nearly impossible to carefully examine the foundations of mathematics without running headlong into philosopical questions.

Basically, the objects that can't be constructed with sets in mathematics are objects that make sense grammatically but run into logical issues because they're "too large" i.e. they are collections that are too broadly defined to be able to consistently specify what elements they contain. The most famous example of this - but by no means the only one - is Russell's paradox, which asks about the set of all sets which are not members of themselves. Let's call it S. If you think carefully about this, it blows up: If S is not a member of itself, then it must be in S. But if it's in S, then it can't be a member of itself and we have a contradiction. You can find more information about it here: http://en.wikipedia.org/wiki/Russell%27s_paradox It was this and other related paradoxes that motivated mathematicans to create axiomatic formulations of set theory that have axioms that strictly limit the kinds of collections that are "legal" in a given set theory.

This may seem like just wordplay, but the limitations of Russell's paradox are felt in many areas of modern mathematics at the most basic level. Consider the set of all topological spaces or the set of all groups. The reason these collections cannot be sets is because they can be put in one to one correspondence with the set of all ordinal numbers. It can be shown using logic that this is eqivalent to considering the set of all objects with a certain mathematical property i.e. is a member of itself, which leads us back to Russell's paradox! It actually leads to a more formal but equivalent version of the paradox called the Burali-Forti paradox. That doesn't necessarily mean we discard collections like this - there are versions of set theory which include collections like this and call them proper classes. These are collections which are perfectly sensible but too large to be sets. In category theory, where these kinds of collections turn up all the time, these kinds of set theoretic issues have haunted mathematicians from the beginning. Which is why some of them propose disposing of set theory altogether and basing the foundations of mathematics on category theory. To me, this is the proverbial throwing the baby out with the bathwater. That being said - you can see how complex and thorny these issues can be, which is probably why most practicing mathematicians just ignore them. If these questions intrigue you - as they do me - hit google and begin doing your research!

I think there is an argument to be made that nothing in mathematics can not be formalized in set theory unless you are deliberately going out of your way to push the limits of set theory in unnatural ways. I will explain what this means later.

First, you should ask what it means to be an object of mathematics. Except for finite or recursive objects, it is difficult to make an argument for in what sense certain objects exist. For example, is the first uncountable ordinal $\omega_1$ really just an object described by set theory or that its existence is entirely dependent on the underlying set theory. Or if you are familiar with algebra, most of the interesting theorem are about subrings, subgroups, etc and rarely about the multiplication of the ring or group elements. You are not exactly proving these theorems in the theory of rings or groups, so where exactly are the familiar theorems of algebra being proved. Anyway these examples are to show that ordinary mathematics is already using some of the set construction principles perhaps formalized in something like $\mathsf{ZFC}$. Upon some ontological inspection, several particular parts of mathematics have very strict rules, how come one take it for granted that one is allowed to take unions of sets, pairs elements together, define subsets using certain properties, etc. If one tries to apply mathematical rigor to this area, then one may realizes that one was already using certain rules for the manipulation of diverse mathematical objects. Of the many possible alternative, $\mathsf{ZFC}$ set theory (and its extensions) has been a particularly use and natural such foundation.

Finally, I am usually very skeptical when someone asserts that something can not be handled in first order set theory and asserts that one should work in class set theory or some foundational category theory. A common argument is that the object of study is a proper class (not a set) and hence totally beyond the capabilities of set theory. Generally saying something is a proper class means it is first order describable in the language of set theory.

For the most part, if someone asserts that a theorem of a algebra, number theory, etc (anything not about set theory or category theory) and he or she states that the problem can not be handled in set theory, I generally think it is an elaborate example of: "I can not classify the finite groups of order 2 in set theory because the class of groups of order 2 is a proper class".

There's an area called Category Theory. It deals with the notion of a category, consisting of a class of objects (which in general is $NOT$ a set and morphisms between these objects.

For example, there is a category called $set$ in which the objects are all the sets in math, and its morphisms are maps between sets.

Google it up. Also "proper class".

There's also the notion of cardinal of a set. intuitively, given a set $X$, $card X$ denotes the "quantity" of elements of $X$. The class of all cardinals is not a set.