Why is a function space considered to be a "vector" space when its elements are not vectors?

A vector space is just a set in which you can add and multiply by elements of the base field. You can add polynomials together and multiply them by real numbers (in a way satisfying the axioms,) so polynomials form a vector space. A vector is nothing more or less than an element of a vector space, so polynomials can be seen as vectors.

The formal definition of "vector space" follows a pattern that is seen very often in modern mathematics. You start with some particular object(s) - in the case of vector spaces these are the $n$-dimensional euclidean spaces - and figure out the essential laws governing these object(s). You then make these laws the axioms of some newly defined class of objects (vector spaces in this case), and see whether you find other object(s) besides the one you already knew which follow these laws.

In the case of vector spaces, these laws (or axioms) are essentially

• That any two elements can be added, and that addition is commutative and associative (i.e. that $a + b = b + a$ and $a + (b + c) = (a + b) + c$).
• That there's a neutral element $\mathbf{0}$ for which $a + \mathbf{0}$ for every $a$.
• That elements can be multiplied with elements from some field. Fields are the class of objects which follow the same basic laws of arithmetic that hold for the real numbers. The elements of the field are generally called scalars.
• That this multiplication operation is compatible with the addition and multiplication operations of the field. In other words, if $x,y$ are elements of the field and $a,b$ are vectors, we require that $(x+y)\cdot a = x\cdot a + y\cdot a$, that $1\cdot a = a$, and that $0\cdot a = \mathbf{0}$.

Any set $X$ that (together with a field of scalars $F$) obeys these axioms is called a vector space.

I think you are confused between the idea of a vector (ie. element of a set that forms a vector space) and the coordinate vectors of real numbers usually used to keep track of these vectors. More abstractly, vector spaces can be over any field.

In the case of polynomials, after choosing a basis (this is always possible with finite dimension), we can write down the coordinate vectors in $R$, if our polynomials have real coefficients.

For example, we may associate the polynomial $f=\sqrt 2x^2 +2x +\pi$ with $(\sqrt 2,2,\pi)\in \mathbb R^3$ after deciding on the ordered basis $\{x^2,x,1\}\subset \{\text {polynomials of degree at most 2}\}$.

To sum this up nicely, we say that for finite dimensional vector space $V$, $V\cong R^n$ where $n$ is the dimension of the vector space over some field, $R$. This is due to the necessary "free-ness" as modules in the category theoretic sense.

But they are vectors!

Generally speaking, a space of functions mapping a topological space into a vector space satisfies the axioms of a vector space under the usual operations of addition and scalar multiplication. You don't need to have a basis to have a vector space, and in fact, it is often profitable to find or construct a basis that suits the problem you're interested in.