What is the mathematical relevance of whether an expression has a closed form?

As others have pointed out this is mostly a matter of convention, but the convention is not completely arbitrary. Note that it has nothing to do with the numerical approximation or infinite series; by that standard you should be just as unhappy with $\sqrt{x}$ or $\sin(x)$ as you are with $e^x$. (Indeed, rational functions are the only functions whose values can be calculated without infinite series.)

Instead, the convention has to do with differential equations. The reason why $e^x$, $\sin x$, and $\cos x$ are considered elementary is that (together with polynomials) they are the only functions you need to write down solutions to differential equations of the form $Dy = 0$ where $D$ is a linear differential operator (i.e. a polynomial in $\frac{d}{dx}$). Since linearization is is the most basic tool in the theory of differential equations in general, exponential functions and trigonometric functions play a central role. Logarithms (and inverse trigonometric functions) appear as inverses of elementary functions.

This discussion is the starting point for a larger theory. One defines a differential field to be a field $F$ equipped with a derivation $d$; the standard example is the field of rational functions equipped with differentiation. Just as one extends ordinary fields by adjoining a root of a polynomial, one can extend differential fields by adjoining a solution to an equation involving the derivation. It turns out that two specific differential field extensions are most fundamental:

• Exponential: adjoin a solution to $dy = y \cdot df$ for $f \in F$
• Logarithmic: adjoin a solution to $dy = \frac{df}{f}$ for $f \in F$.

This language can help answer questions about the structure of solutions to linear differential equations (for instance, which functions have elementary antiderivatives).

It isn't. The concept of "elementary" is rather arbitrary, and most often not relevant. In particular, from an advanced point of view, the exponential is defined as a series.