Mathematical notation for the maximum of a set of function values

Your notation looks fine. You could also use the more informal $\alpha = \max(\{f(x_1),\ldots,f(x_n)\})$ or even $\alpha = \max(f(x_1),\ldots,f(x_n))$.

Finally, you could say that $\alpha$ is the maximum (or maximal) value among $f(x_1),\ldots,f(x_n)$, or that $\alpha$ is the maximum (or maximal) value attained by $f$ on the points $x_1,\ldots,x_n$.

The most concise notation for this is just

$$\max f[n]$$

where $f[A]$ is the image of $A$ under $f$ and $n = \{m \mid m < n\}$ is the ordinal definition of numbers (assuming you start at 0 rather than 1).

According to Wikipedia you don't need the commas: $$\alpha = \max \{ f(x) : x = 1 .. n \}$$ Alternatively: $$\alpha = \max \{ f(x) : x \in \mathbb{Z} \land 1 \leq x \leq n \}$$