What is a formal polynomial?

If $f(x) \in F[x]$, then $f(x)$ is a finite sum $a_0+a_1x+a_2x^2+\cdots+a_nx^n$, where all the $a_i \in F$ and the $x$ is just a "formal" symbol. So, no, the $x$ is not in $F$, but is just a symbol, often called an indeterminate.

Yes, if $\alpha \in F$, then by $f(\alpha)$ we just mean the polynomial obtained by replacing each occurence of $x$ by $\alpha$. Think of the well-known $\mathbb{R}[x]$ (you've been working in this ring since high school), and the polyomial $f(x)=2x+4 \in \mathbb{R}[x])$. Then, for example, if $2 \in \mathbb{R}$ (this is your $\alpha$), $f(2)=8 \in \mathbb{R}$.

To make this completely formal without having to worry about what the variable $x$ means, we can define the ring of polynomials over some given ring $R$ to be the following:

The underlying set consists of all sequences of elements from $R$ (starting with the index $0$ for convenience), which are $0$ (ie, the $0$-element in $R$) from some point.

The addition is given term-wise and the multiplication is given by the following:

If we write $ab = c$ then $$c_i = \sum_{k + l = i}a_kb_l$$

This of course gives us exactly the same rules for addition and multiplication that we are used to, and it does not really help with anything but formalism (for example, the only way I can remember how the multiplication goes is through the usual way of writing polynomials).

Some additions:

As mentioned, we can see $x$ as the sequence $(0,1,0,\dots)$.

If $\alpha\in R$ and $f = (f_0,f_1,\dots,f_n,0,\dots)$ is some element in the ring of polynomials, then we can define the element $f(\alpha)\in R$ by $f(\alpha) = \sum_{i=0}^nf_i\alpha^i$ which then corresponds to the usual way we would evaluate a polynomial in the element $\alpha$.