What does the symbol $\operatorname{Tr}$ in the Yang-Mills action mean?

We choose the physics convention $F_{\mu\nu} = \partial_\mu A_\nu - \partial_\nu A_\mu - i g [A_\mu,A_\nu]$, where the generators of the Lie algebra are Hermitian. In this case the integral has a minus sign, $\mathcal{J} = -\int \operatorname{Tr}(F \wedge * F)$.

The first thing to note is that $F$ has vector and Lie algebra indices. The trace is over the Lie algebra, not over the vector indices. The vector indices are just like those of the field strength in QED. In Yang-Mills the field strength (also called the curvature form) is Lie algebra valued. In this case $F_{\mu\nu} = F_{\mu\nu}^a T^a$, where the summation convention is used, and where the $T^a$ are the generators of $\mathfrak{su}(n)$. To be explicit, $F$ has not only tensor components but matrix components $$(F_{\mu\nu})_{ij} = F_{\mu\nu}^a T^a_{ij}.$$

The inner product of $F$ with itself is by definition $(F,F) = \int F\wedge * F$ where $*$ is the Hodge $*$ operator. Thus, we are calculating $\mathcal{J} = -\operatorname{Tr}(F,F)$. It is a standard exercise to find the exterior product of two $r$-forms. We find $$F\wedge * F = \frac{1}{2!} F_{\mu\nu} F^{\mu\nu} dx^1\wedge\cdots\wedge dx^4.$$ Note that the differential forms don't "know" the Lie algebra. The algebra hasn't come into the calculation yet.

From here we must calculate the trace of $F_{\mu\nu}F^{\mu\nu}$, $$\begin{eqnarray} \operatorname{Tr} \left(F_{\mu\nu}F^{\mu\nu}\right) &=& \operatorname{Tr} \left( F_{\mu\nu}^a T^a F^{\mu\nu b}T^b\right) \\ &=& \operatorname{Tr}\left(T^a T^b\right) F_{\mu\nu}^a F^{\mu\nu b} \\ &=& \frac{1}{2} \delta^{a b} F_{\mu\nu}^a F^{\mu\nu b} \\ &=& \frac{1}{2} F_{\mu\nu}^a F^{\mu\nu a}, \end{eqnarray}$$ where we have used the standard normalization convention for the $T^a$, $\operatorname{Tr} T^a T^b = \frac{1}{2}\delta^{ab}$. (This comes from the fact that we want $\mathfrak{su}(2)$ to live naturally in $\mathfrak{su}(n)$, and the generators of $\mathfrak{su}(2)$ are taken to be $T^a = \sigma^a/2$, where $\sigma^a$ are the Pauli matrices.) Thus, we find $$\begin{eqnarray} \mathcal{J} &=& -\int \operatorname{Tr}(F \wedge * F) \\ &=& -\frac{1}{4} \int d^4x \, F_{\mu\nu}^a F^{\mu\nu a}. \end{eqnarray}$$

It's the trace and it's being taken over the vector space of the fields $F$.

As others have mentioned this is just the trace on matrices. But the following viewpoint may help to see where this is coming from. $F$ lives in the tensor product space $su(2) \otimes \Omega^2(M)$ of the Lie algebra $su(2)$ with $2$-forms. $\Omega^2(M)$ has an inner product given by the $L^2$ inner product and $su(2)$ has a unique (up to scale) Ad-invariant inner product given by the Killing form, which is really just $(A,B) \mapsto tr(AB)$ viewing $A, B \in su(2)$ as $2 \times 2$ matrices. Taking the induced inner product on the tensor product, the expression you wrote is the norm squared of $F$.