How to find signature of $\phi(A,B)=tr(AB)$

As you note, the canonical inner product on $\Bbb R^{n \times n}$ is given by $\langle A,B \rangle = \operatorname{tr}(AB^T)$. As such, every bilinear form $\phi$ can be uniquely expressed as $$ \phi(A,B) = \langle \Phi(A), B \rangle $$ for some linear transformation $\Phi: \Bbb R^{n \times n} \to \Bbb R^{n \times n}$. If the bilinear form is symmetric, then $\Phi$ must be self-adjoint. The signature of a symmetric bilinear form corresponds to the signs of the eigenvalues of the corresponding map.

In our case, we find that $\Phi(A) = A^T$. That is, our bilinear form can be written as $$ \phi(A,B) = \langle A^T,B \rangle $$ The map $\Phi$ has eigenspaces $\ker(\Phi - \operatorname{id}) = S_n$ of dimension $n(n+1)/2$ and $\ker(\Phi + \operatorname{id}) = A_n$ of dimension $n(n-1)/2$. Thus, we find $$ \sigma(\phi) = \left(\frac{n(n+1)}{2},\frac{n(n-1)}{2},0 \right) $$