Minimize the expected value of the product of 2 normally distributed variables

The formula is correct, but your application of the formula is not. The correct expectation using this formula is $$\begin{align} \operatorname{E}[X^5 Y^3] &= \sigma_1^5 \sigma_2^3 \frac{5! 3!}{2^{(5+3)/2}} \sum_{j=0}^{\lfloor \min(5,3)/2 \rfloor} \frac{(2\rho)^{2j+1}}{(\frac{5-1}{2} - j)! (\frac{3-1}{2} - j)! (2j+1)!} \\ &= (\sqrt{2})^5 (\sqrt{8})^3 (45) \sum_{j=0}^1 \frac{(-1)^{2j+1}}{(2-j)!(1-j)!(2j+1)!} \\ &= 5760 \left( \frac{-1}{2! 1! 1!} + \frac{-1}{1! 0! 3!}\right) \\ &= -5760 \left( \frac{1}{2} + \frac{1}{6} \right) \\ &= -3840. \end{align}$$

That said, there is no guarantee that $(X,Y)$ is bivariate normal. As such, the formula is not applicable because you are interested in the minimum value of this expectation under all distributions that satisfy the given criteria, and it is not assured that the minimum is attained for the case when $(X,Y)$ is in fact bivariate normal.

We can use Cauchy-Schwarz in the form $$|\operatorname{E}[X_1 X_2]|^2 \le \operatorname{E}[X_1^2]\operatorname{E}[X_2^2]$$ to show that $\operatorname{E}[X^5 Y^3] \ge - 5760\sqrt{7}$, but this lower bound is not necessarily attainable.