when does $\det(AB^T+BA^T)\le \det(AA^T+BB^T)$ hold?

I assume that $A,B$ are real $n\times m$ matrices. But the same proof gives the same result as below for complex matrices with the adjoint instead of the transpose.

For every real $n\times m$ matrices $A,B$, we have $$ |\det (AB^T+BA^T)|\leq \det(AA^T+BB^T). $$

Proof: if $x\in\ker (AA^T+BB^T)$, then $\|A^Tx\|^2+\|B^Tx\|^2=0$, whence $A^Tx=B^Tx=0$ and so $x\in \ker(AB^T+BA^T)$. In other words, if $\det(AA^T+BB^T)=0$, we do have $\det(AB^T+BA^T)=0$ and the inequality holds. So we will assume that $AA^T+BB^T$ is invertible from now on.

The matrix $C=(A-B)(A-B)^T$ is positive semidefinite. So $$ 0\leq (Cx,x)=((AA^T+BB^T)x,x)-((AB^T+BA^T)x,x) $$ whence $$ ((AB^T+BA^T)x,x)\leq ((AA^T+BB^T)x,x). $$ Changing $B$ for $-B$, we get altogether

$$ |((AB^T+BA^T)x,x)|\leq ((AA^T+BB^T)x,x). $$

Denote $C:=AB^T+BA^T$ which is symmetric, and $D:=AA^T+BB^T$ which is positive definite and has a unique positive definite square root $D^{1/2}$. Now the inequality above entails, with $x=D^{-1/2}y$: $$ |(D^{-1/2}CD^{-1/2}y,y)|\leq(y,y)=\|y\|^2\qquad\forall y\in\mathbb{R}^n. $$ Since $T=D^{-1/2}CD^{-1/2}$ is symmetric, it is diagonalizable in an orthonormal basis and the above shows that its spectrum is contained in $[-1,1]$. A fortiori $$ |\det (D^{-1/2}CD^{-1/2})|\leq 1\qquad \Rightarrow \qquad |\det C|\leq \det D. $$ QED.

Note: the inequality you wanted initially requires that $A,B$ be square matrices of the same size. As you observed, the inequality is not true in general. The transpose was not at the right place. It follows in particular from the inequality above that the inequality you first wanted holds when $A,B$ are both symmetric, or both skew-symmetric. Of course, it is true also if $AB=0$. One can add some more solutions. All $(A,B)$ such that the symmetric matrix $AB+B^TA^T$ is singular or has an odd number of negative eigenvalues. But I doubt there is a nice general characterization. Although I might be wrong.