$AB$ is not invertible

Since $rank(AB)\le rank(A)\le \min\{m,n\}=n<m$ and $AB$ is an $m\times m$ matrix,

$AB$ is not invertible and therefore $\det(AB)=0$.

Since $m>n$, we must have that $\ker B$ is non-trivial (look at row canonical form, for example). Hence $Bv=0$ for some $v \neq 0$, and so $ABv=0$. It follows that $AB$ is singular and that $\det (AB) = 0$.

If $$ A=(a_{ij}) \in \mathbb{R}^{m\times n}, \ B=(b_{ij})\in \mathbb{R}^{n\times m}, $$ with $m>n$, then $$ C:=AB=(c_{ij}) \in \mathbb{R}^{m\times m}, $$ with $$ c_{ij}=\sum_{k=1}^na_{ik}b_{kj}. $$ For $j=1,\ldots,m$ let $$ \mathbf{c}_j=Ce_j \in \mathbb{R}^m, $$ where $(e_1,\ldots, e_m)$ stands for the canonical basis of $\mathbb{R}^m$. Then we have $$ \mathbf{c}_j=\left[ \begin{array}{c} \sum_{k=1}^na_{1k}b_{kj}\\ \vdots\\ \sum_{k=1}^na_{mk}b_{kj} \end{array} \right]=\sum_{k=1}^nb_{kj} \left[ \begin{array}{c} a_{1k}\\ \vdots\\ a_{mk} \end{array} \right]= \sum_{k=1}^nb_{kj}\mathbf{a}_k, $$ with $$ \mathbf{a}_j=\left[ \begin{array}{c} a_{1j}\\ \vdots\\ a_{mj} \end{array} \right] \in \mathbb{R}^n. $$ We now have \begin{eqnarray} \det(C)&=&\det(\mathbf{c}_1,\ldots,\mathbf{c}_m)=\sum_{k_1=1}^n\sum_{k_2=1}^n\ldots\sum_{k_m=1}^nb_{k_1,1}b_{k_2,2}\ldots b_{k_m,m}\det(\mathbf{a}_{k_1},\ldots,\mathbf{a}_{k_m}) \end{eqnarray} Since there are exactly $n$ vectors $\mathbf{a}_j$, and the determinant has $m>n$ entries we have $$ \det(\mathbf{a}_{k_1},\ldots,\mathbf{a}_{k_m})=0 \quad \forall k_1,\ldots,k_m \in \{1,\ldots,n\} $$ because at least two entries must be equal. Hence $$ \det(C)=0. $$