Solution 1:

Noncommutative algebra is filled with examples of this.

For example, take $A=\begin{bmatrix}0&1\\0&0\end{bmatrix}$ and $B=\begin{bmatrix}0&0\\0&1\end{bmatrix}$.

You have $AB\neq 0$ but $BA=0$.

Rings in which $ab=0$ implies $ba=0$ are called reversible rings. That is a particularly strong condition, and is pretty interesting to study. I highly recommend Greg Marks' paper: Reversible and symmetric rings (2002), and P.M. Cohn's paper Reversible rings (1999).

This would imply that $AB=0$ does not necessarily imply $[A,B]=0$.

Solution 2:

What about $$B = \begin{pmatrix}0 & 1 \\ 0 & 0 \end{pmatrix} $$ and $$A = \begin{pmatrix}1 & 0 \\ 0 & 0 \end{pmatrix}$$

Solution 3:

Sure - here's an example, with matrices taking entries in the field $\mathbb{Z}_{2}$:

$$ \begin{bmatrix} 1 & 1\\ 0 & 0\\ \end{bmatrix} \begin{bmatrix} 1 & 0\\ 1 & 0 \end{bmatrix} = \begin{bmatrix} 0 & 0\\ 0 & 0 \end{bmatrix}, $$

but

$$ \begin{bmatrix} 1 & 0\\ 1 & 0\\ \end{bmatrix} \begin{bmatrix} 1 & 1\\ 0 & 0 \end{bmatrix} = \begin{bmatrix} 1 & 1\\ 1 & 1 \end{bmatrix}. $$

Solution 4:

As the other answer show: there are uncountably many pairs of matrices $(A,B)$ such that $AB = 0$ while $BA \neq 0$. If you think of square matrices as linear transformations then it is obvious why this should be so: in $AB$, we can think of the product as recording the image of each of the columns of $B$ under the linear transformation $A$, likewise with $BA$.

A nice question to ask is the following: Given a fixed matrix $A \in \text{Mat}_n\mathbb{R},$ what is the following:

$$\widetilde{A} := \{ X \in \text{Mat}_n\mathbb{R} : AX = 0 \ \wedge \ XA \neq 0\} \, ?$$

Clearly, if $X \in \widetilde{A}$ then $\lambda X \in \widetilde{A}$ for all $\lambda \neq 0.$ Interestingly, this space is not a vector space because the zero matrix $0 \notin \widetilde{A}$ and $X,Y \in \widetilde{A}$ does not imply that $X+Y \in \widetilde{A}.$