How to prove that a nilpotent matrix is not invertible?

Solution 1:

Usually, nilpotent means that $B^m=0$ for some $m>1 $, not necessarily $2$.

A direct way to see that $B $ is singular is $$ 0=\det (B^m)=(\det (B))^m, $$ so $\det (B)=0$.

Another way, without using determinants: if $B $ were invertible, then $$B=(B^{-1})^{m-1}\,B^m=0, $$ a contradiction.

Solution 2:

Suppose $B^{-1}$ exists and $B^k=0$. Then $$ I = B^{-1} B=(B^{-1})^2 B^2=\ldots=(B^{-1})^k B^k = (B^{-1})^k 0 = 0 $$ which is a contradiction.

Solution 3:

Hint: If $AB = I$, then we must also have $A^kB^k = I$ for any $k$.

Or, if you prefer:

Hint: Note that $\det(B^k) = \det(B)^k$.

I don't think your argument is correct. For example, $$ B = \pmatrix{ 0&1\\0&0 } $$ is nilpotent, but doesn't have "two or more rows that are the same".