Suppose R is a commutative ring, M is the maximum ideal of R, and R/M is not a field. Prove: (R/M)² = 0.

It is a well known theorem that if R is a commutative ring with identity not equal to zero then R/M is a field. So R in this question doesn't have a non-zero identity. For example, if R = 2Z, which is a ring with no identity, and whose maximal ideal is 4Z, then 2Z/4Z = {4Z, 2 + 4Z}. I'm not sure what exactly does the notation (R/M)² mean or how it is defined. If (R/M)² means the set {ab∈R/M | a,b∈R/M}, then (2Z/4Z)² = {4Z} (= 0?) Can anyone help to prove?

The question can be rephrased as:

Suppose $R$ is a nonzero rng with no nontrivial ideals, and $R$ has no multiplicative identity. Then $R^2=\{0\}$.

The $R^2$ should be interpreted as the ideal product, that is, the set $\{\sum r_is_i\mid r_i, s_i\in R\}$ with finite sums only, of course.

Since this is always an ideal, we know either $R^2=\{0\}$ or $R^2=R$. If $R^2=R$, we can show $R$ has an identity, and then $R$ must be a field.

Suppose $R^2=R$. Then there must exist an $x$ such that $xR\neq \{0\}$, and in fact, $xR=R$ again since there are no nontrivial ideals. In particular $xr=x$ for some $r\in R$.

Here's an important thing to notice: the annihilator $ann(z):=\{r\in R\mid zr=0\}$ is also an ideal of $R$. It can again be only one of two things, $R$ or $\{0\}$.

At this point, we have above that $x\neq 0$, and $xr\neq 0$, so $ann(x)=\{0\}$. Also, $ann(r)=\{0\}$ since $xr\neq 0$.

Because $xr^2-xr=xr-x=0$, we can see $x(r^2-r)=0$. But as $ann(x)=\{0\}$, it must be that $r^2=r$.

Then for any other $y\in R$, $yr-y\neq 0$ would imply also that $yrr-yr\neq 0$, but of course that isn't the case because $y(r^2-r)=y0=0$. This means then that $yr-y=0$ for all $y$, and that just says that $r$ is the identity of $R$.

So going back to your original statement: if $R$ is a nonzero ring with no nontrivial ideals, and $R$ is not a field, then it must be that $R^2=\{0\}$.