If $ab \equiv r \pmod{p}$, and $x^2 \equiv a \pmod{p}$ then $y^2 \equiv b \pmod{p}$ for which condition of $r$?
The following three facts are relevant.
If $a$ and $b$ are squares then $r$ is a square.
If $a$ and $b$ are not squares then $r$ is a square.
If one of $a,b$ is a square and the other is not then $r$ is not a square.
It's simple to prove the first one: Let $a = x^2$, $b = y^2$ so that $r = ab = (xy)^2$.
As for the next two I will give a "hi-tech" proof (See Martins comment for a direct proof).
Modular arithmetic $\mathbb{Z}/p\mathbb{Z}$ is a ring, but the group of units $(\mathbb{Z}/p\mathbb{Z})^\times$ is a group: It's the multiplicative part of the ring. The only non-unit is $0$ so as a set it's $\{1,2,\cdots,p-1\}$.
This group is cyclic (a very fundamental and strong theorem, see here for some discussion on it) and we can use this to prove the theorems: Let $g$ generate the group, so that every element is of the form $g^k$ for some $0 < k < p-1$,
- First note that $a$ is a square iff $a = g^{2k}$. (Easy to prove).
Let $a = g^u$, $b = g^v$ so $r = ab = g^{u+v}$. All three facts are an easy deduction from this. You can prove them now but I would like to define some good notation.
We can introduce the Legendre symbol now: Define a map $\left(\tfrac{-}{p}\right)$ that takes $g$ to $-1$ and extend it to the whole group: It's a group homomorphism from the units group $\mathbb{Z}/p\mathbb{Z}$ into $\{1,-1\}$. It's $1$ exactly when the number is a square and $0$ otherwise.
So now we have:
If $a$ and $b$ are squares then $r$ is a square because $\left(\tfrac{r}{p}\right) = \left(\tfrac{ab}{p}\right) = \left(\tfrac{a}{p}\right)\left(\tfrac{b}{p}\right) = 1 \cdot 1 = 1$.
If $a$ and $b$ are not squares then $r$ is a square because $\left(\tfrac{r}{p}\right) = \left(\tfrac{ab}{p}\right) = \left(\tfrac{a}{p}\right)\left(\tfrac{b}{p}\right) = -1 \cdot -1 = 1$.
If one of $a,b$ is a square and the other is not then $r$ is not a square because $\left(\tfrac{r}{p}\right) = \left(\tfrac{ab}{p}\right) = \left(\tfrac{a}{p}\right)\left(\tfrac{b}{p}\right) = 1 \cdot -1 = -1$.
What you are observing is simply the addition tables for $\rm\, \mathbb Z/2\, \cong\, G/G^2,\,\ G = \mathbb (Z/p)^{*}.\:$ The squares form a subgroup of index $2$, so they multiply the same as addition mod $2\, $ (parity addition). With notation: $ $ Square $\rm = S \to 0\in \mathbb Z/2,\ $ Nonsquare $\rm = N\to 1\in \mathbb Z/2,\: $ we have
$$\begin{align}\rm S \:\!*\:\! S\: &=\rm\:\! \:S\,\iff 0 + 0\ \equiv\ 0\\ \rm S * N\ &=\rm\ N\iff 0 + 1\ \equiv\ 1\\ \rm N * N\ &=\rm\ S\, \iff 1 + 1\ \equiv\ 0 \end{align}\qquad\qquad$$
You can find further examples in my post a couple days ago, which describes various ways of characterizing subgroups, including the complementary view of a subgroup. When you are first learning elementary number theory and group theory, in order to better comprehend the abstract algebraic structures, it is essential to explore many concrete instances of these abstractions. As you can see from the many examples in my linked post, such subgroup laws (and their complements) are already well-known subconsciously in the familiar number systems. The algebraic abstractions serve to "reify" this intuitive knowledge in a more formal structured manner. This is essential in order to cope with such structure when it arises in much more complex scenarios.
We are working in the multiplicative group $G=(\mathbb{Z}/p\mathbb{Z})^*$. This group is cyclic with order $p-1$, which is even for $p \neq 2$. Then $G^2$, consisting of the squares of elements in $G$, is an index 2 subgroup. The set of nonsquares is just the coset of $G^2$ in $G$.
We can immediately conclude:
- The product of two squares is a square.
- The product of a square and a nonsquare is a nonsquare.
- The product of two nonsquares is a square.