Number Theory: Solutions of $ax^2+by^2\equiv1 \pmod p$

Note: A caveat: It is to be noted that the analysis below only works for $p>2$ an odd prime. In any case, the result to be proved does not hold for $p=2$, and we shall only concern ourselves with $p>2$.

We have $(ax)^2 \equiv -aby^2+a \pmod{p}$, so we get $2$ solutions for $x$ when $(\frac{-aby^2+a}{p})=1$, $1$ solution when $(\frac{-aby^2+a}{p})=0$, and $0$ solutions when $(\frac{-aby^2+a}{p})=-1$. Thus the number of solutions is $$\sum_{y=0}^{p-1}{\left(1+\left(\frac{-aby^2+a}{p}\right)\right)}=p+\sum_{y=0}^{p-1}{\left(\frac{-aby^2+a}{p}\right)}=p+\left(\frac{-ab}{p}\right)\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)}$$

It is fairly well known that $\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)}=-1$, so this will give the number of solutions as $p-(\frac{-ab}{p})$ indeed. I shall prove that $\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)}=-1$ below:

Proof of $\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)}=-1$: By Euler's criterion, $$\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)} \equiv \sum_{y=0}^{p-1}{(y^2-b^{-1})^{\frac{p-1}{2}}} \equiv \sum_{y=0}^{p-1}{(y^{p-1}+Q(y))} \pmod{p}$$

where $Q(y)$ is a polynomial in $y$ with degree $\leq p-2$, over $\mathbb{Z}_p$.

It is easy to prove that $\sum_{y=0}^{p-1}{y^n} \equiv \begin{cases} 0 \pmod{p} & n=0 \, \text{or} \, p-1 \nmid n \\ -1 \pmod{p} & n>0, p-1 \mid n \end{cases}$.

Indeed, we have $\sum_{y=0}^{p-1}{y^0}=\sum_{y=0}^{p-1}{1} \equiv 0 \pmod{p}$, and for $n>0$, consider a primitive root $g \pmod{p}$, then $$\sum_{y=0}^{p-1}{y^n}=\sum_{y=1}^{p-1}{y^n}\equiv \sum_{i=0}^{p-2}{(g^i)^n} \equiv \begin{cases} \frac{1-(g^n)^{p-1}}{1-g^n} \equiv 0 \pmod{p}& p-1 \nmid n \\ \sum_{y=1}^{p-1}{1} \equiv -1 \pmod{p} & p-1 \mid n \end{cases}$$

Thus we now have

$$\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)} \equiv \sum_{y=0}^{p-1}{(y^{p-1}+Q(y))} \equiv -1\pmod{p}$$

Note that $$\left|\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)}\right| \leq \sum_{y=0}^{p-1}{\left|\left(\frac{y^2-b^{-1}}{p}\right)\right|} \leq \sum_{y=0}^{p-1}{1}=p$$

Therefore we must have $\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)}=-1 \, \text{or} \, p-1$. If $\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)}=p-1$, then we must have $p-1$ terms equal to $1$ and exactly $1$ term $\left(\frac{c^2-b^{-1}}{p}\right)$ which is $0$. However $\left(\frac{(-c)^2-b^{-1}}{p}\right)=0$ as well, which is only possible if $c \equiv -c \pmod{p}$, i.e. $c=0$. This gives $\left(\frac{-b^{-1}}{p}\right)=0$, a contradiction. Therefore $\sum_{y=0}^{p-1}{\left(\frac{y^2-b^{-1}}{p}\right)}=-1$, as desired.

Note: The above method can be modified to help evaluate $\sum_{y=0}^{p-1}{\left(\frac{f(y)}{p}\right)}$ for any polynomial $f(y)$.

For the first part: If $-ab =c^2 \pmod{p}$ then

$$(ax)^2+aby^2 = a \pmod {p} \Leftrightarrow (ax)^2-(cy)^2 = a \pmod {p} $$

Now, $ax-cy=\alpha \neq 0 \pmod{p}$ and $ax+cy = \alpha^{-1}a$. You get exactly one solution for each $0 \neq \alpha \pmod{p}$.