Why does dividing both sides of this system of equations to each other yields infinite "incorrect solutions"?

This might be basic, but I'm really bad at basic math. I'm trying to solve the following system of equations: $$\sqrt{x^2+y^2}\cdot \left(x-5\right)=6x+y \tag{1},$$$$\\\sqrt{x^2+y^2}\cdot \left(y-1\right)=6y-x-2 \tag{2}$$ I put them in Wolfram Alpha to test the result, and it yields 3 solutions, which I assume is true. All nice and dandy.

But then I couldn't find out how to move forward. So I decide to divide $(2)$ by $(1)$. which gives: $$\frac{y-1}{x-5}=\frac{6y-x-2}{6x+y} \tag{3}$$

After a few calculations, I figured that this is an equation for a circle. $$y^2+29y=-x^2+9x+10 \tag{4}$$ which implies that there's an infinite number of solutions. Which means that I'm wrong.

So can anybody tell me what I did wrong? And if possible teach me how to solve the system of equations please? Thank you! :D

Dividing (or, in general, "combining") several equations can be used to derive new equations that must necessarily hold, but they are not equivalent to the original system. Consider for example:

$$ \begin{cases} x = 1 \\ y = 1 \end{cases} $$

The above is a well defined system of two equations with two unknowns, and obviously has the unique solution $x=y=1\,$. Dividing the equations, however, gives $\,\dfrac{x}{y}=1 \iff x=y\,$ with infinitely many solutions. But not all of those satisfy the original system, in fact just one does.

Let $\,r=\sqrt{x^2+y^2} \ge0\,$, then the system can be written as:

$$ \begin{cases} \begin{align} r(x-5) &= 6x+y \\ r(y-1) &= 6y-x-2 \end{align} \end{cases} \quad\iff\quad \begin{cases} (r-6)x - y = 5r \\ x + (r-6)y = r - 2 \end{cases} $$

Solving for $\,x,y\,$ gives:

$$ \begin{cases} x = \dfrac{5 r^2 - 29 r - 2}{r^2 - 12 r + 37} \\[10px] y = \dfrac{r^2 - 13 r + 12}{r^2 - 12 r + 37} \end{cases} \tag{*} $$

Substituting the above into $\,r^2=x^2+y^2\,$ then gives, successively:

$$ \begin{alignat}{2} && r^2(r^2 - 12 r + 37)^2 &= (5 r^2 - 29 r - 2)^2 + (r^2 - 13 r + 12)^2 \\ && r^6 - 24 r^5 + 192 r^4 - 572 r^3 + 355 r^2 + 196 r - 148 &= 0 \\ && (r - 1) (r^2 - 12 r + 37) (r^3 - 11 r^2 + 4) &= 0 \end{alignat} $$

The latter factorization gives the root $\,r=1\,$, and the cubic factor gives two more positive real roots. Substituting those back into $\,(*)\,$ gives the solutions in $\,x,y\,$.

When you divide the two equations, you're producing another consistent equation that must be satisfied (except, perhaps, where a denominator of the resulting equation vanishes, where the division operation entails dividing by zero) but not all solutions of the new equation need be a solution of the original equations.

In this case, you've found a new condition, namely, that all solutions must lie on the circle with the given equation, which may help you in finding the solution(s) of the system.

From this comment:

it produces only one equation while the original system had two. One equation generally removes one degree of freedom from the in this case original 2 degrees, so you get a 1-dimensional solution set.

Take an easier example:

$$ \begin{cases} x+y = 1 \\ x-y = 1 \end{cases} $$

The only solution is $(x,y)=(1,0)$. Normally you would get to the solution by adding both sides to get

$$ 2x=2 \tag{*}$$

and solve it as

$$ x=1 $$

But then you get back to (the one of) the original equations, substitute $x$ you already know and solve for $y$. This means when writing (*) you do really mean something like:

$$ \begin{cases} x+y = 1 \\ 2x = 2 \end{cases} $$

or

$$ \begin{cases} 2x = 2 \\ x-y = 1 \end{cases} $$

Note if your original system included a parameter $\alpha$ like this

$$ \begin{cases} x+y = 1 +\alpha \\ x-y = 1 -\alpha \end{cases} \tag{**} $$

then you could get the same (*)! When you solve it and get $x=1$, the form may suggest a single solution. But your domain is still two-dimensional and what you really get is

$$1x+0y=1$$

solved by

$$\{(x,y):x=1, y \in R \}$$

(or it may be $C$ instead of $R$ if you allow complex numbers). This is not a single solution. Choose any single solution from within this set and there is a certain value of $\alpha$ that makes the chosen solution solve (**).

In fact (*) aggregates solutions to all possible versions of (**). By going from (**) to (*) and solving the sole latter, you lose the information what $\alpha$ is.

So where is a similar parameter in your case, since you divide your original equations? Here:

$$ \begin{cases} \sqrt{x^2+y^2}\cdot \left(x-5\right)=\left(6x+y\right)\cdot\beta \\ \sqrt{x^2+y^2}\cdot \left(y-1\right)=\left(6y-x-2\right)\cdot\beta \\ \beta\neq0 \end{cases} \tag{***} $$

It leads to the circle you got but the circle doesn't depend on $\beta$. I haven't investigated if every single point of the circle solves (***) for some value of $\beta$ but it seems to me that every solution of (***) for any possible $\beta$ belongs to the circle. The circle aggregates solutions to different systems of equations.

Taking a single point that belongs to the circle, you don't know if it solves (***) for $\beta=1$ (which is your case) until you confront it with something that specifies $\beta=1$ somehow. It may be any of your two original equations, they implicitly set $\beta=1$.

Others already gave you the answer on how to solve it, this is an additional remark. Observe the following: Suppose we have the folowing equation, it has a solution set associated (shown on the right):

$$x=\pi \tag*{$S_1=\{\pi\}$}$$

Now what happens when we square both sides?

$$x^2=\pi^2 \tag*{$S_2=\{\pi,-\pi\}$}$$

What if we take the first equation and apply the sine function in both sides?

$$\sin(x)=\sin(\pi)=0 \tag*{$S_3=\{\pi n : n\in \Bbb{Z}\}$}$$

So, applying a function to both sides of the equation can mess up the solutions and come up with more or less solutions: There are $2$ solutions for $S_2$, but infinite solutions for $S_3$. Observe that $3\pi$ won't be a solution to the first equation, nor to the second. You should try "keep track" of the solution set as you transform the equations.