$AB=Y = y + y'(x) (X-x)$ $PG= Y = y - \frac{1}{y'(x)} (X-x)$ Find the curves such that $PM=MG$.
Solution 1:
Here is an approach that follows the line of reasoning you demonstrate in your post.
The equation of line normal to your unknown curve $C$ as the point $P=(a,f(a))$ is $$y=-\frac{1}{f'(a)}(x-a)+f(a)$$ By finding the $x$ and $y$ intercepts of this line we determine the points $M$ and $G$ to be $$M=\Big(0,f(a)+\frac{a}{f'(a)}\Big)$$ $$G=\Big(f'(a)f(a)+a,0\Big)$$ Then $$(PM)^2=(GM)^2 \iff f'(a)=-{2a \over f(a)}$$ In other words, your unknown function $f$ must satisfy the differential equation $y'=-\frac{2x}{y}$ providing you with the same solution curves as @JeanMarie.
Solution 2:
It is difficult for me to find another error than the one I have already found.
In fact, there is a simpler approach.
Let me say first what is the solution, the proof being given hereafter.
The solution is any ellipse (at least its arc in the first quadrant) with implicit equation :
$$x^2+\frac{y^2}{2}=k \tag{1}$$
or, using a parametric representation:
$$\begin{cases}x&=&\cos(t)\\y&=&\sqrt{2} \sin(t)\end{cases}$$
Here is a Desmos representation where $k=1$ that one can animate by playing on the cursor for $s$.
Now, let us give the proof.
Let $P'$ be the projection of $P$ onto the $x$ axis. The length of line segment $GP'$ is classically called the "subnormal" of the curve ; it is given by a classical formula $|yy'|$. Taking into account the fact that $(x,y)$ is in the first quadrant $x>0, \ y>0$ and that the slope is negative, we have: $|yy'|=-yy'.$
The important remark now is that we need only consider abscissas. Indeed, condition $PM=MG$ gives by projection $P′O=OG$ ($O$ being the origin) itself equivalent to $$GP'=2OP' \ \ \ \iff \ \ \ -yy'=2x,$$
a differential equation whose integration gives (1), $k$ being an integration constant.
Remark: the solution curve, defined at first for $x>0,y>0$ is in fact valid for $|x|>0,|y|>0$.
Historical corner: issues on subnormals and subtangents were rather common in the 17th and 18th centuries. They were sometimes used as challenges between mathematicians just before and still after the beginning of Calculus.