Ellipse and hyperbola have the same foci

The $125$ and the $100$ in your hyperbola equation should have been swapped: $$\frac{x^2}{125}-\frac{y^2}{100}=1$$

This gives $P$ as $\left( \dfrac{25\sqrt5}3,\dfrac{40}{3} \right)$ and $PA \times PB \,$ yields the given answer of $500$.

Here's a geometric solution.

For any point $P$ on the ellipse, the sum of distances $PA+PB$ to the foci $A$ and $B$ is constant and equals the length of the major axis. So $PA+PB=50$.

Also, for an ellipse the distance $f$ from the center of the ellipse to one of the foci satisfies the equation $f^2+b^2=a^2$, where $2a$ and $2b$ are the lengths of the major and minor axes respectively. So $f^2=25^2-20^2=225$ and therefore $f=15$.

For a hyperbola the distance $f_H$ from the center to one of the foci satisfies the equation $f_H^2=a^2+b^2$, where now $2a$ and $2b$ stand for the lengths of the transverse and conjugate axes respectively. Since the ellipse and hyperbola share the same foci, we know $f_H=f=15$. But we are given that $2b=20$, so $a^2=15^2-10^2=125$ and therefore $a=5\sqrt 5$.

Finally, for a hyperbola the difference of distances $PA-PB$ is constant and equals the length of the transverse axis, i.e., $PA-PB=10\sqrt 5$.

We now have two equations in two unknowns: $PA+PB=50$, and $PA-PB=10\sqrt 5$. Solve these and obtain $PA\cdot PB=500$.

Using elliptic coordinates:

\begin{align*} z &= c\cosh (\alpha+\beta i) \\ (x,y) &= (c\cosh \alpha \cos \beta, c\sinh \alpha \sin \beta) \end{align*}

Ellipse: $$\frac{x^2}{c\cosh^2 \alpha}+\frac{y^2}{c\sinh^2 \alpha}=1$$

In this case, $$\ \begin{array}{rcccl} \text{major axis} &=& 2c\cosh \alpha &=& 50 \\ \text{minor axis} &=& 2c\sinh \alpha &=& 40 \end{array}$$

Hyperbola:

$$\frac{x^2}{c\cos^2 \beta}-\frac{y^2}{c\sin^2 \beta}=1$$

In this case, $$\ \begin{array}{rcccl} \text{transverse axis} &=& 2c\cos \beta &=& 2\sqrt{15^2-10^2} \\ \color{red}{\text{conjugate axis}} &=& 2c\sin \beta &=& 20 \end{array}$$

Now, \begin{align*} |z+c|+|z-c| &= 2c\cosh \alpha \\ |z+c|-|z-c| &= \pm 2c\cos \beta \\ 4|z+c||z-c| &= (|z+c|+|z-c|)^2-(|z+c|-|z-c|)^2 \\ &= 4c^2(\cosh^2 \alpha-\cos^2 \beta) \\ |z+c||z-c| &= c^2(\cosh^2 \alpha-\cos^2 \beta) \\ &= c^2(\sinh^2 \alpha+\sin^2 \beta) \\ &= \left( \frac{40}{2} \right)^2+ \left( \frac{20}{2} \right)^2 \\ &= 500 \end{align*}