How to simplify a square root
Solution 1:
If you're faced with a question that says "Prove that $\sqrt{27-10\sqrt{2}}$ $=5 - \sqrt{2}$", then it's just a matter of squaring $5 - \sqrt{2}$ and seeing that you get $27-10\sqrt{2}$. But suppose the question your faced with is to find a square root of $27-10\sqrt{2}$ of the form $a+b\sqrt{2}$, where $a$ and $b$ are rational. Then you have $$ 27-10\sqrt{2}=\left(a+b\sqrt{2}\right)^2 = a^2 + 2ab\sqrt{2} + 2b^2 $$ so \begin{align} 27 & = a^2+2b^2 \\[8pt] -10 & = 2ab \end{align} From the second equation we get $a=-5/b$, then the first equation becomes $$ 27 = \frac{25}{b^2} + 2b^2 $$ or $$ 2(b^2)^2 -27b^2 + 25 = 0. $$ A solution is $b^2=1$, and you can go on from there to find $b$ and then $a$.
(And remember that the number will have two square roots.)
Later note: In order for all this to work, we have to rely on the fact that $\sqrt{2}$ is irrational. That enables us to conclude that the rational parts are equal and the irrational parts are equal, so we have two equations.
Solution 2:
No lucky guesses are needed, there is a simple denesting algorithm for $\rm\:\sqrt{a+b\sqrt{n}}$
Simple Denesting Rule $\rm\ \ \ \color{#0A0}{subtract\ out}\ \sqrt{norm}\:,\ \ then\ \ \color{brown}{divide\ out}\ \sqrt{trace} $
$\begin{array}{lll}\rm Recall\ \ \ w = a + b\sqrt{n}\ \ \ has\!\!\! &{\bf norm} &\!\!\!\rm=\: w\:\cdot\: w' = &\!\!\!\!\rm(a + b\sqrt{n})\ \cdot\: (a - b\sqrt{n})\ =\: a^2 - n\: b^2 \\ \\ \rm and,\ furthermore,\ \ w\ \ has \!\!\!&{\bf trace} &\!\!\!\!\rm =\: w+w' = &\!\!\!\!\rm (a + b\sqrt{n}) + (a - b\sqrt{n})\: =\: 2\:a\end{array}$
Here $\:27-10\sqrt{2}\:$ has norm $= 23^2.\,$ $\rm\, \color{#0A0}{Subtracting\ out}\ \sqrt{norm}\ = -23\ $ yields $\ 50-10\sqrt{2}\:$
and this has $\rm\ \sqrt{trace}\: =\: 10,\ \ hence\ \ \ \color{brown}{dividing\ it\ out}\ $ of this yields $\rm\ 5 - \sqrt{2} =\:$ sought sqrt.
Remark $\ $ The sign of the norm sqrt was chosen to make the trace sqrt rational. The same answer would arise using the opposite sign, but with slightly more work (rationalizing a denominator). $\ $ For many further examples see other posts on radical denesting.