square root of $1/2 + \sqrt3/2?$

Simpler than undetermined coefficients is the following rule I discovered as a teenager.

Simple Denesting Rule $\rm\ \ \ \color{blue}{subtract\ out}\ \sqrt{norm}\:,\ \ then\ \ \color{brown}{divide\ out}\ \sqrt{trace} $

Recall $\rm\: w = a + b\sqrt{n}\: $ has norm $\rm =\: w\:\cdot\: w' = (a + b\sqrt{n})\ \cdot\: (a - b\sqrt{n})\ =\: a^2 - n\: b^2 $

and, furthermore, $\rm\:w\:$ has trace $\rm\: =\: w+w' = (a + b\sqrt{n}) + (a - b\sqrt{n})\: =\: 2\:a$

Here $\:1+\sqrt{3}/2\:$ has norm $= 1/4.\:$ $\rm\ \: \color{blue}{subtracting\ out}\ \sqrt{norm}\ = 1/2\ $ yields $\ 1/2+\sqrt{3}/2\:$

and this has $\rm\ \sqrt{trace}\: =\: 1,\ \ thus,\ \ \ \color{brown}{dividing\ it\ out}\ $ of this yields the sqrt: $\:1/2+\sqrt{3}/2.$

Below is another example.

Note $\:9-4\sqrt{2}\:$ has norm $= 49.\:$ $\rm\ \: \color{blue}{subtracting\ out}\ \sqrt{norm}\ = 7\ $ yields $\ 2-4\sqrt{2}\:$

and this has $\rm\ \sqrt{trace}\: =\: 2,\ \ so,\ \ \ \color{brown}{dividing\ it\ out}\ $ of this yields the sqrt: $\:1-2\sqrt{2}.$

See here for many more examples, and see this answer for general radical denesting algorithms.

For rationals $x,y,a,b,k$ where $k$ is square-free and $y \ne 0$, $(a + b \sqrt{k})^2 = x + y \sqrt{k}$ if and only if $a^2 + k b^2 = x$ and $2 a b = y$, and thus $b = y/(2a)$ and $a^2 + k y^2/(4 a^2) = x$, i.e. $(2 a^2 - x)^2 = x^2 - k y^2$. Thus $x^2 - k y^2$ must be the square of a rational, and if it is $r^2$ then $(x \pm r)/2$ must be the square of a rational.

In your example, $x=1, y=1/2, k=3$, $x^2 - k y^2 = 1 - 3/4 = 1/4 = (1/2)^2$ and $(1 - 1/2)/2 = (1/2)^2$, so $1 + \sqrt{3}/2$ is a square.

On the other hand, for $x=3, y=1, k=5$, $x^2 - k y^2 = 9 - 5 = 2^2$ but $(3 - 2)/2 = 1/2$ and $(3+2)/2 = 5/2$ are not squares, so $3 + \sqrt{5}$ is not the square of an expression of the form $a + b \sqrt{5}$ with $a$ and $b$ rational (on the other hand, it is the square of $\sqrt{10}/2 + \sqrt{2}/2$).

The method is correct. Unfortunately, the square roots of $s+t\sqrt{k}$, where $s$ $t$ and $k$ are rational, are usually not of the form $x+y\sqrt{k}$ with rational $x$ and $y$. When you use your method, the problem gets revealed when the system of equations analogous to the one you produced has no rational solution.

The system of equations can be turned into a quadratic in $x^2$. If the discriminant is not the square of a rational, there are no rational $x$ and $y$. And even if the discriminant is the square of a rational, giving rational $x^2$, that does not guarantee the rationality of $x$.