Showing there exists a unique $\theta$ for 2D rotation matrix
Solution 1:
I'll pick things up from the partial progress in the comments. As you have established, if $A = \pmatrix{a&b\\c&d}$, then $A^TA = I$ implies that $$ a^2 + c^2 = b^2 + d^2 = 1, \quad ab + cd = 0. $$ On the other hand, $AA^T = 0$ implies that $a^2 + b^2 = c^2 + d^2 = 1$. With that, we have $$ a^2 + c^2 = c^2 + d^2 \implies a^2 = d^2 \implies d = a \quad \text{or } \quad d = -a. $$ Similarly, $a^2 + b^2 = a^2 + c^2$ implies that $b^2 = c^2$, so that either $c = b$ or $c = -b$. Now, from the determinant of $A$, we have $$ ad - bc = 1 \implies a(\pm a) - b(\pm b) = 1 = a^2 + b^2. $$ Argue that this equality can only hold if $d = a$ and $c = -b$. Thus, $A$ is a matrix of the form $$ A = \pmatrix{a & b\\ -b & a}, $$ where $a^2 + b^2 = 1$. From there, you should be able to argue that such a $\theta$ exists and is unique.
Solution 2:
The question says, or rather, should say:
Suppose we are given a matrix $A$ with the properties: $A^T A = I$ and $\det A = 1$.
Show that there exists a unique $\theta\in [0,2\pi)$ such that $A$ is of the form $$ A = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta &\cos\theta\end{bmatrix}. $$
$$$$
Let $$A = \begin{bmatrix} a & b \\ c &d\end{bmatrix}.$$
$A^T A = I\ \implies$
$a^2 + c^2 = b^2 + d^2 = 1\quad (1),\ $ and
$ab+cd = 0\quad (2).$
$\det A = 1\implies$
$ad-bc = 1\quad (3).$
If $\ b=0,\ (1)\implies d=\pm 1 \overset{(2)}{\implies} c=0\overset{(3)}{\implies} a=\pm 1.$
If $\ b\neq 0,\ $ then $\ (2)\implies a= \frac{-cd}{b} \overset{(1)}{\implies} \frac{c^2d^2}{b^2} + c^2 = 1 \implies c^2 \left( \frac{d^2+b^2}{b^2}\right) = 1 \implies c^2 = b^2$
$\implies c=\pm b \implies a = \mp d \overset{(3)}{\implies} c=-b\ $ and $\ a=d.$
In either case,
$$A = \begin{bmatrix} a & b \\ -b &a\end{bmatrix},$$
where $\ a^2+b^2=1\ $ is the only restriction on $\ a\ $ and $\ b.$
They want us to write this as
$$ A = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta &\cos\theta\end{bmatrix} $$
and show that for each pair $\ (a,b)\ $ with $\ a^2+b^2=1,\ $ this can only be done with one value of $\ \theta\in [0,2\pi).$
To this end:
If $\ -b\ (=\sin\theta) >0,\ $ then $\ 0<\theta<\pi,\ $ and since $\ \cos\ $ is injective on this interval, there is only one $\ \theta\ $ such that $\ \cos\theta = a\ $ also.
Else if $\ b\ (=\sin\theta) = 0\ $ then $\ \theta\ $ can only equal $\ 0\ $ or $\ \pi.\ a=\cos\theta\ $ has a different value in both cases.
Else $\ -b\ (=\sin\theta) <0,\ $ in which case $\ \pi<\theta<2\pi,\ $ and since $\ \cos\ $ is injective on this interval, there is only one $\ \theta\ $ such that $\ \cos\theta = a\ $ also.