Eliminate $\theta$ from $\sin3\theta=a\cos\theta$ and $\cos3\theta=b\sin\theta$
Solution 1:
My first idea after seeing the problem is to say that $$ a^2\cos^2\theta+b^2\sin^2\theta=1 $$ From this we get $$ a^2(1+\cos2\theta)+b^2(1-\cos2\theta)=2 $$ whence $$ \cos2\theta=\frac{2-a^2-b^2}{a^2-b^2} $$ But we also have $$ \frac{\sin^23\theta}{a^2}+\frac{\cos^23\theta}{b^2}=1 $$ whence $$ b^2\sin^23\theta+a^2\cos^23\theta=a^2b^2 $$ and, similarly to the above, $$ \cos6\theta=\frac{2a^2b^2-a^2-b^2}{a^2-b^2} $$ Now use the identity for $\cos3x$ in terms of $\cos x$.
Solution 2:
I used WolframAlpha to get the following relation between $a$, $b$
$$4 - 3 a^2 - 2 a b + a^3 b - 3 b^2 + 2 a^2 b^2 + a b^3=0$$ which is equivalent to $$(a-b)^2 = \frac{ ((a+b)^2 -4)^2}{(a+b)^2 + 4}$$ or $$v^2 = \frac{ (u^2 -1)^2}{u^2 + 1}$$ where $u=\frac{a+b}{2}$ and $v=\frac{a-b}{2}$.
The contour curve is the union of two graphs $$v = \pm \frac{ u^2 -1}{\sqrt{u^2 + 1}}$$
$\bf{Added:}$ With trigonometry, one checks that for $$a = \frac{\sin 3 \theta}{\cos \theta}\\ b= \frac{\cos 3 \theta}{\sin \theta}$$ we have $$\frac{a+b}{2} = \cot 2 \theta \\ \frac{a b -1}{2} = \cos 4 \theta$$
and from here we get a relation between $a$, $b$