Eliminating $\theta$ from the system $x\sin\theta-y\cos\theta=-\sin4\theta$, $x\cos\theta+y\sin\theta=\frac52-\frac32\cos4\theta$

This question, like your previous one, seems almost cruel. There don't seem to be particularly elegant ways to proceed, and the solutions aren't particularly pretty. (Who's inflicting these problems on you?)

In these situations, I often just convert to complex exponentials, via $$\cos u = \frac12\left(e^{iu}+e^{-iu}\right) \qquad \sin u = \frac{1}{2i}\left(e^{iu}-e^{-iu}\right)$$ and let Mathematica do all the symbol-crunching.

Here, defining $p := e^{i\theta}$, the equations become $$\begin{align} p^8 + \phantom{2}p^5 ( x - i y)\phantom{-10p^4\;\,} -\phantom{2} p^3 (x+iy) - 1 &= 0 \\ 3 p^8 + 2 p^5 (x-iy) - 10 p^4 + 2 p^3 (x+iy) + 3 &= 0 \end{align}$$

At this point, "all we need to do" is eliminate $p$. In a polynomial system, this can be accomplished using the method of resultants, conveniently implemented by Mathematica's Resultant function (because no one should be applying this method by hand). The method yields this $\theta$-free relation:

$$\begin{align} 0 &= x^{10}+5 x^8 y^2+10 x^6 x^4 + 10 x^4 y^6+5 x^2 y^8+y^{10} \\ &-705 x^8+12180 x^6 y^2 -24230 x^4 y^4+12180 x^2 y^6-705 y^8 \\ &+122560 x^6-112320 x^4 y^2 -112320 x^2 y^4 +122560 y^6 \\ &+599040 x^4-1361920 x^2 y^2 +599040 y^4 \\ &+327680 x^2+327680 y^2 \\ &-1048576 \end{align}$$

It happens that we can find a lot of $(x^2+y^2)$ groupings, and write, say,

$$\begin{align} 0 &= \left(x^2 + y^2\right)^5 - 705 \left( x^2 + y^2\right)^4+122560\left(x^2+y^2\right)^3+599040\left(x^2+y^2\right)^2+327680\left(x^2+y^2\right)\\ &+5000x^2y^2\left(3\left(x^2 + y^2\right)^2 -96 \left(x^2+y^2\right) -512 -10x^2y^2 \right) \\ & -1048576 \\ \end{align} \tag{$\star$}$$

Which is marginally better, I suppose. There may be additional clever groupings, but it hardly seems worth the trouble to look for them. (Note: I can't even guarantee that I transcribed everything properly.) Even so, I hope this helps. $\square$

Addendum. The appearance of complex conjugates in $(2)$ suggests that there may be some benefit in expressing things in terms of $z := x+iy$ and $\overline{z}:=x-iy$. It even helps to work towards this from the outset. Writing $(1)$ and $(2)$ for the original equations, we can obtain these conjugate equations: $$\begin{align} (1) + i(2): &\quad \phantom{5}p^8 - 10 p^4\phantom{z} + \phantom{1}4 p^3 z + 5 = 0 \\ (1) - i(2): &\quad 5 p^8 + \phantom{1}4 p^5 \overline{z} - 10 p^4\phantom{z} + 1 = 0 \\ \end{align}$$ The equations are a little nicer-looking, but they don't really make the task of eliminating $p$ any easier. A computer algebra system doesn't care anyway. Be that as it may, applying the method of resultants yields $$\begin{align} 0 &= 16777216 - 5242880 z \overline{z} - 4464640 z^2 \overline{z}^2 - 1000960 z^3 \overline{z}^3 + 30 z^4 \overline{z}^4 - 16 z^5 \overline{z}^5 \\ &+2500\left(z^4+\overline{z}^4\right)(-1024-192z\overline{z}+z^2\overline{z}^2) \\ &+ 3125 \left(z^8 + \overline{z}^8\right) \end{align}$$ The powers of $z\overline{z}$ are nice to see, because they give us $x^2+y^2$. Also, we can manipulate the expression to include powers of $z^2+\overline{z}^2=2(x^2-y^2)$; doing so is surprisingly effective, as the left-over $z\overline{z}$ terms factor: $$16 (z \overline{z}-16)^5 -3125 (z^2 + \overline{z}^2)^4 + 10000\left(z^2 + \overline{z}^2\right)^2 \left(256 + 48 z\overline{z} + z^2 \overline{z}^2\right)= 0$$ Abbreviating with $u:= z\overline{z}=x^2+y^2$ and $v:=\frac12(z^2+\overline{z}^2)=x^2-y^2$, this is

$$(u-16)^5 + 2500 v^2\left(u^2+48u+256\right) - 3125 v^4 = 0 \tag{$\star\star$}$$

This might be about as good as things get.

My believe that one of the coefficients of $x,y$ in Eliminating $\theta$ from trigonometric system will be negative has been corroborated by the current question.

Solving for $x,y$

$$\dfrac x{-\cos t(3\cos4t-5)-2\sin t\sin4t}=\dfrac y{2\cos t\sin4t-\sin t(3\cos4t-5)}=\dfrac12$$

$$4x=10\cos t-5\cos3t-\cos5t\iff x=5c-4c^5$$

$$4y=-\sin5t+5\sin3t+10\sin t\iff y=5s-4s^5$$ where $c=\cos t,s=\sin t$ using this and this

$$x^2+y^2=25-40(1-2c^2s^2)+16(1-3c^2s^2)$$

$$=1+32c^2s^2=1+8\sin^22t=1+4(1-\cos4t)=5-4\cos4t$$

$$\cos4t=\dfrac{5-x^2-y^2}4\ \ \ \ (1)$$

Again squaring & adding $$4(x^2+y^2)=4\sin^24t+(5-3\cos4t)^2=25+4(1-\cos^24t)+9\cos^24t-30\cos4t\ \ \ \ (2)$$

Replace the value of $\cos4t$ in $(2)$ from $(1)$