Given that $a\cos\theta+b\sin\theta+A\cos 2\theta+B\sin 2\theta \leq 1$ for all $\theta$, prove that $a^2+b^2\leq 2$ and $A^2+B^2 \leq 1$

We have $$0 \leq F(\theta) + F(\theta + \pi) = 2 - 2(A \cos 2\theta + B \sin 2\theta).$$

Let $u$ be the vector $(A,B)$. Pick $\theta$ so that $v = (\cos 2\theta, \sin 2\theta)$ is a unit vector in the same direction as $u$. Then

$$\sqrt{A^2 + B^2} = |u| = u \cdot v = A \cos 2\theta + B \sin 2\theta \leq 1.$$

For the second part, write $$0 \leq F(\theta) + F(\theta + \pi/2) = 2 - (b + a)\cos \theta - (b- a)\sin \theta.$$

Now pick $\theta$ so that $v = (\cos \theta,\sin \theta)$ is a unit vector in the same direction as $u = (b + a,b - a)$. Then $$\sqrt{2}\sqrt{a^2 + b^2} = \sqrt{(b + a)^2 + (b - a)^2} = |u| = u \cdot v \leq 2.$$

Here is another (?) approach: define $c=a-bi$, $d=A-Bi$. Then we are given that $$ \operatorname{Re}(cz+dz^2)\le 1,\quad\forall z\in\Bbb C\colon |z|=1.\tag{1} $$ Replacing $z$ with $-z$ and with $iz$ in (1) we get two more inequalities \begin{eqnarray} z\mapsto -z:\quad\operatorname{Re}(-cz+dz^2)\le 1,\quad\forall z\in\Bbb C\colon |z|=1,\tag{2}\\ z\mapsto iz:\quad\ \operatorname{Re}(icz-dz^2)\le 1,\quad\forall z\in\Bbb C\colon |z|=1.\tag{3} \end{eqnarray} Adding (1) and (2) gives $$ \operatorname{Re}(dz^2)\le 1,\quad\forall z\in\Bbb C\colon |z|=1\quad\Leftrightarrow\quad |d|\le 1. $$ Similarly adding (1) and (3) gives $$ \operatorname{Re}((1+i)cz)\le 2,\quad\forall z\in\Bbb C\colon |z|=1\quad\Leftrightarrow\quad |(1+i)c|\le 2. $$