Proving $(ax+by-1)^2 \ge (x^2+y^2-1)(a^2+b^2-1)$
This is a quadratic inequality in $x,y$, so all you have to do is complete the squares.
Formally, if $\Delta=(ax+by-1)^2 -(x^2+y^2-1)(a^2+b^2-1)$ then you have
$$ (1-a^2)\Delta=(1-a)^2(1+ay-bx)^2+(1-(a^2+b^2))(x-a)^2 $$