Calculate angle between two vectors, given their rotation w.r.t. a third vector.
Solution 1:
Assuming that $a, b, c$ have norm 1, we have \begin{align} a' &= 2(a\cdot c)a - c\\ b' &= 2(b\cdot c)b - c\\ \end{align} Indeed, this implies $a'\cdot a = a\cdot c$ and $a'\cdot c = 2(a\cdot c)^2-1$ which is the cosine of the double angle. Alternatively, it is obvious geometrically that $\frac{a'+c}{2}$ is the orthogonal projection of $c$ on $a$, that is to say $(a\cdot c) a$.
Hence \begin{equation} a'\cdot b' = 4 (a\cdot c)(b\cdot c)(a\cdot b) - 2(a\cdot c)^2 - 2(b\cdot c)^2 +1\end{equation}