Rotation matrix in spherical coordinates

I think what you might be looking for is Rodrigues' Rotation Formula. Using spherical coordinates:

Your arbitrary point on the unit sphere is: $$ \mathbf{a} = (\sin\theta\cos\phi, \sin\theta\sin\phi, \cos\theta) $$

Your arbitrary axis is represented by the unit vector: $$ \hat{\mathbf{k}} = (\sin\Theta\cos\Phi, \sin\Theta\sin\Phi, \cos\Theta) $$

Then the result of rotating $\mathbf{a}$ around $\hat{\mathbf{k}}$ by the angle $\beta$, using the right-hand-rule, is given by

$$ \mathbf{b} = \cos\beta\,\mathbf{a} + \sin\beta\,(\hat{\mathbf{k}} \times \mathbf{a}) + (\hat{\mathbf{k}}\cdot\mathbf{a})(1-\cos\beta)\,\hat{\mathbf{k}} $$

Of course, now $\mathbf{b}$'s Cartesian coordinates need to be converted to spherical: $$ \tan\phi' = \frac{b_y}{b_x} \qquad\mbox{and}\qquad \tan\theta' = \frac{\sqrt{b_x^2 + b_y^2}}{b_z} $$ so that $$ \mathbf{b} = (\sin\theta'\cos\phi', \sin\theta'\sin\phi', \cos\theta') $$

The same article on Rodrigues' Formula also discusses a matrix representation of the rotation operation in question.


Rotations in spherical coordinates are affine transformations so there isn't a matrix to represent this on the standard basis $(\theta,\phi)$, you'll need to introduce another coefficient here: $(\theta,\phi,1)$, the rotation matrix in the $\theta$ direction is then, for example, rotating by $\alpha$ is;
$R(\alpha) = \left( \begin{array}{ccc} 1 & 0 & \alpha \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right) $