Deriving form of general rotation matrix along an axis using calculus

I think the direct approach using standard vector algebra is much better in this case than calculus. You can derive the rotation matrix very easily as follows:

1. Let the axis of rotation be the unit vector $a$, and let the vector you want to rotate be $v$.

2. Decompose $v$ into $v_1$ along the axis and $v_2$ perpendicular to the axis, then

$v_1 = (v \cdot a) a = a a^T v $ and $v_2 = v - v_1 = (I - a a^T) v $

3. Construct the vector that has the same magnitude as $v_2$ but orthogonal to both the axis and $v_2$. This vector is $v_3 = a \times v_2 = a \times (I - a a^T) v = a \times v$

4. Now the rotated vector is a combinaton of $v_1, v_2$ and $v_3$ as follows:

$v' = R v = v_1 + \cos \theta v_2 + \sin \theta v_3 $

Substitute what $v_1, v_2, v_3$ are and you're almost done. Hence,

$v' = R v = a a^T v + \cos \theta (I - a a ^T ) v + \sin \theta (a \times v) \hspace{20pt} (1) $

This last equation is known as the Rodrigues' rotation formula for a vector $v$ about an axis $a$. To extract the rotation matrix we need to express $a \times v$ in matrix form. Writing the components of $a \times v$ in terms of the components of $a$ and $v$, one deduces that

$ a \times v = S_a v \hspace{20pt} (2) $

where

$S_a = \begin{bmatrix} 0 && - a_z && a_y \\ a_z && 0 && - a_x \\ -a_y && a_x && 0 \end{bmatrix} $

Using this last equation $(2)$ into $(1)$ and factoring $v$ out, we get the rotation matrix as

$ R = a a ^T + ( I - a a ^T ) \cos \theta + S_a \sin \theta \hspace{20pt}(4) $

That's all to it.

Now, let's check if the matrix derived by the OP matches what I got here.

The final matrix obtained by the OP is

$ M_\phi = I + ( (1 - \cos \phi) I - B \sin \phi ) B^2 $

Now, $B^2 v = n \times (n \times v) = n (n^T v) - v (n^T n) = n n^T v - v = (n n^T - I ) v$

Replacing $B^2 $ with $(n n^T - I)$ in the above formula gives

$ M_\phi = I + ( (1 - \cos \phi) I - B \sin \phi ) (n n^T - I ) $

This expands into

$ M_\phi = I + (1 - \cos \phi) (n n^T - I) + B \sin \phi $

(Because $B n = 0 $ )

Simplifing further,

$ M_\phi = I + n n^T - I + \cos \phi (I - n n^T ) + B \sin \phi $

And this equal to

$ M_\phi = n n^T + \cos \phi (I - n n^T ) + B \sin \phi \hspace{20pt}(*)$

Comparing this with the result I derived earlier ( equation $(4)$ ) , we find that $(*)$ is identical to $(4)$ once we make the substitutions $n = a$ , $B = S_a$ and $\phi = \theta$.