Show that an orthogonal group is a $ \frac{n(n−1)}2 $-dim. $ C^{\infty} $-Manifold and find its tangent space

They've already given you the smooth map $\phi(X) = X^{t}X - I_n,$ and $O(n) = \phi^{-1} (0),$ so it is automatically a manifold (if the rank is constant). To see the dimension, you need to calculate the nullity of $\phi^{\prime}.$ See below for a full solution.

The point here is that $\phi$ takes $n\times n$ matrices to symmetric matrices since $(X^{t}X- I)^t = X^{t}(X^{t})^{t} - I = X^t X - I.$ Take the directional derivative $\phi_v (X) = \lim_{h\to 0} \frac{(X+hv)^t(X+hv) - I - X^t X + I}{h} = X^t v + v^t X = (X^t v) + (X^t v)^t.$ That is, $\phi^{\prime}(X)$ takes $v$ to $(X^t v)+(X^t v)^t.$ Then we note that this is symmetric, and $\phi^{\prime}$ is onto the space of symmetric matrices since $X$ is invertible. The space of symmetric matrices has dimension $n(n+1)/2$ (you can do this yourself), whence the dimension of $O(n)$ is $n^2 - n(n+1)/2 = n(n-1)/2.$

To calculate the tangent space at $I$, you just want to look at $\phi^{\prime} (I)$ and calculate its null space.