How does the lie algebra capture compactness of the lie group?

This is a soft question.

Let $V,\rho$ be a representation of the lie algebra $\mathfrak{so}_3(\mathbb{R})$. Then if I understand everything right, $V$ is necessarily completely reducible, because the representation $$\rho:\mathfrak{so}_3\rightarrow \mathfrak{gl}(V)$$ is the derivative of a representation $$P: G\rightarrow GL(V)$$ where $G$ is either $SO_3$ or its universal cover $SU_2$. Because the latter is compact, there is a $G$-invariant inner product on $V$, obtained by beginning with an arbitrary inner product and averaging it over $G$ with respect to a Haar measure. Then the orthogonal complement of any $P$-subrepresentation of $V$ will also be a $P$-subrepresentation; thus $V$ is completely reducible as a representation of the lie group $G$. But a $P$-subrepresentation is also a $\rho$-subrepresentation and vice versa, so $V$ is also completely reducible as a representation of the lie algebra $\mathfrak{so}_3$.

Here's what's bugging me. In my mind, the fact that any representation of $SO_3$ is completely reducible is a consequence of the topological fact that its universal cover is compact, as in the above paragraph. But this same fact is reflected in the representations of $\mathfrak{so}_3$, which does not have any topological differences from any other three-dimensional lie algebra. So the topological information about $SO_3$ must be being captured in purely algebraic information about $\mathfrak{so}_3$.

What is that information? And how is it related to compactness?

Apologies for the softness of this question. I am unsatisfied but I am not sure in advance what kind of answer will satisfy me. Any thoughts you have for me are appreciated. Thanks so much.

Unipotent groups are never compact, and clearly the Lie algebra does not detect compactness for tori, so let's restrict to semisimple $G$. By Sylvester's law of inertia, the Killing form on the Lie algebra $\mathfrak{g}$ of $G$ can be diagonalized with diagonal entries equal to $1$, $-1$, or $0$. By Cartan's criterion the Killing form is nondegenerate, so no $0$s appear, and I claim that the Killing form is negative definite (i.e. only $-1$s appear) if and only if $G$ is compact.

If the Killing form is negative definite, then minus the Killing form is a $G$-invariant inner product on the adjoint representation $\mathfrak{g}$. This means that the action map $G \to \text{GL}(\mathfrak{g})$, which has finite kernel, lands in the (compact) orthogonal group for this inner product.

Conversely, if $G$ is compact then by a standard averaging argument we can find an inner product on the adjoint representation $\mathfrak{g}$ which is $G$-invariant, and in particular $\mathfrak{g}$ acts on itself by skew-symmetric transformations, i.e. $\text{ad}(x)^{\top} = -\text{ad}(x)$. But now for $x \neq 0$ we have $$\text{tr}(\text{ad}(x)\text{ad}(x)) = -\text{tr}(\text{ad}(x)^{\top}\text{ad}(x)) < 0.$$

Edit: In the negative definite implies compact direction, I should explain why the adjoint representation has finite kernel, since the way I said it might be confusing. The kernel of the adjoint representation is the center of $G$, which is discrete, having trivial Lie algebra. Thus the adjoint representation is an isogeny onto the image of $G$, which is a compact semisimple group and therefore has finite fundamental group.

First, I should comment that I think this is a great question, and that you may post it also on mathoverflow. Second, although I have an answer to yr question, I still find the phenomenon you describe intriguing and am interested in other people's answers. Third, I suspect that my answer is contained somehow in the answer of Justin Campbell, so perhaps you can use my answer as an aid to follow his.

Anyway, the situation is as follows. Every compact Lie groups is, up to a finite cover, the direct product of a torus and some simple Lie groups. If the torus factor is absent, ie the group is covered by the product of simple groups, the group is called semisimple. For example, $SO(3)$ and $SO(n)$ for $n>4$ are simple, $SO(4)$ is semisimple (it is covered 2:1 by $SU(2)\times SU(2)$). The Lie algebra of a compact semisimple Lie group is characterized by an algebraic property called "compactness" (surprise). This is the property that the Killing form, a quadratic form on the Lie algebra given by $tr(ad(x)^2)$, is negative definite.

What about an abelian compact group (a torus)? well, the Lie algebra is abelian, same as for $\mathbb R^n$, so has no chance of recording the compactness. And indeed, there are representations of an abelian algebra which are not completely reducible.

And so it turns out that algebraicaly, what is important for complete reducibility, is not the compactness, but the semisimplicity. Any finite dimensional representation of a semisimple algebra is completely reducible. This can be shown purely algebraicaly, but here is a sketch of a proof that does use compactness. Any (real) semisimple algebra $\mathfrak{g}$ can be complexified to obtain a complex semisimple algebra $\mathfrak{g}_{\mathbb C}$. Now inside $\mathfrak{g}_{\mathbb C}$ one can find a compact real form, ie a real subalgebra $\mathfrak{k}\subset \mathfrak{g}_{\mathbb C}$ of a compact semisimple group $K$, whose complexification is $\mathfrak{g}_{\mathbb C}$. Now given a representation of $\mathfrak{g}$, extend it linearly to $\mathfrak{g}_{\mathbb C}$ and restrict to $\mathfrak{k}$. Now decompose into irreducibles, by compactness of $K$. Then this decomposition is preserved when we extend back to $\mathfrak{g}_{\mathbb C}$ and restrict to $\mathfrak{g}$. (This proof is called the "unitary trick").

Let $\mathfrak g$ be a semisimple real Lie algebra. One can show that the following are equivalent:

i) For every $x \in \mathfrak g$, the linear map $ad(x) : \mathfrak g \rightarrow \mathfrak g$ is semisimple. ("Every element of $\mathfrak g$ is $ad$-semisimple.")

ii) The only $x \in \mathfrak g$ such that $ad(x)$ is a nilpotent endomorphism is $x=0$. ("$\mathfrak g$ has no nonzero $ad$-nilpotent elements.")

iii) The only $x \in \mathfrak g$ such that $ad(x)$ is diagonalisable (over $\mathbb R$) is $x=0$. ("$\mathfrak g$ has no nonzero $ad$-diagonalisable elements.")

iv) The only parabolic subalgebra of $\mathfrak g$ (in the sense of https://math.stackexchange.com/a/4135409/96384) is $\mathfrak g$ itself.

These equivalences actually hold over any field of characteristic $0$ and are part of Lemma 3.2.1. of my thesis. In the case of the field $\mathbb R$, one can further show equivalence to

v) The Killing form of $\mathfrak g$ is negative definite.

Combine this with Justin Campbell's answer, and i)-iv) give you a bunch of very algebraic criteria indeed. I like to advertise them because they seem to be not well-known enough. Look e.g. how easy my answer to Showing the Lie Algebras $\mathfrak{su}(2)$ and $\mathfrak{sl}(2,\mathbb{R})$ are not isomorphic. is if one knows criterion ii.