The difference between applying a rotation matrix to a vector (points) and to a matrix (transformation)

Suppose that the rotation matrix is defined as $\mathbf{R}$. Then in order to rotate a vector and a matrix, the following expressions are, respectively, used

$\mathbf{u'}=\mathbf{R} \mathbf{u}$

and

$\mathbf{U'}=\mathbf{R} \mathbf{U} \mathbf{R}^T$,

where $\mathbf{u}$ and $\mathbf{U}$ are, respectively, an arbitrary vector and an arbitrary matrix.

For me, the first one is obvious since you simply multiply the rotation matrix by the vector (for example a point coordinate in 3D) and obtain the rotated vector (rotated point coordinate in 3D). However, the second one is not clear for me and why the rotation should be multiplied from both sides and how this expression is derived.

P.S. The matrix $\mathbf{U}$ can be interpreted as a stretch matrix in 3D.

Here the matrix $U$ is considered not as a bunch of column vectors, but as a (matrix of the) linear map $F\colon {\Bbb R}^n\to {\Bbb R}^n$ $$ y=F(x)=Ux. $$ What happens if we rotate both $y$ and $x$ by $R$? We get (since $R^TR=I$ for rotations) $$ y=Ux\quad\Rightarrow\quad Ry=RUx\quad\Rightarrow\quad Ry=\underbrace{RUR^T}_{U'}Rx\quad\Rightarrow\quad y'=U'x'. $$ Thus the matrix $U'=RUR^T$ corresponds to the same linear map $F$ in the new coordinates after rotation ($x'\mapsto y'$).

In general, for any change of the basis $x'=Sx$, $y'=Sy$ the corresponding change of the matrix $U$ is $$ Sy=\underbrace{SUS^{-1}}_{U'}Sx\quad\Rightarrow\quad y'=U'x'. $$ It means that the class of all similar matrices $\{SUS^{-1}\colon S\text{ invertible}\}$ is exactly the class of all matrices that describe the same linear map in different bases.

Using your example where $U$ is a stretch matrix in 3D, if you want to "rotate" this matrix, you essentially want this stretch action to occur in a different direction / axis. Suppose you have some shape aligned to this new axis. You want to know what the $U'$ is that stretches the shape parallel to this axis. To do this, you use $R^T$ to rotate everything back to the original orientation. Then you do the original stretch transformation $U$. Then you rotate this back using $R$. So $U'=RUR^T$.

One thing that may be instructive is to recall that every matrix can be represented as the linear combination of a series of dyadic/outer products between two vectors, $U = \sum_i a_i \otimes b_i = \sum_i a_i b_i^T$ where $a_i$ and $b_i$ are a sequence of column vectors.

When changing the basis of the matrix, we are in effect applying the vector rule for changing bases to both sequences of vectors:

$$U’ = \sum_i a_i’ \otimes b_i’ = \sum_i Ra_i (Rb_i)^T = \sum_iRa_ib_i^T R^T = RUR^T$$

Hope this helps!