What is this vector notation called?
If $\hat{j}=\begin{bmatrix}-\frac{1}{\sqrt{2}}\\-\frac{1}{\sqrt{2}}\\0\end{bmatrix}$ and $R'=\begin{bmatrix}10\\0\\0\end{bmatrix}$, then
$$\hat{j}\cdot R'=-\frac{1}{\sqrt{2}}\times 10-\frac{1}{\sqrt{2}}\times 0+0\times 0=-\frac{10}{\sqrt{2}}$$
and so:
$$\hat{j}(\hat{j}\cdot R')=-\frac{10}{\sqrt{2}}\begin{bmatrix}-\frac{1}{\sqrt{2}}\\-\frac{1}{\sqrt{2}}\\0\end{bmatrix}=\begin{bmatrix}5\\5\\0\end{bmatrix}$$
Note that $\hat{j}\cdot R'$ is a scalar $-\frac{10}{\sqrt{2}}$, not a vector.