How to know if vector is in column space of a matrix?

You could form the projection matrix, $P$ from matrix $A$:

$$P = A(A^TA)^{-1}A^T$$

If a vector $\vec{x}$ is in the column space of $A$, then

$$P\vec{x} = \vec{x}$$

i.e. the projection of $\vec{x}$ unto the column space of $A$ keeps $\vec{x}$ unchanged since $\vec{x}$ was already in the column space.

$\therefore$ check if $P\vec{u} = \vec{u}$

You did perform the operation correctly.

The property you are taking advantage of with this method is the fact that the column space of a matrix is the same as the range of the corresponding matrix transformation (i.e. $x \mapsto A \vec{x}$). By definition of the range of a function, $\vec{u}$ is in the range if and only if there exists some $\vec{x}$ such that $A \vec{x} = \vec{u}$. So you attempted to solve that matrix equation, and determined that there was no solution (by producing an inconsistency).