How do I exactly project a vector onto a subspace?

I am trying to understand how - exactly - I go about projecting a vector onto a subspace.

Now, I know enough about linear algebra to know about projections, dot products, spans, etc etc, so I am not sure if I am reading too much into this, or if this is something that I have missed.

For a class I am taking, the proff is saying that we take a vector, and 'simply project it onto a subspace', (where that subspace is formed from a set of orthogonal basis vectors).

Now, I know that a subspace is really, at the end of the day, just a set of vectors. (That satisfy properties here). I get that part - that its this set of vectors. So, how do I "project a vector on this subspace"?

Am I projecting my one vector, (lets call it a[n]) onto ALL the vectors in this subspace? (What if there is an infinite number of them?)

For further context, the proff was saying that lets say we found a set of basis vectors for a signal, (lets call them b[n] and c[n]) then we would project a[n] onto its signal subspace. We project a[n] onto the signal-subspace formed by b[n] and c[n]. Well, how is this done exactly?..

Thanks in advance, let me know if I can clarify anything!

P.S. I appreciate your help, and I would really like for the clarification to this problem to be somewhat 'concrete' - for example, something that I can show for myself over MATLAB. Analogues using 2-D or 3-D space so that I can visualize what is going on would be very much appreciated as well.

Thanks again.

Solution 1:

I will talk about orthogonal projection here.

When one projects a vector, say $v$, onto a subspace, you find the vector in the subspace which is "closest" to $v$. The simplest case is of course if $v$ is already in the subspace, then the projection of $v$ onto the subspace is $v$ itself.

Now, the simplest kind of subspace is a one dimensional subspace, say the subspace is $U = \operatorname{span}(u)$. Given an arbitrary vector $v$ not in $U$, we can project it onto $U$ by $$v_{\| U} = \frac{\langle v , u \rangle}{\langle u , u \rangle} u$$ which will be a vector in $U$. There will be more vectors than $v$ that have the same projection onto $U$.

Now, let's assume $U = \operatorname{span}(u_1, u_2, \dots, u_k)$ and, since you said so in your question, assume that the $u_i$ are orthogonal. For a vector $v$, you can project $v$ onto $U$ by $$v_{\| U} = \sum_{i =1}^k \frac{\langle v, u_i\rangle}{\langle u_i, u_i \rangle} u_i = \frac{\langle v , u_1 \rangle}{\langle u_1 , u_1 \rangle} u_1 + \dots + \frac{\langle v , u_k \rangle}{\langle u_k , u_k \rangle} u_k.$$

Solution 2:

Take a basis $\{v_1, \dots, v_n\}$ for the "signal subspace" $V$. Let's assume $V$ is finite dimensional for simplicity and practical purposes, but you can generalize to infinite dimensions. Let's also assume the basis is orthonormal.

The projection of your signal $f$ onto the subspace $V$ is just

$$\mathrm{proj}_V(f) = \sum_{i=1}^n \langle f, v_i \rangle v_i$$

and $f = \mathrm{proj}_V(f) + R(f)$, where $R(f)$ is the remainder, or orthogonal complement, which will be 0 if $f$ lies in the subspace $V$.

The $i$-th term of the sum, $\langle f, v_i\rangle$, is the projection of $f$ onto the subspace spanned by the $i$-th basis vector. (Note, if the $v_i$ are orthogonal, but not necessarily orthonormal, you must divide the $i$-th term by $\|v_i\|^2$.)

Solution 3:

The projection of the vector is the vector belonging to the subspace that best approximates the former, i.e. such that the (squared) norm of the difference is the smallest.

WLOG take a subspace spanned by three vectors. We minimize for the coordinates $u,v,w$

$$\epsilon=(\vec d-u\vec a-v\vec b-w\vec c)^2.$$

This is achieved by canceling the gradient (factor $2$ omitted),

$$\begin{cases} \dfrac{d\epsilon}{du}=\vec a\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0,\\ \dfrac{d\epsilon}{dv}=\vec b\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0,\\ \dfrac{d\epsilon}{dw}=\vec c\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0.\\ \end{cases}$$ giving a linear system of three equations in three unknonws.

In the special case that $\vec a,\vec b,\vec c$ form an orthonormal basis, we simply have

$$u=\vec d\cdot\vec a,v=\vec d\cdot\vec b,w=\vec d\cdot\vec c,$$ and the projection is

$$\vec u^*=(\vec d\cdot\vec a)\vec a+(\vec d\cdot\vec b)\vec b+(\vec d\cdot\vec c)\vec c.$$