Meaning/Justification for Describing Functions as 'Orthogonal'

When introducing Fourier series, my lecturer stated that 2 periodic functions, $f$ and $g$, with period $2L$ are orthogonal iff $$\int^{L}_{-L}{f(x)g(x)}\mathrm dx=0$$ Wikipedia agrees, even defining the above integral to be the inner product of $f$ and $g$, completely analogous with the terminology from linear algebra. My lecturer hinted that there is a reason for the terminology being the way it is, but personally I'm having a bit of a hard time seeing what the above integral has to do with orthogonality as I would understand it.

On a related note, does this connection (between orthogonal functions and orthogonality more generally, assuming there is one) in some way 'explain' why the sine and cosine functions can be used to construct any continuous functions in a Fourier series, while other sets of functions cannot. Currently it seems the orthogonality plays an almost magical role in deriving the coefficient equations, I'm wondering if there's something deeper there.

Thanks!

Solution 1:

Indeed, there is deep theory involved behind all of this. When you learn more about inner product/Hilbert spaces, you will see spaces that have a certain operation on them called an inner product whose axioms/properties you can find in the link. These spaces are interesting because inner products give us a sense of "angle" or "direction" (note how its properties generalize the dot product of Euclidean space). Further, we can define a norm on that space with this inner product: $ \|x\| = \sqrt{\langle x,x \rangle}$ and then this norm gives us an idea of the "length" of a vector in this space.

Then we define a metric on this space with $ d(x,y) = \|x-y\|$ and this metric gives us an idea of distance between two points (which leads us to think about convergence of sequences and other analysis-type questions). This metric also then allows us to generalize open sets which endows a sense of whether some points are "close together" or "split apart", and these open sets then induce a topology, which allows us to study topological properties such as continuity and connectedness.

Clearly that is just an overload of new information, so you may wonder - what does it all mean? Well essentially each of these stages are abstractions that are structurally similar to the well known Euclidean space. Hilbert spaces are the most similar to $\mathbb{R}^n$ - in them we have all the ideas above, of angle, direction, magnitude of a vector, distance between vectors, neighbourhoods of vectors, and convergence of every Cauchy sequence. This is why Hilbert spaces are so interesting - we have plenty of intuition about how they should behave, they have enough structure on them to give us plenty of useful results, but they are general enough to be substantially different from $\mathbb{R}^n$.

One example of a Hilbert space is the Lebesgue square-integrable functions with the inner-product $\langle f,g \rangle = \int_X f\cdot \bar g\; \mathrm d\mu$, and you are considering one of them. The integral in your post corresponds to the inner product of this space (remember, this is the generalization of a dot product) and just as when the dot product of Euclidean vectors was $0$ we declared them orthogonal, we do the same here. As anon commented, when two functions are orthogonal they are linearly independent, which is a familiar property. If you have enough linearly independent vectors chosen wisely to form a basis, we should be able to form any vector in this space as a linear combination of the basis vectors - that is a Fourier series.

Solution 2:

An Inner Product Space is a vector space on which an inner product has been defined. An inner product is a positive definite bilinear form, $\left<x,y\right>$; that is, a bilinear form where $\left<x,x\right>\ge0$ and $\left<x,x\right>=0$ if and only if $x=0$. Orthogonality is defined using an inner product; that is, $x$ and $y$ are orthogonal when $\left<x,y\right>=0$. The length of a vector is also defined in terms of the inner product: $|x|^2=\left<x,x\right>$. Using a process called Gram-Schmidt Orthonormalization, one can turn any basis for a finite dimensional inner product space into an Orthonormal basis, a basis $\{e_i\}_{i=1}^n$ for which $\left<e_i,e_j\right>=0$ when $i\not=j$ and $\left<e_i,e_i\right>=1$.

In an inner product space, a vector can be written as a linear combination of the basis elements $$ v=\sum_{i=1}^na_ie_i\tag{1} $$ If the basis is orthonormal, we can easily compute the coefficients in $(1)$ using the inner product: $$ \begin{align} \left<v,e_i\right> &=\left<\sum_{j=1}^na_je_j\;,\;e_i\right>\\ &=\sum_{j=1}^na_j\left<e_j\;,\;e_i\right>\\ &=a_i\tag{2} \end{align} $$ We also have the Pythagorean Theorem for lengths $$ \begin{align} |v|^2 &=\left<v,v\right>\\ &=\left<\sum_{i=1}^na_ie_i\;,\;\sum_{j=1}^na_je_j\right>\\ &=\sum_{i=1}^n\sum_{j=1}^na_ia_j\left<e_i,e_j\right>\\ &=\sum_{i=1}^na_i^2\tag{3} \end{align} $$ Many inner products can be defined on an inner product space. For functions on $(-L,L]$, the simplest in form is $$ \left<f,g\right>=\int_{-T}^Tf(x)g(x)\;\mathrm{d}x\tag{4} $$ With respect to this inner product, the trigonometric functions, $\{\sin(\frac{\pi nx}{L})\}_{n=1}^\infty$ and $\{\cos(\frac{\pi nx}{L})\}_{n=0}^\infty$, or equivalently $\{e^{i\pi nx/L}\}_{n=-\infty}^\infty$, are orthonormal.

Solution 3:

Consider the vectors ${\bf x}=(x_n)$, ${\bf y}=(y_n)$ in $\mathbb{R}^{N+1}$ defined by

$$x_n = \sqrt{h} \,f(nh), \qquad y_n=\sqrt{h}\, g(nh), \qquad h = \frac{2L}{N}$$

for $n=0,1,\dots,N$.

The standard Euclidean inner product of ${\bf x}$ and ${\bf y}$ is given by

$${\bf x\cdot y} = h\sum_{n=1}^N f(nh) g(nh)$$

But this is also the Riemann sum corresponding to the integral of $f(x)g(x)$ over the interval $[-L,L]$. In the limit $N\to\infty$ the sum converges (assuming nice properties of $f$ and $g$) to

$$\int_{-L}^L f(x) f(x) dx$$

which is the expression given by your lecturer for the inner product between two functions.