Why is the maximum Rayleigh quotient equal to the maximum eigenvalue?
(Note: I'm only interested in real-valued matrices here, so I'm using "transpose" and "symmetric" instead of the more general "transjugate" and "Hermitian" in the hope that it will simplify the proof. But the theorem apparently holds for complex-valued matrices as well.)
The Rayleigh quotient $R(M,v)$ of a symmetric matrix $M$ and a vector $v$ is defined as $\frac{v^T M v}{v^T v}$, where $x^T$ is the matrix transpose of $x$.
I've been told that the vector $v$ which gives the largest Rayleigh quotient is, in fact, the eigenvector corresponding to the largest eigenvalue of $M$. And furthermore, the value of the quotient in this case is equal to that eigenvalue. However, I've been unable to find a full proof of this fact, or an explanation of why it should work this way.
Why is there this connection between the Rayleigh quotient and the eigenvalues? Anything from an intuitive explanation to a formal proof would be appreciated.
First, note that R does not depend on the length of v, so we might as well impose the constraint $\mid v\mid^2=1$.
We maximize $R$ subject to this constraint by using a Lagrange multiplier: $v^tMv+\lambda (\mid v\mid^2-1)$, and differentiating with respect to the components of v, we obtain the equation $Mv+\lambda v=0$, so the extrema are precisely the eigenvectors of $M$. If $v$ is an eigenvector, then it follows immediately that the value of $R$ is the corresponding eigenvalue.
The matrix $M$ describes a linear map $M:\>{\mathbb R}^n=V\to V$ of the euclidean vector space $V$ in terms of the standard basis of $V$. The Rayleigh quotient $$R(M,v):={\langle v, Mv\rangle \over |v|^2}$$ is defined independently of the chosen basis, and for orthonormal bases is given by the formula you quote. Now for a symmetric matrix $M$ there is an orthonormal basis that diagonalizes $M$. With respect to such a basis we have $$R(M,v)={\sum_{i=1}^n \lambda_i v_i^2\over\sum_{i=1}^nv_i^2}\ ,$$ and this is maximal when $v$ is an element of the eigenspace $E_\lambda$ corresponding to the eigenvalue $\lambda:=\max{\rm spec}(M)$.
I really enjoy the answers above, and they help me gain some geometrical intuition. I cannot comment under the answer, so I share them here. Hopefully, someone could help me to correct or modify my answer. Many thanks.
1) Pre-multiplying a symmetric matrix $M$ to a vector $v$ is the same as rotating and stretching this vector in the original space. The result should be $ku$, $u$ is a vector with some direction in the same space and $k$ is a scalar.
2) The eigenvector of $M$ gives a direction in which a vector will remain the same after pre-multiplying by $M$.
3) The Rayleigh quotient can be viewed as the cosine value of the angle between the original vector $v$ and $Mv$, multiplied by a scalar $k$.
Then the answer can be implied by those facts.