Basic question about $\sup_{x\neq 0}{} \frac{\|Ax\|}{\|x\|} = \sup_{\|x\| = 1}{\|Ax\|} $, $x \in\mathbb{R}^n$

Solution 1:

$$\frac{ \|Ax\| }{\|x\|} = \left|\left| A \frac{x}{\|x\|} \right|\right|$$ Running through all $x \neq 0$ is equivalent to running through all $y:= \frac{x}{\|x\|} $ with $\| y \|=1$.

Solution 2:

As julien pointed out, what follows is not a proof that

$$\sup_{x\neq0}\frac{||Ax||}{||x||}=\sup_{||y||=1}||Ay||,\quad\quad(1)$$

it is a discussion about the consequences of $(1)$. For a proof of $(1)$ see MichaelNgelo's answer.

$(1)$ implies that if we know that a given vector achieves either of the two supremums in $(1)$, then we can deduce from it a vector that achieves the other supremum.

Specifically, $x^*$ is such that

$$\frac{||Ax^*||}{||x^*||}=\sup_{x\neq0}\frac{||Ax||}{||x||}\quad\quad(*)$$

if and only if $y^*:=\frac{x^*}{||x^*||}$ is such that

$$||Ay^*||=\sup_{||y||=1}||Ay||.\quad\quad(**)$$

The above is useful because it allows to compute one of the supremums by computing the other instead, in particular, we can substitute a supremum over $\mathbb{R}\backslash \{0\}$ with one over the unit sphere.

An easy way to argue the above is by contradiction. For example, one direction follows from:

Suppose that $x^*$ satisfies $(*)$, but $y^*:=\frac{x^*}{||x^*||}$ does not satisfy $(**)$. Then, there exists some $\hat{y}\neq y^*$ such that $||\hat{y}||=1$ and

$$||A\hat{y}||>||Ay^*||\Rightarrow \frac{||A\hat{y}||}{1}=\frac{||A\hat{y}||}{||\hat{y}||}>||Ay^*||=\left|\left|A\frac{x^*}{||x^*||}\right|\right|=\frac{||Ax^*||}{||x^*||}=\sup_{x\neq0}\frac{||Ax||}{||x||}$$

which gives a contradiction.

EDIT: The $\sup$ is taken in order to make the function $\sigma$, that maps from the vector space of real $n\times n$ matrices, $M_n(\mathbb{R})$, to $[0,\infty)$, and is defined as

$$\sigma(A):=\sup_{x\neq 0}\frac{||Ax||}{||x||},$$

be a norm (that is, satisfy the norm axioms) on $M_n(\mathbb{R})$. If we drop the sup and instead use a fixed $x$, then $\sigma$ doesn't satisfy the first axiom ($\sigma(A)=0\Leftrightarrow A=0)$ any more.