What does double vertical-line means in linear algebra?

Double bars (or sometimes even single bars) tend to denote a norm in Mathematics. Most likely, the double bars here are denoting the Euclidean norm. This is just the length of the vector. So for example, the vector (I shall write it horizontally for compactness) $(1,2,3)$ has length $$ \|(1,2,3) \|=\sqrt{1^2+2^2+3^2}=\sqrt{14} $$ and the vector $$ \|(3,-1,2) \|=\sqrt{3^2+(-1)^2+2^2}=\sqrt{14} $$ Notice that $A\mathbf{x}$ is just a vector, so $\|A\mathbf{x}\|$ is just the length of the vector. $\|\mathbf{x}\|$ is just the length of $\mathbf{x}$. So here you are looking for scaling of $\mathbf{x}$ under transformation by $A$ to be between $m$ and $M$. (Look at $\frac{\|A\mathbf{x}\|}{\|\mathbf{x}\|}$ and think about what it means 'pictorially' to see what I am talking about).