Geometry of 2-D Linear Discriminant Function [closed]
I am studying ML through the Bishop book myself, and I could not understand why the distance of a given point $\mathbb{x}$ is given by $y(\mathbb{x})/||w||$. Could you help me on deriving this formula from scratch? Thank you.
Let $H = \{ \mathbf{x} \in\mathbb{R}^n ~|~ \mathbf{w}^\top \mathbf{x} = w_0\}$ be a hyperplane (or linear decision surface). Moreover, the Eucledian distance between a point $\mathbf{q}$ and a point $\mathbf{x}$ on the plane is $$\lVert \mathbf{x} - \mathbf{q}\rVert = \sqrt{\sum_{i=1}^n (x_i - q_i)^2 }, \mathbf{x} \in H$$
What we want to find is the shortest such distance, which we can do by solving the equivalent minimization problem \begin{eqnarray} \min_\mathbf{x} f(\mathbf{x})= \frac{1}{2} \lVert \mathbf{x} - \mathbf{q}\rVert^2,\\\text{s. t.} \\ \mathbf{w}^\top\mathbf{x} = w_0. \end{eqnarray}
Using Lagrange multipliers, we can solve this by solving the system: \begin{eqnarray} \nabla f(\mathbf{x}) = \lambda \mathbf{w},\\ \mathbf{w}^\top\mathbf{x} = w_0. \end{eqnarray}
Can you take it from here?
Suppose the hypeplane is defined by the equation $y(\mathbf{x}) = \mathbf{w}^T \mathbf{x} - w_0=0$.
It holds \begin{eqnarray*} \mathbf{x}-\mathbf{x}_\perp &=& \alpha \mathbf{w} \\ \mathbf{w}^T\mathbf{x} - \mathbf{w}^T\mathbf{x}_\perp &=& \alpha \| \mathbf{w} \|^2 \\ \mathbf{w}^T\mathbf{x} - w_0 &=& \alpha \| \mathbf{w} \|^2 \end{eqnarray*}
The (signed) orthogonal distance of a general point $\mathbf{x}$ from the decision surface is given by definition $$ d(\mathbf{x}) = \alpha \| \mathbf{w} \| = \frac{\mathbf{w}^T\mathbf{x} - w_0}{\| \mathbf{w} \|^2} \| \mathbf{w} \| = \frac{\mathbf{w}^T\mathbf{x} - w_0}{\| \mathbf{w} \|} = \frac{y(\mathbf{x})}{\| \mathbf{w} \|} $$
For the origin, the distance is thus $- w_0/\| \mathbf{w} \|$.