Geometric interpretation of duality and Slater's condition

Solution 1:

A key idea in convex analysis is to think of a set (such as $\mathcal G$) in terms of the half-spaces that contain it.

For a given $\lambda$, you could imagine all the hyperplanes of the form $\lambda u + t = \text{const}$ for which $\mathcal G$ is contained in the upper half space.

And what is the "best" choice of the constant on the right hand side?

The "best" choice is $g(\lambda)$, because that is the largest constant such that $\mathcal G$ is contained in the upper half space of $\lambda u + t = \text{const}$.

So, you can think of $\lambda u + t = g(\lambda)$ as being a hyperplane for which $\mathcal G$ is contained in upper half space. Moreover, for this value of $\lambda$, this is the "best" hyperplane, in the sense that the containment is as tight as possible. If you shifted this hyperplane up any higher, containment would be violated.

Solution 2:

See lecture 8 of Stanford course convex optimization. At time 1.04 (one hour and 4 minutes from start) Stephen Boyd start explaining the geometric interpretation.

http://www.youtube.com/watch?v=FJVmflArCXc

You also have his book (free from his web site) with more formal details.

Solution 3:

As you said $g(\lambda)$ for a given $\lambda$ is a scalar. Suppose $\lambda = 3$ and $g(\lambda) = 5$. That defines the line $3u+t=5$. The dual is $5$ (not $3u+t=5$). The intercept of $3u+t=5$ line is $5$ which is $g(\lambda)$. You get the dual function by varying the $\lambda$s (must be nonnegative). Therefore, the dual function defines a family of lines; find the largest intercept among all. That is $d^*$ which satisfies $d^* \leq p^*$ according to weak duality. In case there is more than one constraint you no longer have supporting lines but supporting hyperplanes.

In summary, the lines are not the duals. A dual (which is a scalar) helps to define the line.