Equality Constrained Non Negative Linear Least Squares (Unit Simplex Constraint)
I have the following constrained linear least-squares problem:
$$\min_{x \in \mathbb{R}^n} \frac{1}{2}||Ax-b||_2^2,$$
$$\text{subject to } \sum_{i=1}^n x_i = 1 \text{ and } x_i \geq 0, \text{ for } i=1,...,n.$$
If we discard the equality constraint, we have a standard non-negative least-squares (NNLS) problem, which can be solved by any NNLS solver.
On the other hand, if we discard the non-negativity constraints, we have a least-squares problem with an equality constraint, which can be solved in closed form (up to a matrix inverse calculation) by using Lagrange multipliers, for instance.
However, considering both constraints, I do not see any method that is particularly suitable for the problem. Of course, I could use the projected gradient descent method, but this one is pretty generic and does not exploit the linear least-squares nature of the problem.
My questions are:
1- Is there any numerical optimization algorithm that is particularly suitable for this problem?
2- Or can I reformulate the problem in a less constrained form so that can it be solved by a standard NNLS solver?
My attempt:
The Lagrangian is:
$$L(x,\lambda,\mu) = \frac{1}{2}||Ax-b||_2^2 + \lambda (\mathbf{1}^Tx - 1) - \mu^T x,$$ where $\lambda$ is a scalar, $\mu$ is $n$-dimensional and $\mathbf{1}$ is an $n$-dimensional vector of ones.
Thus, writing the KKT conditions, we get:
$$\begin{pmatrix} A^TA & \mathbf{1} & -I \\ \mathbf{1}^T & 0 & \mathbf{0}^T \end{pmatrix}\begin{pmatrix} x \\ \lambda \\ \mu\end{pmatrix} = \begin{pmatrix} A^Tb \\ 1 \end{pmatrix}, \text{ } \mu_i \geq 0, \text{ and } \mu_i x_i = 0, \text{ for } i=1,...,n.$$
I don't know how to proceed from here, due to the complementary slackness constraints...
Your problem can be formulated as convex quadratic programming problem with a linear equality constraint and nonnegativity constraints on $x$.
$\min \frac{1}{2}\left( x^{T}(A^{T}A)x - 2(b^{T}A)x + b^{T}b \right)$
subject to
$\sum_{i=1}^{n} x_{i}=1$
$x \geq 0$.
As a practical matter, you're probably best off using a well-written library routine for linearly constrained quadratic programming. There are many of them available in a variety of programming languages. If you need to implement this yourself, then implementing an active set QP solver is relatively straight forward.