A sufficient condition of infinite horizon HJB equation

I found a lecture note and book describing Hamilton-Jacobi-Bellman (HJB) equation. In the references, the sufficient condition of HJB for optimality seems C1 condition of the value function (optimal cost).

However, when considering linear quadratic regulation (LQR) problem, the matrix used in value function should be "unique positive definite solution" of continuous algebraic Riccati equation (ARE) (described as CARE here). Also, several papers in control theory usually assumes admissibility of controller.

In this context, I don't understand why "other non-positive-definite solutions" of CARE cannot be optimal solution while satisfying HJB equation (in LQR problem, CARE). Is it due to the boundary condition? If so, is there any reference about sufficient condition for optimality of infinite-horizon HJB equation?

Solution 1:

In the lecture notes by Daniel Liberzon, the sufficient conditions for optimality are clearly outlined in Section 5.1.4. Pertinent to your question are: the Hamiltonian maximization condition in Equation 5.22 and the answer to Exercise 5.4.

In finite horizon problems, any $\mathcal{C}^1$ solution of the HJB equation that also satisfies the Hamiltonian maximization condition is the optimal value function of the corresponding optimal control problem.

In infinite horizon problems, we need more. In addition to continuous differentiability and Hamiltonian maximization, we need one of the following:

• $\hat{V}(x(t)) \to 0$ as $t\to\infty$ along every trajectory with bounded infinite-horizon cost, OR
• $\hat{V}(0) = 0$ and all bounded cost trajectories converge to zero.

It can be shown that under general conditions, positive definiteness and continuous differentiability of a solution of the HJB equation imply Hamiltonian maximization and the second of the two conditions detailed above, and as such, are sufficient to conclude that it is in fact, the optimal value function. There is a complete proof in the corrigendum of this book.