Eigenvalues of product of two hermitian matrices
Solution 1:
I cannot follow the argument from your book, at least the way it's written here.
The way I would prove the fact is that, being positive definite, we can write $A=F^*F$ for some matrix (you can take $F=A^{1/2}$ if you know functional calculus, but the point is that such an $F$ exists).
And then use the fact that the eigenvalues of a product do not change if you write the product the other way. So the eigenvalues of $AB=F^*FB$ are the same as those of $FBF^*$. This last matrix is clearly Hermitian, so it has real eigenvalues.
Edit: I noticed that I never explained what happens with the reasoning in the question. What happens, if one actually use the $A,B$ given in the question with $\lambda=i$, is that $$ \langle BABx,x\rangle=\langle ABx,Bx\rangle=\lambda\,\langle x,Bx\rangle=0. $$ So one cannot conclude that $\lambda$ is real.
Solution 2:
Let $A$ be positive definite and $B$ be Hermitian. If $x$ is an eigenvector of $AB$ with eigenvalue $\lambda$, then $$\langle ABx,Bx \rangle = \lambda \langle x,Bx \rangle $$ But if we denote $v = Bx$ we can see that since $A$ is positive definite, either $v=0$ and $ABx = Av = 0$ meaning $\lambda = 0$, or $$\langle ABx,Bx \rangle = \langle Av,v \rangle > 0 \text{ and real}$$ This tells us two things: $$\lambda \neq 0 \text{ , } \langle x,Bx \rangle \neq 0 $$ and if $\langle x,Bx \rangle$ is real then $\lambda$ is real. But since B is Hermitian, $\langle x,Bx \rangle$ is real! This can be seen by decomposing x into the eigenbasis $\{v_n\}$ of B: $$x= \sum_n{c_nv_n}$$ $$\langle x,Bx \rangle = \sum_n{|c_n|^2\lambda_nv_n}$$ which is real since each component is real. Thus every eigenvalue $\lambda$ of $AB$ is real.
Solution 3:
Generally, we write $B^{1/2}AB^{1/2}$ to explain $AB$'s eigenvalues are real. The positivity of $B$ is used in the existence of $B^{1/2}$.