Spectrum of a product of operators on a Banach space
Let $A$ and $B$ be two operators on a Banach space X.
I am interested in the relationship between the spectra
of $A$, $B$ and $AB$.
In particular, are there any set theoretic inclusions
or everything can happen in general like:
$\sigma(A) \subset \sigma(AB)$, and conversely,
$\sigma(B) \subset \sigma(AB)$, and conversely?
If we know the spectra $\sigma(A)$, $\sigma(B)$ of
$A$ and $B$, can we determine the spectrum of $AB$?
I would appreciate any comment or reference.
If $A$ and $B$ commute, then $\sigma(AB)\subseteq \sigma(A)\sigma(B)$. This can be seen by applying the Gelfand transform to the commutative Banach algebra consisting of all operators that commute with all operators that commute with $A$ and $B$. (The reason to take the double commutant rather than simply the Banach algebra generated by $A$, $B$, and the identity is that the spectra of $A$, $B$, and $AB$ may be larger relative to the latter algebra.)
If $A$ and $B$ do not commute, things aren't as nice. E.g., if $A$ is a nonzero nilpotent (or quasinilpotent) operator on a Hilbert space and $B=A^*$, then $\sigma(A)=\sigma(B)=\{0\}$, but the spectral radius of $AB$ is $\|A\|^2\neq0$. For a very concrete example consider $A=\begin{bmatrix}0 & 1 \\ 0 & 0 \end{bmatrix}$ and $B=\begin{bmatrix}0 & 0 \\ 1 & 0 \end{bmatrix}$.
The spectra of $A$ and $B$ can also be much larger than that of $AB$, even in the commutative case. The spectrum of an invertible operator $A$ can be an arbitrary compact subset of the complex plane not containing $0$, but $\sigma(AA^{-1})=\{1\}$. For a noncommutative example, the spectrum of a nonunitary isometry $S$ on Hilbert space (like the unilateral shift) is the closed unit disk, while $\sigma(S^*S)=\{1\}$ and $\sigma(SS^*)=\{0,1\}$.
Here's a little more detail on the first paragraph. Let $a$ and $b$ be commuting elements of a unital Banach algebra $\mathcal{A}$ (for instance, $\mathcal{A}$ could be the algebra of all bounded operators on a Banach space). In order to compare the spectrum of $ab$ to that of $a$ and $b$, we want to apply the Gelfand transform to a commutative Banach algebra $\mathcal{B}$ containing $a$ and $b$. One must be careful, however, because if $a,b,1_\mathcal{A}\in\mathcal{B}\subset\mathcal{A}$, there may be elements of $\mathcal{B}$ that are invertible in $\mathcal{A}$ but whose inverses lie in $\mathcal{A}\setminus\mathcal{B}$. To ensure that this doesn't affect the spectra of $a$, $b$, or $ab$, one can either just stipulate that $\mathcal{B}$ also contains all elements of the form $(a-\lambda)^{-1}$, $(b-\lambda)^{-1}$, and $(ab-\lambda)^{-1}$, as $\lambda$ ranges over the resolvent sets of $a$, $b$, and $ab$ respectively, or just take the double commutant of $\{a,b\}$ in $\mathcal{A}$, which will automatically contain all of these elements.
Once the problem is reduced to the case of elements $a$ and $b$ of a commutative unital Banach algebra $\mathcal{B}$, the Gelfand transform gives you a homomorphism $\Gamma:\mathcal{B}\to C(X)$, where $C(X)$ is the (Banach) algebra of continuous complex-valued functions on the maximal ideal space of $\mathcal{B}$ with pointwise operations (and $\sup$ norm, but that isn't very important here). One of the most fundamental properties of the Gelfand transform is that it preserves spectra; that is, for all $c\in\mathcal{B}$, $\sigma(c)=\Gamma(c)(X)$. Using this fact, the inclusion $\sigma(ab)\subseteq\sigma(a)\sigma(b)$ is translated to a tautological fact about the range of a pointwise product of functions.
To prove that $\sigma(c)=\Gamma(c)(X)$, remember that $X$ can be identified with (or is defined to be, depending on your perspective) the set of nonzero homomorphisms from $\mathcal{B}$ to $\mathbb{C}$. If $\phi\in X$, then $\phi(c-\phi(c))=0$, which, since unital homomorphisms send invertible elements to invertible elements, shows that $c-\phi(c)$ is not invertible. This shows that $\Gamma(c)(\phi)$ is in the spectrum of $c$, so $\Gamma(c)(X)\subseteq\sigma(c)$. Conversely, if $\lambda\in\sigma(c)$, then there is a maximal ideal $M$ of $\mathcal{B}$ containing $c-\lambda$, and by the Mazur-Gelfand theorem there is a unique isomorphism $\mathcal{B}/M\cong \mathbb{C}$. Composing with the quotient map gives a $\phi\in X$ with kernel $M$, so in particular $\phi(c-\lambda)=\phi(c)-\lambda = 0$, showing that $\lambda=\Gamma(c)(\phi)$, so that $\sigma(c)\subseteq\Gamma(c)(X)$.
To learn more about this, I recommend consulting any good textbook on functional analysis or Banach algebras. For example, Rudin, Douglas, and Rickart are good sources for the Gelfand theory that vary widely in emphasis and additional topics covered. As for your question about whether $\sigma(T^{-1}ATB)\subseteq\sigma(A)\sigma(B)$ for commuting $A$ and $B$, the answer is that this need not hold if $T^{-1}AT$ and $B$ do not commute. For example, let $A=B=\begin{bmatrix} 0 & 1\\ 0 & 0\end{bmatrix}$ and $T=\begin{bmatrix}0 & 1 \\ 1 & 0 \end{bmatrix}$.