Invariant Subspace of Two Linear Involutions

Solution 1:

Consider the linear transformation $AB:V\to V$. Since $V$ is a complex vector space, $AB$ has at least an eigenvector, call it $X$, i.e. $ABx=\lambda x$. Note, $AABx=Bx=A\lambda x=\lambda Ax$ and then $x=BBx=BA\lambda x=\lambda BAx$ So consider the space $\langle x, Ax\rangle$. Note, this space is invariant under $A$, clearly, $Bx=\lambda Ax$ and $BAx=x$ so it's invariant under $B$. This space is at least one dimensional.

Solution 2:

Note: First couple of paragraphs relate to some statements made in a previous version of the question.

If the two operators are the identity, then it certainly has a subspace which is both invariant under both, and $1$ dimensional. The phrasing of the question does not suggest that there are no larger subspaces which are invariant under both (after all, $\mathbf{V}$ itself is always invariant under both $A$ and $B$, so you always have larger dimensional invariant subspaces. In fact, what you want to find is the smallest possible nontrivial subspace which is invariant under both $A$ and $B$.

The reason you want this is that if you can find a $1$-dimensional subspace that is invariant under an operator, that subspace gives you an eigenvector and an eigenvalue. If you can find a $1$-dimensional subspace that is invariant under both $A$ and $B$, then this tells you that there is a vector which is an eigenvector for both $A$ and $B$, which is a very useful condition (note that the same vector may be an eigenvector for both $A$ and $B$, but corresponding to different eigenvalues). Unfortunately, you don't always have this, which is why the problem says that sometimes you need to take a $2$-dimensional subspace instead.

For example, if you take $A$ to be the linear transformation on $\mathbb{C}^2$ that sends $(1,0)$ to $(-1,0)$ and $(0,1)$ to $(0,1)$; and you take $B$ to be the linear transformation that sends $(1,0)$ to $(0,-1)$ and $(0,1)$ to $(-1,0)$, then $A$ and $B$ have no common eigenvector, so you will not be able to find a $1$-dimensional subspace that is invariant under both $A$ and $B$.

You are correct that the only possible eigenvalues of $A$ and of $B$ are $1$ and $-1$; in fact, more can be said: unless the characteristic of the field is $2$, then $A$ and $B$ are diagonalizable, because their minimal polynomials divide $t^2-1$, which splits and is square free.

Let $E^A_{1}$ and $E^A_{-1}$ be the eigenspaces of $A$ associated to $1$ and $-1$, respectively (one of them could be trivial, for example if $A=I$ then $E^A_{-1}=\{\mathbf{0}\}$, and similarly if $A=-I$; that's fine); and similarly with $E^B_{1}$ and $E^B_{-1}$. If $E^A_{\lambda}\cap E^B_{\mu}\neq\{\mathbf{0}\}$ for some $\lambda,\mu\in\{1,-1\}$, then you'll be done: their intersection contains a $1$-dimensional subspace that is invariant for both $A$ and $B$.

So you are on the right track in trying to find a nontrivial intersection of eigenspaces.

Unfortunately, you cannot assert that you will necessarily have $E^A_{1}$ and $E^B_{1}$ intersecting nontrivial (what if $A=-I$ and $B=I$, for example? Then $E^A_{1}={\mathbf{0}}$).

Since $A$ and $B$ have to be diagonalizable, you know that $\dim(\mathbf{V}) = \dim(E^A_1)+\dim(E^A_{-1}) = \dim(E^B_1)+\dim(E^B_{-1})$. So try to figure out what would happen if all four intersections are trivial (we know it can happen, as I already gave an example above where it happens).

Added. Okay, let's finish this, with the assumption that $\mathbf{V}$ is a complex vector space.

If $E^A_{i}\cap E^B_{j}\neq\{\mathbf{0}\}$ for some $i,j\in\{1,-1\}$, then we are done: just take a vector which is an eigenvector of both $A$ and $B$, and this gives a one dimensional invariant subspace. So we may assume that $E^A_{i}\cap E^B_{j}=\{\mathbf{0}\}$, $i,j\in\{1,-1\}$. That is, we must be in the situation:

$\dim(\mathbf{V})= 2n$, $\dim(E^A_{i}) = \dim(E^B_j) = n$, and $E^A_i\cap E^B_{j} = \{\mathbf{0}\}$ for $i,j\in\{1,-1\}$.

(The dimensions follow because if either $E^A_i$ has dimension more than half the dimension of $\mathbf{V}$, then it must intersection one $E^B_j$ nontrivially by simple dimension considerations, and symmetrically for the $E^B_j$; so the dimension of $\mathbf{V}$ must be even, and the eigenspaces must each have dimension exactly half).

Since $A$ and $B$ are diagonalizable, we know that $$\mathbf{V}=E^A_1\oplus E^A_{-1} = E^B_1\oplus E^B_{-1}.$$ Given $\mathbf{v}\in \mathbf{V}$, we can write it uniquely as $\mathbf{v}=\mathbf{y}_1+\mathbf{y}_{-1}$, with $\mathbf{y}_j\in E^B_j$; define $p^B_{j}\colon \mathbf{V}\to E^B_{j}$ by $p^B_j(\mathbf{y}_1+\mathbf{y}_{-1}) = \mathbf{y}_j$.

The restriction of $p^B_{1}$ to $E^A_1$ is injective, since $\mathrm{ker}(p^B_1) = E^B_{-1}$; likewise for $p^B_{-1}$. Let $p^B_{1,1}$ be the restriction of $p^B_{1}$ to $E^A_1$, let $p^B_{1,-1}$ be the restriction of $p^B_{-1}$ to $E^A_{-1}$; and likewise let $p^B_{-1,1}$ be the restriction of $p^B_{1}$ to $E^A_{-1}$, and $p^B_{-1,-1}$ be the restriction of $p^B_{-1}$ to $E^A_{-1}$. Then $p^B_{i,j}\colon E^A_i\to E^B_j$ is an isomorphism, being a one-to-one linear transformation between spaces of the same dimension.

Let $T\colon E^A_1\to E^A_1$ be the operator defined by $$T = \left(p^B_{1,-1}\right)^{-1}\circ p^B_{-1,-1}\circ\left(p^B_{-1,1}\right)^{-1}\circ p^B_{1,1}.$$ Then since $E^A_1$ is a complex vector space, $T$ has an eigenvector $\mathbf{x}_1$, corresponding to some eigenvalue $\lambda$.

Let $\mathbf{x}_1 = \mathbf{y}_1 + \mathbf{y}_{-1}$, with $\mathbf{y}_j \in E^B_{j}$. Note that $\mathbf{y}_j\neq \mathbf{0}$ (if either one is zero, then we get a vector which is an eigenvector for both $A$ and $B$, and we already ruled out this case). Let $$\mathbf{x}_{-1} = (p^B_{-1,1})^{-1}(\mathbf{y}_1) = (p^B_{-1,1})^{-1}\circ p^B_{1,1}(\mathbf{x}_1)\in E^A_{-1}.$$ Then $\mathbf{x}_{-1}\neq\mathbf{0}$. Therefore $\mathbf{x}_1,\mathbf{x}_{-1}$ are linearly independent. Let $\mathbf{W}=\mathrm{span}(\mathbf{x}_1,\mathbf{x}_{-1})$. Since $\mathbf{x}_i$ is an eigenvector of $A$, that means that $\mathbf{W}$ has a basis of eigenvectors of $A$, and therefore $\mathbf{W}$ is $A$-invariant.

Write $\mathbf{x}_{-1} = \mathbf{z}_1+\mathbf{z}_{-1}$, with $\mathbf{z}_j\in E^B_j$. Then $$\mathbf{z}_1 = p^B_{-1,1}(\mathbf{x}_{-1}) = p^B_{-1,-1}\left((p^B_{-1,1})^{-1}(\mathbf{y}_1)\right) = \mathbf{y}_1.$$ And notice that since $T(\mathbf{x}_1) = \lambda\mathbf{x}_1$, then $$\mathbf{y}_{-1} = p^B_{1,-1}(\mathbf{x}_1) = \frac{1}{\lambda}p^B_{1,-1}(T(\mathbf{x}_1)) = \frac{1}{\lambda}p^B_{-1,-1}\circ (p^B_{-1,1})^{-1}\circ p^B_{1,1}(\mathbf{x}_1) = \frac{1}{\lambda}p^B_{-1,-1}(\mathbf{x}_{-1}) = \frac{1}{\lambda}\mathbf{z}_{-1}.$$ Therefore, $\mathbf{z}_{-1} = \lambda\mathbf{y}_{-1}$. That is, we have: $$\begin{align*} \mathbf{x}_1 &= \mathbf{y}_1 + \mathbf{y}_{-1},\\ \mathbf{x}_{-1} &= \mathbf{y}_1 + \lambda\mathbf{y}_{-1}. \end{align*}$$ Finally, note that $\lambda\neq 1$, because $\mathbf{x}_1$ and $\mathbf{x}_{-1}$ are linearly independent. But this mean that $$\mathbf{W} = \mathrm{span}(\mathbf{x}_1,\mathbf{x}_{-1}) = \mathrm{span}(\mathbf{y}_1+\mathbf{y}_{-1},\mathbf{y}_1+\lambda\mathbf{y}_{-1}) = \mathrm{span}(\mathbf{y}_1,\mathbf{y}_{-1}).$$ The last equality follows since we can replace $\mathbf{y}_1+\lambda\mathbf{y}_{-1}$ with the difference between the two basis vectors, which is a nonzero multiple of $\mathbf{y}_{-1}$; then we can replace this multiple with $\mathbf{y}_{-1}$ itself, and then we can replace $\mathbf{y}_1+\mathbf{y}_{-1}$ with $\mathbf{y}_1$ by itself, since $\mathbf{y}_{-1}$ is also in the span.

Thus, there is a basis for $\mathbf{W}$, namely $[\mathbf{y}_1,\mathbf{y}_{-1}]$, consisting of eigenvectors of $B$. Therefore, $\mathbf{W}$ is $B$-invariant.

In summary, if there are no $1$-dimensional subspaces that are both $A$- and $b$-invariant, then there is a $2$-dimensional subspace that is both $A$- and $B$-invariant.

Note. Although there seems to be some lack of symmetry on $T$, since it depends on the projections on $B$, note that $(p^B_{ij})^{-1} = p^A_{ji}$, so in fact the argument is very symmetric relative to $A$ and $B$.

Added 2. (And simplified) To answer your question in comments in Plop's response ("Why are they necessarily collinear?"), suppose that you have

$\dim(\mathbf{V})=2n$, $\dim(E^A_{i})=\dim(E^B_{j})=n$, $E^A_i\cap E^B_{j}=\{\mathbf{0}\}$

and that $\mathbf{W}$ is a $2$-dimensional subspace of $\mathbf{V}$ that is $A$-invariant, and $B$-invariant.

Remember that the restriction of a diagonalizable operator to an invariant subspace is diagonalizable. So $\mathbf{W}$ must have a basis of eigenvectors of $A$ and of $B$.

If $\mathbf{W}\subseteq E^A_i$, then the eigenvectors of $B$ contained in $\mathbf{W}$ would be common eigenvectors of $A$ and $B$, which we are ruling out. So $\mathbf{W}$ cannot be contained in $E^A_i$; symmetrically, it cannot be contained in $E^B_j$. Since the restrictions of both $A$ and $B$ are diagonalizable, we can find a basis for $\mathbf{W}$ consisting of eigenvectors of $A$, and one consisting of eigenvectors of $B$. But that means that there is a basis $\mathbf{x}_1,\mathbf{x}_{-1}$ of $\mathbf{W}$ with $\mathbf{x}_i\in E^A_i$, and a basis $\mathbf{y}_1,\mathbf{y}_{-1}$ with $\mathbf{y}_j\in E^B_j$.

If we write $\mathbf{x}_i$ as $$\begin{align*} \mathbf{x}_1 &= \mathbf{z}_1+\mathbf{z}_{-1}\\ \mathbf{x}_{-1}&=\mathbf{w}_1 + \mathbf{w}_{-1} \end{align*}$$ with $\mathbf{z}_j,\mathbf{w}_j\in E^B_j$, then since the $\mathbf{y}_j$ are a basis for $\mathbf{W}$ we must have that $\mathbf{z}_j,\mathbf{w}_j\in\mathrm{span}(\mathbf{y}_j)$, so the $E^B_j$ components of $\mathbf{x}_{1}$ and $\mathbf{x}_{-1}$ are multiples of the same vector; that is, they are collinear. So $p^B_{1,1}(\mathbf{x}_1)$ and $p^B_{-1,1}(\mathbf{x}_{-1})$ are collinear; likewise with their images in $E^B_{-1}$.

Symmetrically, if we write the $\mathbf{y}_j$ as a sum of vectors in $E^A_i$, then the $E^A_1$ components must both be multiples of $\mathbf{x}_1$, and the $E^A_{-1}$ components must both be multiples of $\mathbf{x}_{-1}$, hence they are collinear.

In particular, using the argument given by Plop, we see that the existence of a 2-dimensional subspace of $\mathbf{V}$ which is both $A$- and $B$-invariant, then it is not contained in $E^A_i$ or $E^B_j$, $i,j\in\{1,-1\}$, and the existence of an eigenvector for the operator $T$ defined above follows.

In summary, putting together the two parts, we have the following:

Theorem. Suppose $\mathbf{V}$ is a vector space (over a field of characteristic different from $2$), and $A$ and $B$ are operators such that $A^2=B^2=I$. Let $E^A_i$ be the eigenspace of $A$ corresponding to $i$, $E^B_j$ the eigenspace of $B$ corresponding to $j$, with $i,j\in\{1,-1\}$. Suppose further that $\dim(\mathbf{V})=2n$, $\dim(E^A_i)=\dim(E^B_j)=n$, and $E^A_i\cap E^B_j = \{\mathbf{0}\}$. Then $\mathbf{V}$ has a $2$-dimensional subspace that is both $A$-invariant and $B$-invariant if and only if the operator $$T = \left(p^B_{1,-1}\right)^{-1}\circ p^B_{-1,-1}\circ\left(p^B_{-1,1}\right)^{-1}\circ p^B_{1,1} = p^A_{-1,1}\circ p^B_{-1,-1}\circ p^A_{1,-1}\circ p^B_{1,1}$$ on $E^A_1$ has an eigenvector, where $p^A_{j,i}$ is the restriction of the projection $p^A_i\colon \mathbf{V}\to E^A_i$ along $E^A_{-i}$ to $E^B_j$, and $p^B_{i,j}$ is the restriction of the projection $p^B_j\colon \mathbf{V}\to E^B_{j}$ along $E^B_{-j}$ to $E^A_i$.

The reason we need the characteristic to be different from $2$ is so that we can use the fact that $A$ and $B$ are both diagonalizable; if the characteristic is $2$, then we don't know that, because the minimal polynomial could be $(t+1)^2$.

Solution 3:

Arturo's answer can be condensed to the following:

Let $U_1$, $\ldots$, $U_4$ be the eigenspaces of $A$ and $B$. Letting the simple cases aside we may assume that $U_i\oplus U_j=V$ for all $i\ne j$. We have to produce four nonzero vectors $x_i\in U_i$ that lie in a two-dimensional plane.

For $i\ne j$ denote by $P_{ij}:V\to V$ the projection along $U_i$ onto $U_j$; whence $P_{ij}+P_{ji}=I$. The map $T:=P_{41}\circ P_{32}$ maps $V$ to $U_1$, so it leaves $U_1$ invariant. It follows that $T$ has an eigenvector $x_1\in U_1$ with an eigenvalue $\lambda\in{\mathbb C}$. Now put $$\eqalign{x_2&:=P_{32}x_1\in U_2\ ,\cr x_3&:=x_1-x_2=P_{23} x_1\in U_3\ ,\cr x_4&:=x_2-\lambda x_1=x_2- P_{41}P_{32}x_1=x_2-P_{41}x_2=P_{14}x_2\in U_4\ .\cr}$$ It is easily checked that all four $x_i$ are nonzero.