Subgroups and invariants in a unitary group U(3)
Solution 1:
The answer (and the method) is the same as the previous question.
The subgroup $K$ of $U(3)$ containing invariant matrices are isomorphic to the finite group $$ \mathbb{Z}_4\times S_4 \cong\langle i\rangle\times D(2,3,4) $$ where $\langle i\rangle=\{\pm I,\pm iI\}\cong\mathbb{Z}_4$ and $D(2,3,4)$ is the von Dyck group which is isomorphic to $S_4$.
More specifically, $D(2,3,4)=\langle a,b,c \mid a^2=b^3=c^4=abc=I\rangle$ is represented in $U(3)$ as follows: $$ a = \begin{pmatrix} -1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}, \quad b = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}, \quad c = \begin{pmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix} $$
It is a fact from elementary linear algebra that $P_a$ and $k^TP_ak$ have the same rank because $k\in U(3)$ is non-singular. Notice that $P_1,P_2,P_3$ are of rank $2$, but $P_4,P_5,P_6$ are of rank $1$. Thus, if $k^TP_ak=\pm P_b$, then either $a,b\in\{1,2,3\}$ or $a,b\in\{4,5,6\}$.
All the matrices preserving $\{P_i\mid 1\leq i\leq 6\}$ also preserve $\{P_i\mid 1\leq i\leq 3\}$.
Because we already know all the matrices preserving $\{P_i\mid 1\leq i\leq 3\}$ in the previous question, it suffices to check whether those matrices preserve $\{P_i\mid 4\leq i\leq 6\}$ or not.
Note that the three generators $a,b,c$ preserve $\{P_i\mid 4\leq i\leq 6\}$ as follows $$ \begin{gather*} a^TP_4a=P_4, \quad b^TP_4b=P_5, \quad c^TP_4c=P_5 \\ a^TP_5a=P_6, \quad b^TP_5b=P_6, \quad c^TP_5c=P_4 \\ a^TP_6a=P_5, \quad b^TP_6b=P_4, \quad c^TP_6c=P_6 \end{gather*} $$
It is trivial that $\langle i\rangle=\{\pm I,\pm iI\}$ preserve $\{P_i\mid 4\leq i\leq 6\}$.
Therefore we have the same solution as the previous question.