Non-Abelian subgroups and invariants in a unitary group 2
The answer (and the method) is the same as the previous question.
Answer. The subgroup $K$ of $U(3)$ containing invariant matrices are isomorphic to the finite group $$ \mathbb{Z}_4\times S_4 \cong\langle i\rangle\times D(2,3,4) $$ where $\langle i\rangle=\{\pm I,\pm iI\}\cong\mathbb{Z}_4$ and $D(2,3,4)$ is the von Dyck group which is isomorphic to $S_4$.
More specifically, $D(2,3,4)=\langle a,b,c \mid a^2=b^3=c^4=abc=I\rangle$ is represented in $U(3)$ as follows: $$ a = \begin{pmatrix} -1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}, \quad b = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}, \quad c = \begin{pmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix} $$
Here is more detailed calculation:
Claim 1. The matrices in $U(3)$ satisfying $k^TP_1k=\pm P_1$ are $$ \begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & \gamma \end{pmatrix} \quad\text{or}\quad \begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & \gamma \end{pmatrix} $$ where $|\alpha|=|\beta|=|\gamma|=1$ and $\alpha\beta=\pm1$.
Proof. Let us find $k=(k_{ij})$ such that $k^TP_1k=\pm P_1$. It implies the following equations $$ \begin{gather*} k_{11}k_{21} = k_{12}k_{22} = k_{13}k_{23} = 0, \\ k_{11}k_{22} + k_{12}k_{21} = \pm1, \tag{*} \\ k_{11}k_{23} + k_{13}k_{21} = k_{12}k_{23} + k_{13}k_{22} = 0. \end{gather*} $$ Note that $k_{13}=0$. Otherwise $k_{21}=k_{22}=k_{23}=0$, and it contradicts to $k\in U(3)$. Similarly, $k_{23}=0$. Moreover, $k_{31}=k_{32}=0$ since $k\in U(3)$. Now the remaining equations in (*) $$ k_{11}k_{21} = k_{12}k_{22} = 0, \quad k_{11}k_{22} + k_{12}k_{21} = \pm1 $$ give the matrices in the claim depending on $k_{11}\neq0$ or $0$.
Fix a permutation matrix $Q=\begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}$. Then we have $$ QP_1Q^T=P_2 \quad\text{and}\quad Q^2P_1(Q^2)^T=Q^TP_1Q=P_3 $$
In a similar way as the previous question, we have $$ \begin{align*} K_{00} \equiv \{k\in U(3) \mid k^TP_1 k=\pm P_1\} &= K_{00} \\ K_{02} \equiv \{k\in U(3) \mid k^TP_1 k=\pm P_2\} &= K_{00}Q^T \\ K_{01} \equiv \{k\in U(3) \mid k^TP_1 k=\pm P_3\} &= K_{00}Q \\ K_{10} \equiv \{k\in U(3) \mid k^TP_2 k=\pm P_1\} &= QK_{00} \\ K_{12} \equiv \{k\in U(3) \mid k^TP_2 k=\pm P_2\} &= QK_{00}Q^T \\ K_{11} \equiv \{k\in U(3) \mid k^TP_2 k=\pm P_3\} &= QK_{00}Q \\ K_{20} \equiv \{k\in U(3) \mid k^TP_3 k=\pm P_1\} &= Q^TK_{00} \\ K_{22} \equiv \{k\in U(3) \mid k^TP_3 k=\pm P_2\} &= Q^TK_{00}Q^T \\ K_{21} \equiv \{k\in U(3) \mid k^TP_3 k=\pm P_3\} &= Q^TK_{00}Q \end{align*} $$
Finally, we have the following 16×6 invariant matrices:
Claim 2. For $\alpha,\beta,\gamma\in\mathbb{C}$ satisfying $\alpha\beta=\pm1$, $\alpha\gamma=\pm1$, and $\beta\gamma=\pm1$, $$ \begin{align*} K_{00}\cap K_{12}\cap K_{21} &\Rightarrow \begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & \gamma \end{pmatrix} \quad K_{00}\cap K_{11}\cap K_{22} \Rightarrow \begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & \gamma \end{pmatrix} \\ K_{02}\cap K_{10}\cap K_{21} &\Rightarrow\begin{pmatrix} \alpha & 0 & 0 \\ 0 & 0 & \beta \\ 0 & \gamma & 0 \end{pmatrix} \quad K_{02}\cap K_{11}\cap K_{20} \Rightarrow\begin{pmatrix} 0 & 0 & \alpha \\ \beta & 0 & 0 \\ 0 & \gamma & 0 \end{pmatrix} \\ K_{01}\cap K_{10}\cap K_{22} &\Rightarrow\begin{pmatrix} 0 & \alpha & 0 \\ 0 & 0 & \beta \\ \gamma & 0 & 0 \end{pmatrix} \quad K_{01}\cap K_{12}\cap K_{20} \Rightarrow\begin{pmatrix} 0 & 0 & \alpha \\ 0 & \beta & 0 \\ \gamma & 0 & 0 \end{pmatrix} \end{align*} $$