For which values does the Matrix system have a unique solution, infinitely many solutions and no solution?

Note that your system is equivalent to the matrix equation $$ \begin{bmatrix} 1 & 3 & -3\\ 0 & 1 & 2 \\ 2 & 5 & a^2-9 \end{bmatrix} \begin{bmatrix} x\\ y\\ z \end{bmatrix} = \begin{bmatrix} 4\\ a\\9 \end{bmatrix} $$ Since $$ \det\begin{bmatrix} 1 & 3 & -3\\ 0 & 1 & 2 \\ 2 & 5 & a^2-9 \end{bmatrix}=a^2-1 $$ this system is guaranteed a unique solution for $a\neq\pm 1$ (do you know why?).

Now the augmented systems for $a=1$ is $$ \begin{bmatrix} 1&3&-3&4\\ 0&1&2&1\\ 2&5&-8&9 \end{bmatrix} $$ Row-reducing this matrix gives $$ \DeclareMathOperator{rref}{rref}\rref\begin{bmatrix} 1&3&-3&4\\ 0&1&2&1\\ 2&5&-8&9 \end{bmatrix} = \begin{bmatrix} 1&0&-9&0\\ 0&1&2&0\\ 0&0&0&1 \end{bmatrix} $$ This system is not consistent (why?) so the original system has no solution for $a=1$.

Can you repeat the process for $a=-1$?

Addendum. You mention in your question that you're having trouble taking determinants. To find the determinant computed above we can expand about the first column: \begin{align*} \det\begin{bmatrix} \color{blue}1 & \color{red}3 & \color{red}{-3}\\ \color{blue}0 & \color{green}1 & \color{green}2 \\ \color{blue}2 & \color{purple}5 & \color{purple}{a^2-9} \end{bmatrix} &= (\color{blue}{1})\cdot\det \begin{bmatrix}\color{green}1&\color{green}2\\\color{purple}5 & \color{purple}{a^2-9} \end{bmatrix}-(\color{blue}0)\det\begin{bmatrix}\color{red}3 & \color{red}{-3}\\ \color{purple}5 & \color{purple}{a^2-9}\end{bmatrix}+(\color{blue}{2})\det\begin{bmatrix}\color{red}3 & \color{red}{-3}\\ \color{green}1 & \color{green}2 \end{bmatrix} \\ &= \left(a^2-9-10\right)-(0)+2\,\left(6+3\right) \\ &= a^2-19+18 \\ &= a^2-1 \end{align*}

Simply by Gauss-Jordan elimination you get: $$\left(\begin{array}{ccc|c} 1 & 3 & -3 & 4\\ 0 & 1 & 2 & a\\ 2 & 5 & a^2-9 & 9 \end{array}\right)\sim \left(\begin{array}{ccc|c} 1 & 3 & -3 & 4\\ 0 & 1 & 2 & a\\ 0 &-1 & a^2-3 & 1 \end{array}\right)\sim \left(\begin{array}{ccc|c} 1 & 3 & -3 & 4\\ 0 & 1 & 2 & a\\ 0 & 0 & a^2-1 & a+1 \end{array}\right)\sim \left(\begin{array}{ccc|c} 1 & 3 & -3 & 4\\ 0 & 1 & 2 & a\\ 0 & 0 & (a-1)(a+1) & a+1 \end{array}\right)$$

Now we would like to divide by $(a-1)(a+1)$. But we can do this only if this expression is non-zero. So we have to deal with the cases $a=\pm1$ separately.

If $a=1$, the last row is $\left(\begin{array}{ccc|c} 0 & 0 & 0 & 2 \end{array}\right)$, which corresponds to the equation $0x+0y+0z=2$. The equation $0=2$ clearly has no solution.

If $a=-1$ you get the system $$\left(\begin{array}{ccc|c} 1 & 3 &-3 & 4\\ 0 & 1 & 2 & -1\\ 0 & 0 & 0 & 0 \end{array}\right)$$ which has infinitely many solutions. (You can also compute them if you want.)

In all other cases (i.e., for $a\ne\pm1$) you can continuo and you get $$ \left(\begin{array}{ccc|c} 1 & 3 & -3 & 4\\ 0 & 1 & 2 & a\\ 0 & 0 & (a-1)(a+1) & a+1 \end{array}\right)\sim \left(\begin{array}{ccc|c} 1 & 3 & -3 & 4\\ 0 & 1 & 2 & a\\ 0 & 0 & 1 & \frac1{a-1} \end{array}\right)\sim \left(\begin{array}{ccc|c} 1 & 0 &-9 & 4-3a\\ 0 & 1 & 2 & a\\ 0 & 0 & 1 & \frac1{a-1} \end{array}\right)\sim \left(\begin{array}{ccc|c} 1 & 0 & 0 & \frac{-3a^2+7a+5}{a-1}\\ 0 & 1 & 0 & \frac{a^2-a-2}{a-1}\\ 0 & 0 & 1 & \frac1{a-1} \end{array}\right)$$ So in these cases the solution is $x=\frac{-3a^2+7a+5}{a-1}$, $y=\frac{a^2-a-2}{a-1}$, $z=\frac{1}{a-1}$.