For $\det(A)=0$, how do we know if $A$ has no solution or infinitely many solutions?
Solution 1:
For any square linear system $\,A\vec x=\vec b\,$ over some field, there exists a unique solution iff $\,\det A\neq 0\,$ , as then we can use the inverse matrix:
$$A\vec x=\vec b\Longleftrightarrow A^{-1}A\vec x=A^{-1}\vec b\Longleftrightarrow A^{-1}\vec b=\vec x $$
As for (a) and your "main question": if $\,\det A=0\,$ one still may have to check whether there are no solutions or infinite solutions (assuming we're working on an infinite field). For example, if the system is homogeneous (over an infinite field) it must have infinite solutions, whereas if the system is non-homogeneous it may have no solutions or several:
$$\begin{cases}x+y=1\\x+y=1\end{cases} \Longleftrightarrow \begin{pmatrix}1&1\\1&1\end{pmatrix}\binom{x}{y}=\binom{1}{1}\longrightarrow\,\,\text{infinite solutions}$$
$$\begin{cases}x+y=1\\x+y=0\end{cases} \Longleftrightarrow \begin{pmatrix}1&1\\1&1\end{pmatrix}\binom{x}{y}=\binom{1}{0}\longrightarrow\,\,\text{no solutions at all}$$
and, of course, in both cases above we have $\,\det A=0\,$
Solution 2:
For the case of a linear system of non-homogeneous equations, you need to consider the augmented matrix and compare its rank to the rank of the coefficient matrix of the system.
Solution 3:
There are two cases actually: If the vector b is not in the column space of the matrix A, it will have no solutions. If the b is in the column space of A, and since det(A)=0, then it will have infinitely many solutions. Hoping this can be a good starting point for you.
Solution 4:
Let be the system of equations: $$ \left\{ \begin{array}{l} ax+by=e \\ cx+dy=f \end{array} \right. $$ If we multiply the determinant by $x$ and use some properties of determinants we get: $$x\begin{array}{|cc|} a & b \\ c & d \\ \end{array}= \begin{array}{|cc|} ax & b \\ cx & d \\ \end{array} \Rightarrow$$
$$\Rightarrow x\begin{array}{|cc|} a & b \\ c & d \\ \end{array}= \begin{array}{|cc|} e-by & b \\ f-dy & d \\ \end{array} \Rightarrow$$ $$\Rightarrow x\begin{array}{|cc|} a & b \\ c & d \\ \end{array}= \begin{array}{|cc|} e & b \\ f & d \\ \end{array}+ \begin{array}{|cc|} -by & b \\ -dy & d \\ \end{array} \Rightarrow$$ $$ \Rightarrow x\begin{array}{|cc|} a & b \\ c & d \\ \end{array}= \begin{array}{|cc|} e & b \\ f & d \\ \end{array}+ 0 \Rightarrow$$ $$\Rightarrow x\begin{array}{|cc|} a & b \\ c & d \\ \end{array}= \begin{array}{|cc|} e & b \\ f & d \\ \end{array} $$ If we have a system where $\begin{array}{|cc|} a & b \\ c & d \\ \end{array}=0$ and $\begin{array}{|cc|} e & b \\ f & d \\ \end{array}=0$ then $x$ can be any real number.
We can do the same with $y$, but in that case we get: $$\Rightarrow y\begin{array}{|cc|} a & b \\ c & d \\ \end{array}= \begin{array}{|cc|} a & e \\ c & f \\ \end{array} $$ Again if $\begin{array}{|cc|} a & b \\ c & d \\ \end{array}=0$ and $\begin{array}{|cc|} a & e \\ c & f \\ \end{array}=0$, $y$ can assume many values depending on $x$ values or vice versa.