Help finding the determinant of a 4x4 matrix?

Here is how you should write it down in practice. $$ A = \begin{pmatrix} 5 & -7 & 2 & 2 \\ 0 & 3 & 0 & -4 \\ -5 & -8 & 0 & 3 \\ 0 & 5 & 0 & -6 \\ \end{pmatrix} $$ so that $$ \det A = \begin{vmatrix} 5 & -7 & 2 & 2 \\ 0 & 3 & 0 & -4 \\ -5 & -8 & 0 & 3 \\ 0 & 5 & 0 & -6 \\ \end{vmatrix} \\ = 2 \cdot \begin{vmatrix} 0 & 3 & -4 \\ -5 & -8 & 3 \\ 0 & 5 & -6 \\ \end{vmatrix} \\ = 2 \cdot -(-5) \cdot \begin{vmatrix} 3 & -4 \\ 5 & -6 \\ \end{vmatrix} \\ = 2 \cdot 5 \cdot (-18 -(-20)) \\ = 20. $$ I expanded along the third column in the first case and then along the first column, because these are the ones with the most zeros, so it saves a lot of computations. I am assuming that's what's in your book.

If you want to expand along other columns or rows, just keep track of the appropriate minors without making any computation mistakes (I guess that's the hard part ; the only trick is to make it slowly and be careful). So here we go (along the first row): $$ \det A = \begin{vmatrix} 5 & -7 & 2 & 2 \\ 0 & 3 & 0 & -4 \\ -5 & -8 & 0 & 3 \\ 0 & 5 & 0 & -6 \\ \end{vmatrix} = 5 \begin{vmatrix} 3 & 0 & -4 \\ -8 & 0 & 3 \\ 5 & 0 & -6 \\ \end{vmatrix} -(-7) \begin{vmatrix} 0 & 0 & -4 \\ -5 & 0 & 3 \\ 0 & 0 & -6 \\ \end{vmatrix} + 2 \begin{vmatrix} 0 & 3 & -4 \\ -5 & -8 & 3 \\ 0 & 5 & -6 \\ \end{vmatrix} -2 \begin{vmatrix} 0 & 3 & 0 \\ -5 & -8 & 0 \\ 0 & 5 & 0 \\ \end{vmatrix}. $$ I don't know how you defined the determinant, but in any definition you chose it should be obvious that a determinant of a matrix with a column of zeros is zero. (If it is not clear to you, feel free to tell me your definition and I will happily answer in the comments.) So the only non-zero term in this sum is $$ 2 \begin{vmatrix} 0 & 3 & -4 \\ -5 & -8 & 3 \\ 0 & 5 & -6 \\ \end{vmatrix}. $$ Note that this is the same determinant as when I expanded along the third column. So you can finish this computation by looking at what I did before.

A useful trick to remember the signs in the Laplace expansion (that's the name of the trick of expanding along a row or a column) is the following matrix : $$ \begin{vmatrix} + & - & + & - \\ - & + & - & + \\ + & - & + & - \\ - & + & - & + \end{vmatrix} $$ It works for any determinant size, just make sure that the coordinate of the matrix in the top left is a $+$ sign.

Hope that helps,

For M3 it is $-90+100$ rather than $-90-100$ which gives $10 \times 2=20$ as the final answer

Steve explained where you made a mistake in your calculations. And Patrick explained how you can save computations by judiciously choosing the rows/ columns you expand along. Just for fun, I'll explain a different way of evaluating the determinant. I'm just going to use the relationship between the elementary row/ column operations and the determinant.

Here are those relationships:

1. Swapping two rows/ columns of a matrix will give a factor of $-1$ to the determinant. Let $a_k$ be the $k$th row (or column) of the matrix $A$. Then $$\det(A) = \det(a_1,\dots, a_i, \dots, a_j, \dots, a_n) = -\det(a_1,\dots, a_j, \dots, a_i, \dots, a_n)$$
2. A common factor can be "pulled out" of a row/ column. $$\det(a_1,\dots,ka_i,\dots,a_n) = k\det(a_1,\dots,a_i,\dots,a_n)$$
3. Adding a scalar muliple of one row/ column to another will not change the determinant at all. $$\det(a_1,\dots, a_i, \dots, a_j, \dots, a_n) = \det(a_1,\dots, a_i, \dots, a_j+ka_i, \dots, a_n)$$

Let's use these properties of the determinant to calculate the determinant of your matrix:

$$\begin{align}\begin{vmatrix} 5&-7&2&2\\ 0&3&0&-4\\ -5&-8&0&3\\ 0&5&0&-6\\ \end{vmatrix} &= \enspace\ \frac 12\begin{vmatrix} 5 & -7 & 2 & 2 \\ 0 & 6 & 0 & -8 \\ 0 & -15 & 2 & 5 \\ 0 & -1 & 0 & 2\end{vmatrix} &{\begin{pmatrix}R_2 \to 2R_2 \\ R_3 \to R_3+R_1 \\ R_4 \to R_4-R_2\end{pmatrix}} \\ &= -\frac 12\begin{vmatrix} 5 & -7 & 2 & 2 \\ 0 & -1 & 0 & 2 \\ 0 & -15 & 2 & 5 \\ 0 & 6 & 0 & -8\end{vmatrix} & \begin{pmatrix}R_2 \leftrightarrow R_4\end{pmatrix} \\ &= -\frac 12\begin{vmatrix} 5 & -7 & 2 & 2 \\ 0 & -1 & 0 & 2 \\ 0 & 0 & 2 & -25 \\ 0 & 0 & 0 & 4\end{vmatrix} & \begin{pmatrix}R_3\to R_3-15R_2 \\ R_4\to R_4+6R_2\end{pmatrix} \\ &\stackrel{(*)}= -\frac 12(5)(-1)(2)(4) \\ &= \enspace\ 20\end{align}$$

where $(*)$ is due to the fact that the determinant of a triangular matrix is the product of the diagonal elements.

If you look at where the $0$'s are in your matrix, it is fairly easy to see that your matrix becomes block-triangular when transformed to (for instance) the ordered basis $[e_1,e_3,e_2,e_4]$, in other words the standard basis but with the second and third vectors interchanged. The change of basis swaps the second and third rows and the second and third columns, and gives you $$A'= \pmatrix{5&2&-7&2\\-5&0&-8&3\\0&0&3&-4\\0&0&5&-6}. $$ Since change of basis does not affect the determinant, and the determinant of a block-triangular matrix is the product of the determinants of the diagonal blocks, you get $$ \det(A)=\det(A') =\left|\matrix{5&2\\-5&0}\right|\times\left|\matrix{3&-4\\5&-6}\right| =10\times2=20. $$