Solving $D=\begin{vmatrix}a&\omega b&\omega^2c\\\omega^2b&c&\omega a\\\omega c&\omega^2a&b\end{vmatrix}$

We know that multiplying a row of a determinant by a constant changes the determinant by that constant. Therefore, $$ \begin{vmatrix} a&\omega b&\omega^2c\\\omega^2b & c & \omega a\\\omega c & \omega^2 a & b \end{vmatrix} = \frac{1}{\omega^3}\begin{vmatrix} a&\omega b&\omega^2c\\b & \omega c & \omega^2 a\\ c & \omega a & \omega^2b \end{vmatrix} $$ where we multiplied the second row by $\omega$ and third row by $\omega^2$. But actually $\omega^3=1$ so the constant factor in front is just $1$. Now, multiplying a column by a constant also has the same effect, so we multiply the second column by $\omega^2$ and third column by $\omega$ to conclude that, in fact,

$$D = \begin{vmatrix} a & b & c\\ b & c & a\\ c & a & b \end{vmatrix}.$$

So really $D=D'$. It is also easier to now compute the determinant by directly expanding it to get $D = 3abc-a^3-b^3-c^3 = -(a+b+c)(a^2+b^2+c^2-ab-bc-ca)$.

Note that the final form of the determinant is a circulant matrix, for which there is a standard formula to compute the determinant. So, this problem can be generalised to $n\times n$ matrices and you can still write out an algebraic expression for the determinant explicitly.

Also, it is generally a horrible idea to write $\omega$ out in its real and imaginary part (i.e. write $\omega = (-1+i\sqrt{3})/2$) and try to simplify from there. What you need for things to simplify is just the property that $\omega^3=1$, and you are just making your life harder by writing out the real and imaginary parts. It really won't help at all.

Using the Rule of Sarrus we obtain, with $\omega^3=1$, $$ \det (D)=\det(D')= - a^3 + 3abc - b^3 - c^3. $$

Throughout this solution, I shall be making use of the fact that $\omega^3=1$ as $\omega$ is the complex cube root of unity. Now, consider the determinant $$D_1=\begin{vmatrix} 1 & 0 & 0\\ 0 & \omega & 0\\ 0 & 0 & \omega^2 \end{vmatrix} = 1\cdot \omega \cdot \omega^2=1$$ Multiplying the matrix of $D'$ with the matrix of determinant $D_1$ we get, $$\begin{bmatrix} a & b\omega & c\omega^2\\ b & c\omega & a\omega^2\\ c & a\omega & b\omega^2 \end{bmatrix}$$ Taking the determinant of that matrix and using the property that $det(A\cdot B)=det(A)\cdot det(B)$ we obtain $$D'=\begin{vmatrix} a & b\omega & c\omega^2\\ b & c\omega & a\omega^2\\ c & a\omega & b\omega^2 \end{vmatrix}$$ I shall be multiplying with $\omega^2$ on both sides to get - $$\omega^2\cdot D'=\omega^2\cdot\begin{vmatrix} a & b\omega & c\omega^2\\ b & c\omega & a\omega^2\\ c & a\omega & b\omega^2 \end{vmatrix}$$ Now from the property of the determinant that any scalar factor may be multiplied across any one row or column of the determinant, I am multiplying the outside factor with the second row ($R_2$) on the RHS - $$\omega^2\cdot D'=\begin{vmatrix} a & b\omega & c\omega^2\\ b\omega^2 & c & a\omega\\ c & a\omega & b\omega^2 \end{vmatrix}$$ Here the property of $\omega^3=1$ was used twice. Now multiplying with $\omega$ on both sides, we get - $$1\cdot D'=\omega\cdot\begin{vmatrix} a & b\omega & c\omega^2\\ b\omega^2 & c & a\omega\\ c & a\omega & b\omega^2 \end{vmatrix}$$ Again I shall be multiplying the scalar factor outside with the third row of the determinant ($R_3$) - $$ D'=\begin{vmatrix} a & b\omega & c\omega^2\\ b\omega^2 & c & a\omega\\ c\omega & a\omega^2 & b \end{vmatrix}$$ Again the property $\omega^3=1$ was used. If you look carefully, the determinant on the RHS is $D$. Therefore, we have - $$\boxed{D=D'}$$ Therefore, the correct answer to the question must be (3).