Have I found a formula for the area of a triangle?

Since $\frac12ab\sin C=A$, you accidentally multiplied $a^2b^2\cos^2C$ by $A^2$ instead of $1$ in one of your $=$ signs. We can correct this as$$\frac{(a^2+b^2-c^2)^2}{4}=a^2b^2\cos^2C\frac{2A}{ab\sin C}=\frac{2Aab\cos^2C}{\sin C}\implies A=\frac{(a^2+b^2-c^2)^2\sin C}{8ab\cos^2C}$$or infinitely many alternatives, including$$\frac{(a^2+b^2-c^2)^2}{4}=a^2b^2\cos^2C\frac{ab\sin C}{2A}\implies A=\frac{2a^3b^3\cos^2C\sin C}{(a^2+b^2-c^2)^2}.$$The first of these is @SammyBlack's result. They are both dimensionally consistent in that they have length dimension $2$, as an area should, be it $2\times2-2=2$ or $3\times2-2\times2=2$ (because e.g. $a^n$ has length dimension $n$). Your formula, by contrast, has length dimension $2\times2-3\times2=-2$.

Great idea! But you need to be careful where you introduce the area $A$ using the SAS formula $$ A = \frac{1}{2} ab \sin C $$

Here's how we can repair the calculation. We have $$ \bigl( c^2 - (a^2 + b^2) \bigr)^2 = 4 a^2 b^2 \cos^2 C. $$ To avoid extra parentheses, noting that $(-u)^2 = u^2$ for all $u$, we can write: $$ ( a^2 + b^2 - c^2 )^2 = 4 a^2 b^2 \cos^2 C. $$ Now, multiply by $$ \frac{\sin C}{8ab \cos^2 C} $$ to obtain $$ A = \frac{1}{2} ab \sin C = \frac{( a^2 + b^2 - c^2 )^2 \sin C}{8ab \cos^2 C}. $$ Also, since $$ \frac{\sin C}{\cos^2 C} = \frac{1}{\cos C} \frac{\sin C}{\cos C} = \sec C \tan C, $$ we can rewrite the formula as $$ A = \frac{( a^2 + b^2 - c^2 )^2 \sec C \tan C}{8ab}. $$

Note: you have to exclude the case of right triangles so as to avoid division by $0$. This formula applies to all other triangles though!