Is it allowed to use augmented matrix technique in solving system of non-linear equations
You can apply the technique, but you must be careful when interpreting its results.
In particular, what you can do is this. You can introduce $4$ new variables like so: $a=x, b=x^2,c=xy, d=y$. Now, any solution to the original set of equations will also be the solution to the set of linear equations
$$a+2b+3c+0d = 6\\ 2a+b+3c+d=5\\ a-b+0c+d=7$$
and because the linear system above has no solutions (this part can be proven using the augmented matrix technique), you can conclude that the original system has no solutions.
However, this line of reasoning is a one way street. In general, the linear system might have solutions even if the original system does not have them. A trivial example would be the equation $$x^2=-1$$ which has no solutions in $\mathbb R$, but if you introduce a new variable $a=x^2$, the linear equation $a=-1$ clearly has a solution.
A less trivial example would be the system of equations as follows:
$$\begin{align} 1\cdot x^2+2\cdot xy+1\cdot y^2 + 0\cdot x + 0\cdot y = 1\\ 0\cdot x^2+0\cdot xy + 0\cdot y^2 + 1\cdot x+1\cdot y=2 \end{align}$$
- If you transform the system to $$1\cdot a + 2\cdot b + 1\cdot c + 0\cdot d + 0\cdot e = 1\\ 0\cdot a + 0\cdot b + 0\cdot c + 1\cdot d + 1\cdot e = 2$$ then this system has infinitely many solutions.
- The original system has no solutions, because the first equation translates to $(x+y)^2=1$, which is incompatible with the second equation which is $(x+y)=2$.
So, TL;DR: you can use linear equations to prove there is no solution, but the existence of a linear solution does not mean that a solution exists.