How to solve non-linear recurrence relation in general?
For linear recurrence, we can use generating function. So is there a general technique to solve non-linear recurrence or it depends on a specific sequence? For example, $$a_{n+1} = \dfrac{a_n(a_n - 3)}{4}$$ for $a_0 = a$
As already mentioned, currently for non-linear recurrence relations, there are no known general techniques for obtaining a closed form solution.
For the very few non-linear recurrences that are actually solvable, the techniques used to solve them are strongly dependent on the specific recurrence that your dealing with.