Can non-linear transformations be represented as Transformation Matrices?
Solution 1:
As Harry says, you can't (the example of affine transformations can be tweaked to work because they're just linear ones with the origin translated). However, approximating a nonlinear function by a linear one is something we do all the time in calculus through the derivative, and is what we often have to do to make a mathematical model of some real-world phenomenon tractable.
Solution 2:
As others have already mentioned, the Jacobian determinant transforms one coordinate system to another by relating infinitesimal areas (or volumes) from one system to another. Consider going from Cartesian to Polar coordinates:
\begin{align} J &= \det\frac{\partial(x,y)}{\partial(r,\theta)} =\begin{vmatrix} \frac{\partial x}{\partial r} & \frac{\partial x}{\partial \theta} \\\\ \frac{\partial y}{\partial r} & \frac{\partial y}{\partial \theta} \\\\ \end{vmatrix} \\&=\begin{vmatrix} \cos\theta & -r\sin\theta \\\\ \sin\theta & r\cos\theta \\\\ \end{vmatrix} =r\cos^2\theta + r\sin^2\theta = r \end{align}
This is useful because:
$$\mathrm{d}A = J\;\mathrm{d}r\,\mathrm{d}\theta = r\,\mathrm{d}r\,\mathrm{d}\theta$$
$$\iint_\mathbf{R} f(r,\theta)\,\mathrm{d}A = \int_a^b \int_0^{r(\theta)} f(r,\theta) r\,\mathrm{d}r\,\mathrm{d}\theta$$
Which tells you that if you have a function $f(r, \theta)$ you can compute the integral as long as you remember to add a factor of $r$. The common transformations have all been worked out and can be found here on Wikipedia.
Solution 3:
You can't represent a non linear transformation with a matrix, however there are some tricks (for want of a better word) available if you use homogenous co-ordinates. For example, $3\text{D}$ translation is a non-linear transformation in a $3\times3$ $3\text{D}$ transformation matrix, but is a linear transformation in $3\text{D}$ homogenous co-ordinates using a $4\times4$ transformation matrix. The same is true of other things like perspective projections. This is why $4\times4$ matrices are used in $3\text{D}$ graphics as the homogenous co-ordinate system simplifies things a lot.
To clarify - using homogenous co-ordinates increases the range of transformations representable using matrices from plain linear transformations to affine transformations and some projections, but it doesn't make all non-linear transformations representable using matrices. The non-linear transformation provided as an example is still beyond representation as an affine transformation (Thanks to @Harry for prompting this clarification in the comments)
Solution 4:
You can represent some non-linear transforms (like translation) of an $n$-dimensional vector with an $(n+1)$-dimensional matrix. However, converting the vector to its $(n+1)$-dimensional homogeneous version and back is not a linear transformation and also not representable as a matrix.
More is explained here.