Fourier transform of function composition

Given two functions $f$ and $g$, is there a formula for the Fourier transform of $f \circ g$ in terms of the Fourier transforms of $f$ and $g$ individually?

I know you can do this for the sum, the product and the convolution of two functions. But I haven't seen a formula for the composition of two functions.

There is no such rule in general. The key here is variable substitution: If $g$ is a bijection and smooth enough then, if all integrals exist: $$ (\widehat{f\circ g})(\xi) = \int f(g(x))\exp(ix\xi)dx = \int f(y)\exp(ig^{-1}(y)\xi)|\det g'(y)|^{-1}dy.$$ This does only rarely lead to something interesting, e.g. in the case of scaling (i.e. linear transformation of the variable): Working in $\mathbb{R}^d$ with $A\in\mathbb{R}^{d\times d}$ invertible: $$ (\widehat{f\circ A})(\xi) = |\det A^{-1}|\widehat{f}(A^{-T}\xi). $$

Have a look at Bergner et al. 2006 "A Spectral Analysis of Function Composition and Its Implications for Sampling in Direct Volume Visualization" IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 12, NO. 5. Sec. 3.

Unfortunately it's not quite as simple as the transform of a convolution or even the derivative. The short answer is, in one dimension let \begin{equation} P(k,l) = \int_{x \in \mathcal{R}} e^{i2\pi(lg(x) - kx)}\,dx. \end{equation} The FT of the composite function is \begin{equation} \text{FT}\left[f\left(g(x)\right)\right](k) = \int_{l\in\mathcal{R}} \hat{f}(l)P(k,l) \,dl, \end{equation} where $\hat{f}(l)$ is the Fourier transform of $f(x)$. As you can see the transformation involves the inner product of $\hat{f}(l)$ with a slightly awkward two dimensional function. In the discrete case this would be implemented as a matrix multiplication.