Integral of Derivative squared
While there is no explicit formula of the exact kind you desire, if one is willing to reparametrize, there is an integral-free parametrization of the curves of the form $\bigl(x,f(x),g(x)\bigr)$ for which $g'(x) = (f'(x))^2$: As long as $f''(x)$ is non-vanishing, one can reparametrize such a curve in the form $$ \bigl(x,f(x),g(x)\bigr) = \bigl(u''(t),\ t\,u''(t){-}u'(t),\ t^2u''(t){-}2tu'(t){+}2u(t)\,\bigr) $$ where $u$ is a function of $t = f'(x)$. Conversely, if $u$ is an arbitrary function of $t$ for which $u'''(t)$ is nonvanishing, the curve on the right hand side of the equation is always of the form $\bigl(x,f(x),g(x)\bigr)$ for some functions $f$ and $g$ satisfying $g'(x) = (f'(x))^2$.
Perhaps this is the sort of thing that one can use when one is trying to get an integral-free description of the relationship between the two functions $f$ and $g$ (and $x$, of course).
Let $f(x)=e^{x^2}$, so $(f'(x))^2=4x^2e^{2x^2}$. But $\int4x^2e^{2x^2}\,dx$ can't be evaluated in terms of elementary functions (exponentials, trig functions, polynomials, $n$th roots, etc.).