A third order nonlinear ordinary differential equation.

As Adrian suggests, let $y'=z$, to get the second order equation $$ z''+a\,(z')^2+b\,z^3=0. $$ Since the independent variable $t$ does not appear explicitly in the equation (I am assuming $a$ and $b$ are constants), we let $$ z'=p,\quad z''=\frac{dp}{dt}=\frac{dp}{dz}\,\frac{dz}{dt}=p\,\frac{dp}{dz}. $$ This gives the first order equation $$ p\,\frac{dp}{dz}+a\,p^2+b\,z^3=0, $$ which written as $$ \frac{dp}{dz}=-a\,p-b\,z^3\,p^{-1} $$ is a Bernoulli equation. To solve it, let $u=p^2$. This gives the linear equation $$ \frac{du}{dz}=-a\,u-b\,z^3. $$ I have not done the calculations, but my impression is that you will not be able to get an explicit solution in terms of elementary functions.

$$y'''(t)+a(y''(t))^2+b(y'(t))^3=0$$ Substitute $z=y'$ $$z''(t)+a(z'(t))^2+bz^3=0$$ Substitute $p=z'$ $$\frac {dp}{dz}p+ap^2+bz^3=0$$ $$\frac 12(p^2)'+ap^2+bz^3=0$$ Finally substitute $w=p^2$ $$\frac 12w'+aw+bz^3=0$$ Bernouilli's equation

As a more general solution, if you have an equation of the form $$ x''(t) + a(x(t))x'(t)^2+b(x(t)) = 0 $$ then you can make the substitution $f(x) = x'(t)^2$ to arrive at the equation $$ \frac{1}{2}f'(x) + a(x)f(x)+b(x)= 0 $$ Letting $\mu(x) = \exp\left[\int a(x) dx\right]$, we can solve for $f(x)$: $$ f(x) = \mu(x)^{-1}\left(C_1-2\int\mu(x)b(x)dx\right) $$ which can be substituted back for $x(t)$: $$ x' = \mu(x)^{-1/2}\left(C_1-2\int\mu(x)b(x)dx\right)^{1/2} $$ and solved implicitly: $$ C_2 + t - \int \left[ \mu(x)\left(C_1-2\int\mu(x)b(x)dx\right)^{-1} \right]^{1/2} dx = 0 $$