Evaluation of $\int\frac{5x^3+3x-1}{(x^3+3x+1)^3}\,dx$
Evaluate the integral $$\int\frac{5x^3+3x-1}{(x^3+3x+1)^3}\,dx$$
My Attempt:
Let $f(x) = \frac{ax+b}{(x^3+3x+1)^2}.$ Now differentiate both side with respect to $x$, and we get
$$ \begin{align} f'(x) &= \frac{(x^3+3x+1)^2\cdot a-2\cdot (x^3+3x+1)\cdot (3x^2+3)\cdot (ax+b)}{(x^3+3x+1)^4}\\ &= \frac{5x^3+3x-1}{(x^3+3x+1)^3}\cdot\frac{(x^3+3x+1)\cdot a-6\cdot (x^2+1)\cdot (ax+b)}{(x^3+3x+1)^3}\\ &= \frac{5x^3+3x-1}{(x^3+3x+1)^3} \frac{-5ax^3+6bx^2-3ax+(a-6b)}{(x^3+3x+1)^3}\\ &= \frac{5x^3+3x-1}{(x^3+3x+1)^3} \end{align} $$
for $a = -1$ and $b = 0$. Thus, by the Fundamental Theorem of Calculus,
$$ \begin{align} \int\frac{5x^3+3x-1}{(x^3+3x+1)^3}\,dx &= \int f'(x)\,dx\\ &= f(x)\\ &= -\frac{x}{(x^3+3x+1)^2}+\mathcal{C} \end{align} $$
How we can solve the above integral directly (maybe by using the substitution method)?
Ostrogradsky-Hermite method.
To integrate a rational function $P(x)/Q(x)$ without decomposing it into partial fractions and without finding the roots of the denominator, we can use the Ostrogradski-Hermite method, which generalizes your ansatz
$$ \int\frac{5x^3+3x-1}{(x^3+3x+1)^3}dx= \frac{ax+b}{(x^3+3x+1)^2}+C.$$
You can find a description of this method in section 2.1 of Gradshteyn and Ryzhik's Table of Integrals, Series, and Products, where the identity $(2)$ below is given. The formula $(1)$ appears also on the Ostrogradsky's Wikipedia page.
Assume that $\deg P(x)<\deg $ $Q(x)$. There exist polynomials $P_{1}(x)$, $P_{2}(x)$, $Q_{1}(x)$ and $Q_{2}(x)$, with $Q_{1}(x)=\gcd \left\{ Q(x), Q^{\prime }(x)\right\}$ and $Q_{2}(x)=Q(x)/Q_{1}(x)$, $\deg P_{1}(x)<\deg Q_{1}(x)$, $\deg P_{2}(x)<\deg Q_{2}(x)$, such that
\begin{equation} \int \frac{P(x)}{Q(x)}dx=\frac{P_{1}(x)}{Q_{1}(x)}+\int \frac{P_{2}(x)}{ Q_{2}(x)}dx.\tag{1} \end{equation}
Then
\begin{eqnarray*} P(x) &=&\frac{P_{1}^{\prime }(x)Q_{1}(x)-P_{1}(x)Q_{1}^{\prime }(x)}{\left\{ Q_{1}(x)\right\} ^{2}}Q(x)+\frac{P_{2}(x)}{Q_{2}(x)}Q(x) \\ &=&P_{1}^{\prime }(x)\frac{Q(x)}{Q_{1}(x)}-P_{1}(x)\frac{Q_{1}^{\prime }(x)}{Q_{1}(x)}\frac{Q(x)}{Q_{1}(x)}+P_{2}(x)\frac{Q(x)}{Q_{2}(x)} \\ &=&P_{1}^{\prime }(x)Q_{2}(x)-P_{1}(x)\left\{ \frac{Q_{1}^{\prime }(x)}{Q_{1}(x)}Q_{2}(x)\right\} +P_{2}(x)Q_{1}(x) \end{eqnarray*}
or
\begin{equation} P(x)=P_{1}^{\prime }(x)Q_{2}(x)-P_{1}(x)\left\{ T(x)-Q_{2}^{\prime }(x)\right\}+P_{2}(x)Q_{1}(x),\tag{2} \end{equation}
with $T(x)=Q^{\prime }(x)/Q_{1}(x)$, because from
\begin{equation*} Q^{\prime }(x)=\left\{ Q_{1}(x)Q_{2}(x)\right\} ^{\prime }=Q_{1}^{\prime }(x)Q_{2}(x)+Q_{1}(x)Q_{2}^{\prime }(x)=T(x)Q_{1}(x) \end{equation*}
we obtain
\begin{equation*} \frac{Q_{1}^{\prime }(x)}{Q_{1}(x)}Q_{2}(x)+Q_{2}^{\prime }(x)=T(x). \end{equation*}
To find the coefficients of the polynomials $P_{1}(x)$ and $P_{2}(x)$ equate the coefficients of like powers of $x$.
Application to
\begin{equation*} \frac{P(x)}{Q(x)}=\frac{5x^{3}+3x-1}{\left( x^{3}+3x+1\right) ^{3}}. \end{equation*}
Since
\begin{eqnarray*} Q(x) &=&\left( x^{3}+3x+1\right) ^{3} \\ Q^{\prime }(x) &=&9\left( x^{3}+3x+1\right) ^{2}\left( x^{2}+1\right) \\ Q_{1}(x) &=&\gcd \left\{ Q(x),Q^{\prime }(x)\right\} =\left( x^{3}+3x+1\right) ^{2} \end{eqnarray*}
and \begin{equation*} Q_{2}(x)=\frac{Q(x)}{Q_{1}(x)}=\frac{\left( x^{3}+3x+1\right) ^{3}}{\left( x^{3}+3x+1\right) ^{2}}=x^{3}+3x+1, \end{equation*}
we write
\begin{equation} \int \frac{5x^{3}+3x-1}{\left( x^{3}+3x+1\right) ^{3}}dx=\frac{P_{1}(x)}{ \left( x^{3}+3x+1\right) ^{2}}+\int \frac{P_{2}(x)}{x^{3}+3x+1}dx,\tag{3} \end{equation}
where
\begin{eqnarray*} P_{1}(x) &=&Ax^{5}+Bx^{4}+Cx^{3}+Dx^{2}+Ex+F \\ P_{2}(x) &=&Fx^{2}+Gx+H. \end{eqnarray*}
The identity $(2)$, with
\begin{equation*} T(x)=\frac{Q^{\prime }(x)}{Q_{1}(x)}=9\left( x^{2}+1\right), \end{equation*}
yields
\begin{eqnarray*} 5x^{3}+3x-1 &=&\left( 5Ax^{4}+4Bx^{3}+3Cx^{2}+2Dx+E\right) \left( x^{3}+3x+1\right) \\ &&-\left( Ax^{5}+Bx^{4}+Cx^{3}+Dx^{2}+Ex+F\right) \left\{ 9\left( x^{2}+1\right) -\left( 3x^{2}+3\right) \right\} \\ &&+\left( Gx^{2}+Hx+I\right) \left( x^{3}+3x+1\right) ^{2} \\ &=&Gx^{8}+\left( -A+H\right) x^{7}+\left( -2B+6G+I\right) x^{6} \\ &&+\left( 6H-3C+2G+9A\right) x^{5} \\ &&+\left( -4D+6B+9G+6I+2H+5A\right) x^{4} \\ &&+\left( 3C-5E+4B+6G+9H+2I\right) x^{3} \\ &&+\left( 3C-6F+G+6H+9I\right) x^{2} \\ &&+\left( 6I+H+2D-3E\right) x+\left( E+I-6F\right). \end{eqnarray*}
By equating coefficients we find
\begin{equation*} A=B=C=D=F=G=H=I=0,E=-1.\tag{4} \end{equation*}
Consequently,
\begin{eqnarray*} P_1(x) &=&-x \\ P_2(x) &=&0 \end{eqnarray*} and finally,
\begin{equation*} \int \frac{5x^{3}+3x-1}{\left( x^{3}+3x+1\right) ^{3}}dx=-\frac{x}{\left( x^{3}+3x+1\right) ^{2}}+C,\tag{5} \end{equation*}
as evaluated by you.
$$ \begin{aligned}\int\frac{5x^3 + 3x - 1}{\left(x^3 + 3x + 1\right)^3}\,\mathrm{d}x &=\int\frac{5x^3 + 3x-1}{\left(\sqrt{x}\left(x^{5/2}+3\sqrt{x}+x^{-1/2}\right)\right)^3}\,\mathrm{d}x\\&=\int\frac{5x^3 + 3x-1}{x^{3/2}\left(x^{5/2}+3\sqrt{x}+x^{-1/2}\right)^3}\,\mathrm{d}x\\&=\int\frac{5x^{3/2}+3x^{-1/2}-x^{-3/2}}{\left(x^{5/2}+3\sqrt{x}+x^{-1/2}\right)^3}\,\mathrm{d}x\\&=\int\frac{2\,\mathrm{d}(x^{5/2}+3\sqrt{x}+x^{-1/2})}{\left(x^{5/2}+3\sqrt{x}+x^{-1/2}\right)^3}\\&=2\int\frac{\mathrm{d}\tau}{\tau^3}\\&=-\frac{1}{\tau^2}+C\\&=-\frac{1}{\left(x^{5/2}+3\sqrt{x}+x^{-1/2}\right)^2}+C\\&=\cdots\end{aligned} $$