Integration by parts: $\int e^{ax}\cos(bx)\,dx$

So you have $$\int e^{ax}\cos(bx)dx = \frac{1}{a}e^{ax}\cos(bx) + \frac{b}{a}\left(\frac{1}{a}e^{ax}\sin(bx) - \frac{b}{a}\int e^{ax}\cos(bx)\,dx\right).$$ Multiplying out you get $$\int e^{ax}\cos(bx)\,dx = \frac{1}{a}e^{ax}\cos(bx) + \frac{b}{a^2}e^{ax}\sin(bx) - \frac{b^2}{a^2}\int e^{ax}\cos(bx)\,dx.$$ At this point, you should move that last integral on the right hand side to the left hand side and add in the constant of integration on the right.

Moving the last integral to the left hand side, you get $$\left(1 + \frac{b^2}{a^2}\right)\int e^{ax}\cos(bx)\,dx = \frac{1}{a}e^{ax}\cos(bx) + \frac{b}{a^2}e^{ax}\sin(bx) + C,$$ and I think this is where you made your mistake.

You tried to clear that $1 + \frac{b^2}{a^2}$ by multiplying through by $\frac{a^2}{1+b^2}$. But this is incorrect: $$1 + \frac{b^2}{a^2} = \frac{a^2+b^2}{a^2} \neq \frac{1+b^2}{a^2}$$ so that what you multiplied through did not clear that factor. You need to multiply by $\frac{a^2}{a^2+b^2}$ (or, more horribly, by $$\frac{1}{1+\frac{b^2}{a^2}}$$ which is too horrible for words) for things to cancel out.

If you do that, from $$\frac{a^2+b^2}{a^2}\int e^{ax}\cos(bx)\,dx = \frac{1}{a}e^{ax}\cos(bx) + \frac{b}{a^2}e^{ax}\sin(bx) + C,$$ multiplying both sides by $\frac{a^2}{a^2+b^2}$, we get: $$\int e^{ax}\cos(bx)\,dx = \frac{a}{a^2+b^2}e^{ax}\cos(bx) + \frac{b}{a^2+b^2}e^{ax}\sin(bx) + C$$ the intended answer.

By the way: you don't need to have the "$+C$" on the right hand side until there are no more indefinite integrals there; the constant of integration is implicit in the indefinite integral, so for example, in your penultimate displayed equation, the "$+C$" is superfluous.


Another solution to your original question can be using complex numbers. $$\begin{align} I&=\displaystyle\int e^{ax}\cos {bx}dx \\ &=\Re\left(\displaystyle\int e^{ax}(\cos {bx}+i\sin {bx})\right)dx\\ &=\Re\left(\displaystyle\int e^{(a+ib)x}dx\right)\\ &=\Re\left(\dfrac{e^{(a+ib)x}}{a+ib}\right)\\ &=\Re\left(\dfrac{e^{ax}(\cos {bx}+i\sin {bx})}{a+ib}\right)\\ &=\Re\left(\dfrac{e^{ax}}{a^2+b^2}(\cos {bx}++i\sin {bx})(a-ib)\right)+C\\ \therefore I&=\dfrac{e^{ax}}{a^2+b^2}(a\cos {bx}+b\sin {bx})+C \end{align}$$