Why should we get rid of indefinite integration?

Indefinite integration is most often used to denote anti-differentiation, which leads students to believe that integration and anti-differentiation are the same thing, which is absolutely not true. There are plenty of functions which have anti-derivatives but are not integrable, and functions which are integrable but don't have anti-derivatives. The use of anti-derivatives isn't the problem, it's the term "indefinite integral" and the use of an integral symbol for them that's the problem.

Edit: I assume we're talking about Riemann integration here. Take any function $f$ which is differentiable on some interval such that $f'$ is unbounded. Then $f'$ has an antiderivative, namely $f$, but is not integrable on that interval since integrable functions must be bounded. The issue is that students come to believe that the Fundamental Theorem of Calculus tells us how to compute all integrals, when in reality this only applies to certain integrals.

Primitives are useful (Barrow's rule!). I agree that the name "indefinite integral" and the symbol used can be confusing for some (clumsy) students.