What's the idiom for creating an ever more complex solution to a problem?

It means to take a (flawed) existing solution/explanation to a problem and modify it in ever more convoluted ways to make it work when the current version fails.

It comes from pre-Newtonian times when there were people working on a theory of orbital mechanics, thinking that orbits were circular rather than elliptical, and every time someone made an observation of some moon that the theory didn't handle, they bolted on a bit more math to try and account for that observation.

It was "working on circles" or "calculating orbits" or something like that.

It's known as adding epicycles:

Adding epicycles is an expression denoting ad hoc attempts to make a theory conform to observations by making said theory needlessly or absurdly complicated.

The expression comes from a technique used by Astronomers from antiquity to the early modern period (generally working with the Ptolemaic system) to explain observations that planets would often reverse direction in the sky. These observations made sense in a Sun-centered view of the solar system, but not an Earth-centered one (Heliocentric systems had greater observational problems of their own, which prevented their general acceptance in antiquity).