Word describing that things become more difficult to understand rather than easier

Try convoluted.

Convoluted adjective 1 (Especially of an argument, story, or sentence) extremely complex and difficult to follow - ODO

Putting this into your sample sentence, we get:

His explanation for that math problem was very convoluted. I'm even more puzzled now!

Is this answer fitting?

Obfuscate

To make so confused or opaque as to be difficult to perceive or understand:

"A great effort was made ... to obscure or obfuscate the truth" (Robert Conquest).

As in:

His explanation for that math problem was very befogging. I'm even more puzzled now!

befog TFD

tr.v. befogged, befogging, befogs 1. To cover or obscure with or as if with fog. 2. To cause confusion in; muddle.