How can I physically destroy data from a failed HDD?

You're correct; physical destruction is the only good way to do this (you'd need a magnet so strong that it's not feasible to get one for most people unless you're on staff at the Large Hadron Collider). Professional disposal operations generally do this with an industrial metal shredder. For you, bending the platters with a hammer, sandpapering them, and then running a drill through them in multiple points is sufficient to stop anything but advanced forensic data recovery. If you're really concerned about even that, or you just want style points, you might try thermite. It is usually sufficient to melt the platters entirely.

Save time, I just use a 5lb hammer.

Note: wear safety goggles.

Or buy one of these

On newer drives, even the high-power magnets I used to use don't work anymore to erase the disk.

If you have a spike, or similar object (hard metal that is pointed, think of a nail that is much bigger) and drive it through the center of the chassis with a hammer. That will bend the platers and damage the heads making it extremely difficult to recover.

A land stake from the hardware store should work for this.

Dispose of the board separate from the drive. This way any encoding information about the drive recorded on the controller will not be available making recovery exponentially more difficult.

After all that, you can safely send the remains to a computer recycler for shredding.

Apparently the method a certain british university uses is drilling a hole through it. Failing which, thermite.