How to stop spinning rocket in space
It's hard to say what the actual cause is, but I suspect you've somehow managed to get your vessel into a glitched state where the parts are exerting an unbalanced (and thus unphysical) acceleration and/or torque on each other.
(You might be able to tell whether it's a pure torque or an off-center acceleration by seeing whether your orbit changes as the vessel starts to spin. Of course, once it's spinning fast enough, any further net acceleration will tend to cancel out over the course of the rotation.)
Some potential triggers for such bugs that I've seen include:
-
Overuse of autostruts (and possibly other factors, especially with large and complex vessels) can sometimes create glitched internal stresses that may cause your vessel to spin and/or shake itself apart. This is often referred to as a "kraken attack" by KSP players, although that term is also used for other game glitches.
(Note that, even if you don't deliberately use autostruts on your vessel, some parts like wheels and landing gears are always autostrutted to the heaviest part on your vessel and can thus trigger autostrut-related glitches.)
-
Robotic or otherwise movable parts (like wheels, landing gears, cargo bays, etc.) with same-vessel interactions enabled and pushing against other parts on the same vessel can sometimes exert forces without a matching counterforce, creating a non-zero net force on the vessel. This can and has been exploited to create reactionless "kraken drive" mechanisms that can allow a vessel to reach unlimited velocities without using any fuel, or do other silly things like drive up vertical walls.
-
As a specific case of the above, pairs of nearby docking ports with unequal docking force can create an unbalanced force, apparently even without same-vessel interactions. When used as a deliberate exploit to create reactionless thrust, this is often called a "Zompi drive" or a "Z-drive".
In particular, your observation that quicksaving and loading stopped the torque, but caused some of your parts to move, is consistent with (but not necessarily proof of) a kraken attack caused by unbalanced internal stresses. Part connections permanently stretching under sufficient stress is, as far as I know, a deliberate feature of the physics engine, and can sometimes relieve the stress by allowing the vessel to inelastically deform. But if the stress is caused by the physics engine glitching out, it's unlikely to be sufficient. However, when you reload your vessel, the game will recalculate things like autostruts, and if the vessel was sufficiently deformed, its new configuration may no longer trigger the glitch.