What exactly happens when a heavy soldier misses with a rocket shot?

When the shots fail, the rocket will launch in a random trajectory and detonate if it hits something. Typically the direction the rocket heads will still be in the general vincinity of the original intended target. This CAN cause friendly fire.

Once you obtain the blaster launcher, your rockets will no longer be able to miss (on top of causing more damage) as they become guided rockets instead.

As for enemy rockets, the only ones that use a similar system are the EXALT agents. I suppose theoretically they can miss too but typically I don't give them a chance to use their rocket on me so I have limited experience and I've never seen one fail yet.

When the shots fail, they will fire at the same level and elevation that you aimed it at, but be off by a fixed angular offset (from RNG seed). This will cause it to explode w/ regular collision rules.

It is thus not only possible to cause friendly fire, but for the heavy to blow him/herself up.

To test this, if you saved right before firing the miss, you can in fact get the rocket to hit completely centered if you compensate for the offset when trying again. Note: this is assuming the Save Scum Second Wave option from Enemy Within is not enabled.

Unsure about exact mechanics of enemy rockets as they don't typically live long enough to try.