Why does overflow hidden stop floating elements escaping their container?

It creates a block formatting context.

Block formatting contexts are important for the positioning (see float) and clearing (see clear) of floats. The rules for positioning and clearing of floats apply only to things within the same block formatting context. Floats do not affect the layout of things in other block formatting contexts, and clear only clears past floats in the same block formatting context.

see also: http://www.w3.org/TR/CSS2/visuren.html#block-formatting

It's correct that the overflow style is intended to control what happens to overflowing content.

The effect on the floating elements is a side effect of the overflow style creating a block formatting context for the element.

When you don't specify a size for the containing element, the block formatting context gets its size from the elements that it contains, so that is the size that the containing element gets.