Why is H.264 video so much faster in an FLV container than in an MP4 container?

Solution 1:

TL;DR: MP4 is used when the video site stores more metadata in the video than FLV supports, or uses an audio codec that FLV doesn't support. FLV's simplicity and for-streaming design makes it a good choice if you don't have a good reason to use MP4.

As for flash's timeline scrubbing, I have no idea why it does that since I never coded flash, but it's possible that it's a knob it uses, or something that works specifically with adobe's streaming server in order to seek in the file. It also works as a way to stop the pesky user from keeping the file on his disk.

Some stuff you already knew:

There are fundamental differences between the FLV and MP4 (aka isomedia) containers. FLV was devised by Adobe from the start as a streaming container, and it's really damn simple. All it does is send a video packet, then audio packet, then video packet... However, it only supports very few codecs, and no metadata other than timestamps in miliseconds. Unless you need MP4-specific features, you would do just fine with FLV.

ISO media, on the other hand, is based on Apple's MOV container. It's separated in atoms, and there is a particular atom, moov, that needs to be decoded before any other atoms can be read. The issue you're having with MP4 is because the moov atom is written at the end of file, which is a lot easier to do for encoding programs. There are tools, such as qtfaststart, which will do the necessary munging to put the moov atom at the beginning of the file. Thus, the file will start playback as soon as it has data, instead of needing to be fully downloaded before starting.