Can someone point me to resources to reverse engineer ancient Mac software?

"ResEdit" was the basic utility to inspect/edit resources such as images, video, and sound in Mac OS applications and games. It runs in Mac OS or Classic environments and can be downloaded from http://www.mac.org/utilities/resedit/

"ResFork" runs on OS X 10.2 and provides similar functionality to ResEdit.

Using ResEdit or ResFork you should be able to find the codec identifier. You can then usually work from the codec identifier to a video converter. Perhaps focusing on converters from the same timeframe as the application you are working with.

If you don't have a machine that can run a Mac Classic environment, then the emulator "SheepShaver" will be useful as a debug/reverse engineer environment. SheepShaver can be downloaded from: http://sheepshaver.cebix.net/