Why do older games play on my new machine if I buy them from Steam but not if I install from the original disc?

Most of the games available for Steam have been modified to install or work on the newer OS. When they haven't, such as with Railroad Tycoon 3 notes it is not compatible with Windows Vista nor Windows 7. If you view Railroad Tycoon 2 Platinum, it doesn't have a disclaimer because it works with Windows 7, Windows Vista, and older OSes. This is because it has been updated. For whatever reason, Railroad Tycoon 3 hasn't.

The games you have a physical copy cannot have their code updated (the disc doesn't magically change with newer code, after all). So if the game code or its installer needs changes to work on the newer OSes, then the physical copy simply will not work.

If you have an older game that you want to play but won't work on your computer's current OS, you might be able to get it to install and play using an emulator (such as DosBox for MS-DOS games) or by installing virtual machine software and building a virtual machine that runs that older OS.