Can you boot into recovery mode on an older iMac running a newer version of OS X (10.8)?

I believe the issue is that recovery mode was introduced in 2011, you are indeed correct.

If you have access to another Mac you could create a bootable macOS installer and boot from that