How does a Reverse DNS lookup work with regards to spam filters?

The test isn't normally comparing against the SMTP envelope (like you describe as your eyeball-based process), but finding the reverse DNS hostname of the connecting host (if any), running that through forward DNS and seeing if it resolved back to the original IP number.

So what you need is 1) to have reverse DNS set up (for the first step) and 2) to have it set up correctly (for the second step).

There's nothing to stop anyone from comparing with the SMTP envelope as well, even if that's frequently seen as too paranoid, so you'd also want anyplace your hostname appears in your mail transmission to use the DNS name that appears in your reverse DNS.

Exactly so. You check whether the mail server who just talked to you is not pretending to be someone else by corelating the forward DNS query (the A record of the domain) with the reverse query (the PTR record of the IP address).

If they match, bravo. If they don't, boo, hiss.

This, of course, implies that any server with multiple host names on their IP address should just use the address specified in the PTR when sending mails.