What exactly is handling hosts.allow and hosts.deny?

Solution 1:

it's called TCP Wrappers - a program is compiled to make use of the features (such as 'portmap', part of the NFS world) which then hooks into hosts.allow/deny and such.

http://en.wikipedia.org/wiki/TCP_Wrapper

https://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.5/html/Deployment_Guide/s1-services-tcp-wrappers.html

Solution 2:

This is handled by a Library called TCP Wrappers.
This library is a wrapper to standard Unix/Linux networking function adding ACL capabilities based on hosts.allow and hosts.denied.
It's a pretty old system.