How does Mac OS keep track of moved files?

From a higher point of view (in opposite to the low level implementation) this is probably related to NSURL's bookmark capabilities.

A bookmark provides a persistent reference to a file-system resource. When you resolve a bookmark, you obtain a URL to the resource’s current location. A bookmark’s association with a file-system resource (typically a file or folder) usually continues to work if the user moves or renames the resource, or if the user relaunches your app or restarts the system.

Most Apps don't use the Bookmarks (mentioned by Matthieu) for this purpose, because Bookmarks are meant to locate a file after the app has been quit in between), but rather either keep the file open while the app runs, or reference the file using its File ID (aka CNID or objID), or simply by using a NSURL object that keeps track of the file.

The secret to this working is that HFS(+) volumes assign each file and folder a unique number (the CNID), and so even if the file is renamed or moved, it can be looked up by that number again. Bookmarks also make use of this number by storing it for this purpose.

This also means that if you place your file on a file system that does not support file IDs, e.g. on a SMB network volume or a FAT formatted disk, renaming and moving of such files may NOT be detected by those apps, because there's no alternative tracking mechanism in this case.

See also Apple's Technical QA1113: The /.vol/ dir and volFS.

APFS, Apple's new volume format, does support File IDs as well, BTW.