How do private BitTorrent trackers monitor how much users upload/download?

The BitTorrent clients report back to the tracker what they are doing. There are some clients that lie but this can be detected by the fact the reports from the two sides consistently disagree. That's why you'll see some site banning the lying clients.

Some private trackers will identify users by IP address but most give each user a different .torrent file where the announce URL inside the .torrent file is unique for each user to identify them when connecting to the tracker.

Combined with the fact that bittorrent clients constantly inform the tracker about how much data they have downloaded and uploaded, this allows the tracker to record a users stats between sessions and even dynamic IP changes.

My understanding is that a BitTorrent tracker is merely a registry of users that are currently downloading/seeding

If you take a look at the specification, you'll see that clients are required to report back a bit more information than just that. Two of those things are how much data has been downloaded and uploaded.

The client can, of course, fake this, which is why tracker staff take other means to determine if you are "stat-hacking".