How to manage rack keys?
Buy a key safe, put it in the server room, give the key(s) to the key safe to your trusted minion(s) and one or two helpful backup folks (friendly and helpful owners, or the accounting people who are already trusted with cash and such).
The trusted folks are the only ones who lock and relock racks, and they write themselves in on a log.
Since the safe is in the server room, it should be covered by whatever security you've already got there.
Put them in a lock box secured by a digital passcode for each user, which then logs entry to the box.