How to set up a secure system to allow only specific clients to access specific services on a server?

Use a VPN. Assign certificates (OpenVPN) or keys (WireGuard) to people and provide specific IP addresses to people too. Then use your firewall to only allow access to services from specified IP addresses. I would probably use Wireguard for this today.