How can I remove my users from local admin rights with a GPO?

Restricted Groups

Computer configuration > Windows Settings > Security Settings > Restricted Groups.

If you set that up with Using the "Members" Restricted Group Portion of Policy, it will remove anything else that is listed locally. It will only apply the group that is in your policy. You can read more about it here