At least points 3 and 4 are invalid/insecure.

  1. Created normal user accounts for my clients (ex. client1, client2...) and then added all of them to serveradmin group

What does a normal user account has to do with server admin? Absolutely nothing, even naming wise. What is the point of having each site user belong to the serveradmin group?

  1. Set nginx user to serveradmin

Likewise. NGINX should run under its own user, typically nginx (CentOS/RHEL distros) or www-data (Debian-based systems). It has nothing to do with server administration.

I can't go into details on how the current setup is insecure as this needs far more details like socket ownership for PHP, file locations, etc.

A truly secure setup assumes as much user separation as possible, and and this pertains to services. So start with allocating (or using the one provided by package) separate user for NGINX and follow through secure permissions for PHP-FPM. It is NGINX's user that should be in each site user's group, not the other way around:

usermod -a -G client1 nginx
usermod -a -G client2 nginx
...

Now nginx user belongs to client1 and client2 groups. Why, is because the nginx user really has to be able to read each and every website files.

PHP-FPM pools, on the other hand are fine to be bound to client1:client1 both in runtime and socket listen options.