What is the difference between the system group and netgroup?

"System groups" (often just "groups") contain users^[1], and are one of the basic security concepts in UNIX and many other operating systems. Each user belongs to a primary group and any number of secondary groups; a group, like a user, can be allowed to access or modify certain files and directories using the usual chown, chmod, chgrp, setfacl tools.

"Netgroups", on the other hand, are part of NIS/YP directory services system used only in Unix-like systems. A netgroup contains entries in the form (hostname, user, nis-domain) (with all three items optional), and can describe a user on any host, any user on a specific host, and such. Netgroups are not used with files directly, but only by NIS-aware programs: NFS exported filesystems, sudo, system login rights.

^[1] Other operating systems may allow more advanced configurations; e.g. Windows NT supports groups as group members.