From command line on macOS, how to add a user group to the Sharing and Permissions list for a directory?
Yes, you can do it, but it's a bit complicated. Permissions on macOS are rather complex; the Finder hides most of the complexity, but at the command line it's fully exposed and you have to deal with it.
Really short answer: use chmod +a to add access control entries, ls -le to view them, and man chmod and man ls for details.
Medium-length answer: to add (or remove) Read only access for the group mygroup to the folder at /path/to/folder:
chmod +a "group:mygroup allow list,search,readattr,readextattr,readsecurity" /path/to/folder
chmod -a "group:mygroup allow list,search,readattr,readextattr,readsecurity" /path/to/folder
For a file, Read only access would be:
chmod +a "group:mygroup allow read,readattr,readextattr,readsecurity" /path/to/file.txt
To add Read & Write access:
chmod +a "group:mygroup allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity" /path/to/folder
and for a file:
chmod +a "group:mygroup allow read,write,append,readattr,writeattr,readextattr,writeextattr,readsecurity" /path/to/file.txt
To examine these ACL and check your work:
ls -le /path/to/file.txt
Advanced usage you can call chmod =a# will rewrite the numbered rule instead of add or remove granularly the permissions.
To just remove an entry of the ACL chmod -a# wipes that entire entry like the - control does in the GUI.
Long answer: macOS has two different types of file permissions: standard POSIX (unix-like) permissions, and access control lists (ACLs) consisting of one or more access control entries (ACEs). All files and folders have POSIX permissions, consisting of one user (the owner), one group, and everyone else, and for each of those some combination of read, write, and execute (don't ask) access. They can (but usually don't) have a list of ACEs that allow (or deny, but don't worry about that) access to additional users and/or groups, and have much more detailed control over what access is being allowed (/denied).
The Finder hides the distinction between POSIX permissions and ACEs, but anytime you have more than one user or group, the additional ones are ACEs. So to add access for another group, you need to add an ACE. chmod +a does this. You also need to specify a full list of types of read and/or write (or other) access are being granted. The Finder's idea of "Read only" access corresponds to read,readattr,readextattr,readsecurity, and its "Read & Write" access corresponds to read,write,append,readattr,writeattr,readextattr,writeextattr,readsecurity.