Why does chmod(1) on the group affect the ACL mask?

It would not be convenient for you if chmod() didn't have this behaviour.

It would be highly inconvenient, because things that people traditionally expect to work on Unixes would break. This behaviour serves you well, did you but know it.

It's a shame that IEEE 1003.1e never became a standard and was withdrawn in 1998. In practice, fourteen years on, it's a standard that a wide range of operating systems — from Linux through FreeBSD to Solaris — actually implement.

IEEE 1003.1e working draft #17 makes for interesting reading, and I recommend it. In appendix B § 23.3 the working group provides a detailed, eight page, rationale for the somewhat complex way that POSIX ACLs work with respect to the old S_IRWXG group permission flags. (It's worth noting that the TRUSIX people provided much the same analysis ten years earlier.) I'm not going to copy it all here. Read the rationale in the draft standard for details. Here is a very brief précis:

• The SunOS manual is wrong. It should read

  If you use the chmod(1) command to change the file group owner permissions on a file with ACL entries, either the file group owner permissions or the ACL mask are changed to the new permissions.

  This is the behaviour that you can see happening, despite what the current manual page says, in your question. It's also the behaviour specified by the draft POSIX standard. If a CLASS_OBJ (Sun's and TRUSIX's terminology for ACL_MASK) access-control entry exists, the group bits of a chmod() set it, otherwise they set the GROUP_OBJ access-control entry.

• If this weren't the case, applications that did various standard things with `chmod()`, expecting it to work as `chmod()` has traditionally worked on old non-ACL Unixes, would either leave gaping security holes or see what they think to be gaping security holes:

  • Traditional Unix applications expect to be able to deny all access to a file, named pipe, device, or directory with chmod(…,000). In the presence of ACLs, this only turns off all user and group permissions if the old S_IRWXG maps to CLASS_OBJ. Without this, setting the old file permissions to 000 wouldn't affect any USER or GROUP entries and other users would, surprisingly, still have access to the object.

    Temporarily changing a file's permission bits to no access with chmod 000 and then changing them back again was an old file locking mechanism, used before Unixes gained advisory locking mechanisms, that — as you can see — people still use today.

  • Traditional Unix scripts expect to be able to run chmod go-rwx and end up with only the object's owner able to access the object. Again — as you can see — this is still the received wisdom twelve years later. And again, this doesn't work unless the old S_IRWXG maps to CLASS_OBJ if it exists, because otherwise that chmod command wouldn't turn off any USER or GROUP access control entries, leading to users other than the owner and non-owning groups retaining access to something that is expected to be accessible only to the owner.

  • A system where the permission bits were otherwise separate from and anded with the ACLs would require file permission flags to be rwxrwxrwx in most cases, which would confuse the heck out of the many Unix applications that complain when they see what they think to be world-writable stuff.

    A system where the permission bits were otherwise separate from and ored with the ACLs would have the chmod(…,000) problem mentioned before.

Further reading

• Winfried Trümper (1999-02-28). Summary about Posix.1e
• Portable Applications Standards Committee of the IEEE Computer Society (October 1997). Draft Standard for Information Technology—Portable Operating System Interface (POSIX)—Part 1: System Application Program Interface (API)— Amendment #: Protection, Audit and Control Interfaces [C Language] IEEE 1003.1e. Draft 17.
• Craig Rubin (1989-08-18). Rationale for Selecting Access Control List Features for the Unix System. NCSC-TG-020-A. DIANE Publishing. ISBN 9780788105548.