How does an operating system prevent an unprivileged process from executing a privileged instruction?
Solution 1:
chmod is a filesystem operation, not a privileged instruction. Filesystem permissions are not handled at the hardware level. The software (specifically the OS) sees that the process invoking the system call does not have sufficient permissions to perform the operation on the filesystem object and the system call returns with a permission error.
Solution 2:
Actually, you are asking two different questions:
- How does an operating system prevent an unprivileged process from executing a privileged instruction?
- Why can an unprivileged process not invoke
chmod 777 /?
Answer for 2:
chmod internally invokes a function from the libc (conveniently also called chmod()). This function checks whether the caller has sufficient privileges for the operation - if not, it returns error EPERM.
The answer for 1 is more interesting:
The exact mechanism depends on OS and hardware platform, but basically it's like this: All modern processors have built-in security features. This allows the OS to tell the processor: "run this program, but do not let it execute these privileged instructions". So the processor itself will enforce the restriction on allowed instructions. If the programm tries to execute a privileged instruction, the processor will pass control back to the OS, which will usually terminate the misbehaving program. For details, see e.g. https://en.wikipedia.org/wiki/Ring_%28computer_security%29