rsync "Operation not permitted (1)" error only when run in launchd [duplicate]

Solution 1:

I've spent a few weeks trying to sort this out. I agree that the currently accepted answer is not really a solution -- it's not much better than just disabling SIP altogether.

My solution is a totally hacky workaround, but doesn't require whitelisting bash entirely. Update: Slightly less hacky workaround below.

Update 20190226: As detailed in the Restic issue linked below, this seems to have stopped working. The original binaries I created with this method continue to work without errors for some strange reason, but I can't give access to new binaries directly using this method.

Overview

Package a standard MacOS application that runs a script (a bash script in my case, which in turn sets up an environment and runs a separate binary that requires FDA)
Add that application to FDA
Run the application by opening the .app (either by double clicking via GUI, or open /path/to/MyApp.app but not by directly using the executable stored in e.g. MyApp.app/Contents/Resources/)

For Step 1, you can easily package your script as an app by using tools built into MacOS such as Automator.app or Script Editor, or Platypus also works.

Example contents of such an app might be a simple AppleScript (in Script Editor) such as:

on run
    do shell script "/usr/local/bin/bash /path/to/myscript.sh"
end run

From Script Editor, use the dropdown in the save menu to save as an application, then CLOSE SCRIPT EDITOR.

Add this app to Full Disk Access through System Preferences -> Security & Privacy.

NB: If you didn't save and close Script Editor prior to adding to FDA as instructed, it seems that some kind of invisible process (automated background saves?) will change something (some kind of timestamp or hash?) that is required for Full Disk Access, which can some intermittent errors that were a major headache to figure out. So if you didn't do this, remove your app from FDA, save and close Script Editor, then add back to FDA.

For your LaunchAgent, use something like:

<string>/usr/bin/open</string>
<string>/path/to/MyApp.app</string>

Root access

If your backup script needs root access (e.g. to back up root-owned 0600 files), you're in for another set of hacky workarounds, since /usr/bin/open won't seem to run anything as root (even if you specify the UserName key in in a root-owned /Library/LaunchDaemons/ plist. (I'd be happy to open this as a separate question if appropriate, since I think the below workaround leaves much to be desired.)

One option for this is to add with administrator privileges in your AppleScript, but this requires manually typing in your password, defeating the purpose of automated backup. My current workaround is to:

sudo visudo and give my unprivileged user the ability to run my backup script as sudo with NOPASSWD: (I also specify the hash of the script to hopefully improve security, e.g. myuser ALL=(ALL) NOPASSWD: sha256:hashgoeshere /path/to/myscript.sh)
sudo chown root myscript.sh
sudo chmod 0740 myscript.sh (4 so it can still be added to VCS)
Change the AppleScript to do shell script "sudo -n /path/to/myscript.sh" and resave as MyApp.app
Add MyApp.app to FDA
Change my launchd script to open /path/to/MyApp.app
Reload launchd script with launchctl and test to make sure it seems to be working

UPDATE:

After some preliminary testing, it looks like a slightly less hacky workaround is to compile a binary (using a compiled language) that calls your bash script. Add that binary to FDA and it seems to work. Add to the root-owned /Library/LaunchDaemons plist, and you have a way to call it from root without all the craziness above.

Example script in Go:

// Runrestic provides a binary to run my restic backup script in MacOS Mojave with Full Disk Access
package main

import (
    "log"
    "os"
    "os/exec"
    "path/filepath"
)

func main() {
    ex, err := os.Executable()
    if err != nil {
        log.Fatal(err)
    }
    dir := filepath.Dir(ex)
    script := filepath.Join(dir, "restic-backup.sh")
    cmd := exec.Command("/usr/local/bin/bash", script)
    if err := cmd.Run(); err != nil {
        log.Fatal(err)
    }
}

For security, I sudo chown root and sudo chmod 0700 the resulting binary before adding to Full Disk Access (although admittedly an attacker could just change the bash script that this calls if it were left unprotected).

Solution 2:

I solved this as follows as follows:

Allow Bash to have Full Disk Access

Open Preferences
Go to Security & Preferences
Select Full Disk Access in the list on the left
Click the lock to make changes
Click the + button on the list on the right
Navigate to the root of your HD
Press CMD+Shift+. to show all the hidden items
Select /bin/bash
Quit Preferences
Restart the mac (I am not sure if this is really necessary)

Run your script correctly

The mistake I made was that the launch agent ran the script like this:

<key>ProgramArguments</key>
<array>
    <string>/Users/channing/bin/backup.sh</string>
</array>

Do this instead

<key>ProgramArguments</key>
<array>
    <string>/bin/bash</string>
    <string>/Users/channing/bin/backup.sh</string>
</array>

Restart your agent:

launchctl unload ~/Library/LaunchAgents/backup.plist
launchctl load ~/Library/LaunchAgents/backup.plist

Rejoice.

Solution 3:

The currently accepted answer is a bit hard to follow with all of its updates. Here's a short summary of what currently does and doesn't work, plus a new tip.

Adding scripts or binaries to the "Full Disk Access" list no longer works. The only thing that works is adding an actual macOS app. As Channing mentions, Automator.app, Script Editor, or Platypus can be used to create one.

Tip: the whitelist applies to any script or binary inside the app's directory. So, you don't have to launch the app directly. In fact, the app itself is completely irrelevant -- it simply acts as a container for whitelisting. You can copy your script to an arbitrary app, whitelist that app, then run your script. The only caveat is that you have to remove and re-add the app to the whitelist every time you change your script or binary within it.

Solution 4:

Had the same issue on macOS Catalina trying to schedule Borg backups.

I created an app using "Script Editor" that runs /usr/local/bin/borg-backup.sh using zsh.

do shell script "zsh /usr/local/bin/borg-backup.sh"

I then exported the app to /Applications/borg-backup.app clicking "File" then "Export..." choosing "Application" for "File Format".

Finally, I updated ~/Library/LaunchAgents/local.borg-backup.plist.

<key>ProgramArguments</key>
<array>
  <string>open</string>
  <string>/Applications/borg-backup.app</string>
</array>

The first time the launch agent ran, a prompt asked me to grant borg-backup.app access to ~/Documents.

Solution 5:

Edit:

I did some more testing and can't seem to give any normal† program full disk access anymore. I wrote a minimal shell script [1], a minimal binary calling that shell script [2], and a binary trying to access a secure location [3]. I then gave all these scripts/executables Full Disk Access, and also to /bin/sh for good measure. Calling any of these directly via the shell gives me an error.

I then stumbled across an Apple dev forums discussion about The Rules for Full Disk Access. It looks like you need an app bundle to give Full Disk Access permissions, which explains why granting full disk access to a terminal app enables that app to call ls ~/Library/Mail successfully.
However it does not explain why you can grant access to /bin/bash and then use that in your launchd.plist file to have full disk access in your shell script.

† simple binary, not an App Bundle living in /Applications

[1] /Users/me/access-test.sh:

#!/bin/sh
ls /Users/me/Library/Mail

[2] /Users/me/access-test.c:

#include <unistd.h>

int main(int argc, char *const argv[]) {
    const char *file = "/Users/me/access-test.sh";
    return execvp(file, argv);
}

[3] /Users/me/access-test-2.c:

#include <stdio.h>
#include <dirent.h>

int main(void) {
    DIR *dp;
    struct dirent *ep;

    dp = opendir("/Users/me/Library/Mail");
    if (dp == NULL) {
        perror("Couldn't open directory");
        return 1;
    } else {
        while ((ep = readdir(dp))) {
            puts(ep->d_name);
        }
        closedir (dp);
    }
}

My initial response, which turned out to be wrong:

~~Allow your backup script full disk access.~~

Go to System Preferences → Security & Privacy → Privacy → Full Disk Access. Then click the lock 🔒 to make changes, click on the + button and add your script.

The next time launchd launches your script it will have full disk access and pass these permissions on to any sub-processes it spawns – like rsync in your case.

This works for any executable you want to launch using a launch agent. Just add the executable (given as value for <key>Program</key> or the first value of <key>ProgramArguments</key> in your plist file) to the list of programs having full disk access.