How do I increase the max open files in macOS Big Sur?

/etc/sysctl.conf is no longer supported with Big Sur. To make persistent changes systctl can be issued on boot time using a launch daemon in /Library/LaunchDaemons/com.startup.sysctl.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.startup.sysctl</string>
    <key>LaunchOnlyOnce</key>
    <true/>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/sbin/sysctl</string>
        <string>kern.maxfiles=40480</string>
        <string>kern.maxfilesperproc=28000</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

and install it with

chown root:wheel /Library/LaunchDaemons/com.startup.sysctl.plist
launchctl load /Library/LaunchDaemons/com.startup.sysctl.plist

I have used the following technique with success on Big Sur v11.6

So it seems that there are multiple settings at play here.

The numbers returned by ulimit -Sn and ulimit -Hn for soft and hard per process limits.

The the numbers returned by launchctl limit maxfiles the first of which is a soft per process limit and the second a hard per process limit.

Then the results of sysctl -a | grep maxfiles which shows kern.maxfiles and kern.maxfilesperproc where kern.maxfiles should be the hard limit for all processes and kern.maxfilesperproc should be the hard limit for a single process.

It seems I can permanently set all of these by adding a file at /Library/LaunchDaemons/limit.maxfiles.plist

with the following content, where 524288 corresponds to the soft limit and 16777216 to the hard limit.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
      <string>launchctl</string>
      <string>limit</string>
      <string>maxfiles</string>
      <string>524288</string>
      <string>16777216</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>ServiceIPC</key>
    <false/>
  </dict>
</plist>

In order for the file to take effect you need to ensure that it's owned by root has group wheel.

sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist

You also need to load the daemon.

sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist

And finally to have this affect the output of ulimit -Sn and ulimit -Hn i needed to restart my machine.

Note: It seems like a mistake that setting a per process soft and hard limit via launchctl affects the kernel max per proc and max file limits, but that is what happens, so I've just chosen values large enough that I think its unlikely I will run into any trouble with them.