Alter ulimit for Terminal without changing kernel parameters

Solution 1:

This answer requires a bit of explanation about how these limits actually work:

In general these types of resource limits come with a soft limit and a hard limit. The soft limit means that a process will receive a signal notifying it whenever it exceeds that limit. The hard limit means that the process cannot exceed this limit no matter how hard it tries.

A process is allowed to change its own resource limits at runtime. The soft limit can be raised up to the hard limit, and lowered as much as wanted. However, the hard limit can generally only be lowered. Only the superuser ("root") is allowed to raise the hard limit.

The limit on the maximum number of simultaneously open files is enforced by the kernel. Whenever a process tries to open a file (or duplicate a file descriptor, etc) the kernel will check whether this means that the process exceeds the current limit. If so, the process will either receive a signal (for the soft limit) - or the attempt to open/duplicate/etc will fail (for the hard limit).

Note that this means that when you change the limit during runtime, nothing really happens besides the kernel storing the new limit in its data structures. If the limit is lowered and that means that processes are now exceeding their new limit, nothing happens. The processes will not receive a signal for the soft limit, and files will not be closed or the process terminated for the hard limit. However, when the process attempts to open new files, they will either receive a signal or the open attempt will fail.

Another important property of these resource limits is that they're inherited when a new process is forked. This means that when a process with a limit of X starts a new child-process C1, that child-process C1 will also have a limit of X. If the process later changes its limit to Y, and starts a new child-process C2, this child-process C2 will also have a limit of Y.

The macOS kernel is structured so that the lowest level is a Mach kernel - on top of this is layered a BSD subsystem for enabling a Unix userland. Note that these resource limits only apply to processes in this BSD subsystem of the kernel. All the processes you normally interact with on a daily basis are such BSD processes.

During boot of the system, the Mach kernel initializes the BSD subsystem in a function known as bsd_init(). In this function, the kernel sets the default (i.e. starting) limits for new processes. Among these it sets the default soft limit for number of open files per process to be 256. This value is hard coded into the kernel executable and cannot be configured on runtime. This is why on a default system, you'll see that launchctl limit will report maxfiles as 256.

You can configure that value by compiling your own kernel. Again, this will typically wreak havoc with various integrity protections such as Secure Boot, but it is doable. After downloading the kernel source code, but before compiling it, you'll want to change the file bsd/sys/param.h in approx. line 101, which says:

#define NOFILE  256  /* default max open files per process */

Note that this will change the default limits for not only Terminal.app, but also everything else.

In addition to the limit values stored by the kernel in its data structures for each process, there's also a set of matching kernel parameters. In this scenario, that's for example the kern.maxfilesperproc parameter that can be changed with sysctl. This parameter sets the limit for the first process created in the system - i.e. the root of the inheritance tree. Even though the process is running with superuser privileges, it won't be able to change its own resource limits using setrlimit() above that ceiling (without first raising the ceiling using sysctl).

In Unix-like systems and thus also BSD systems, this specific process is generally known as the "init" process. Historically this was a program named initd. On macOS the role of the init process is actually performed by launchd.

This is why when you use launchctl to set a new limit for maxfiles, you actually end up changing these kernel parameters. Afterall, those are the limits to be enforced for the init process (launchd). As they're stored as kernel parameters, instead of in a per-process data structure, they will be persisted even if the init process (launchd) should crash and be restarted. launchd is itself responsible for this, as it doesn't just call the usual setrlimit() system call to change its resource limits when you use the limit parameter with launchctl - it actually also runs sysctl() directly.

launchd being the root of the inheritance tree means that when you change its limits, this will actually have an effect on all processed to be started on the system. As you want to target the Terminal process specifically, this is not what you want.

If you take a look at Activity Monitor, search for the Terminal process and click the info-button - you'll see that it actually lists launchd as the direct parent of Terminal. This means that there's nothing "in between" launchd and Terminal whose resource limits you could have changed to get an effect on Terminal.

On Linux it is possibly via the prlimit() system call (commonly using the command line tool prlimit) to change the resource limits of an another given process. Sadly, Linux is quite alone in having this system call - it is not found on other Unix systems, including macOS. If we had this system call on macOS, you could just use a tool like prlimit to set the limits you wanted for the ´Terminal`.

You could possibly try changing the resource limits for the program that ultimately runs Terminal.app for you. For example you could change /System/Library/LaunchDaemons/com.apple.WindowServer.plist and add a SoftResourceLimits key with a dictionary holding a NumberOfFileskey set to the integer value of the limit you want. However that limit would apply to every process run by the graphical subsystem, and not only Terminal.app.

All this boils down to the fact that if you want to change the resource limits for Terminal specifically, you need to get the Terminal process to do that itself (i.e. it needs to call setrlimit() itself). Unfortunately there's no option in its Settings or anything like that to set this resource limit.

That leaves you with only one clear solution: You need to change the code of Terminal in order to set the resource limit. That is often done using so called "code injection". This could for example take the form of injecting a dynamic library using DYLD environment variables. However, it is generally a pain to do this on more modern macOS systems, as you will be hindered by SIP (System Integrity Protection) - and various other types of protections, especially on Apple's own programs. It is however doable if you're willing to disable these protections.

Commonly it is not the worth the time, effort and risk involved - and it is easier to simply set the general resource limit so high that you can open the number of Terminal tabs you want - even though that also means that other processes will have a similar higher limit.

A practical work-around might be to simply copy Terminal.app so that you can have 2 sets of Terminals running. It's not perfect, but it would allow you double the amount of tabs without changing any system settings.

You could also live with having to launch Terminal.app via another program that sets the resource limit. For example you could create a small shell script that sets the limit and then launches Terminal.app. It would mean that this shell is kept around while you have the Terminal open, but might be worth it for your scenario.