Exporting DYLD_INSERT_LIBRARIES for libgmalloc not working on Big Sur?
I've been struggling with a very weird issue. I'm trying to use libgmalloc (eg per man 3 libgmalloc) on Big Sur (I'm on 11.3.1) to trouble shoot a memory issue in my code. However, I can't set the environment variable DYLD_INSERT_LIBRARIES. It sets as a shell variable, but not as an environment variable that can be seen by child processes:
stuarts@iMac-Pro ~ % export DYLD_INSERT_LIBRARIES="/usr/lib/libgmalloc.dylib"
stuarts@iMac-Pro ~ % echo $DYLD_INSERT_LIBRARIES
/usr/lib/libgmalloc.dylib
stuarts@iMac-Pro ~ % env |grep DYLD_INSERT_LIBRARIES
stuarts@iMac-Pro ~ %
The variable doesn't show up in the env listing (and isn't detected by my program using getenv() either). I started fooling around setting other randomly named variables, and found that I can export "DYLD" and "A_B", but not "DYLD_A" (or other things that start with "DYLD_").
This seems an extremely strange silent filter in export.
How should one use libgmalloc in Big Sur?
On my (M1) machine:
~$ export DYLD_INSERT_LIBRARIES=foo
~$ env | grep DYLD_INSERT_LIBRARIES
~$ /opt/homebrew/bin/bash
dyld: could not load inserted library 'foo' because image not found
Abort trap: 6
So (as you saw), the variable is being set - but since env is a system binary, it is silently removed. Testing with my homebrew installed version of bash, you can see the value is actually set.
See also this StackOverflow answer.
TL;DR works fine for non-system binaries, silently removed from the environment for system binaries.
Given the security features identified in @negacao's answer, the injection of libgmalloc via environment variable is very fragile (it works directly off the command line, but doesn't work when invoked via scripts, or within the debugger).
Instead, building the application with -lgmalloc in the Makefile worked well both from the command line and in the debugger (as hinted at in the comment from @mmmmmm) and I was able to find my bug in the debugger this way.
Apple needs to update the libgmalloc man page.