dlopen from memory?

I'm looking for a way to load generated object code directly from memory.

I understand that if I write it to a file, I can call dlopen to dynamically load its symbols and link them. However, this seems a bit of a roundabout way, considering that it starts off in memory, is written to disk, and then is reloaded in memory by dlopen. I'm wondering if there is some way to dynamically link object code that exists in memory. From what I can tell there might be a few different ways to do this:

1. Trick dlopen into thinking that your memory location is a file, even though it never leaves memory.

2. Find some other system call which does what I'm looking for (I don't think this exists).

3. Find some dynamic linking library which can link code directly in memory. Obviously, this one is a bit hard to google for, as "dynamic linking library" turns up information on how to dynamically link libraries, not on libraries which perform the task of dynamically linking.

4. Abstract some API from a linker and create a new library out its codebase. (obviously this is the least desirable option for me).

So which ones of these are possible? feasible? Could you point me to any of the things I hypothesized existed? Is there another way I haven't even thought of?

I needed a solution to this because I have a scriptable system that has no filesystem (using blobs from a database) and needs to load binary plugins to support some scripts. This is the solution I came up with which works on FreeBSD but may not be portable.

void *dlblob(const void *blob, size_t len) {
    /* Create shared-memory file descriptor */
    int fd = shm_open(SHM_ANON, O_RDWR, 0);
    ftruncate(fd, len);
    /* MemMap file descriptor, and load data */
    void *mem = mmap(NULL, len, PROT_WRITE, MAP_SHARED, fd, 0);
    memcpy(mem, blob, len);
    munmap(mem, len);
    /* Open Dynamic Library from SHM file descriptor */
    void *so = fdlopen(fd,RTLD_LAZY);
    close(fd);
    return so;
}

Obviously the code lacks any kind of error checking etc, but this is the core functionality.

ETA: My initial assumption that fdlopen is POSIX was wrong, this appears to be a FreeBSD-ism.

I don't see why you'd be considering dlopen, since that will require a lot more nonportable code to generate the right object format on disk (e.g. ELF) for loading. If you already know how to generate machine code for your architecture, just mmap memory with PROT_READ|PROT_WRITE|PROT_EXEC and put your code there, then assign the address to a function pointer and call it. Very simple.

There is no standard way to do it other than writing out the file and then loading it again with dlopen().

You may find some alternative method on your current specific platform. It will up to you to decide whether that is better than using the 'standard and (relatively) portable' approach.

Since generating the object code in the first place is rather platform specific, additional platform-specific techniques may not matter to you. But it is a judgement call - and in any case depends on there being a non-standard technique, which is relatively improbable.

You don't need to load the code generated in memory, since it is already in memory!

However, you can -in a non portable way- generate machine code in memory (provided it is in a memory segment mmap-ed with PROT_EXEC flag).

^{(in that case, no "linking" or relocation step is required, since you generate machine code with definitive absolute or relative addresses, in particular to call external functions)}

Some libraries exist which do that: On GNU/Linux under x86 or x86-64, I know of GNU Lightning (which generates quickly machine code which runs slowly), DotGNU LibJIT (which generates medium quality code), and LLVM & GCCJIT (which is able to generate quite optimized code in memory, but takes time to emit it). And LuaJit has some similar facility too. Since 2015 GCC 5 has a gccjit library.

And of course, you can still generate C code in a file, fork a compiler to compile it into a shared object, and dlopen that shared object file. I'm doing that in GCC MELT , a domain specific language to extend GCC. It does work quite well in practice.

addenda

If performance of writing the generated C file is a concern (it should not be, since compiling a C file is much slower than writing it) consider using some tmpfs file system for that (perhaps in /tmp/ which is often a tmpfs filesystem on Linux)

We implemented a way to do this at Google. Unfortunately upstream glibc has failed to comprehend the need so it was never accepted. The feature request with patches has stalled. It's known as dlopen_from_offset.

The dlopen_with_offset glibc code is available in the glibc google/grte* branches. But nobody should enjoy modifying their own glibc.