Abort in glibc while trying to use sbrk to reduce the size of the data segment
While working with glibc I tried to reduce the data segment using sbrk using a negative parameter, and found a most strange behaviour.
I first malloc, then free it, then reduce data segment with sbrk, and then malloc again with same size as the first one.
The issue is, if the malloc size (both mallocs with same size) is small enough (32k, or eight 4k pages) then everything works fine. But when I increase a little the malloc-free-malloc size (to nine 4k pages) then I get the core dump. What is even more strange is that when I raise the malloc size to cross the mmap threshold (128k) then I get the adjust-abort behaviour.
The C code:
#define _GNU_SOURCE 1
#include <stdio.h>
#include <malloc.h>
#include <unistd.h>
// set MMAP_ALLOC_SIZE to 8 4k-pages it will work,
// set it to 9 4k-pages it raises a 'segmentation fault (core dumped)'
// set it to 33 4k-pages it raises a 'break adjusted to free malloc space' and 'abort (core dumped)'
#define MMAP_ALLOC_SIZE (33 * 4096)
#define PRINT_MEM { \
struct mallinfo mi; \
mi = mallinfo(); \
printf("ptr %p\n", ptr); \
printf("brk(0) %p\n", sbrk(0)); \
printf("heap %d bytes\n", mi.arena); \
printf("mmap %d bytes\n\n", mi.hblkhd); \
}
int main(int argc, char *argv[])
{
void *ptr;
ptr = NULL; PRINT_MEM
printf("1) will malloc > MMAP_THRESHOLD (128 KiB) ...\n");
ptr = malloc(MMAP_ALLOC_SIZE); PRINT_MEM
printf("2) will free malloc ...\n");
free(ptr); PRINT_MEM
printf("3) will reduce brk ...\n");
ptr = sbrk(-100000); PRINT_MEM
printf("4) will malloc > MMAP_THRESHOLD (128 KiB) ... \n");
ptr = malloc(MMAP_ALLOC_SIZE); PRINT_MEM
printf("5) completion.\n"); // never happens if MMAP_ALLOC_SIZE is > 8 4k-pages
return 0;
}
Compiled with:
gcc -Wall testbrk.c -o testbrk
Which gives the successful output for MMAP_ALLOC_SIZE (8 * 4096):
ptr (nil)
brk(0) 0xf46000
heap 0 bytes
mmap 0 bytes
1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr 0xf25670
brk(0) 0xf46000
heap 135168 bytes
mmap 0 bytes
2) will free malloc ...
ptr 0xf25670
brk(0) 0xf46000
heap 135168 bytes
mmap 0 bytes
3) will reduce brk ...
ptr 0xf46000
brk(0) 0xf2d960
heap 135168 bytes
mmap 0 bytes
4) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr 0xf25670
brk(0) 0xf2d960
heap 135168 bytes
mmap 0 bytes
5) completion.
The following abort output for MMAP_ALLOC_SIZE (9 * 4096):
ptr (nil)
brk(0) 0x1b7f000
heap 0 bytes
mmap 0 bytes
1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr 0x1b5e670
brk(0) 0x1b7f000
heap 135168 bytes
mmap 0 bytes
2) will free malloc ...
ptr 0x1b5e670
brk(0) 0x1b7f000
heap 135168 bytes
mmap 0 bytes
3) will reduce brk ...
ptr 0x1b7f000
brk(0) 0x1b66960
heap 135168 bytes
mmap 0 bytes
4) will malloc > MMAP_THRESHOLD (128 KiB) ...
Segmentation fault (core dumped)
And the following adjust-abort output for MMAP_ALLOC_SIZE (33 * 4096):
ptr (nil)
brk(0) 0x1093000
heap 0 bytes
mmap 0 bytes
1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr 0x7fdd1c7f6010
brk(0) 0x1093000
heap 135168 bytes
mmap 139264 bytes
2) will free malloc ...
ptr 0x7fdd1c7f6010
brk(0) 0x1093000
heap 135168 bytes
mmap 0 bytes
3) will reduce brk ...
ptr 0x1093000
brk(0) 0x107a960
heap 135168 bytes
mmap 0 bytes
4) will malloc > MMAP_THRESHOLD (128 KiB) ...
break adjusted to free malloc space
Aborted (core dumped)
So the sbrk reduction call works without error, but the subsequent malloc raises a core dump even if enough memory was still available.
I'm I doing something wrong or is this a limitation in the data segment resize?
EDIT: Besides the code solution using malloc_trim() I've posted below, very welcome from the accepted answer, there are some important things to know about this issue, as recovered from chat:
First, the man page says avoid using sbrk, but the glibc manual does not.
malloc.c from glibc does contains comments on sbrk possibly being called with signed integer - and thus negative integer - values of memory parameter reduction, and does have provisions on the callings of brk and sbrk to be coherent with malloc. The workings of malloc are not unsensitive to sbrk, they are not from "different levels" of coding, they are supposed to work together in harmony, at least by the code comments. Also, my first test case does work well, which means that sbrk is not an issue itself to work with malloc, but is only not treated in some specific cases.
And, finally, it matters that someone can break glibc allocation, it might be a security loophole. For instance, a hacker could use some instance of the glibc being accessed by another layer of inderection in order to summon sbrk in order to cause library crash. I'm not a security expert, but given that glibc has so many different uses around the planet, it could be in principle that some malicious programmer use this sbrk crash to gain access to unprotected systems. Not sure, but sure it should be investigated by glibc developers.
And I'm sure this is not a frivolous question.
Solution 1:
It is well-documented that glibc malloc uses sbrk internally. Absent a statement that says otherwise, it can also use memory obtained with sbrk for internal bookkeeping purposes. It is neither documented nor guessable where exactly this internal bookkeeping data is stored. Thus, taking away any memory obtained by malloc (via sbrk or otherwise) can invalidate this data.
It follows that sbrk with a negative argument should never be used in a program that also uses malloc (and of course any library function that might use malloc, such as printf). A statement to this effect probably should have been included in the glibc documentation, to make the reasoning above unnecessary. There is a statement that cautions against the use of brk and sbrk in general though:
You will not normally use the functions in this section, because the functions described in Memory Allocation are easier to use. Those are interfaces to a GNU C Library memory allocator that uses the functions below itself. The functions below are simple interfaces to system calls.
If you want to release unused memory at the end of the glibc malloc arena, use malloc_trim() (a glibc extension, not a standard C or POSIX function).