Why is malloc not "using up" the memory on my computer?

malloc() does not use memory. It allocates it.

After you allocate the memory, use it by assigning some data.

size_t Size = 256 * 1024 * 1024;
p = malloc(Size);
if (p != NULL) {
  memset(p, 123, Size);
}

Some platforms implement malloc() is such a way that the physical consumption of memory does not occur until that byte (or more likely a byte within a group or "page" of bytes) is accessed.

calloc() may or may not truly use the memory either. A system could map lots of memory to the same physical zeroed memory, at least until the data gets interesting. See Why malloc+memset is slower than calloc?

The memory may not be really available, especially that you didn't do anything using p in your example except check for if it's NULL. From man malloc

By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available. In case it turns out that the system is out of memory, one or more processes will be killed by the OOM killer. For more information, see the description of /proc/sys/vm/overcommit_memory and /proc/sys/vm/oom_adj in proc(5), and the Linux kernel source file Documentation /vm/overcommit-accounting.

The calloc on your system† actually touches the memory by clearing it, and on many systems memory is not really allocated (and thus “used up”) until it is touched by the process to which it is allocated. So just doing malloc does not “use” the memory until you, well, use it.

† See comments