Why can't QEMU allocate the memory if the Linux caches are too big?
Solution 1:
Not all cached data can be discarded immediately. For example, cached dirty pages have to be written back to disk before they can be removed from RAM. You have no swap, so until those writes complete, there simply isn't enough available space for QEMU.
You really should add a reasonable amount of swap. You can't expect the memory manager to do a good job with one hand tied behind its back.