Use disk-backed buffer between pipes
I just found mbuffer
.
Apparently the option -t
uses a temporary file for huge buffer, which is exactly what I was looking for. Alternatively with -T /path/to/file
I can choose which of my mounted filesystems will hold the file.
I also note the file gets deleted just after it's opened, so ls
doesn't show it. Still ls -l /proc/<PID of mbuffer>/fd
confirms it's there.
So my example becomes:
command1 | mbuffer -T file.buffer -m 200G | command2