Split large log file WITHOUT keeping the original (splitting in-place)

Solution 1:

What might work is to stream parts of it directly into xz - I guess you can compress a log file good enough to fit both the original and the compressed parts into your space left.

1. Get the number of lines:

   wc -l myFile.txt
   

2. Divide this into as many parts as you like, e.g. 10k lines per part.

3. Use sed to pipe the part you want into xz:

   sed -n '1,10000p' myFile.txt | xz > outfile01.xz 
   sed -n '10001,20000p' myFile.txt | xz > outfile02.xz
   

etc. This could be done by a script of course.

But honestly, do as EEAA said...

Solution 2:

You could do successive incarnations of tail and truncate to carve off chunks from the end of the massive file.

Something like

tail -n 10000 myFile.txt > myFile.001.txt
truncate -s -$(wc -c myFile.001.txt) myFile.txt
xz myFile.001.txt
rm myFile.001.txt

You could also script that. It'll probably take a while to run, though, and it'd be much better to just deal with it off-box.