Are there any tools in linux for splitting a file in place?

Solution 1:

#!/bin/bash
# (c) whitequark 2010

set -e

if [ $# != 2 ]; then
  echo "Usage: $0 <filename> <part size>"
  echo "  This script will split file to multiple parts, starting from"
  echo "  the end, and truncating the original file in process."
  echo "  Part size is specified in bytes."
  echo "  Use at your own risk."
  exit 0
fi

filename=$1
partsize=$2

size=$(stat -c '%s' "${filename}")
parts=$(($size / $partsize))

do_split() {
  _part=$1
  _size=$2

  echo "Splitting part $_part"
  echo $(($partsize * ($_part - 1)))
  dd if="${filename}" of="${filename}.$(printf '%04d' $_part)" \
      count=1 bs=$partsize skip=$(($_part - 1))
  echo "Truncating source file"
  truncate "${filename}" --size="-$_size"
}

lastsize=$(($size % $partsize))
if [ $lastsize != 0 ]; then
  do_split $(($parts + 1)) $lastsize
fi

for i in $(seq $parts -1 1); do
  do_split $i $partsize
done

rm "${filename}"

gedit has successfully ran after disassembling and assembling it again.

Solution 2:

I found @whitequark script really useful. But I wanted to split a 500GB disk image into some big chunks of about 50GB each. This way, the script failed, since dd can't handle such a big bs parameter.

So I customized the script for making bs=1M and asking for megabytes instead of bytes. Now I can split in place and in really big chunks using, for example, 50000 for 50GB.

#!/bin/bash
# (c) whitequark 2010
# (c) dertalai 2015 (minimal modifications)

set -e

if [ $# != 2 ]; then
  echo "Usage: $0  "
  echo "  This script will split file to multiple parts, starting from"
  echo "  the end, and truncating the original file in process."
  echo "  Part size is specified in megabytes (1 MB = 1048576 bytes)."
  echo "  Use at your own risk."
  exit 0
fi

filename=$1
#partsize=$2
partsizeMB=$2
partsize=$(($2 * 1048576))

size=$(stat -c '%s' "${filename}")
parts=$(($size / $partsize))

do_split() {
  _part=$1
  _size=$2

  echo "Splitting part $_part"
  echo $(($partsize * ($_part - 1)))
  dd if="${filename}" of="${filename}.$(printf '%04d' $_part)" \
      count=$partsizeMB bs=1M skip=$((($_part - 1) * $partsizeMB))
  echo "Truncating source file"
  truncate "${filename}" --size="-$_size"
}

lastsize=$(($size % $partsize))
if [ $lastsize != 0 ]; then
  do_split $(($parts + 1)) $lastsize
fi

for i in $(seq $parts -1 1); do
  do_split $i $partsize
done

rm "${filename}"

Solution 3:

Do you actually have the 500GB file yet? If you're generating a 500GB file by archiving a folder or disk, and then trying to split it, you can split it on-the-fly by piping the output of tar (or whatever you're using) into split:

sudo tar cvjsp /Volumes/BackupDisk/Backups.backupdb/ | \
     split -d -b 4480m - Backups.backupdb.tar.bz2.

This will make DVD-sized splits of an archive of my Time machine database. However, it does make them all at once, which means that it really doesn't do what you're looking for.

See my question here for more info. Whitequark's script could be useful over there with some slight modification! I'll have to try it.