Randomly shuffle rows in a large text file

Solution 1:

You can use the shuf command from GNU coreutils. The utility is pretty fast and would take less than a minute for shuffling a 1 GB file.

The command below might just work in your case because shuf will read the complete input before opening the output file:

$ shuf -o File.txt < File.txt

Solution 2:

Python one-liner:

python -c 'import sys, random; L = sys.stdin.readlines(); random.shuffle(L); print "".join(L),'

Reads all the lines from the standard input, shuffles them in-place, then prints them without adding an ending newline (notice the , from the end).

Solution 3:

If like me you came here to look for an alternate to shuf for macOS then use randomize-lines.

Install randomize-lines(homebrew) package, which has an rl command which has similar functionality to shuf.

brew install randomize-lines

Usage: rl [OPTION]... [FILE]...
Randomize the lines of a file (or stdin).

  -c, --count=N  select N lines from the file
  -r, --reselect lines may be selected multiple times
  -o, --output=FILE
                 send output to file
  -d, --delimiter=DELIM
                 specify line delimiter (one character)
  -0, --null     set line delimiter to null character
                 (useful with find -print0)
  -n, --line-number
                 print line number with output lines
  -q, --quiet, --silent
                 do not output any errors or warnings
  -h, --help     display this help and exit
  -V, --version  output version information and exit

Solution 4:

For OSX the binary is called gshuf.

brew install coreutils
gshuf -o File.txt < File.txt