Reading very large files in PHP

Solution 1:

Are you sure that it's fopen that's failing and not your script's timeout setting? The default is usually around 30 seconds or so, and if your file is taking longer than that to read in, it may be tripping that up.

Another thing to consider may be the memory limit on your script - reading the file into an array may trip over this, so check your error log for memory warnings.

If neither of the above are your problem, you might look into using fgets to read the file in line-by-line, processing as you go.

$handle = fopen("/tmp/uploadfile.txt", "r") or die("Couldn't get handle");
if ($handle) {
    while (!feof($handle)) {
        $buffer = fgets($handle, 4096);
        // Process buffer here..
    }
    fclose($handle);
}

Edit

PHP doesn't seem to throw an error, it just returns false.

Is the path to $rawfile correct relative to where the script is running? Perhaps try setting an absolute path here for the filename.

Solution 2:

Did 2 tests with a 1.3GB file and a 9.5GB File.

1.3 GB

Using fopen()

This process used 15555 ms for its computations.

It spent 169 ms in system calls.

Using file()

This process used 6983 ms for its computations.

It spent 4469 ms in system calls.

9.5 GB

Using fopen()

This process used 113559 ms for its computations.

It spent 2532 ms in system calls.

Using file()

This process used 8221 ms for its computations.

It spent 7998 ms in system calls.

Seems file() is faster.