How to search file text for a pattern and replace it with a given value

I'm looking for a script to search a file (or list of files) for a pattern and, if found, replace that pattern with a given value.

Thoughts?

Solution 1:

Disclaimer: This approach is a naive illustration of Ruby's capabilities, and not a production-grade solution for replacing strings in files. It's prone to various failure scenarios, such as data loss in case of a crash, interrupt, or disk being full. This code is not fit for anything beyond a quick one-off script where all the data is backed up. For that reason, do NOT copy this code into your programs.

Here's a quick short way to do it.

file_names = ['foo.txt', 'bar.txt']

file_names.each do |file_name|
  text = File.read(file_name)
  new_contents = text.gsub(/search_regexp/, "replacement string")

  # To merely print the contents of the file, use:
  puts new_contents

  # To write changes to the file, use:
  File.open(file_name, "w") {|file| file.puts new_contents }
end

Solution 2:

Actually, Ruby does have an in-place editing feature. Like Perl, you can say

ruby -pi.bak -e "gsub(/oldtext/, 'newtext')" *.txt

This will apply the code in double-quotes to all files in the current directory whose names end with ".txt". Backup copies of edited files will be created with a ".bak" extension ("foobar.txt.bak" I think).

NOTE: this does not appear to work for multiline searches. For those, you have to do it the other less pretty way, with a wrapper script around the regex.

Solution 3:

Keep in mind that, when you do this, the filesystem could be out of space and you may create a zero-length file. This is catastrophic if you're doing something like writing out /etc/passwd files as part of system configuration management.

Note that in-place file editing like in the accepted answer will always truncate the file and write out the new file sequentially. There will always be a race condition where concurrent readers will see a truncated file. If the process is aborted for any reason (ctrl-c, OOM killer, system crash, power outage, etc) during the write then the truncated file will also be left over, which can be catastrophic. This is the kind of dataloss scenario which developers MUST consider because it will happen. For that reason, I think the accepted answer should most likely not be the accepted answer. At a bare minimum write to a tempfile and move/rename the file into place like the "simple" solution at the end of this answer.

You need to use an algorithm that:

1. Reads the old file and writes out to the new file. (You need to be careful about slurping entire files into memory).

2. Explicitly closes the new temporary file, which is where you may throw an exception because the file buffers cannot be written to disk because there is no space. (Catch this and cleanup the temporary file if you like, but you need to rethrow something or fail fairly hard at this point.

3. Fixes the file permissions and modes on the new file.

4. Renames the new file and drops it into place.

With ext3 filesystems you are guaranteed that the metadata write to move the file into place will not get rearranged by the filesystem and written before the data buffers for the new file are written, so this should either succeed or fail. The ext4 filesystem has also been patched to support this kind of behavior. If you are very paranoid you should call the fdatasync() system call as a step 3.5 before moving the file into place.

Regardless of language, this is best practice. In languages where calling close() does not throw an exception (Perl or C) you must explicitly check the return of close() and throw an exception if it fails.

The suggestion above to simply slurp the file into memory, manipulate it and write it out to the file will be guaranteed to produce zero-length files on a full filesystem. You need to always use FileUtils.mv to move a fully-written temporary file into place.

A final consideration is the placement of the temporary file. If you open a file in /tmp then you have to consider a few problems:

• If /tmp is mounted on a different file system you may run /tmp out of space before you've written out the file that would otherwise be deployable to the destination of the old file.

• Probably more importantly, when you try to mv the file across a device mount you will transparently get converted to cp behavior. The old file will be opened, the old files inode will be preserved and reopened and the file contents will be copied. This is most likely not what you want, and you may run into "text file busy" errors if you try to edit the contents of a running file. This also defeats the purpose of using the filesystem mv commands and you may run the destination filesystem out of space with only a partially written file.

  This also has nothing to do with Ruby's implementation. The system mv and cp commands behave similarly.

What is more preferable is to open a Tempfile in the same directory as the old file. This ensures that there will be no cross-device move issues. The mv itself should never fail, and you should always get a complete and untruncated file. Any failures, such as device out of space, permission errors, etc., should be encountered during writing the Tempfile out.

The only downsides to the approach of creating the Tempfile in the destination directory are:

• Sometimes you may not be able to open a Tempfile there, such as if you are trying to 'edit' a file in /proc for example. For that reason you might want to fall back and try /tmp if opening the file in the destination directory fails.
• You must have enough space on the destination partition in order to hold both the complete old file and the new file. However, if you have insufficient space to hold both copies then you are probably short on disk space and the actual risk of writing a truncated file is much higher, so I would argue this is a very poor tradeoff outside of some exceedingly narrow (and well-monitored) edge cases.

Here's some code that implements the full-algorithm (windows code is untested and unfinished):

#!/usr/bin/env ruby

require 'tempfile'

def file_edit(filename, regexp, replacement)
  tempdir = File.dirname(filename)
  tempprefix = File.basename(filename)
  tempprefix.prepend('.') unless RUBY_PLATFORM =~ /mswin|mingw|windows/
  tempfile =
    begin
      Tempfile.new(tempprefix, tempdir)
    rescue
      Tempfile.new(tempprefix)
    end
  File.open(filename).each do |line|
    tempfile.puts line.gsub(regexp, replacement)
  end
  tempfile.fdatasync unless RUBY_PLATFORM =~ /mswin|mingw|windows/
  tempfile.close
  unless RUBY_PLATFORM =~ /mswin|mingw|windows/
    stat = File.stat(filename)
    FileUtils.chown stat.uid, stat.gid, tempfile.path
    FileUtils.chmod stat.mode, tempfile.path
  else
    # FIXME: apply perms on windows
  end
  FileUtils.mv tempfile.path, filename
end

file_edit('/tmp/foo', /foo/, "baz")

And here is a slightly tighter version that doesn't worry about every possible edge case (if you are on Unix and don't care about writing to /proc):

#!/usr/bin/env ruby

require 'tempfile'

def file_edit(filename, regexp, replacement)
  Tempfile.open(".#{File.basename(filename)}", File.dirname(filename)) do |tempfile|
    File.open(filename).each do |line|
      tempfile.puts line.gsub(regexp, replacement)
    end
    tempfile.fdatasync
    tempfile.close
    stat = File.stat(filename)
    FileUtils.chown stat.uid, stat.gid, tempfile.path
    FileUtils.chmod stat.mode, tempfile.path
    FileUtils.mv tempfile.path, filename
  end
end

file_edit('/tmp/foo', /foo/, "baz")

The really simple use-case, for when you don't care about file system permissions (either you're not running as root, or you're running as root and the file is root owned):

#!/usr/bin/env ruby

require 'tempfile'

def file_edit(filename, regexp, replacement)
  Tempfile.open(".#{File.basename(filename)}", File.dirname(filename)) do |tempfile|
    File.open(filename).each do |line|
      tempfile.puts line.gsub(regexp, replacement)
    end
    tempfile.close
    FileUtils.mv tempfile.path, filename
  end
end

file_edit('/tmp/foo', /foo/, "baz")

TL;DR: That should be used instead of the accepted answer at a minimum, in all cases, in order to ensure the update is atomic and concurrent readers will not see truncated files. As I mentioned above, creating the Tempfile in the same directory as the edited file is important here to avoid cross device mv operations being translated into cp operations if /tmp is mounted on a different device. Calling fdatasync is an added layer of paranoia, but it will incur a performance hit, so I omitted it from this example since it is not commonly practiced.