Rename returns "bareword not allowed" when trying to lowercase parts of multiple filenames

I have two files in a folder on my Ubuntu 16.04:

a1.dat
b1.DAT

I want to rename b1.DAT to b1.dat so I would have following files as a result in the folder:

a1.dat
b1.dat

I tried (unsuccessfully):

$ rename *.DAT *.dat
Bareword "b1" not allowed while "strict subs" in use at (user-supplied code).
Bareword "DAT" not allowed while "strict subs" in use at (user-supplied code).

and

$ find . -iname "*.DAT" -exec rename DAT dat '{}' \;
Bareword "DAT" not allowed while "strict subs" in use at (user-supplied code).
Bareword "DAT" not allowed while "strict subs" in use at (user-supplied code).

Searching for this resulted in no meaningful solution...

Solution 1:

That error looks like it comes from Perl rename. You need to use quoting, but you only need to specify the part you want to change, search-and-replace style. The syntax is like this:

rename -n 's/\.DAT/\.dat/' *

Remove -n after testing to actually rename the files.

To include hidden files, change glob setting before running the command:

shopt -s dotglob

If you want to rename files recursively, you can use

shopt -s globstar
rename -n 's/\.DAT/\.dat/' **

or if there are many paths in or below the current directory that do not end with .DAT, you had better specify those paths in the second command:

rename -n 's/\.DAT/\.dat/' **/*.DAT

This will be faster if your files have various different names not ending in .DAT.^[1]

To turn off these settings you can use shopt -u, eg shopt -u globstar, but they are off by default and will be off when you open a new shell.

If this results in an excessively long argument list, you can use, for example, find:

find -type f -name "*.DAT" -exec rename -n -- 's/\.DAT/\.dat/' {} \;

or better

find -type f -name "*.DAT" -exec rename -n -- 's/\.DAT/\.dat/' {} +

Using find ... -exec with + is faster than using \; because it constructs an argument list from the found files. Originally I thought that it wouldn't be possible for you to use it, because you mentioned that you were having an argument list too long issue, but now I know that the list will also be cleverly broken into multiple invocations of the command as needed to avoid that problem^[2].

Since rename will process every filename in the same way, it does not matter how long the argument list is as it can be safely split across multiple invocations. If the command you are using with -exec does not accept multiple arguments or requires its arguments to be in a particular order or for any other reason splitting the argument list will cause something unwanted to happen, you can use \;, which causes the command to be invoked once for every file found (if the argument list was too long for other methods, this will take a long time!).

Many thanks to Eliah Kagan for the very useful suggestions to improve this answer:

^[1]Specifying the filenames when globbing.
^[2]find ... -exec with + splits the argument list.

Solution 2:

You can do:

rename -n 's/DAT$/\L$&/' *.DAT

Drop -n for actual renaming to take place.

• glob pattern *.DAT matches all files end in .DAT in the current directory

• in the rename's substitution, DAT$ matches DAT at the end

• \L$& makes the whole match lowercased; $& refers to the whole match

If you just want to do for b1.DAT:

rename -n 's/DAT$/\L$&/' b1.DAT

Example:

% rename -n 's/DAT$/\L$&/' *.DAT
rename(b1.DAT, b1.dat)

Solution 3:

Other answers have addressed one of the two major aspects of this question: that of how to successfully carry out the renaming operation you needed. The purpose of this answer is to explain why your commands did not work, including the meaning of that weird "bareword not allowed" error message in the context of the rename command.

The first section of this answer is about the relationship between rename and Perl and how rename uses the first command-line argument you pass it, which is its code argument. The second section is about how the shell performs expansions--specifically globbing--to construct an argument list. The third section is about what is going on in Perl code that gives "Bareword not allowed" errors. Finally, the fourth section is a summary of all the steps that take place between entering the command and getting the error.

1. When rename gives you weird error messages, add "Perl" to your search.

In Debian and Ubuntu, the rename command is a Perl script that performs file renaming. On older releases--including 14.04 LTS, which is still supported as of this writing--it was a symbolic link pointing (indirectly) to the prename command. On newer releases, it points instead to the newer file-rename command. Those two Perl rename commands work mostly the same way, and I'll just refer to them both as rename for the rest of this answer.

When you use the rename command, you're not just running Perl code someone else has written. You're also writing your own Perl code and telling rename to run it. This is because the first command-line argument you pass to the rename command, other than option arguments like -n, consists of actual Perl code. The rename command uses this code to operate on each of the pathnames that you pass it as subsequent command-line arguments. (If you don't pass any pathname arguments, then rename reads pathnames from standard input instead, one per line.)

The code is run inside a loop, which iterates once per pathname. At the top of each iteration of the loop, before your code runs, the special $_ variable is assigned the pathname currently being processed. If your code causes the value of $_ to be changed to something else, then that file is renamed to have the new name.

Many expressions in Perl operate implicitly on the $_ variable when they are not given any other expression to use as an operand. For example, the substitution expression $str =~ s/foo/bar/ changes the first occurrence of foo in the string held by the $str variable to bar, or leaves it unchanged if it does not contain foo. If you just write s/foo/bar/ without explicitly using the =~ operator, then it operates on $_. This is to say that s/foo/bar/ is short for $_ =~ s/foo/bar/.

It's common to pass an s/// expression to rename as the code argument (i.e., the first command-line argument), but you don't have to. You can give it any Perl code you wish for it to run inside the loop, to examine each value of $_ and (conditionally) modify it.

This has a lot of cool and useful consequences, but the vast majority of them are well beyond the scope of this question and answer. The main reason I bring this up here--in fact, the main reason I decided to post this answer--is to make the point that, because the first argument to rename is actually Perl code, anytime you get a weird error message and you have trouble finding information about it by searching, you can add "Perl" to the search string (or even replace "rename" with "Perl", sometimes) and you'll often find an answer.

2. With rename *.DAT *.dat, the rename command never saw *.DAT!

A command like rename s/foo/bar/ *.txt does not typically pass *.txt as a command-line argument to the rename program, and you don't want it to, unless you have a file whose name is literally *.txt, which hopefully you do not.

rename does not interpret glob patterns like *.txt, *.DAT, *.dat, x*, *y, or * when passed to it as pathname arguments. Instead, your shell performs pathname expansion on them (which is also called filename expansion, and also called globbing). This happens before the rename utility is run. The shell expands globs into potentially multiple pathnames and passes them all, as separate command-line arguments, to rename. In Ubuntu, your interactive shell is Bash, unless you have changed it, which is why I've linked into the Bash reference manual above.

There is one situation where a glob pattern may be passed as a single unexpanded command-line argument to rename: when it does not match any files. Different shells exhibit different default behavior in this situation, but Bash's default behavior is to simply pass the glob literally. However, you rarely want this! If you did want it, then you should ensure that the pattern is not expanded, by quoting it. This goes for passing arguments to any command, not just to rename.

Quoting is not just for globbing (filename expansion), because there are other expansions that your shell performs on unquoted text and, for some of them but not others, also on text enclosed in " " quotes. In general, anytime you want to pass an argument that contains characters that may be treated specially by the shell, including spaces, you should quote it, preferably with ' ' quotes.

The Perl code s/foo/bar/ does not contain anything treated specially by the shell, but it would have been a good idea for me to have quoted that too--and written 's/foo/bar/'. (In fact, the only reason I didn't was that it would be confusing to some readers, as I had not yet talked about quoting.) The reason I say this would have been good is because it's very common that Perl code does contain such characters, and if I were to change that code, I might not remember to check if quoting was needed. In contrast, if you want the shell to expand a glob, it must not be quoted.

3. What the Perl interpreter means by "bareword not allowed"

The error messages you showed in your question reveal that, when you ran rename *.DAT *.dat, your shell expanded *.DAT into a list of one or more filenames, and that the first of those filenames was b1.DAT. All the subsequent arguments--both any others expanded from *.DAT and any expanded from *.dat--came after that argument, so they would have been interpreted as pathnames.

Because what actually ran was something like rename b1.DAT ..., and because rename treats its first non-option argument as Perl code, the question becomes: why does b1.DAT produce these "bareword not allowed" errors when you run it as Perl code?

Bareword "b1" not allowed while "strict subs" in use at (user-supplied code).
Bareword "DAT" not allowed while "strict subs" in use at (user-supplied code).

In a shell, we quote our strings to protect them from unintended shell expansions that would otherwise transform them into other strings automatically (see the section above). Shells are special purpose programming languages that work very differently from general-purpose languages (and their very weird syntax and semantics reflect that). But Perl is a general-purpose programming language and, like most general-purpose programming languages, the main purpose of quoting in Perl is not to protect strings, but to mention them at all. This is actually a way most programming languages are similar to natural language. In English, and supposing you have a dog, "your dog" is a two-word phrase, while your dog is a dog. Similarly, in Perl, '$foo' is a string, while $foo is something whose name is $foo.

However, unlike just about every other general-purpose programming language, Perl will also sometimes interpret unquoted text as mentioning a string--a string that is the "same" as it, in the sense that it is made up of the same characters in the same order. It will only try to interpret code that way if it is a bareword (no $ or other sigil, see below), and after it couldn't find any other meaning to give it. Then it will take it as a string, unless you tell it not to by enabling restrictions.

Variables in Perl typically begin with a punctuating character called a sigil, which specifies the broad type of the variable. For example, $ means scalar, @ means array, and % means hash. (There are others.) Don't worry if you find that confusing (or boring), because I'm only bringing it up to say that when a valid name appears in a Perl program but is not preceded by a sigil, that name is said to be a bareword.

Barewords serve various purposes, but they usually signify a built-in function or a user-defined subroutine that has been defined in the program (or in a module used by the program). Perl has no built-in functions called b1 or DAT, so when the Perl interpreter sees the code b1.DAT, it tries to treat b1 and DAT as the names of subroutines. Assuming no such subroutines have been defined, this fails. Then, provided restrictions have not been enabled, it treats them as strings. This will work, though whether or not you actually intended this to happen is anybody's guess. Perl's . operator concatenates strings, so b1.DAT evaluates to the string b1DAT. That is, b1.DAT is a bad way to write something like 'b1' . 'DAT' or "b1" . "DAT".

You can test this out yourself by running the command perl -E 'say b1.DAT', which passes the short Perl script say b1.DAT to the Perl interpreter, which runs it, printing b1DAT. (In that command, the ' ' quotes tell the shell to pass say b1.DAT as a single command-line argument; otherwise, the space would cause say and b1.DAT to be parsed as separate words and perl would receive them as separate arguments. perl does not see the quotes themselves, as the shell removes them.)

But now, try writing use strict; in the Perl script before say. Now it fails with the same kind of error you got from rename:

$ perl -E 'use strict; say b1.DAT'
Bareword "b1" not allowed while "strict subs" in use at -e line 1.
Bareword "DAT" not allowed while "strict subs" in use at -e line 1.
Execution of -e aborted due to compilation errors.

This happened because use strict; prohibited the Perl interpreter from treating barewords as strings. To prohibit this particular feature, it would really be sufficient to enable just the subs restriction. This command produces the same errors as above:

perl -E 'use strict "subs"; say b1.DAT'

But usually Perl programmers will just write use strict;, which enables the subs restriction and two others. use strict; is generally recommended practice. So the rename command does this for your code. That is why you get that error message.

4. In summary, this is what happened:

1. Your shell passed b1.DAT as the first command-line argument, which rename treated as Perl code to run in a loop for each pathname argument.
2. This was taken to mean b1 and DAT connected with the . operator.
3. b1 and DAT were not prefixed with sigils, so they were treated as barewords.
4. Those two barewords would have been taken as names of built-in functions or any user-defined subroutines, but no such thing had either of those names.
5. If "strict subs" had not been enabled, then they would have been treated like the string expressions 'b1' and 'DAT' and concatenated. That's far from what you intended, which illuminates how this feature is often not helpful.
6. But "strict subs" was enabled, because rename enables all restrictions (vars, refs, and subs). Therefore, you got an error instead.
7. rename quit due to this error. Because this sort of error happened early, no file renaming attempts were made, even though you had not passed -n. This is a good thing, which often protects users from unintentional filename changes and occasionally even from actual data loss.

Thanks go to Zanna, who helped me address several important shortcomings in an ealier draft of this answer. Without her, this answer would have made way less sense, and might not be posted at all.