Why are Perl source filters bad and when is it OK to use them?

It is "common knowledge" that source filters are bad and should not be used in production code.

When answering a a similar, but more specific question I couldn't find any good references that explain clearly why filters are bad and when they can be safely used. I think now is time to create one.

1. Why are source filters bad?
2. When is it OK to use a source filter?

Solution 1:

Why source filters are bad:

1. Nothing but perl can parse Perl. (Source filters are fragile.)
2. When a source filter breaks pretty much anything can happen. (They can introduce subtle and very hard to find bugs.)
3. Source filters can break tools that work with source code. (PPI, refactoring, static analysis, etc.)
4. Source filters are mutually exclusive. (You can't use more than one at a time -- unless you're psychotic).

When they're okay:

1. You're experimenting.
2. You're writing throw-away code.
3. Your name is Damian and you must be allowed to program in latin.
4. You're programming in Perl 6.

Solution 2:

Only perl can parse Perl (see this example):

@result = (dothis $foo, $bar);

# Which of the following is it equivalent to?
@result = (dothis($foo), $bar);
@result = dothis($foo, $bar);

This kind of ambiguity makes it very hard to write source filters that always succeed and do the right thing. When things go wrong, debugging is awkward.

After crashing and burning a few times, I have developed the superstitious approach of never trying to write another source filter.

I do occasionally use Smart::Comments for debugging, though. When I do, I load the module on the command line:

$ perl -MSmart::Comments test.pl

so as to avoid any chance that it might remain enabled in production code.

See also: Perl Cannot Be Parsed: A Formal Proof

Solution 3:

I don't like source filters because you can't tell what code is going to do just by reading it. Additionally, things that look like they aren't executable, such as comments, might magically be executable with the filter. You (or more likely your coworkers) could delete what you think isn't important and break things.

Having said that, if you are implementing your own little language that you want to turn into Perl, source filters might be the right tool. However, just don't call it Perl. :)

Solution 4:

It's worth mentioning that Devel::Declare keywords (and starting with Perl 5.11.2, pluggable keywords) aren't source filters, and don't run afoul of the "only perl can parse Perl" problem. This is because they're run by the perl parser itself, they take what they need from the input, and then they return control to the very same parser.

For example, when you declare a method in MooseX::Declare like this:

method frob ($bubble, $bobble does coerce) {
  ... # complicated code
}

The word "method" invokes the method keyword parser, which uses its own grammar to get the method name and parse the method signature (which isn't Perl, but it doesn't need to be -- it just needs to be well-defined). Then it leaves perl to parse the method body as the body of a sub. Anything anywhere in your code that isn't between the word "method" and the end of a method signature doesn't get seen by the method parser at all, so it can't break your code, no matter how tricky you get.

Solution 5:

The problem I see is the same problem you encounter with any C/C++ macro more complex than defining a constant: It degrades your ability to understand what the code is doing by looking at it, because you're not looking at the code that actually executes.