Writing robust R code: namespaces, masking and using the `::` operator

Short version

For those that don't want to read through my "case", this is the essence:

1. What is the recommended way of minimizing the chances of new packages breaking existing code, i.e. of making the code you write as robust as possible?

2. What is the recommended way of making the best use of the namespace mechanism when

   a) just using contributed packages (say in just some R Analysis Project)?

   b) with respect to developing own packages?

3. How best to avoid conflicts with respect to formal classes (mostly Reference Classes in my case) as there isn't even a namespace mechanism comparable to :: for classes (AFAIU)?

The way the R universe works

This is something that's been nagging in the back of my mind for about two years now, yet I don't feel as if I have come to a satisfying solution. Plus I feel it's getting worse.

We see an ever increasing number of packages on CRAN, github, R-Forge and the like, which is simply terrific.

In such a decentralized environment, it is natural that the code base that makes up R (let's say that's base R and contributed R, for simplicity) will deviate from an ideal state with respect to robustness: people follow different conventions, there's S3, S4, S4 Reference Classes, etc. Things can't be as "aligned" as they would be if there were a "central clearing instance" that enforced conventions. That's okay.

The problem

Given the above, it can be very hard to use R to write robust code. Not everything you need will be in base R. For certain projects you will end up loading quite a few contributed packages.

IMHO, the biggest issue in that respect is the way the namespace concept is put to use in R: R allows for simply writing the name of a certain function/method without explicitly requiring it's namespace (i.e. foo vs. namespace::foo).

So for the sake of simplicity, that's what everyone is doing. But that way, name clashes, broken code and the need to rewrite/refactor your code are just a matter of time (or of the number of different packages loaded).

At best, you will know about which existing functions are masked/overloaded by a newly added package. At worst, you will have no clue until your code breaks.

A couple of examples:

• try loading RMySQL and RSQLite at the same time, they don't go along very well
• also RMongo will overwrite certain functions of RMySQL
• forecast masks a lot of stuff with respect to ARIMA-related functions
• R.utils even masks the base::parse routine

(I can't recall which functions in particular were causing the problems, but am willing to look it up again if there's interest)

Surprisingly, this doesn't seem to bother a lot of programmers out there. I tried to raise interest a couple of times at r-devel, to no significant avail.

Downsides of using the :: operator

1. Using the :: operator might significantly hurt efficiency in certain contexts as Dominick Samperi pointed out.
2. When developing your own package, you can't even use the :: operator throughout your own code as your code is no real package yet and thus there's also no namespace yet. So I would have to initially stick to the foo way, build, test and then go back to changing everything to namespace::foo. Not really.

Possible solutions to avoid these problems

1. Reassign each function from each package to a variable that follows certain naming conventions, e.g. namespace..foo in order to avoid the inefficiencies associated with namespace::foo (I outlined it once here). Pros: it works. Cons: it's clumsy and you double the memory used.
2. Simulate a namespace when developing your package. AFAIU, this is not really possible, at least I was told so back then.
3. Make it mandatory to use namespace::foo. IMHO, that would be the best thing to do. Sure, we would lose some extend of simplicity, but then again the R universe just isn't simple anymore (at least it's not as simple as in the early 00's).

And what about (formal) classes?

Apart from the aspects described above, :: way works quite well for functions/methods. But what about class definitions?

Take package timeDate with it's class timeDate. Say another package comes along which also has a class timeDate. I don't see how I could explicitly state that I would like a new instance of class timeDate from either of the two packages.

Something like this will not work:

new(timeDate::timeDate)
new("timeDate::timeDate")
new("timeDate", ns="timeDate")

That can be a huge problem as more and more people switch to an OOP-style for their R packages, leading to lots of class definitions. If there is a way to explicitly address the namespace of a class definition, I would very much appreciate a pointer!

Conclusion

Even though this was a bit lengthy, I hope I was able to point out the core problem/question and that I can raise more awareness here.

I think devtools and mvbutils do have some approaches that might be worth spreading, but I'm sure there's more to say.

Solution 1:

GREAT question.

Validation

Writing robust, stable, and production-ready R code IS hard. You said: "Surprisingly, this doesn't seem to bother a lot of programmers out there". That's because most R programmers are not writing production code. They are performing one-off academic/research tasks. I would seriously question the skillset of any coder that claims that R is easy to put into production. Aside from my post on search/find mechanism which you have already linked to, I also wrote a post about the dangers of warning. The suggestions will help reduce complexity in your production code.

Tips for writing robust/production R code

1. Avoid packages that use Depends and favor packages that use Imports. A package with dependencies stuffed into Imports only is completely safe to use. If you absolutely must use a package that employs Depends, then email the author immediately after you call install.packages().

Here's what I tell authors: "Hi Author, I'm a fan of the XYZ package. I'd like to make a request. Could you move ABC and DEF from Depends to Imports in the next update? I cannot add your package to my own package's Imports until this happens. With R 2.14 enforcing NAMESPACE for every package, the general message from R Core is that packages should try to be "good citizens". If I have to load a Depends package, it adds a significant burden: I have to check for conflicts every time I take a dependency on a new package. With Imports, the package is free of side-effects. I understand that you might break other people's packages by doing this. I think its the right thing to do to demonstrate a commitment to Imports and in the long-run it will help people produce more robust R code."

2. Use importFrom. Don't add an entire package to Imports, add only those specific functions that you require. I accomplish this with Roxygen2 function documentation and roxygenize() which automatically generates the NAMESPACE file. In this way, you can import two packages that have conflicts where the conflicts aren't in the functions you actually need to use. Is this tedious? Only until it becomes a habit. The benefit: you can quickly identify all of your 3rd-party dependencies. That helps with...

3. Don't upgrade packages blindly. Read the changelog line-by-line and consider how the updates will affect the stability of your own package. Most of the time, the updates don't touch the functions you actually use.

4. Avoid S4 classes. I'm doing some hand-waving here. I find S4 to be complex and it takes enough brain power to deal with the search/find mechanism on the functional side of R. Do you really need these OO feature? Managing state = managing complexity - leave that for Python or Java =)

5. Write unit tests. Use the testthat package.

6. Whenever you R CMD build/test your package, parse the output and look for NOTE, INFO, WARNING. Also, physically scan with your own eyes. There's a part of the build step that notes conflicts but doesn't attach a WARN, etc. to it.

7. Add assertions and invariants right after a call to a 3rd-party package. In other words, don't fully trust what someone else gives you. Probe the result a little bit and stop() if the result is unexpected. You don't have to go crazy - pick one or two assertions that imply valid/high-confidence results.

I think there's more but this has become muscle memory now =) I'll augment if more comes to me.

Solution 2:

My take on it :

Summary : Flexibility comes with a price. I'm willing to pay that price.

1) I simply don't use packages that cause that kind of problems. If I really, really need a function from that package in my own packages, I use the importFrom() in my NAMESPACE file. In any case, if I have trouble with a package, I contact the package author. The problem is at their side, not R's.

2) I never use :: inside my own code. By exporting only the functions needed by the user of my package, I can keep my own functions inside the NAMESPACE without running into conflicts. Functions that are not exported won't hide functions with the same name either, so that's a double win.

A good guide on how exactly environments, namespaces and the likes work you find here: http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/

This definitely is a must-read for everybody writing packages and the likes. After you read this, you'll realize that using :: in your package code is not necessary.