How can a non-imported method in a not-attached package be found by calls to functions not having it in their namespace?

An R namespace acts as the immediate environment for all functions in its associated package. In other words, when function bar() from package foo calls another function, the R evaluator first searches for the other function in <environment: namespace:foo>, then in "imports.foo", <environment: namespace:base>, <environment: R_GlobalEnv>, and so on down the search list returned by typing search().

One nice aspect of namespaces is that they can make packages act like better citizens: unexported functions in <environment: namespace:foo> and functions in imports:foo are available only: (a) to functions in foo; (b) to other packages that import from foo; or (c) via fully qualified function calls like foo:::bar().

Or so I thought until recently...

The behavior

This recent SO question highlighted a case in which a function well-hidden in its package's namespace was nonetheless found by a call to a seemingly unrelated function:

group <- c("C","F","D","B","A","E")
num <- c(12,11,7,7,2,1)
data <- data.frame(group,num)

## Evaluated **before** attaching 'gmodels' package
T1 <- transform(data, group = reorder(group,-num))

## Evaluated **after** attaching 'gmodels
library(gmodels)
T2 <- transform(data, group = reorder(group,-num))

identical(T1, T2) 
# [1] FALSE

Its immediate cause

@Andrie answered the original question by pointing out that gmodels imports from the the package gdata, which includes a function reorder.factor that gets dispatched to inside the second call to transform(). T1 differs from T2 because the first is calculated by stats:::reorder.default() and the second by gdata:::reorder.factor().

My question

How is it that in the above call to transform(data, group=reorder(...)), the dispatching mechanism for reorder finds and then dispatches to gdata:::reorder.factor()?

(An answer should include an explanation of the scoping rules that lead from a call involving functions in the stats and base packages to a seemingly well-hidden method in gdata.)

Further possibly helpful details

1. Neither gdata:::reorder.factor, nor the gdata package as a whole are explicitly imported by gmodels. Here are the import* directives in gmodels' NAMESPACE file:

   importFrom(MASS, ginv)
   importFrom(gdata, frameApply)
   importFrom(gdata, nobs)
   

2. There are no methods for reorder() or transform() in <environment: namespace:gmodels>, nor in "imports:gmodels":

   ls(getNamespace("gmodels"))
   ls(parent.env(getNamespace("gmodels")))
   

3. Detaching gmodels does not revert reorder()'s behavior: gdata:::reorder.factor() still gets dispatched:

   detach("package:gmodels")
   T3 <- transform(data, group=reorder(group,-num))
   identical(T3, T2)
   # [1] TRUE
   

4. reorder.factor() is not stored in the list of S3 methods in the base environment:

   grep("reorder", ls(.__S3MethodsTable__.))
   # integer(0)
   

R chat threads from the last couple of days include some additional ideas. Thanks to Andrie, Brian Diggs, and Gavin Simpson who (with others) should feel free to edit or add possibly impt. details to this question.

Solution 1:

I'm not sure if I correctly understand your question, but the main point is that group is character vector while data$group is factor.

After attaching gmodels, the call for reorder(factor) calls gdata:::reorder.factor. so, reorder(factor(group)) calls it.

In transform, the function is evaluated within the environment of the first argument, so in T2 <- transform(data, group = reorder(group,-num)), group is factor.

UPDATED

library attaches the import packages into loaded namespace.

> loadedNamespaces()
 [1] "RCurl"     "base"      "datasets"  "devtools"  "grDevices" "graphics"  "methods"  
 [8] "stats"     "tools"     "utils"    
> library(gmodels) # here, namespace:gdata is loaded
> loadedNamespaces()
 [1] "MASS"      "RCurl"     "base"      "datasets"  "devtools"  "gdata"     "gmodels"  
 [8] "grDevices" "graphics"  "gtools"    "methods"   "stats"     "tools"     "utils"    

Just in case, the reorder generic exists in namespace:stats:

> r <- ls(.__S3MethodsTable__., envir = asNamespace("stats"))
> r[grep("reorder", r)]
[1] "reorder"            "reorder.default"    "reorder.dendrogram"

And for more details

The call of reorder will search the S3generics in two envs:

see ?UseMethod

first in the environment in which the generic function is called, and then in the registration data base for the environment in which the generic is defined (typically a namespace).

then, loadNamespace registers the S3 functions to the namespace.

So , in your case, library(gmodels) -> loadNamespace(gdata) -> registerS3Methods(gdata).

After this, you can find it by:

> methods(reorder)
[1] reorder.default*    reorder.dendrogram* reorder.factor*    

   Non-visible functions are asterisked

However, as the reorder.factor is not attached on your search path, you cannot access it directly:

> reorder.factor
Error: object 'reorder.factor' not found

Probably this is whole scenario.