Join inherited scope with by in data.table
I'm on data.table 1.9.3, and maybe I'm wrong, but I don't recall the following to be expected previously.
I build 2 data.tables, dta and dtb
> dta
idx vala fdx
1: 1 2 a
2: 2 4 a
3: 3 6 b
> dtb
idx valb
1: 1 3
2: 4 6
> dput(x = dta)
structure(list(idx = c(1, 2, 3), vala = c(2, 4, 6), fdx = c("a",
"a", "b")), .Names = c("idx", "vala", "fdx"), row.names = c(NA,
-3L), class = c("data.table", "data.frame"), .internal.selfref =
<pointer: 0x0000000000110788>, sorted = "idx")
> dput(x = dtb)
structure(list(idx = c(1, 4), valb = c(3, 6)), .Names = c("idx",
"valb"), row.names = c(NA, -2L), class = c("data.table", "data.frame"
), .internal.selfref = <pointer: 0x0000000000110788>, sorted = "idx")
The key is idx in both cases.
The following works, of course
> dta[dtb, sum(valb)]
[1] 9
However this doesn't
> dta[dtb, sum(valb), by = fdx]
Error in `[.data.table`(dta, dtb, sum(valb), by = fdx) :
object 'valb' not found
But this does
> dta[dtb][, sum(valb), by = fdx]
fdx V1
1: a 3
2: NA 6
If we see the intermediate step
> dta[dtb]
idx vala fdx valb
1: 1 2 a 3
2: 4 NA NA 6
I would have expected
dta[dtb, sum(valb), by = fdx] == dta[dtb][, sum(valb), by = fdx]
Where have I gone wrong?
Just a guess
library(data.table)
dta <- data.frame(idx=c(1,2,3),
vala=c(2,4,6),
fdx=c('a','a','b'))
dta <- data.table(dta)
dtb <- data.frame(idx=c(1,4),
valb=c(3,6))
dtb <- data.table(dtb)
setkey(dta,idx)
setkey(dtb,idx)
So when you call
dta[dtb, sum(valb)]
it's kind of like calling
tmp <- dta[dtb]
attach(tmp)
sum(valb)
detach(tmp)
However, if you call
dta[dtb, sum(valb), by=fdx]
then it's kind of like calling
tmp <- dta[dtb]
# attach(tmp) : attach doesn't happen
sum(valb)
# detach(tmp) : detach doesn't happen
The function doesn't know what to do with the additional arguments. For instance, this would also throw an error:
dta[dtb, class(fdx), sum(valb)]
However, this works
dta[dtb][, sum(valb), by=fdx]
which is kind of like
tmp <- dta[dtb]
tmp[, sum(valb), by=fdx]
Like I said, this is just a guess as to why the function may not be working as expected.