Read gzipped csv directly from a url in R
Solution 1:
I am almost certain I answered this question once before. The upshot is that Connections API of R (file()
, url()
, pipe()
, ...) can do decompression on the fly, I do not think you can do it for remote http objects.
So do the very two-step you have described: use download.file()
with a tempfile()
result as second argument to fetch the compressed file, and then read from it. As tempfile()
object, it will get cleaned up automatically at the end of your R session so the one minor fix I can suggest is to skip the unlink()
(but then I like explicit cleanups, so you may as well keep it).
Edit: Got it:
con <- gzcon(url(paste("http://dumps.wikimedia.org/other/articlefeedback/",
"aa_combined-20110321.csv.gz", sep="")))
txt <- readLines(con)
dat <- read.csv(textConnection(txt))
dim(dat)
# [1] 1490 19
summary(dat[,1:3])
# aa_page_id page_namespace page_title
# Min. : 324 Min. :0 United_States : 79
# 1st Qu.: 88568 1st Qu.:0 2011_NBA_Playoffs : 52
# Median : 2445733 Median :0 IPad_2 : 43
# Mean : 8279600 Mean :0 IPod_Touch : 38
# 3rd Qu.:16179920 3rd Qu.:0 True_Grit_(2010_film): 38
# Max. :31230028 Max. :0 IPhone_4 : 26
# (Other) :1214
The key was the hint the gzcon
help that it can put decompression around an existing stream. We then need the slight detour of readLines
and reading via textConnection
from that as read.csv
wants to go back and forth in the data (to validate column width, I presume).
Solution 2:
Using data.table::fread :
x <- data.table::fread("http://dumps.wikimedia.org/other/articlefeedback/aa_combined-20110321.csv.gz")
dim(x)
[1] 1490 19
x[, 1:2]
# aa_page_id page_namespace
# 1: 26224556 0
# 2: 31653 0
# 3: 26224556 0
# 4: 26224556 0
# 5: 1058990 0
# ---
# 1486: 619464 0
# 1487: 19283361 0
# 1488: 19006979 0
# 1489: 5078775 0
# 1490: 30209619 0
Solution 3:
This function generalizes Dirk's answer:
R <- function(file_url) {
con <- gzcon(url(file_url))
txt <- readLines(con)
return(read.csv(textConnection(txt)))
}