Read gzipped csv directly from a url in R

Solution 1:

I am almost certain I answered this question once before. The upshot is that Connections API of R (file(), url(), pipe(), ...) can do decompression on the fly, I do not think you can do it for remote http objects.

So do the very two-step you have described: use download.file() with a tempfile() result as second argument to fetch the compressed file, and then read from it. As tempfile() object, it will get cleaned up automatically at the end of your R session so the one minor fix I can suggest is to skip the unlink() (but then I like explicit cleanups, so you may as well keep it).

Edit: Got it:

con <- gzcon(url(paste("http://dumps.wikimedia.org/other/articlefeedback/",
                       "aa_combined-20110321.csv.gz", sep="")))
txt <- readLines(con)
dat <- read.csv(textConnection(txt))

dim(dat)
# [1] 1490   19

summary(dat[,1:3])
# aa_page_id       page_namespace                 page_title  
# Min.   :     324   Min.   :0      United_States        :  79  
# 1st Qu.:   88568   1st Qu.:0      2011_NBA_Playoffs    :  52  
# Median : 2445733   Median :0      IPad_2               :  43  
# Mean   : 8279600   Mean   :0      IPod_Touch           :  38  
# 3rd Qu.:16179920   3rd Qu.:0      True_Grit_(2010_film):  38  
# Max.   :31230028   Max.   :0      IPhone_4             :  26  
# (Other)              :1214  

The key was the hint the gzcon help that it can put decompression around an existing stream. We then need the slight detour of readLines and reading via textConnection from that as read.csv wants to go back and forth in the data (to validate column width, I presume).

Solution 2:

Using data.table::fread :

x <- data.table::fread("http://dumps.wikimedia.org/other/articlefeedback/aa_combined-20110321.csv.gz")

dim(x)
[1] 1490   19

x[, 1:2]
#       aa_page_id page_namespace
#    1:   26224556              0
#    2:      31653              0
#    3:   26224556              0
#    4:   26224556              0
#    5:    1058990              0
#   ---                          
# 1486:     619464              0
# 1487:   19283361              0
# 1488:   19006979              0
# 1489:    5078775              0
# 1490:   30209619              0

Solution 3:

This function generalizes Dirk's answer:

R <- function(file_url) {
  con <- gzcon(url(file_url))
  txt <- readLines(con)
  return(read.csv(textConnection(txt)))
}