Calculate the mean of one column from several CSV files

I have over 300 CSV files in a folder (named 001.csv, 002.csv and so on). Each contains a data frame with a header. I am writing a function that will take three arguments: the location of the files, the name of the column you want to calculate the mean (inside the data frames), and the files to use in the calculation.

Here is my function:

pollutantmean2 <- function(directory = getwd(), pollutant, id = 1:332) {

    # add one or two zeros to ID so that they match the CSV file names
    filenames <- sprintf("%03d.csv", id)

    # path to specdata folder
    # if no path is provided, default is working directory
    filedir <- file.path(directory, filenames)

    # get the data from selected ID or IDs from the specified path
    dataset <- read.csv(filedir, header = TRUE)

    # calculate mean removing all NAs
    polmean <- mean(dataset$pollutant, na.rm = TRUE)

    # return mean
    polmean

}

It appears there are two things wrong with my code. To break it down, I separated the function into two separate function to handle the two tasks: 1) get the required files and 2) calculate the mean of the desired column (aka pollutant).

Task 1: Getting the appropriate files - It works as long as I only want one file. If I select a range of files, such as 1:25 I get an error message that says Error in file(file, "rt") : invalid 'description' argument. I have Googled this error but still have no clue how to fix it.

# function that obtains csv files and stores them
getfile <- function(directory = getwd(), id) {
    filenames <- sprintf("%03d.csv", id)
    filedir <- file.path(directory, filenames)
    dataset <- read.csv(filedir, header = TRUE)
    dataset
}

If I run getfile("specdata", 1) it works fine, but if I run getfile("specdata", 1:10) I get the following error: Error in file(file, "rt") : invalid 'description' argument.

Task 2: Calculating mean of specified named column - Assuming I have a usable data frame, I then try to calculate the mean with the following function:

calcMean <- function(dataset, pollutant) {
    polmean <- mean(dataset$pollutant, na.rm = TRUE)
    polmean
}

But if I run calcMean(mydata, "sulfate") (where mydata is a data frame I loaded manually) I get an error message: Warning message: In mean.default(dataset$pollutant, na.rm = TRUE) : argument is not numeric or logical: returning NA

The odd thing is that if I run mean(mydata$sulfate, na.rm = TRUE) in the console, it works fine.

I have researched this for several days and after endless tweaking, I have run out of ideas.

Solution 1:

You do not need more functions. The solution can be simpler from my understanding in 6 lines:

pollutantmean <- function(directory, pollutant, id = 1:10) {
filenames <- sprintf("%03d.csv", id)
filenames <- paste(directory, filenames, sep="/")
ldf <- lapply(filenames, read.csv)
df=ldply(ldf)
# df is your list of data.frames
mean(df[, pollutant], na.rm = TRUE)
}

Solution 2:

I think your major problem is listing the files in your working directory and reading them into R. Try list.files function in R Example code which may work for you is

  files <- list.files(pattern = ".csv") ## creates a vector with all file names in your folder
polmean <- rep(0,length(files))
for(i in 1:length(files)){
   data <- read.csv(files[i],header=T)
   polmean[i] <- mean(data$pollutant)
 }
result <- cbind(files,polmean)
write.csv(result,"result_polmeans.csv")

This program gives you the data with name of file in the first column and corresponding means in the second column.