How to add a number of observations per group and use group mean in ggplot2 boxplot?

Is this anything like what you're after? With stat_summary, as requested:

# function for number of observations 
give.n <- function(x){
  return(c(y = median(x)*1.05, label = length(x))) 
  # experiment with the multiplier to find the perfect position
}

# function for mean labels
mean.n <- function(x){
  return(c(y = median(x)*0.97, label = round(mean(x),2))) 
  # experiment with the multiplier to find the perfect position
}

# plot
ggplot(mtcars, aes(factor(cyl), mpg, label=rownames(mtcars))) +
  geom_boxplot(fill = "grey80", colour = "#3366FF") +
  stat_summary(fun.data = give.n, geom = "text", fun.y = median) +
  stat_summary(fun.data = mean.n, geom = "text", fun.y = mean, colour = "red")

Black number is number of observations, red number is mean value. joran's answer shows you how to put the numbers at the top of the boxes

hat-tip: https://stackoverflow.com/a/3483657/1036500

I think this is what you're looking for maybe?

myboxplot <- ddply(mtcars,
                    .(cyl),
                    summarise,
                    min = min(mpg),
                    q1 = quantile(mpg,0.25),
                    med = median(mpg),
                    q3 = quantile(mpg,0.75),
                    max= max(mpg),
                    lab = length(cyl))
ggplot(myboxplot, aes(x = factor(cyl))) + 
    geom_boxplot(aes(lower = q1, upper = q3, middle = med, ymin = min, ymax = max), stat = "identity") + 
    geom_text(aes(y = max,label = lab),vjust = 0)

I just realized I mistakenly used the median when you were asking about the mean, but you can obviously use whatever function for the middle aesthetic you please.