Animated rgl graphs with knitr

I've got an example based on yours that works using knitr on Windows 8.1 within RStudio 0.99.441.

This produces a 40-frame animation of the plot. It uses the built-in hook_plot_custom to include the plots which are generated manually by animating the 3d plot. The code for the animation was based on those in the help and source of play3d and movie3d. movied3d itself cannot be used because it is too inflexible in its file naming.

I've put this up on github at https://github.com/NikNakk/testAnimateRgl/ . The pdf is at https://github.com/NikNakk/testAnimateRgl/raw/master/rglKnitr.pdf

\documentclass{article}
\usepackage{animate}

<< label = setup, include = FALSE>>=
library("rgl")
library("car")
library("knitr")
knit_hooks$set(rgl = hook_plot_custom)
@ 
\begin{document}

<< label=rgl1, rgl=TRUE, fig.show='animate', fig.width=5, fig.height=5, out.width='.6\\linewidth', dev='png', fig.num = 40, interval=0.1>>=
scatter3d(prestige ~ income + education, data=Duncan)
M <- par3d("userMatrix")
par3d(windowRect = 100 + opts_current$get("dpi") *
        c(0, 0, opts_current$get("fig.width"), 
        opts_current$get("fig.height")))
spinFunc <- par3dinterp(userMatrix=list(M,
                             rotate3d(M, pi/2, 1, 0, 0),
                             rotate3d(M, pi/2, 0, 1, 0)))
for(i in 1:40) {
  par3d(spinFunc(i / 10))
  Sys.sleep(0.05)
  rgl.snapshot(fig_path(".png", number = i), fmt = "png")
}
@

\end{document}

Edit: New version

Here's another version which demonstrates the use of custom chunk options to set the parameters for the rather simpler spin3d. Note that with this version, the chunk is just a single line (the scatter3d plot). spin3d.axis is used to set the axis parameter to spin3d; spin3d.rpm is used to set the rpm parameter. The number of images and the interval between images is set using the standard fig.num and interval parameters.

\documentclass{article}
\usepackage{animate}

<< label = setup, include = FALSE>>=
  library("rgl")
library("car")
library("knitr")
hook_rgl_spin <- function(before, options, envir) {
  if (!before) {
    par3d(windowRect = 100 + options$dpi *
          c(0, 0, options$fig.width, 
            options$fig.height))
    if (!is.null(options$spin3d.axis)) {
      spin3d.axis <- options$spin3d.axis
    } else {
      spin3d.axis <- c(0, 0, 1)
    }
    if (!is.null(options$spin3d.rpm)) {
      spin3d.rpm <- options$spin3d.rpm
    } else {
      spin3d.rpm <- c(0, 0, 1)
    }
    spinFunc <- spin3d(axis = spin3d.axis, rpm = spin3d.rpm)
    for(i in 1:options$fig.num) {
      par3d(spinFunc(i * options$interval))
      Sys.sleep(0.05)
      rgl.snapshot(fig_path(".png", number = i), fmt = "png")
    }

    hook_plot_custom(before, options, envir)
  }
}
knit_hooks$set(rgl = hook_rgl_spin)
@ 
  \begin{document}

<< label=rgl1, rgl=TRUE, fig.show='animate', fig.width=5, fig.height=5, out.width='.6\\linewidth', dev='png', fig.num = 40, interval=0.1, spin3d.axis=c(0, 0, 1), spin3d.rpm = 20>>=
  scatter3d(prestige ~ income + education, data=Duncan)
@

  \end{document}