How do I call the `function` function?

This is because function is a special primitive:

typeof(`function`)
#> [1] "special"

The arguments are not evaluated, so you have actually passed quote(formals(mean)) instead of the value of formals(mean). I don't think there's a way of calling function directly without evaluation tricks, except with an empty formals list which is just NULL.

For completeness’ sake, Lionel’s answer hints at a way of calling `function` after all. Unfortunately it’s rather restricted, since we cannot pass any argument definition except for NULL:

mean5 = `function`(NULL, mean(x, ...))
formals(mean5) = formals(mean)

(Note the lack of quoting around the body!)

This is of course utterly unpractical (and formals<- internally calls as.function anyway.)

After digging a little bit through the source code, here are a few observations:

1. The actual function creation is done by mkCLOSXP(). This is what gets called by function() {}, by as.function.default() and by .Primitive("function") (a.k.a. `function`)

2. as.function.default() gets routed to do_asfunction(), which also calls CheckFormals(). However, it directly constructs these formals a few lines above that.

3. As you pointed out, the other place where CheckFormals() gets called is inside do_function(). However, I don't think do_function() gets called by anything other than .Primitive("function"), so this is the only situation where CheckFormals() is called on the user's input.

4. CheckFormals() does actually correctly validate a pairlist object.

You can check the last point yourself by running parts of the CheckFormals() function using inline::cfunction

inline::cfunction( c(x="ANY"),
  'Rprintf("is list?: %d\\nTag1 OK?: %d\\nTag2 OK?: %d\\nTag3 NULL?: %d\\n",
     isList(x), TYPEOF(TAG(x)) == SYMSXP, TYPEOF(TAG(CDR(x))) == SYMSXP,
     CDR(CDR(x)) == R_NilValue); return R_NilValue;' )( formals(mean) )

# is list?: 1
# Tag1 OK?: 1
# Tag2 OK?: 1
# Tag3 NULL?: 1

So, somewhere between you passing formals(means) to .Primitive("function") and it getting forwarded to CheckFormals() by do_function(), the argument loses its validity. (I don't know the R source well enough to tell you how that happens.) However, since do_function() is only called by .Primitive("function"), you don't encounter this situation with any other examples.