What does a JVM have to do when calling a native method?

Calling a JNI method from Java is rather expensive comparing to a simple C function call. HotSpot typically performs most of the following steps to invoke a JNI method:

1. Create a stack frame.
2. Move arguments to proper register or stack locations according to ABI.
3. Wrap object references to JNI handles.
4. Obtain JNIEnv* and jclass for static methods and pass them as additional arguments.
5. Check if should call method_entry trace function.
6. Lock an object monitor if the method is synchronized.
7. Check if the native function is linked already. Function lookup and linking is performed lazily.
8. Switch thread from in_java to in_native state.
9. Call the native function
10. Check if safepoint is needed.
11. Return thread to in_java state.
12. Unlock monitor if locked.
13. Notify method_exit.
14. Unwrap object result and reset JNI handles block.
15. Handle JNI exceptions.
16. Remove the stack frame.

The source code for this procedure can be found at SharedRuntime::generate_native_wrapper.

As you can see, an overhead may be significant. But in many cases most of the above steps are not necessary. For example, if a native method just performs some encoding/decoding on a byte array and does not throw any exceptions nor it calls other JNI functions. For these cases HotSpot has a non-standard (and not known) convention called Critical Natives, discussed here.