Under what circumstances can I use rundll32 to invoke a function in a DLL?

There are almost certainly going to be problems.

What functions can I use rundll32 on?

Well, optimally, you wouldn't use it at all—it's deprecated:

Rundll32 is a leftover from Windows 95, and it has been deprecated since at least Windows Vista because it violates a lot of modern engineering guidelines. If you run something via Rundll32, then you lose the ability to tailor the execution environment to the thing you’re running.

rundll32 is only designed to call a very specific type of function, specifically, functions that create a window and process window messages. This Microsoft document on the rundll32 interface explains that the function you're calling must take an HWND, a HINSTANCE, a LPSTR, and an int (in that order), and must use the _stdcall/CALLBACK calling convention.

Consult the MSDN documentation on the function you want to call to see whether it has that signature. For example, ExitWindowsEx doesn't fit. In fact, I can't find a single function with that signature that's documented.

If you're not sure what all of that means, you can skip the next section and go on to the final one.

What can go wrong if I use it on other functions?

Remember that whole function signature thing? rundll32 assumes the function you're trying to call has the right signature, no matter whether that's actually the case. When its assumption is wrong, seriously bizarre stack-related stuff can go on. (The stack is a memory location that stores parameters passed between functions.)

Mismatched function signatures will result in the data provided by rundll32 being interpreted as something different because of where the function expects its parameters to be on the stack. The first datum rundll32 supplies is a window handle, which are assigned in a nondeterministic fashion. Therefore, if you misuse rundll32, you're effectively passing random data as the first parameter. Further reading: What can go wrong when you mismatch the calling convention?

The rest of the parameters don't work out any better. Even if you supply a command line to the function via rundll32, it's not going to get parsed into the data types appropriate for the function. Instead, rundll32 just takes the entire "extra" text, jams it into a string variable, and passes a pointer to that data to the function as a later parameter. It's up to the function to parse the text. Therefore, there's no hope of ever invoking arbitrary functions with user-supplied arguments with rundll32.

It's not a complete disaster if the target function expects no parameters, but you're still messing up the stack. There is code in rundll32 to deal with that, but that doesn't give you an excuse to keep abusing it.

If you're using rundll32 to invoke a compatible function in a third-party DLL that's designed to be used this way, you could still have problems. rundll32 always opts in to all new Windows features (e.g. Data Execution Prevention), which can introduce changes that the third-party code wasn't designed for; that's why the behavior is opt-in.

Fortunately, there's a better way to do what you want.

What can I do instead?

Trying to lock the workstation (user32.dll,LockWorkStation)? Just run tsdiscon.

Trying to shut down the system (user32.dll,ExitWindowsEx)? Use the shutdown utility with the appropriate flags—see shutdown /? for help.

Are you an application developer, and trying to invoke a DLL function of yours without writing a new EXE? Write that EXE, probably in C++.

Trying to run something else? My open-source utility SprintDLL supports user-specified parameter lists, several calling conventions, and scripting to work with multiple functions and their outputs. If you prefer to only use Windows-included tools, you can do P/Invoke with PowerShell.

This question and answer were written as a reference usable for correcting materials that suggest using rundll32 in a dubious fashion. If you see people abusing rundll32, give them a link to this page!