How do I delete a folder that's in use?

Sometimes when I'm working in Windows I'll get this prompt when I try to delete a directory:

Folder In Use
The action can't be completed because the folder is open in another program
Close the folder and try again.

However, the folder isn't "in use" at all, and I don't have it open. So how can I delete it short of rebooting?

This answer is old, but seems to still get traffic. As noted below, there is a built in tool in Windows now:

Start → All Programs → Accessories → System Tools → Resource Monitor (or Run resmon.exe)

You can get more info in the answers below this one. The rest of this answer is no longer as relevant.

Process Explorer from Sysinternals will let you search through the handles and DLLs. Search for the directory and find out which process is "using" the folder.

From there, you have the ability to either shut down that process, or just close that particular file handle. Be sure about what you're doing when working with the file handle directly, and make sure it's not being used by some important process.

There's a native GUI for Windows:

Start>>All Programs>>Accessories>>System Tools>>Resource Monitor (or Run resmon.exe)

You can search for the "Associated Handles" using the searchbox (circled in red), and right click the process you want to end.

As an example, in the image below I could not delete my Eclipse directory. Searching for the Eclipse associated handles showed that the adb.exe had a handle to the directory. After ending the adb process, I could then delete the Eclipse directory.

Unlocker and WhoLockMe don't work on 64-bit Windows. I use LockHunter.

When using tools that forces file handles closed (like the previously-mentioned Unlocker), keep in mind that doing so might cause serious problems. (See this article by Raymond Chen.) Make sure you know which applications are using the file and that none of them are doing anything important.

It's safer to try shutting down the applications first instead of just forcing the handle closed.