When can they use ESC to close windows?

So, I coincidentally noticed the following behavior:

Some windows can be closed by pressing the esc key, like the Preference window in any app or the 'Safari help' window or the 'Fonts' window in TextExit for example.

Other windows can not be closed by pressing the esc key, like the Safari browser window or the 'Special Characters' window.

The default cmd w always seems to work.

What is the rule for keystrokes on closing windows? When are we allowed to use esc ?

What you're seeing, namely, the behavior where the Escape key closes the window, is automatically built in to the NSPanel class, which is a subclass of NSWindow.

From Window Programming Guide > How Panels Work:

A panel is a special kind of window, typically serving an auxiliary function in an application. The NSPanel subclass of NSWindow adds a few special behaviors to windows in support of the role panels play:

...

• If a panel is the key window and has a close button, it closes itself when the user presses the Escape key.

The preferences window (or panel) in many applications is implemented as an NSPanel: for example, Safari, Preview, Terminal, Console. The Fonts panel in Cocoa apps is implemented as an NSPanel. When you choose Safari > Safari Help, it actually launches an invisible background application, HelpViewer.app (/System/Library/CoreServices/HelpViewer.app), whose main window is implemented as a floating NSPanel. For these reasons, those panels will respond to the Escape key by closing the window. Standard general windows, which are instances of NSWindow, don't automatically get this behavior, as only panels are meant to behave that way.

Using the esc key to close a modal dialog is usually the same as pressing the Cancel button for the dialog. This is similar to how pressing the Return or Enter keys confirms the action for the button that is highlighted.

This has been in Apple's Human Interface Guidelines for Macintosh since the very beginning and is not new to OS X. The guideline can be found here under the "Dismissing Dialogs" heading:

https://developer.apple.com/library/mac/documentation/userexperience/conceptual/applehiguidelines/Windows/Windows.html#//apple_ref/doc/uid/20000961-TPXREF56

Search for "escape" on that page. The match is about 3/4 of the way down the page and is explicit about this behavior.

From the Human Interface Guidelines:

In general, include a Cancel button. The Cancel button returns the computer to the state it was in before the dialog appeared. It means “forget I mentioned it.” Also, make sure that the keyboard shortcut Command-period and the Esc (Escape) key are mapped to the Cancel button.

Another useful behavior, is if a dialog accepts Return or Enter as part of a text field, very often pressing Command-Return will press the default button.