How to switch back to previous window on OS X?
Before this is moved to superuser, the shortcut you're looking for is command-`
Command-Tab works nicely between apps, but how to do the same within one application having several windows open?
I was also a bit annoyed at this, until I found the "Move focus to the next window in application" keyboard shortcut option (under System Preferences -> Keyboard -> Keyboard Shortcuts -> Keyboard & Text Input).
I set it to Cmd-§ (⌘§) myself (which is close to Cmd-Tab and I couldn't think of anything better). Shift-Cmd-§ cycles in the opposite direction. It's not ideal, but sort of tolerable for me, although I'd still prefer Cmd-Tab toggling between all windows.
There are also 3rd party utilities that improve OS X's window switching capabilities; check out Witch for instance. I haven't tried these myself though.
When you have to window active (focused), do Command-~.
Hyperswitch (free) and Witch ($14, though more configurable) allow doing this.
If you want to make a custom shortcut, I noticed that macOS Sierra seems to automatically map Shift-[shortcut here] to the reverse of your original shortcut, when it makes sense (e.g. anything that involves "next" probably has a corresponding "previous" version). Possibly because someone expected that a user would expect this to happen?
So if you look in System Preferences > Keyboard > Shortcuts > Keyboard you'll only find "Move focus to next window" bound by default to Command-` and this is why Command-Shift-` triggers "Move focus to last window."
Set "Move focus to next window" to any other combination that doesn't involve Shift, and Shift-[that combination] will do "Move focus to last window".