How to switch select between buttons using only keyboard

I am trying to keep my fingers on the keyboard as much as possible and would like to be completely mouse-free in some applications, such as Sublime Text. E.g., I can initiate a file opening or closing using just the keyboard but I do not know how to navigate close or open file dialogs using the keyboard alone. More specifically, how do I switch between selected buttons? E.g. tabbing works in other OSs I am familiar with (e.g. Windows and Linux KDE) but here I cannot tab between Save and Cancel buttons when I want to close a file.

Using the tab key between save and cancel buttons works for OS X too. You only have to enable it:

1. Go to System Preferences
2. Go To Keyboard
3. Go to the tab Keyboard Short
4. On the bottom of the window, set the radio button to "All controls":

From now on, when you see a OK/Cancel window the OK button is blue and can be pressed using the return key, and the cancel button has a outline which can be pressed using the space bar:

But the real trick is: You are now able to press the tab key to move the outline to another button and hit the space bar to press the highlighted button.

A common shortcut for a Cancel button is Cmd.. (i.e. Command and period a.k.a. full stop).