What is the button in the upper right of windows?

Solution 1:

Yes, it toggles the toolbar in all applications. In many applications, it's shortcut is Option+Command+T, but this is not true for all (Preview uses Command+B).

Solution 2:

For future reference, the Window Elements topic of the Apple Human Interface Guidelines is the definitive reference here.

Solution 3:

In Finder windows, the lozenge also shows and hides the sidebar.