Visual Studio IDE - return to previous

Ctrl+- should do the trick which is the standard keyboard shortcut for Navigate Backwards. Or using the navigation buttons in the standard toolbar.

This is explained on MSDN at https://blogs.msdn.microsoft.com/zainnab/2010/03/01/navigate-backward-and-navigate-forward/

On this MSDN page, you can see the following explanations

The editor remembers locations when you move in a single command more than several lines away from where you are currently working, or if you edit in a particular location that is not adjacent to the last place you edited.

The goal is to remember interesting locations so that you can recall where you have been working without remembering so many locations the feature is not useful (such as every character typed, or every line entering several new lines of code one right after the other).

There was a command which did exactly you asking for: View.PopBrowseContext. It always returns you to the very place where you have pressed F12. It doesn't take in account any movements or edits you did after you have pressed F12.

By default in C# layout it is mapped to Ctrl+Chift+8, but you can assign it to any hotkey you want.

BTW, in VS2015 they decided to get rid of this command.

Update: The extension I've made to emulate this command is available on the marketplace now: PopContext.