How to cancel Ctrl+Tab navigation?

Alt-Tab is handled by the operating system, which implements the behavior you're describing for the escape key. The application that currently has focus will never see it.

However, Ctrl-Tab is handled by the application, and that behavior is determined by the decisions made by the application's developers. That will do anything, but typically will select between multiple sub-windows or tabs because this is the behavior a user expects.

Pressing the escape key while holding down control is also handled by the operating system, not the application, and the application will not see that keypress, therefore cannot react to it.

If you decide after hitting Ctrl-Tab that you'd like to not change tabs, hitting Ctrl-Shft-Tab should back you up the list of tabs to where you just were.

I know this is an old question. However, in new version of navigators, you can move between tabs with Ctrl-PageUp/PageDown (so Ctrl-PageDown is like Ctrl-Tab). Moreover, in the latest version of Firefox, the default behavior of Ctrl-Tab has been changed: it is used to switched between recently used tabs. So just press Ctrl-Tab again to go back to the last tab.

This default behavior is of course could be changed to go back to the old behavior.