SSH escape key ("~") only works when connection is stuck?

When I have an unresponsive SSH connection, I can kill it with <enter>~.. However, when the connection is responding, the ~ escape doesn't work. It just prints a tilda at the console.

So if I want to modify SSH port forwarding and press <enter>~C<enter>, all I get is:

~C: command not found

(From bash, not from ssh.)

What do I need to do so that the SSH escape key works properly?

EDIT: I have found a big clue: the remote shell was actually ash, not bash. When I run bash on the remote machine, the SSH escape key works! When I run ash inside bash inside ash, again, it doesn't work!

But this is very strange. The escape key should be caught by the SSH client and not even forwarded to the remote shell. So why should it matter exactly which remote shell is receiving input from SSH?

Simple workaround: run the cat command, then enter the escape sequence.

The cat command will by default print what is passed in stdin, so while it's running there will be no escape characters sent and you can use the ssh escape key as normal. When done just ctrl-c out of cat back to the shell.

Example If needed, open up a prompt and execute cat by typing cat and pressing enter.

$ 
$ cat 

Now type ~?

~?
Supported escape sequences:
 ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~C   - open a command line
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice

It works! Now just enter any command. Then to return to prompt press control-C.

^C
$

I've figured out the secret!

As I posted in the "edit" above, the remote shell was BusyBox ash, not bash.

From libbb/lineedit.c:2336-2338, in the BusyBox sources:

/* Print out the command prompt, optionally ask where cursor is */
parse_and_put_prompt(prompt);
ask_terminal();

That is used to print out the command prompt in ash. But notice, as soon as it prints the prompt, another function called ask_terminal is called. What does ask_terminal do? It prints out the following characters: <ESCAPE>[6n.

You never see those characters in your terminal. Actually, they are an ANSI terminal control escape code. <ESC>[6n is a "Query Cursor Position" command -- it tells the terminal emulator to send back another ANSI escape code, which tells the shell where the cursor (text insertion point) is located in the terminal window.

So as soon as you press Enter, ash prints out <ESC>[6n, and sshd passes that back to ssh and from there to the terminal emulator. Immediately, before you can press ~, your terminal emulator sends something like <ESC>[47;13R to standard input, and ssh passes that over the connection to sshd and from there to ash, telling ash where your cursor is.

Now, the SSH client doesn't actually know what those ANSI escape codes mean. To SSH, they are all just characters read from standard input. Rather than seeing <ENTER>~C, the SSH client sees <ENTER><ESC>[47;13R~C, and since it doesn't see the ~ right after Enter, it doesn't think that it is an escape code.

The question is what to do about this. It would be nice if OpenSSH understood those ANSI escapes sent by the terminal and would still accept the ~ escape character after an ANSI terminal control command. I may send the OpenSSH guys a patch and see if they are interested in fixing this...