When renaming a file in Windows, CTRL + Backspace does not delete whole words [duplicate]
So the reason for that happening (the box appearing) is that it's a control character. It's one of the several ASCII non-printable characters. Look into AutoHotKey to override the default behavior of the Control and Delete keys. Have a look at this here