Backspace in insert mode in vi doesn't erase the character

Here is the simplest solution.
Open a terminal, go to home directory and type

vi .vimrc

a new file open now add these lines to the file and exit by saving

$ set nocompatible
$ set backspace=2

That's correct behavior for vi, and it does erase the character, it just doesn't show it by replacing it with a blank like in vim. It will be apparent when you overwrite the erased character with another character, or switch back to command mode.

E.g. starting with command mode on an empty line, the following will result in the line containing fo:

ifooBackspaceEsc

Before you hit the Esc key, the line will read foo, but the last o has been tagged as an "erase-column". See http://pubs.opengroup.org/onlinepubs/9699919799/utilities/vi.html#tag_20_152_13_88 (mainly point 4 under that heading, and also browse a little further down to read about <Control>-H)

You need to change to the "insert" mode by moving the cursor with i. There are other ways to delete characters directly. You can download a powerpoint that I use when teaching a class on VI here.

It sounds like you are in the "Command" mode. To move your cursor:
H = Left, J = Up, K = Down, L = Right

Once you have your cursor positioned you can delete text as follows:
x Deletes the character under the cursor
X Deletes the character before the cursor
dw Deletes from the cursor to the next word
dd Deletes the line the cursor is on.

To enter text, you can use one of the text entry modes.
a Adds text to the right of the cursor
A Adds text to the end of the current line
i Adds text to the left of the cursor
I Adds test to the beginin of the current line
o Opens a new line below the current line and places you in text entry mode
O Opens a new line Above the current line and places you in text entry mode

To exit Text entry mode, and return to the Command mode, use Esc.

To Undo changes: (A student favorite)
u Undo the last command entered
U Undo all changes to the ** current line**

To Save/Quit: :w Writes (Saves) the file and remains open
:wq Writes (Saves) the file and exits VI
:q Quits (Exits) if you've made no changes
:q! Quits (Exits) without saving changes
ZZ Writes (Saves) the file and exits VI (same as :wq)

Install the full vim package to get the backspace and arrow key functionality

sudo apt-get update
sudo apt-get install vim