Why is vim leaving temporary file versions all over the place?

Solution 1:

You can disable backup files putting this line in your vimrc file:

set nobackup

But if you still want to backup your files, centralize it to a directory with:

set backupdir=~/tmp

for example.

Solution 2:

First of all I assume those files […] are indeed serving the purpose of temporary backups while editing.

No. They are non-temporary backups made when saving.

The temporary files created while editing are (on UNIX-alike platforms) named .wibble.swp (for a file named wibble).

:help backup
in vim itself gives you the gen on the former, including the rules for when backup files are created and when they are not.
:help swap
gives you the gen on the latter.