Can you delete the contents of the downloads folder?

I noticed that the files and folders that were direct sub-folders of downloading were numbered.

state_17515_206.patch
17515/
state_17585_17581.patch
17585/

These correspond with the steam app ID's of the games that don't exist on my system (17585).

There are no other files which leads me to believe that this is just a bug in Steam (because it is cleaning up files for games I do have).
I.e. it will clean up downloads for games that it knows about, but any other files it may have put there will not be removed if the game is deleted.

I deleted the files, restarted steam, and nothing adverse happened.

It should be ok. If in doubt just move the contents to somewhere else temporarily and see if it has any adverse affect.