Is unit testing viable in game programming?

Often the code doesn't break itself into distinct units.

That's just poor design. The code doesn't break itself into anything. You, the designer, have to impose a structure on the code so that you can demonstrate that it actually works.

However, this function will likely return no value...

So?

...and have the side effect of, player.velocity.y = -height and checkCollisions(player).

Then test for that.

I can't think of a clear unit test to build around this.

Why not? You just gave an excellent specification for the results of the function.

You might need a few mock objects to replace the full-blown Player with a simplified MockPlayer that's easier to test with.

But your specification of behavior was perfect. Just test for the things you described.

Unit tests can be useful for a lot of the low level libraries you may use with game code but I seriously doubt it is of much use for the higher level stuff. Games are usually simulations and rely on massive amounts of shared state which can't meaningfully be mocked up and tested in isolation. Often functions in games do not return any sort of value that you can instantly check but instead set a process in motion which should complete at some point in the future. Testing such behaviour is worthwhile but requires a significantly different approach to the unit test idea of testing pieces of code in isolation.