Finder window reappears if still open

I know of no way to do this if the Finder is the frontmost app at the start of your sequence. It's fairly standard practice that if you close the frontmost window in any app, the next behind it will come to focus.

However, if Finder is not the frontmost app, then Cmd ⌘ -clicking the red dot will close the window without bringing the Finder to the front.

There are a few Cmd ⌘ + action functions that will operate on a window without bringing it to the front - you can drag window edges to resize, minimise, close, etc.

You can kind of 'fake' what you want if you click on the Desktop first, then when you close your first window, the Desktop itself will be 'frontmost'... yet of course will not actually move.