How do I hide empty folders in VSCode after switching Git branches?

I am refactoring a medium-sized code base in Visual Studio Code. I create a new branch, create some new folders, and move some files there. Whenever I commit my work and switch back to the main branch, the files in those folders are gone (as expected), but the empty folders remain!

It seems VSCode does not understand that these directories are part of the branch I am working on. Is there any way to solve this issue?

The problem is: VSCode will not do any type of git cleanup (git clean -fdxs for instance), because that might remove too many untracked elements (not just your empty folders from your other branch).
That would lead to some nasty surprises on each branch switch.

So said empty folders remain in your working tree and, as described here, any git clean should be done carefully.