How do you remove a colonized habitat?

There is no way to do this. Habitats cannot be destroyed by you.

However, there is one option, though this is only realistic in a multiplayer game.

There is the war demand "disassemble Habitat", available to xenophobes or any empire with access to purging populations and cleansing planets. If it is enforced at the end of the war, the chosen habitat will be destroyed.

Relying on the AI to pick this war demand is not a good idea, but if it is multiplayer, you could ask another player to declare war on you, demand the disassembly of your habitats, you surrender and all your habitats are gone.