How do you clear a custom destination?
Click an empty spot on the map, and you'll have the options: Move It, Leave It, Remove It.
I just figured it out by accident. It's so simple that I didn't think to try it. You just click somewhere else on the map and it will ask you if you want to keep your marker or remove it.