How can I stop a worker destroying my town?
There is an option, under one of the menu screens (I don't have the game on this computer, so unfortunately I can't doublecheck where exactly) that does exactly what you seek:
"Automated workers leave existing improvements"
Which means once you've gotten your cities generally where you want them, you can automate the workers, who will finish up, build roads, etc, all the while leaving your own hand-placed improvements.