How do you make a button turn a piston off?
Make a NOT Gate. This type of simple redstone logic circuit is essential for creating a circuit that turns something OFF when activated, instead of ON. Here is a picture:
Just add a button and maybe a repeater for delay, and it will turn off the redstone torch, turning the power OFF.