How to setup Azure traffic manger to switch between a website and a down (maintenance) page, without disruption?

In your scenario you want to use the priority routing method, this will use the endpoint with the highest priority unless it is not available then it will use the low priority. You can then switch the priority when you need to switch to the maintenance page.

Using performance routing will result in people hitting your maintenance page when it is quicker for them to reach that than your main page.

Traffic manager is a DNS based routing tool, so there will always be some delay when you make the change, there isn't really a way around this. If you need quicker response time then you will want to look at something like Front Door, which is not DNS based.