How to "safely" shutdown/restart a Galera cluster?

For a graceful shutdown of the cluster, first verify status of your cluster. For each node check the status. Then if the status is synced then you can shut down the node one at a time. What is tricky is when starting the nodes back it needs to re-create the cluster as shutting down destroys the cluster. If all nodes in the cluster are synchronized (that is it contain the same positive "seqno" values) then any node can start the new cluster. If possible I would test this heavily before running on production.

I know it is late reply to this question. But just in case any one looking for the exact answer about how to safely shutdown and restart the mariadb galera cluster.

For example we have three mariadb galera nodes(1,2,3) running on ubuntu servers. To stop/shutdown the cluster in safe way without destroying the cluster:

1. Make sure no active transactions or connections against the cluster nodes.
2. On node3, run the following command to check whether the node is up to date: SHOW STATUS LIKE 'wsrep_local_state_comment'; you should see ' synced ' as return value
3. run the following command to stop mariadb service: sudo systemctl stop mariadb
4. On node2 and node1, repeat the same steps, first on node2, and then on node1.

Now you stopped the galera cluster in best way and to start again start from node1 as following:

1. on Node1 run following command: galera_new_cluster
2. Then on Node2 sudo systemctl start mariadb
3. on node3 sudo sytsemctl start mariadb