Moving the SecondaryName Node in a Cloudera HBase Cluster

Solution 1:

It should be safe enough to simply remove the Secondary Namenode role, and then add it back again onto a different node on the cluster. In the intermediate period, you may see a warning from Cloudera Manager that the role doesn't exist (which may cause long term issues with the namenode), but not having a Secondary doesn't put your data any more at risk.

Solution 2:

The job of the 2NN is to read the changes on the HDFS filesystem and add them to the fsimage. This decreases NN startup time as during startup the NN reads the fsimage file and then applies all the interim log changes on top of it. The naming is somewhat unfortunate as it is indeed not a standby/backup NN but only a utility to increase NN performance.

  • In CM there's an option to "Roll Edits" (depending on your CM version) on the 2NN, see also "Checkpointing". Make sure to do that before moving.

  • To be safe stop all services

  • Move the 2NN role to a new machine.

  • Restart all services

  • (Optional but worth it): Implement HA