How to properly configure a 2 node glusterfs system?
I am trying to make a 2 node linux servers with high available apache using glusterfs 3.7.6 for data replication and pacemaker+corosync as resource manager. However i'm seeing a problem with gluster in the specific scenario when both nodes get shutdown and one of them comes online first. Even though there's a brick on that node and gluster service is running, there's no brick process.
[root@node1 ~]# gluster volume status data
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data N/A N/A N N/A
NFS Server on localhost N/A N/A N N/A
NFS Server on localhost N/A N/A N N/A
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
And when I start the other node, everything seems to work and I can mount the volume.
[root@node1 ~]# gluster volume status data
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data 49152 0 Y 2674
Brick node2:/gluster_data 49152 0 Y 3086
NFS Server on localhost N/A N/A N N/A
Self-heal Daemon on localhost N/A N/A Y 2796
NFS Server on node2 N/A N/A N N/A
Self-heal Daemon on node2 N/A N/A Y 3085
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
And at this point if I shut down node2, node1 brick process remains active so atleast I can mount and use it.
[root@node1 ~]# gluster volume status data
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data 49152 0 Y 2674
NFS Server on localhost N/A N/A N N/A
Self-heal Daemon on localhost N/A N/A Y 2796
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
So my observation is that for the gluster volume to work, both nodes need to be online atleast for a moment so the bricks can start and then if one node goes down, it won't affect the volume operation. So how can I make it work when one of the nodes come online from a total blackout ?
The problem any cluster-node has when coming up from a full stop is:
Do I have the latest state, or not? I don't want to claim
latestif I'm behind the other down nodes.
This is why clustering very often includes some kind of quorum mechanism, so existing nodes can vote on state and converge on a consensus. Two node clusters can't use this mechanism, since there will never be a 'majority' partition. In the 3.7 release, Gluster gained a quorum feature.
http://gluster.readthedocs.org/en/release-3.7.0beta1/Features/server-quorum/
In that document, they state that 2-node clusters can't use it for the very reason I described above.
In your case, you may want to consider creating some management-only nodes in your Gluster setup. These would be peers that are probed into the cluster, but don't host any storage. Their entire reason to exist would be to maintain cluster-state. These can exist in different racks, datacenters, power phases, to try and ensure they are in a different fault-domain than the storage bricks. This will increase the number of members in a cluster, and will improve your chances of having a majority partition if one of the storage-bricks comes up without the other one.
Unfortuonately, the behavior you're seeing is working as designed and can't be changed without adding more servers into the cluster.