How to get DRBD to automatically start after reboot, mount volume, start service, and configure primary/secondary

You should definitely consider using a product which is made for dealing with such problems.

I described in this post [ Nagios/Icinga: Don't show CRITICAL for DRBD partitions on standby node ] how I manage to do exactly what you expect using opensvc, and it works fine for years.

• no need for fstab entries as mounts are described in the opensvc service configuration file, which is automatically synchronized between opensvc cluster nodes

• no need to setup update-rc.d drbd defaults because opensvc stack is modprobing the drbd module when it sees that you have drbd resources in some services, and then brings up drbd in primary/secondary states

• to reach primary/secondary at boot, just setup the nodes parameter in the DEFAULT section of the opensvc service configuration file.

  If you want server1 as primary and server2 as secondary, just set nodes=server1 server2 using the command svcmgr -s mydrbdsvc set --kw DEFAULT.nodes="server1 server2"

• if you only want the service to be started at boot on server1, set the orchestrate=start parameter using the command svcmgr -s mydrbdsvc set --kw DEFAULT.orchestrate=start

• if you only want the service to be orchestrated in high availability mode (meaning automatic failover between nodes), set the orchestrate=ha parameter using the command svcmgr -s mydrbdsvc set --kw DEFAULT.orchestrate=ha

• to relocate the service from one node to the other, you can use svcmgr -s mydrbdsvc switch command

There's a lot to unpack here, but I will start by saying: That DRBD version is old! You should upgrade it to either 8.4.11 (as of this writing, December 2018), or move to the 9 branch. But the questions you're asking can be solved with the version you're using.

Let's take a look at your three consice questions you posed at the bottom of your post:

1. What FSTAB entries are supposed to be there?

None, ideally. DRBD devices must first be promoted before they can be used. The fstab is not the best option even for DRBD-9 block devices that can autopromote, as in most cases it will trip up the boot process of things aren't all fine. But it can technically work with many caveats.

2. Why does update-rc.d drbd defaults not work?

Don't do it that way. Debian uses systemd, and DRBD has a systemd unit file. You should be using that UNLESS you're using a cluster manager, which I highly recommend. So in the absence of the Pacemaker cluster resource manager, you should be issuing things like # systemctl enable drbd to make it start at boot. Or # systemctl start drbd to start the service after it's stopped. The most common commands are start, stop, restart, enable and disable.

3. Why do I have to reset primary and secondary on both servers after each restart?

Because DRBD has no concept of leader elections. An external system must promote a DRBD resource to Primary, and there can only be one primary at a time. There are a lot of ways to do this.

You can "manually" promote a resource with # drbdadm primary <resource> and then mount things up from there -- but that's what you want to avoid. DRBD can "autopromote" in version 9, which will automatically promote a resource upon attempting to open its block device for access (like with a filesystem mount or a volume group activation) -- but the version you're running can't do that (upgrade? This might be enough for you). OR you could use a finite state leader election system to control promotion actions and ensure the state of both DRBD and the application stack it supports. That's Pacemaker.

You want Pacemaker, I promise. Pacemaker is not overly difficult, but it is extremely large. You won't have to learn much of it in order to accomplish what you want. You can spend a lot of time and energy making a "perfect" cluster that will resist any failure under the sun, and that effort would be rewarded. It's a good system.

A combination of Corosync/Pacemaker, DRBD-9.x or DRBD-8.4.x latest, and whatever you have on top of that should accomplish what you want automatically. There are lots of docs out there which detail how to do this. Here's one that's up to date:

linbit users-guide-9.0

I would recommend reading that entire guide if you have the time. DRBD has undergone some serious evolution in the past few years.