Documentation As-A-Manual vs. Documentation As-A-Checklist

When writing mine I've always devolved into writing two three sets. The get-er-done checklist, with a MUCH LONGER appendix about the architecture of the system including why things are done the way they are, probable sticking points when coming online, and abstract design assumptions. followed by a list of probable problems and their resolutions, followed by a longer section with information about how a system works, why it does it that way, and other information useful for pointing people in the right direction should something unique happen.

At my last job we were required to write doc so that even level-1 helpdesk people could bring things back up. This required checklists, which generally became out of date within 3 months of the writing. We were strongly urged to write troubleshooting guides whenever possible, but when the contingency tree gets more than three branches in it, you just can't write that doc without going abstract.

When leaving my last job, I turned in a 100 page 'how to do my job' manual before I left. It had the abstract stuff in it, design philosophy, as well as integration points. Since I was presumably writing for another sysadmin who was going to replace me, I aimed it at someone who could take abstract notions and turn them into concrete actions.

Five years have passed and I find my opinion on this has shifted somewhat. Both Document as Manual and Document as Checklist have very valuable places in the hierarchy of documentation and both need to be produced. They target very different audiences, though.

Document as Checklist

The target market for this kind of documentation are coworkers who want to how how to do a thing. They come in two types:

• Coworkers who just want to know how to do a thing and don't have time to thumb through a fifteen page manual and figure out the steps for themselves.
• Procedures that are fairly complex in steps, but only need to be run once in a while.

Impatience is the driver for the first kind. Maybe your coworker doesn't actually want to know why the output has to be piped through a 90 character perl regex, just that it has to be in order to close the ticket. Definitely include a statement like, "For a detailed explanation for why this workflow looks like this, follow this link," in the checklist for those that do want to know why.

The second point is for procedures that aren't run often but contain pitfalls. The checklist acts as a map to avoid the Certain Doom of just winging it. If the checklist is kept in a documentation repo, it saves having to search email for the time the old admin sent out a HOWTO.

In my opinion good checklist-documentation also includes sections on possible failure points, and responses to those failures. This can make the document rather large and trigger TL;DR responses in coworkers, so I find that making the failure-modes and their responses a link from the checklist rather than on the page itself makes for an unscary checklist. Embrace hypertextuality.

Document as Manual

The target market for this kind of documentation are people who want to learn more about how a system works. The how-to-do-a-thing style documentation should be able to be derived from this documentation, but more commonly I see it as a supplement to checklist-style documentation to back up the decisions made in the workflow.

This is the documentation where we include such chewy pieces like:

• Explaining why it's configured this way.
  • This section may include such non-technical issues like the politics surrounding how the whole thing was purchased and installed.
• Explaining common failure modes and their responses.
• Explaining any service-level-agreements, both written and de facto.
  • De facto: "if this fails during finals week it's a drop-everything problem. If during summer break, go back to sleep and deal with it in the morning."
• Setting out upgrade and refactoring goals.
  • The politics may be different later, why don't we fix some of the bad ideas that introduced in the beginning?

Which are all very useful for obtaining a comprehensive understanding of the whole system. You don't need a comprehensive understanding to run simple human-automation tasks, you need it to figure out why something broke the way it did and have an idea where to make it not do that again.

You also mentioned Disaster Recovery documentation that has to be a checklist.

I understand, you have my sympathies.

Yes, DR documentation does need to be as checklist-like as possible.
Yes, DR documentation is the most resistant to checklisting due to how many ways things can break.

If your DR checklist looks like:

1. Call Dustin or Karen.
2. Explain the problem.
3. Stand back.

You have a problem. That is not a checklist, that is an admission that the recovery of this system is so complex it takes an architect to figure out. Sometimes that's all you can do, but try to avoid it if at all possible.

Ideally DR documentation contains procedure checklists for a few different things:

• Triage procedures to figure out what went wrong, which will help identify...
• Recovery procedures for certain failure-cases. Which is supported by...
• Recovery scripts written well beforehand to help minimize human error during recovery.
• Manual-style documentation about the failure cases, why they occur and what they mean.

Triage procedures are sometimes all the DR documentation you can make for some systems. But having it means the 4am call-out will be more intelligible and the senior engineer doing the recovery will be able to get at the actual problem faster.

Some failure cases have straight-forward recovery procedures. Document them. While documenting them you may find cases where lists of commands are being entered in a specific order, which is a great use-case for scripting; it can turn a 96 point recovery procedure into a 20 point one. You'll never figure out if you can script something until you map the recovery procedure action by action.

Manual-style documentation for failure cases is the last ditch backstop to be used when there ARE no recovery procedures or the recovery procedures failed. It provides the google-hints needed to maybe find someone else who had that problem and what they did to fix it.

Actually neither, we use Documentation As-a-Testcase

That being said we have written documentation that goes with Documentation As-a-Manual. We had checklists in place but when growing we found them to be error prone and really failing on the system as a whole.

We do however have kind of "Documentation As-a-Checklist" installed, that is - as mentioned above - we extensively monitor our services. There's a saying:

If you aren't monitoring it you aren't managing it

That is so totally true, but another one should be:

If you aren't monitoring it you aren't documenting it

Whenever we need to migrate services we just keep the "Service Group", "Host Group", whatever applies (we use Nagios, as you can guess from the vocabulary) open and it isn't migrated as long as there's a single red point on any of the services.

The tests provide a much better checklist than any hand written check list could provide. It's actually self documenting, as soon as we have some failure that wasn't monitored yet the test will at least be added to Nagios, with that we get 2 Things for free:

• we know when it fails
• another point on the checklist

The "real" documentation is kept in a Wiki mentioning the odds and ends of the specific service or test. If it's missing people will add it as soon as we need to do some migration or need to do some work with it, so far that approach has worked very good.

Also erronous documentation is ironed out really fast, everytime something fails people start looking up the documentation and try to resolve the issue with the HowTos in there, if it's wrong just update it with your findings.

Just think of the errors one could possibly create by following a checklist and not having installed any tests that will give you a green checkbox after applying them. I don't think it's possible to seperate Documentation from Monitoring.

It depends on the target audience for your documentation.

For helpdesk (level 1) types, a checklist is the correct way to go; of course, this presumes that there are higher levels of support with the deeper knowledge you describe.

If the documentation is for the systems group, I always err on the side of more documentation. It's hard enough to have adequate documentation just to get by, if someone (yourself) wants to write more extensive docs with the requisite background information -- no sane individual should stand in your way!

Personally I try and keep documentation as simple as possible. It tends to include:

• What [X] is supposed to do (requirements).
• How [X] has been structured at a high level (design).
• Why I implemented [X] in the way I did (implementation considerations).
• How the implementation of [X] is non-standard (workarounds).
• Common issues with [X] and how to resolve them (issues).

So admittedly my common issues section is likely to be a brief description of what has happened and dot point walkthroughs on how to fix it.

I usually assume some knowledge from the reader of the system in question (unless it is particularly arcane). I don't have time to make most of my technical documentation level 1 or management readable - but a cluey level 3 should be fine.