Getting started with documentation

documentation

We haven't been doing any documentation at my workplace. I'm completely new to it and ask for some guidance getting started.

I have a few questions:

What are the essential documents that a sysadmin should write and maintain? And why are these so important?
How do you keep your documentation in sync with the system? How do you minimize duplication of information?
Recommended guides, best practices, anti-patterns?

since 2003 I'm documenting everything in our inhouse wiki.

Servers

hardware specs
waranty information
network information
and of course installed software and configuration

Workflows

e.g. how-to add or delete a user and give him/her access to all relevant services

Important links

link to all your web interfaces
link to the monitoring URLs (nagios, munin, apc-monitoring...)
link to the wiki (for the printed version!)

Emergency instructions

what to do if intranet server/internet/web server/etc are down

Important:

Choose a wiki engine with easy export to PDF!
Its not useful if you are in holiday, the server running your wiki is down and no one knows what to do because your documentation is offline

Have a look at twiki, docuwiki or mediawiki.

BTW:

there is a OpenOffice.org plugin to write directly to mediawiki - very convenient.

EDIT:

Its also nice to write down some infos to /home/adminuser/maintenance. This is done quick and can be very helpfull, if several admins work on a server. eg:

2009-06-27 -thorsten-
          running aptitude update && aptitude full-upgrade
          everything seems ok
2009-06-25 -andreas-
          cups-pdf wasn't reachable. restarted cups
2009-06-23 -thorsten-
          deleted old log under /var/log/squid
etc.

While you realize that while everyone wants (and needs) documentation, you also need to recognize that no-one has time to read and study the stuff.

So, don't write documentation that needs to be studied - instead, structure your documentation in a way that allows someone to quickly find the information they need, when they need it - which may well be while the system is down and the CTO is breathing down his/her neck.

With this in mind, some suggestions ...

Avoid large blocks of text
Bullet lists are your friend
A clear diagram is golden
Repetition is a good idea (1)
Make it easy to update and extend

(1) Don't create one source of truth and force people to hunt it down. The more important the idea, the more you should repeat it.

Essential docs:

Server documentation - specs/disk layouts/installed software/anything of note
Common procedures - anything that is done that isn't 'trivial' should have a procedure documented, especially if it's something that hasn't been done before.

Keeping documentation in sync can be pretty much a 'fix it as you see mistakes' affair. Along with this needs to come the realization that documentation can and will be out of date, and that it should not be followed blindly without taking this into account. Documentation is there to assist an admin in tasks, not to be a step by step set of rules that replace critical thought.

Minimizing duplication - using something like wikis where you can link documentation together can help with this, instead of repeating information, you just link to it. The problem is that the person writing the documentation needs to know that the information they are about to duplicate already exists. This is usually a question of good organization.