Consulting: Organizing site/environment documentation for customers? [closed]

I've been a partner in a three person contracting / consulting service since June, 2004. We each mainly work our own "accounts", however we need to maintain documentation for each other to allow for "failover" between partners. Most of our Customers have some kind of internal IT staff, many of whom perform some amount of day-to-day maintenance, and we need to communicate documentation to them effectively, too.

My two partners have the advantage (if you can call this that) of having worked as employees under me at another firm and, as a result, they were both indoctrinated to my opinionated way of doing things. The strict consistency (where things, obviously, can be) between Customers' configurations is a godsend. Obviously, products change, so we caucus to discuss new products / versions, etc, and decide on a consistent configuration strategy before we deploy. This wouldn't scale to a large company but, frankly, I see that as a feature rather than a bug. (I won't start ranting about larger "managed services" companies with their employee "engineers" and horrible tendencies for one-offs, half-assed "solutions", and inconsistency between Customers... >smile<)

I am violently against "the dreaded binder". I've never seen physical documentation kept up to date ever. I think it's waste of the Customer's money to spend time making physical copies of documentation. I'd much rather spend time working out how to generate documentation based on "live" data from running configurations.

As an example, I absolutely will not maintain spreadsheets of IP address information. That's what DHCP and DNS (see below for specifics) are for. If those things aren't working then we've got major problems.

We've had Customers request things like "make a document that shows all our Group Policy configuration" and I've dug in my heels and refused to do it. My recurring counter-proposal (which as worked so far) has been to introduce the Customer to administrative tools that can give them "self-service" or using software to generate "live" customer-friendly documentation on-demand.

We try really hard to be meticulous about spelling things out in plain English. A non-technical IT contact can, for example, review a computer's Active Directory group membership and see things like "Software - Install Microsoft Office 2010 Pro" and "Group Policy - Shipping Kiosk Computer Auto-Logon". It doesn't take any documentation to explain what those things mean.

Here's some "live" data we use:

• All IP address allocation is stored in DHCP servers-- this includes statically-addressed devices, too (noted as such in the comments). MAC and IP addresses can be readily queried by scripts or manually and, by definition, the data must be up-to-date if it's being used in production.

• Everything gets a name and PTR record in DNS. Most hosts also get an HINFO record. Things that need verbose descriptions get a TXT record.

• Copious and verbose use of "Notes" fields anywhere possible -- Active Directory, computer descriptions, shared folder descriptions, etc. We're also verbose and clear with things like security group names.

• Comments/remarks in network gear configurations (comments on ACLs, descriptions on ports, SNMP location/contact info, for example).

I'm fairly negative on the idea of free-form information storage in things like text files, wikis, etc. Structure makes for good searching. Whenever I can get a structured storage mechanism to work for me (even if it means I have to write software to query it) I prefer it. Comments that I can parse out of configuration files, databases, etc, always win me over when put up against manually-generated documents that will fall out of date almost immediately.

When we have to store "free-form" information we use our own SVN repository. It contains all the various bits and pieces of static documentation we've created over the years, filed by Customer. We've been using SVN for this since 2004 and it has worked very well as a collaboration tool for us. We version database schemas, sysadmin scripts, Group Policy object backups, etc. I try to check everything I can into version control.

It's very easy to search my checkout with filesystem-based indexing tools. I know that each of us have at least one complete copy of the repository available to us locally at any time. We also made the repository accessible via authenticated WebDAV over SSL in case we absolutely have to get to a data stored there and only have browser access.

We've never been asked to do it, but we'd be happy to create an account on the SVN server to allow a Customer to check-out and interact with their own files (if they have an internal resource who is so-inclined). We use a standardized format for storing all static Customer documentation (software license documentation, purchase records, etc) that's pretty self-explanatory.

Along with the SVN repository, we also self-host our email. All of the incoming/outgoing email has been archived since the company's domain started receiving email. It is available as BSMTP logs to the partners for reference (and, personally, I've found it to be invaluable). The situation has never come up but, I know we'd be happy to give a Customer access to the logs of any correspondence to/from their employees if they ever asked. Furnishing internal communication between the partners would be more difficult because we might well reference multiple Customers in the same message. (We should probably be better about that but we haven't been.)

Passwords are a major "wart" in our process. We're using individual "Password Safe" repositories (with unique combinations) for each Customer to allow the safe file to be shared with the Customer. We keep the master passwords for all the safe files in another safe file, with a combination known only to the partners. This part really needs some work. I think we'd like to have each Customer host an on-site credential vault using a real multi-user password vault application (with an audit trail, etc) but we've been kicking that idea down the beach for almost 10 years now.

Our time tracking records are meticulously detailed and provided to Customers in whatever electronic format they want (which, up to this point, have been ASCII text and PDF). Customers get start/stop times on each billable event, and a verbose description of work performed. We consider these service notes very valuable internally because they allow us to keep up-to-speed with what's happening in the partners' Customer sites. In the event of an issue these records give us a knowledge-based of all prior issues and resolutions we've encountered over the years. I'm not ashamed to say that I've resolved issues for one Customer by finding notes that I forgotten writing for another Customer years before.

A quick and cautionary aside re: producing documentation: At my "old job" (working for someone else years ago) the company got into a legal action against a non-paying Customer. We ended up being on the business-end of a counter-suit from the non-paying Customer. Our internal records and email re: that Customer were subpoenaed and laid-bare before the court. That experience taught me a lot about not storing anything in a fixed medium that you don't want to be made public.

I'd written some emails with some (erm) choice words and phrases in them about my frustrations with this Customer and with some of the other "engineers" at my company. Having to be cross-examined about these things in open court wasn't something that I enjoyed at all.

When we started our current business the partners agreed that all fixed records (email, text messages, voicemail, files in the SVN repository, work records in the time tracker, etc) would be considered "Customer-facing" all the time-- even if they were never intended to end up in the Customers' hands. This has been hard to do and takes a lot of discipline but I think it's worth it. We certainly want to project an air of professionalism to our Customers, and living it is the way to do it. I certainly will never be embarrassed like I was in that courtroom again.

For core information like you're describing (With the exception of a Network Diagram) we use a single Excel workbook, and save this into a network folder with the customers name. While I can understand why people don't like this approach I find it works great because it's a single reference document that I can take with me to site, e-mail around and update quickly.

My biggest complaint would be lack of version control, but I've yet to find anything that works well without making life hard work.

To counter your "I'm shooting for a process that's more portable, secure and elegant than a simple spreadsheet":

More Portable: What can be more portable than a 500 Kilobyte Spreadsheet? I'd be scared of using anything cloud or webbased because internet connectivity can't be guaranteed.

Secure: This one I'll give you and I too would like solutions on making our solution safer.

Elegant: We spent time creating a nice template for our workbook. Like I say, this is a multi-sheet workbook, not just a huge page of scattered information. I think most IT documentation fits nicely into a tabular form and a spreadsheet keeps this very neat and simple.

While I have lots of issues with how we record and store build documentation, I really can't find fault in a solid, well formated spreadsheet. I'll also add that we work with some big IT Services companies and they work in much the same way. Again, it's brilliant - I turn up to site, request the documentation and get a single neat spreadsheet that I can use for the duration.

The big advantage of that dreaded binder is :

• Non-technical clients can understand it, keep it on site and keep it safe like they're used to and already do with all kinds of other important paperwork.
• Your clients can easily hand it over in case they hire a replacement for you, a professional thing to have done and something that would have given you a quick start and reference when you landed them as clients.
• A simple maintenance log in the beginning with the highlights of each site visit is something tangible.
• The binder also holds CD/DVD's and certificates with license codes and activation keys in case the BSA or another vendor visits.

Whatever digital system you use doesn't really matter anymore as longs as you discipline yourself to update the binder whenever you've done something worthwhile.

And the binder is of course completely useless to you if you do nearly all your support remotely.

I have been a big fan of using a wiki because it is easily maintained, hyperlinks to ILO and management interfaces, remote desktop and SSH logins are easily followed. The freeform suits most information I would want to have:

• site description: "the network printer (OKI 1234 MFP) is the one next to John's desk, the HP inkjet printer model 0123 sits in Bobs, the boss's office" makes it a whole lot easier to answer a phone call: " we've run out of ink, what cartridge should we order again?".

• page for each device, ideally with a picture, type and configuration, it's purpose etc.

• There's some decent export modules for Wiki's to generate hardcopy or digital documents for handover and inclusion in that dreaded binder