Source based distros vs Prebuilt package based distros
I maintain more than a fair number of Gentoo machines.
Not because we care for childish funroll-loops speed tuning. What we care about is the flexibility of installing precisely what we want, not every package feature and dependency that a contributor thinks we might one day want. We are perfectly comfortable with how Linux, compiling and libraries work. We don't want everything to be abstracted away from us into a black box.
You should choose to use what you feel most comfortable with. Here are some details that may help to allay your fears of Gentoo specifically.
Build times are relatively negligible on modern hardware. Even more so if your machines are of N+1 redundancy.
You can be picky about how portage behaves, such as placing
MAKEOPTS="-l 1.0"in your make.conf to ensure that new builds backoff when the load average is creeping uncomfortably high.You can use binary packages from Portage if you'd prefer. The mirrors provide a number of common packages or you can roll your own with
--buildpkgonly.If you have a large number of machines then you can benefit from having a nominated build host or distributed compiling.
The latest stable versions of Portage that are now in use are much more unlikely to leave you high and dry when performing upgrades. Tales of conflicts and blocks are pretty much a thing of the past.
If you are looking after so many machines that upgrades become painful then you should be looking at Puppet/BCFG/cfengine anyway ;)
Update in response to question edit:
Whilst I empathise with the situation you describe, it is not something that is symptomatic of source-based distributions or indeed prevented by using a package-based distribution. It is the result of:
- Your own natural unfamilalirity, and
- Your predecessor's poor internal documentation
Truth be told, I haven't extensively used any other Linux distros for quite some time now. For this reason, if you were to place a Debian machine (for example) onto my lap and tell me that OpenLDAP wasn't working after a reboot, then I too might spend 15 minutes or an hour to resolve the issue. Not because I don't have a good understanding of Linux or that Debian isn't a good OS, but primarily because I don't recall the intimate details of Debian's RC scripts or package system. This is why internal documentation is essential within an organisation. It should serve to jump-start the unfamiliar and to fill in the gaps of everyone else. Even if they do know everything.
Just a quick note specifically about Gentoo. Portage's USE flags are incredibly useful and "minimal" is something that I use frequently. I don't for instance want the server binaries of a package to be installed on a machine that will only ever be a client during it's lifetime. Having them unnecessarily present may increase complexity or even be a security concern. It's never an issue of space. You can see what USE flags and dependencies a package is going to adopt (and which it isn't) before it starts compiling by using the -av arguments. So you shouldn't get an surprises.
GNUix> Sorry, snap!
I won't comment on if you should use Gentoo or not - or if building from source 'is worth it'
What I will say is that I use Gentoo and Ubuntu. I used to use Gentoo for all my GNU/Linux machines but decided that Ubuntu on my desktops was easier to manage. I am on the side of admins that think using Gentoo for a server is a great idea -- and use it for almost all of my personal servers.
Apart from all that, what I like more than building stuff from source or tweaking CFLAGS is Portage. Portage is simply hands down the best package and system management tool I have ever used for GNU/Linux. I can create an exact environment with just the things I need and not one thing more and I can do it with out going outside of my package management system. I don't have to have postfix installed simply because some packaging dude decided that it's 'necessary'. I don't need mono installed because I want to run Gnome, I simply don't install Tomboy. Quickly browsing the installed packages on this Ubuntu desktop shows that it has bind installed? Why? It serves no purpose, I'm not running a DNS server on this computer and I certainly don't need the documentation for it, but it's here.
The downside... time. It takes not only the time to build packages but to keep the system maintained. I can be realativly sure that I can run a sudo aptitude update {upgrade} and not have a care in the world... On the other hand you have to be very careful of the small details when you update a Gentoo machine and make sure that you need (and are prepared) for what it wants to do. The 'straw that broke the camels back' per say of why I switched to Ubuntu for my desktops was an upgrade of udev that borked a setup I had -- rather than try and fix it I wanted to get the work I was trying to do done. So I figured I really don't care if bind documentation is taking up space on my desktop because I have the space.
On my servers, however, totally different matter. I want to control everything about the environment, the packages that are installed, every little detail I want to be in control of -- and Gentoo let's me do that with ease.
Life is too short for source based distributions. I have more important things to do. Like using the computer.
You can always use a combination of both methods - use a pre-compiled distribution as a base and have the advantages of a good working integrated environment and fast-painless updates while using self-compiled software where one needs to be at the leading-edge of the development of that software. Either by having some self-discipline (using configure with the prefix option) or by packaging the source yourself. You can always "reuse" the build instructions (rpm spec file or debian subdirectory for example).
I wouldn't use the build mechanism on production servers directly. From my Gentoo days I know that there's a possibility to have a build host and create the packages needed on that host so that you can burn cpu cycles on a dedicated host with the compile flags suitable for your needs.
Overall there isn't too much difference, nearly all of our servers run Debian but we ended up having a build host anyway. We needed a few software packages that weren't available as debs so we set up a reprepro and a buildhost for our needs, the buildhost is pretty much unmaintained (in terms of "does it work") because redeployment is just a matter of a few minutes since we can netboot it and it will automatically redeploy itself.
Either BSD ports/Gentoo/Debian it doesn't matter that much the real profit comes with using a system (as in collection of services on different hosts that provide business value) that will minimize wasted resources and is maintainable. We chose Debian because security updates are a no brainer and they are pre-compiled so that we don't have to "waste" time rebuilding packages whenever a security update comes out. That is about the only drawback I can think of for source based vs. binary based distros.