What should NOT be managed by puppet?

Solution 1:

General rule: If you're using configuration management, manage every aspect of the configuration that you can. The more you centralize the easier it will be to scale your environment out.

Specific examples (cribbed from the question, all "This is why you want to manage it" narratives):

IP Network configuration

OK, sure, you configured an address/gateway/NS on the machine before you dropped it in the rack. I mean if you didn't how would you run puppet to do the rest of the config?

But say now you add another nameserver to your environment and you need to update all your machines -- Don't you want your configuration management system to do that for you?

Or say your company gets acquired, and your new parent company demands that you change from your 192.168.0.0/24 addressing to 10.11.12.0/24 to fit into their numbering system.

Or you suddenly get a massive government contract -- Only catch is you have to turn up IPv6 RIGHT FREAKIN' NOW or the deal is blown....

Looks like network configuration is something we'd like to manage...

IPMI Configuration

Just like with IP addresses, I'm sure you set this up before you put the machine in the rack -- It's just good common sense to enable IPMI, remote console, etc. on any machine that has the capability, and those configurations don't change much...

... Until that hypothetical acquisition I mentioned in IP Configuration above -- The reason you were forced to vacate those 192.168-net addresses is because that's IPMI-land according to your new corporate overlords, and you need to go update all your IPMI cards NOW because they're gonna be trampling on someone's reserved IP space.

OK, it's a bit of a stretch here, but like you said - all of it can be managed with ipmitool, so why not have Puppet run the tool and confirm the configuration while it's doing all of its other stuff? I mean it's not going to hurt anything, so we may as well include IPMI too...

Updates

Software updates are more of a gray area -- In my organization we evaluated puppet for this and found it "sorely lacking", so we use radmind for this purpose. There's no reason Puppet can't call radmind though -- In fact if/when we migrate to Puppet for configuration management that's exactly what's going to happen!

The important thing here is to have all your updates installed in a standard way (either standard across the organization, or standard within platforms) -- There's no reason Puppet shouldn't be launching your update process, as long as you've thoroughly tested everything to ensure that Puppet won't mess up anything.
There's also no reason why Puppet can't call out to a tool that's better suited for this task if you've determined that Puppet can't do a good job on its own...

Solution 2:

Don't reinvent the wheel.

Yes, you can have 50 puppet virtual user resources and realize them as needed in your modules... but if you can, use LDAP.

I speak from bitter experience. Although ldap isn't an option here, yet.

Anouther example is pushing out hosts files, rather than just using DNS.