Can you use configuration management (Puppet/Chef) behind a firewall?

I'm tasked with designing and implementing configuration management (Chef) for a infrastructure where the production servers are completely segmented and isolated behind a jump box.

 ---------------------------------
| Production | Staging, Test, Dev |
 ---------------------------------

The only way to get files into production is with a sftp connection. There are only two people who have the ability to jump into production directly. An exception could be made to allow a tightly controlled automation system (like jenkins) to copy files into production.

Normally, I would recommend setting up a single chef server in the cloud, however the production servers can not have external access. (Without a lot of convincing of some very reluctant people).

One solution I see is to use two Chef servers

 ---------------------------------------
| chef-production | chef-local         |
| Production      | Staging, Test, Dev |
 ---------------------------------------

Using two chef servers has some problems that I'm not sure how to overcome.

• How can I keep the production and the local servers cookbooks in sync?
• How could we add new production nodes to the production chef server, without requiring one of the admins to run the knife command?
• How could we make changes to the databags in production?

I'm open to other suggestions. I'm considering not using a chef server and instead use chef-zero or chef-solo. My understanding is that it isn't the best solution when using multiple environments.

I hate this kind of trustless environment, but I get it; You have to deal with the biz requirements...

You could get around the keeping-cookbooks-in-sync problem with a CI/CD pipeline for your cookbooks, such that they get deployed to both Chef servers automatically. You would want to leverage things like cookbook pinning extensively, but making updates available (even if they're not actively used) is not problematic. Sous-Chef is a tool that my company has open-sourced to help people get started with automating cookbook testing and uploading to a Chef Server. It would have to be modified to meet a case of multiple Chef Servers, but the majority of it is there.

Since Chef12 has both features for "Organizations" and "Role-Based Access Control" (RBAC) (each of which used to only be available in the paid product), you could establish permissions such that only the specified people could modify Production environment, or that Production was a separate Organization within the same Chef server.

Adding new nodes to the Prod Chef server will require some sort of bootstrapping. Details will vary widely depending on your environment, but having the installer local to that environment you could have it set as a run-once job when you kickstart or deploy from a VM Template. If you have tooling around spinning up VMs via API, it shouldn't be too awful to extend it to adding the bootstrapping commands. (Chef is, after all, an API-driven system.)

Your question about databags is too broad for here.

Keep in mind that having multiple chef-servers will require having separate knife configs, so you will either have to remember to pass in a config flag for every one, or wrap commands for knife associating a config file. Unless of course you have sufficient automation to never need knife.

Chef-Zero is intended for local testing where you would use Chef-Solo, but want search. It doesn't really make sense for managing your prod boxes.

Some companies use Chef-Solo for their non-testing environments in a "run-as-needed fashion". Personally, I don't like this. I like Solo for local testing (such as with Vagrant), or if you're setting up systems with the intent that they're immutable. If you're going to have long-running servers (largely stateless, and torn down frequently, standard Chef-Server will help you keep configs from drifting over time.