Distributed Nagios Installation
I'm looking for a plug-in or product that will act as a remote probe and perform tests then send back the results to the central Nagios server.
Reason for this is that I'd like to monitor internal systems and servers at customers, but don't want to allow all the traffic passing the firewalls. Ideally I'd like a soft-probe that would be installed and then perform the tests and send back the results (via SSH) to the central Nagios installation.
Does anyone know of a product or plug-in that would offer such service? If not Nagios, is there any other monitoring system that does such a thing (ideally open-source)?
There are multiple ways to solve this. You can have a secondary server with just nrpe running. In this way it's acting as a proxy. So the main nagios sends a check through the server running nrpe. Example:
From the main nagios server: check_nrpe -H NRPEPROXYHOST -c check_ping -H 10.0.0.3 ....
The NRPEPROXYHOST runs the command as if it were the nagios server and submits the results back to the main server. In this setup the secondary server does not run nagios or any bloated daemons. Just the nrpe daemon, the nagios plugins to be ran. This can even be configured on some sort of gateway device and would not necessarily require a dedicated server be deployed.
======
Method 2 would be configuring a second instance of Nagios at the site and having it perform the active checks and submit the results to the main Nagios server. The main nagios server would have all the checks configured with active checks disabled and passive checks enabled.
This configuration is a true distributed Nagios as documented on their site. It's quite a bit more robust so if you see yourself having to perform several hundred or thousands checks to these server (every 5 minutes) then this is your best choice. In most instances the secondary server is called a "satelite" nagios instance and the results are usually submitted to the main Nagios server via the NSCA protocol (which is encrypted). The Main nagios server listens for these via the nsca daemon and submits them to the external command file for processing by nagios.
The downside is you have to have the config files on two servers and make changes to both sets of configs. You have to have these hosts as passive on the main server and active checks on the satelite server.
This is scalable to no end and the preferred solution for installations with tens of thousands of service checks to be performed. Also, look at building the configs on a central server and keeping them in revision control and have a script on the nagios server periodically checkout the new configs and reload nagios.
=====
Method 3
DNX, http://dnx.sourceforge.net/ an awesome project that patches Nagios so that it can send checks to be performed to "node" nagios servers. To the best of my knowledge though this configuration does not allow you to pick and choose which checks are executed by which node (node affinity), or if they are NOT to be executed by a node. So this solution adds distribution more than it does a proxy into a secondary network.
It's still very much in the development phase, but you've pretty much described the architecture of Reconnoiter.
Reconnoiter is quite different from Nagios, but looks promising if monitoring is more important to you than alerting.
Have you tried PRTG Network Monitor? It offers "remote probes" (that is a small program) that can be installed on remote computers on the network. The "probes" then send information from within other systems to the "core server" which does data analysis. The data connection between probe and core is SSL encrypted. The reason why I'm recommending it is, because its quite easy to set up.
It's explained here (video): http://www.paessler.com/media/flv/1000x700/v2-schemaanimation.flv
It's not open source, but there is a freeware version (with up to 10 sensors, which can be sufficient for monitoring just a view ports of a web server, for example).
Download (Freeware, Trial, etc.) at http://www.paessler.com/prtg