NRPE and Perl - cannot include ANYTHING or it breaks

I've been banging my head off NRPE and Perl for weeks now.

I decided to start from absolute first principles, and create a dummy nagios plugin that does nothing but always returns OK. I called it check_true.pl, installed it on the remote server and configured NRPE to serve it out as check_test.

The entire script is just:

#!/usr/bin/perl

print "OK - this dummy test always returns OK\n";
exit 0;

This works over NRPE without problems.

With this as a starting point, I was going to build up the script I want slowly to see at what point it breaks. I didn't get at all far. The follow breaks over NRPE (but works fine with locally and over SSH):

#!/usr/bin/perl

use strict;

print "OK - this dummy test always returns OK\n";
exit 0;

It gives the dreaded error: NRPE: Unable to read output.

I can't include ANYTHING or I get this error. This makes it impossible to do what I actually need to do!

I thought it might be a problem with the perl include path, but running the following over NRPE shows it is not (gives identical include path as when run on the terminal):

#!/usr/bin/perl

print "OK - Perl include path: ".join(q{, }, @INC)."\n";
exit 0;

Does anyone know why NRPE is behaving so badly with Perl? Can anyone recommend a fix? Or even a workaround?

Update 1: The command is defined in NRPE as follows:

command[check_test]=/usr/lib64/nagios/plugins/check_true.pl

Update 2: I've done some more debugging, and by adding the wrapper below around the perl script I was able to capture STDERR.

#!/bin/sh

out=`/usr/lib64/nagios/plugins/check_true.pl 2>&1`
echo $out

The result makes things even more confusing:

Can't locate strict.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/lib64/nagios/plugins/check_true.pl line 3. BEGIN failed--compilation aborted at /usr/lib64/nagios/plugins/check_true.pl line 3.

A quick search shows that strict.pm is located at /usr/share/perl5/strict.pm, and /usr/share/perl5 IS in the listed @INC!

How can Perl fail to find a file that is right there? This works flawlessly when run as the user nagios in a terminal, so what is NRPE doing to the environment to mess up Perl?

Solution 1:

The culprit was SELinux which on RHEL6 constrains the NRPE process greatly. By default it even blocks access to the core perl modules like strict and warnings.

I haven't yet found a nice solution for turning off SELinux for just NRPE.