I have a perl script that is supposed to run indefinitely. It's being killed... how do I determine who or what kills it?

Solution 1:

Without much in the way of actual knowledge, I'd start looking in dmesg output or assorted syslogs if the OOM killer is running. If so, that's probably it.

Solution 2:

Put in signal handlers for all the signals (TERM, SEGV, INT, HUP, etc) and have them log out whenever they are hit. It wont tell you what is sending the signal, but it will allow you to see what signal it is and perhaps ignore it.

$SIG{'TERM'} = $SIG{'INT'} = sub { print(STDERR "Caught SIG$_[0]. Ignoring\n"); };

That would print out when it caught a sigterm or sigint and then return control back to the program. Of course with all those signals being ignored, the only way to kill it would be to have the program itself exit, or to send it a SIGKILL which cant be caught.

Solution 3:

I realize this isn't exactly an answer to the question you asked, so I apologize if it's somewhat off-topic, but: does your app really need to run continuously, forever? Perl is not the most resource-thrifty environment in the world, and while the overhead of interpreter start-up is not without its drawbacks, extremely long-running scripts can have troubles of their own - memory leaks, often at a level below your control, are the bane of the vanilla-perl developer's existence, which is why folks often mitigate those issues either by running in a more formally resource-conservationist sub-environment like Perl::POE, or by handing over the long-running listener part of the job to a front-end service like xinetd and only executing the perl component when work needs to be done.

I run several perl scripts which run continuously reading and processing the output of our (considerably large) central syslog stream; they suffer from terrible, inexplicable "didn't free up memory despite pruning hash keys" problems at all times, and are on the block to be front-ended by something better suited to continuous high-volume input (an event queue like Gearman, for example), so we can leave perl to the data-munging tasks it does best.

That went on a bit; I do apologize. I hope it's at least somewhat helpful!