chkrootkit scanner detected possible KLM Trojan

Is this anything to worry about?

No, it is a false positive and a long standing bug in chrootkit. You will see the message any time the lkm check reports hidden processes inaccessible by the readdir command. If you have something like ClamAV, MySQL, Exim or MailScanner running you are bound to see this warning.

Easiest check: run chrootkit a couple of times with as many services stopped (ie. mysql, clamav etc). If the results vary it is a clear indicator it is a false positive.

By the way: rkhunter has a better check for lkm.

Some random topics about this message: stackexchange, cpanel.net, Linuxquestions.org and all state this is bogus and a bug.

Something extra: the command ./chkproc -v will show the process ids (PID) for the message you get and you can use that to find the program that is connected to this process with cd /proc/{PID}/ && cat cmdline

In case this is convincing enough you can stop reading. If you want to know about your machine and how things work keep reading.

So what do you need to know about your system to be able to tackle this problem?

Firstly: The best rootkit detector is you. Knowing what services are active on your machine, knowing what scripts are running on your machine is what keeps your system clean and safe. And yes it will take a bit of time to fully understand a Linux system.

Secondly: There is a bit of malware targeting Linux but it is rare. For the simple reason that although Linux is very portable it is not that portable. The differences between distributions (how ever so small), libraries, different kernels and compilers make executing random code on random machines extremely difficult. And the people that are into malware are in it for financial gain. So they focus on Windows for the very simple reasons that it is closed source, has a lot of holes that depend on Microsoft to take action to fix them. (simplified :) )

Now for the warning you see about a possible LKM Trojan installed. LKM stands for Linux Kernel Module and is one of the core modules in Linux. Kernel modules are loaded into matching kernels and if the two do not belong together the modules will not load. This is one of the basic security features of a Linux system that you can use for tracking down malicious code (1).

Some basic things about kernels (2):

• uname -r shows your kernel.

• the kernels installed can be viewed in /boot.

  rinzwind@schijfwereld:/boot$ ls abi-3.16.0-22-generic initrd.img-3.16.0-29-generic abi-3.16.0-23-generic initrd.img-3.16.0-30-generic abi-3.16.0-24-generic memtest86+.bin abi-3.16.0-25-generic memtest86+.elf abi-3.16.0-28-generic memtest86+_multiboot.bin abi-3.16.0-29-generic System.map-3.16.0-22-generic abi-3.16.0-30-generic System.map-3.16.0-23-generic config-3.16.0-22-generic System.map-3.16.0-24-generic config-3.16.0-23-generic System.map-3.16.0-25-generic config-3.16.0-24-generic System.map-3.16.0-28-generic config-3.16.0-25-generic System.map-3.16.0-29-generic config-3.16.0-28-generic System.map-3.16.0-30-generic config-3.16.0-29-generic vmlinuz-3.16.0-22-generic config-3.16.0-30-generic vmlinuz-3.16.0-23-generic grub vmlinuz-3.16.0-24-generic initrd.img-3.16.0-22-generic vmlinuz-3.16.0-25-generic initrd.img-3.16.0-23-generic vmlinuz-3.16.0-28-generic initrd.img-3.16.0-24-generic vmlinuz-3.16.0-29-generic initrd.img-3.16.0-25-generic vmlinuz-3.16.0-30-generic initrd.img-3.16.0-28-generic

• kernel modules are installed in /lib/modules in a subdirectory matching your kernel.

So based on (1) and (2) the next step is to reboot into another kernel. The offending module was compiled against a specific kernel and will not be able to compile itself into the other kernel (simply because the headers do not match).

The amount of directories and files that can be affected when you have a rootkit are limited (a rootkit needs to be started from somewhere). There are 2 directories and a group of files that will be targeted ...

1. /etc/init.d/

   Do an ls -ltr /etc/init.d (it will list them in order they have been last changed) and check for unknown services. Normal services will have sane names. These services can be started by the system or manually.

   rinzwind@schijfwereld:/etc/init.d$ ls acpid hwclock.sh reboot alsa-utils irqbalance resolvconf anacron kerneloops rsync apparmor killprocs rsyslog apport kmod saned atieventsd lightdm sendsigs avahi-daemon lvm2 single bluetooth mountall-bootclean.sh skeleton bootmisc.sh mountall.sh smartmontools brltty mountdevsubfs.sh speech-dispatcher cgmanager mountkernfs.sh sslh cgproxy mountnfs-bootclean.sh sudo checkfs.sh mountnfs.sh thermald checkroot-bootclean.sh networking udev checkroot.sh network-manager udev-finish console-setup ondemand ufw cron php5-fpm umountfs cups pppd-dns umountnfs.sh cups-browsed procps umountroot dbus pulseaudio unattended-upgrades dns-clean rc urandom grub-common rc.local uuidd halt rcS x11-common hostname.sh README

2. /etc/rc*/

   The startup and kill scripts are located in /etc/rc[0-5,S].d. In general the files here have numbers and a sane description (these files are executed in alphabetical order when started and in reversed order during a kill. Watch out for scripts consisting of random numbers and letters. Here is a list (these are valid scripts).

   rinzwind@schijfwereld:/etc$ ls rc*/ rc0.d/: K01alsa-utils K01lightdm K01unattended-upgrades K05umountnfs.sh K01atieventsd K01php5-fpm K01urandom K06networking K01bluetooth K01pulseaudio K01uuidd K07umountfs K01cgmanager K01resolvconf K02avahi-daemon K08umountroot K01cgproxy K01speech-dispatcher K03sendsigs K09halt K01cups-browsed K01sslh K04rsyslog README K01irqbalance K01thermald K05hwclock.sh

   rc1.d/: K01alsa-utils K01irqbalance K01speech-dispatcher README K01atieventsd K01kerneloops K01sslh S01dns-clean K01bluetooth K01lightdm K01thermald S01killprocs K01cgmanager K01php5-fpm K01ufw S01pppd-dns K01cgproxy K01pulseaudio K01uuidd S02single K01cups K01saned K02avahi-daemon K01cups-browsed K01smartmontools K04rsyslog

   rc2.d/: README S01uuidd S02kerneloops S04cups S01apport S02acpid S02rsync S04cups-browsed S01cgmanager S02anacron S02smartmontools S04pulseaudio S01dns-clean S02atieventsd S02speech-dispatcher S04saned S01php5-fpm S02cgproxy S02thermald S05grub-common S01pppd-dns S02cron S03avahi-daemon S05ondemand S01rsyslog S02dbus S03bluetooth S05rc.local S01sslh S02irqbalance S03lightdm

   rc3.d/: README S01uuidd S02kerneloops S04cups S01apport S02acpid S02rsync S04cups-browsed S01cgmanager S02anacron S02smartmontools S04pulseaudio S01dns-clean S02atieventsd S02speech-dispatcher S04saned S01php5-fpm S02cgproxy S02thermald S05grub-common S01pppd-dns S02cron S03avahi-daemon S05ondemand S01rsyslog S02dbus S03bluetooth S05rc.local S01sslh S02irqbalance S03lightdm

   rc4.d/: README S01uuidd S02kerneloops S04cups S01apport S02acpid S02rsync S04cups-browsed S01cgmanager S02anacron S02smartmontools S04pulseaudio S01dns-clean S02atieventsd S02speech-dispatcher S04saned S01php5-fpm S02cgproxy S02thermald S05grub-common S01pppd-dns S02cron S03avahi-daemon S05ondemand S01rsyslog S02dbus S03bluetooth S05rc.local S01sslh S02irqbalance S03lightdm

   rc5.d/: README S01uuidd S02kerneloops S04cups S01apport S02acpid S02rsync S04cups-browsed S01cgmanager S02anacron S02smartmontools S04pulseaudio S01dns-clean S02atieventsd S02speech-dispatcher S04saned S01php5-fpm S02cgproxy S02thermald S05grub-common S01pppd-dns S02cron S03avahi-daemon S05ondemand S01rsyslog S02dbus S03bluetooth S05rc.local S01sslh S02irqbalance S03lightdm

   rc6.d/: K01alsa-utils K01lightdm K01unattended-upgrades K05umountnfs.sh K01atieventsd K01php5-fpm K01urandom K06networking K01bluetooth K01pulseaudio K01uuidd K07umountfs K01cgmanager K01resolvconf K02avahi-daemon K08umountroot K01cgproxy K01speech-dispatcher K03sendsigs K09reboot K01cups-browsed K01sslh K04rsyslog README K01irqbalance K01thermald K05hwclock.sh

   rcS.d/: README S03udev S08checkroot-bootclean.sh S01console-setup S04brltty S08kmod S02alsa-utils S04mountdevsubfs.sh S08urandom S02apparmor S04procps S09mountall.sh S02hostname.sh S04udev-finish S09networking S02mountkernfs.sh S05hwclock.sh S10mountall-bootclean.sh S02resolvconf S05lvm2 S10mountnfs.sh S02ufw S06checkroot.sh S11mountnfs-bootclean.sh S02x11-common S07checkfs.sh S12bootmisc.sh

3. a startup script.

   In general Ubuntu uses bash in a terminal and dash when booting. echo $SHELL will show you what shell is being used. For bash the hidden files to check for weird scripts or weird lines of code are...

   /etc/profile 
   /etc/bashrc
   /etc/bash.bashrc
   
   ~/.profile
   ~/.bash_profile
   

   These are the 5 common ones. Any machine can have more though. Besides that you might also include

   /etc/crontab
   crontab
   

   The last one for both your user and by doing sudo su. "crontab" you can list with crontab -l. Watch for scripts that are not general Linux or created by you.

If you happen to have a second system life becomes a lot easier: you can simply compare all the files above with the second machine.

I know there is already an accepted answer, but I want to show another way on how to debug this, which might help someone else in the future. This helped me to figure out that this was indeed a false positive. I also stumbled upon output like this on Ubuntu 20.04 using chkrootkit v0.53

Checking `lkm'...                                           You have    12 process hidden for readdir command
You have    12 process hidden for ps command
chkproc: Warning: Possible LKM Trojan installed

Note, that the amount of 12 was changing every time I ran it. So to debug this further I ran chkproc -v (usually in /usr/lib/chkrootkit/), which gives output like this:

PID 277881(/proc/277881): not in readdir output
PID 277881: not in ps output
You have     1 process hidden for readdir command
You have     1 process hidden for ps command

Note, that this also changed every time I ran it (different PID, different amount). So it seemed like some (unidentified) short-lived processes were messing with chkproc, which is a known source for false positives, see http://www.chkrootkit.org/faq/. To make my final judgement I needed to identify these short-lived processes and for this I can only recommand using atop, which can track process history over time, so that it can tell you what processes have been active over a period of time.

So to debug all this, here is what you need to do:

1. Run chkproc -v in one shell and let atop run in another shell.
2. When chkproc returns you some PID, wait for atop to update its output and pause it then (usually z key).
3. Have a look at the lower half of the atop output where you will find the process list starting with currently active processes and ending with the ones, that have exited during the past atop recording period. Such processes will be marked with <> in the most right-hand column.
4. Try to find the PID you just saw in chkproc -v from step 1. If it's not in there, then you may have paused atop too early, so try again in this case. If you can find the PID, then have a look at its CMD and RUID column entry. That should give you a good hint, what this process was and should hopefully give you an idea if this is a false positive or not.

Finally I can mention, that I found a lot of such short-lived processes (e.g. lots of sleep), which turned out to be coming Docker containers running web apps on my system. To verify that I stopped all Docker containers temporarily and all these short-lived processes were gone and chkproc as well as chkrootkit were happy again by not finding anything.

So to sum up: Short-lived processes in Docker containers can cause false-positives. However make sure to identify them first instead of ignoring them right away.