sshd running but no PID file

Solution 1:

I have the same problem. I fixed it, temporarily at least, by killing the sshd process and then starting it.

    service sshd status
    openssh-daemon is stopped

(even though I am logged in via ssh)

    rpm -vV openssh openssh-server openssh-clients | grep 'S\.5'
    S.5....T  c /etc/ssh/sshd_config

    netstat -anp | grep sshd
    tcp    0      0 0.0.0.0:22           0.0.0.0:*          LISTEN      17501/sshd

    kill 17501
    service sshd start

    service sshd status
    openssh-daemon (pid  3157) is running...

And now monit is happy, too. :)

Solution 2:

From what you're describing, it almost looks like another process is taking over port 22 and answers your SSH requests instead. Getting a message saying the port is already in use when restarting a service is not normal. Looks like the actual sshd service is killed in favor of that other "phantom" process. Could be that you have installed opensshd twice without changing the port it's using, or (and don't panic here, it's just a possibility) your server has been hacked and the hacker replaced sshd with another daemon of his own.

To see which process is using your port, try this:

netstat -lptun

Then look for any line showing a local address ending with :22, and look at the last column (PID/Program name). Note down any PID using port 22.

Then to find out the full command launched for that PID you do this:

cat /proc/PID/cmdline (where PID = the PID of the process)

If it's not /usr/sbin/sshd, (or whatever opensshd binary it should be) you've got a problem!

Here's a script you can run safely to dump some useful information:

#! /bin/bash

echo -e "Searching for the process listening on port 22...\n"
PORT22_PID=$(netstat -lptun | grep -E ":22\s" | awk '{print $7}' | awk -F/ '{print $1}' | uniq)
if [ ! -n "$PORT22_PID" ]; then
        echo "Error: Was not able to find any process listening on port 22"
        exit 1
fi
echo -e "Found the following PID: $PORT22_PID\n"
echo -e "Command line for PID $PORT22_PID: $(cat /proc/$PORT22_PID/cmdline)\n"
echo -e "Listing process(es) relating to PID $PORT22_PID:\n"
echo "UID        PID  PPID  C STIME TTY          TIME CMD"
ps -ef | grep -E "\s$PORT22_PID\s"
echo
echo -e "Listing RPM information about openssh packages:\n"
RPMS=$(rpm -qa | grep openssh)
for r in $RPMS; do
        rpm -qi $r | sed -n '/^Name/,/^Summary/p'
        echo -e "\n------------------------------------------------------\n"
done

Just paste the output in your original question and it should help. I've tested this script thoroughly on my own Centos server.