What does "<service_name> dead but subsys locked" mean?

I'm trying to run memcached on a centos box and it runs for a while, but then ends up in this state:

memcached dead but subsys locked

netstat shows this:

tcp        0      0 :::11211                    :::*                        LISTEN      
udp        0      0 0.0.0.0:11211               0.0.0.0:*                               

ps shows this:

nobody   21983  0.0  1.8  60272 19912 ?        Ssl  16:46   0:00 memcached -d -p 11211 -u nobody -c 1024 -m 64

Anyone know what that means?

This means the service was running at one time, but has crashed.

When you start a service, it creates a "lock" file to indicate that the service is running. This helps avoid multiple instances of the service. When you stop a service, this lock file is removed.

When a running service crashes, the lock file exists but the process no longer exists. Thus, the message.

Look at the two areas /var/run/*.pid and /var/lock/subsys/*. These are expected to agree with each other. That is, if the (emtpy file) lockfile /var/lock/subsys/crond exists, then the first line of the file /var/run/crond.pid is expected to contain the PID of the process running for this service. If no such process is running, then something is wrong. If a process is indeed running (as you see) but it is not that PID, then something is probably confused.