runaway distnoted process

Sometimes I see a distnoted process suddenly spin up and chew up 100% CPU (on one core) and a ton of memory, often in the neighborhood of 1.5G or so. This happens a few times a day, starting a month or so ago.

The command line is /usr/sbin/distnoted agent, and it's started by launchd, neither of which help much. It's usually been running for somewhere between 4h and 24h before it spins up and pegs the CPU.

Web searches say distnoted manages notification delivery, and lots of other people report the same problem with it, but I haven't yet found a fix. Some people find that closing a culprit application (e.g. Skype) stops it, but I haven't found a culprit on my machine yet. I'm usually only running a few apps: Emacs (24.2 from Homebrew), Firefox, Adium, and Dash.

I'm on Mavericks on a late 2012 13" Retina MBP. Thanks in advance!

Update:

I've turned on distnoted logging in the system log by touching /var/log/do_dnserver_log, but it doesn't help much. I see lines like these (uid 501 is me, 89 I haven't found yet):

distnoted[80011]: # distnote server agent  absolute time: 48754.144787848   civil time: Wed Nov 20 10:52:03 2013   pid: 80011 uid: 501  root: no
distnoted[20]: # distnote server daemon  absolute time: 2.808112262   civil time: Tue Nov 19 09:52:24 2013   pid: 20 uid: 0  root: yes
distnoted[444]: # distnote server agent  absolute time: 16.656997509   civil time: Tue Nov 19 09:52:38 2013   pid: 444 uid: 501  root: no
distnoted[1271]: # distnote server agent  absolute time: 52.518265717   civil time: Tue Nov 19 09:53:14 2013   pid: 1271 uid: 89  root: no
distnoted[689]: Interruption - exiting now.

I've also run sudo dtruss -p PID on a spun-up distnoted process, and it spews lines like this:

kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
...

Solution 1:

Summary from the OP: This was a great tool for debugging. It originally pointed me to Spotlight reindexing the filesystem, but I narrowed down the things it's allowed to index, and I still saw the problem. I ended up setting up a cron job to kill distnoted regularly. See answer farther down.

You can debug distnoted by creating the file /var/log/do_dnserver_log This causes the CFNotificationCenter server (distnoted) to record information about all notifications to the system log.

I would start there, reboot and look at the system log when the CPU spikes up. This should out the culprit easily.

More info on CFNotificationCenter debugging can be found in official Developer docs here: Technical Note TN2124 > CFNotificationCenter

Solution 2:

I've seen this too. Emacs 24.3.1, Mavericks 10.9.

I've found that the distnoted process calms down within seconds after I quit out of Emacs.

I've filed an Emacs bug here: http://permalink.gmane.org/gmane.emacs.bugs/80836

Solution 3:

I know I'm late to the party but this is a memory leak specific to Cocoa emacs on Mavericks that is fixed in the trunk. For now there is a patch you can use to build emacs 24.3 with just the fix.

https://gist.github.com/anonymous/8553178

Solution 4:

I've been having the same problems with distnoted on El Capitan for some time. My solution isn't as harsh as killing it regularly, rather I check for it running out of control (high CPU usage), and then kill it. I use this script:

#!/bin/sh
#
# check for runaway distnoted, kill if necessary
#
PATH=/bin:/usr/bin
export PATH

ps -reo '%cpu,uid,pid,command' | 
    awk -v UID=$UID '
    /distnoted agent$/ && $1 > 100.0 && $2 == UID { 
        system("kill -9 " $3) 
    }
    '

The script is run from cron every minute with this line in crontab:

*   *  *   *  *   sh "$HOME/bin/checkdistnoted"

In practice, the script kills distnotedonce or twice a day, and typically this occurs after backupd starts.

For those not comfortable with the using the OS X shell (command line), the following script will install both the checkdistnoted script and the crontab entry:

#!/bin/sh
#
# install $HOME/bin/checkdistnoted
# setup crontab to run every minute
# 
# MWR Apr 2016
#

INSTALLCMD=bin/checkdistnoted
cd "$HOME"
[ ! -d bin ] && mkdir bin
[ -f $INSTALLCMD ] || {
    cat > $INSTALLCMD <<-"!!"
    #!/bin/sh
    #
    # check for runaway distnoted, kill if necessary
    #

    PATH=/bin:/usr/bin
    export PATH

    ps -reo '%cpu,uid,pid,command' | 
        awk -v UID=$UID '
        /distnoted agent$/ && $1 >= 100.0 && $2 == UID { 
            # kill distnoted agent with >= 100% CPU and owned by me
            system("kill -9 " $3) 
        }
        '
!!
    chmod +x $INSTALLCMD 
    echo installed $INSTALLCMD
}

INSTALLCRON="# check for runaway distnoted every minute:
* * * * * sh \"\$HOME/$INSTALLCMD\""
crontab -l | grep -q '$HOME'/$INSTALLCMD || {
    crontab -l > mycron
    echo "$INSTALLCRON" >> mycron
    crontab mycron
    rm mycron
    echo updated crontab
}

You need to save the above as install_checkdistnoted.sh on your desktop, then run Applications/Utilities/Terminal and type:

cd Desktop
sh install_checkdistnoted.sh 

If it works fully it will print confirmation of each of the steps. The script won't overwrite an existing checkdistnoted script or crontab entry.