What is the "canary thread" and why is it starving?

The following occasionally appears in /var/log/syslog:

rtkit-daemon[1145]: The canary thread is apparently starving. Taking action.
rtkit-daemon[1145]: Demoting known real-time threads.
rtkit-daemon[1145]: Successfully demoted thread 1431 of process 1368 (n/a).
rtkit-daemon[1145]: Successfully demoted thread 1430 of process 1368 (n/a).
rtkit-daemon[1145]: Successfully demoted thread 1368 of process 1368 (n/a).
rtkit-daemon[1145]: Demoted 3 threads.

What's going on here?

Solution 1:

The term "canary" as used here comes from coal mining originally. Coal miners used canaries to detect dangerous gases (if the canary they carried with them died, they knew they had to get out of the shaft/mine ASAP). As a result the term "canary" is now often used for anything that you use to get an (early) warning about a dangerous situation.

In this case it seems like 'rtkit' starts a "normal" thread to test if the threads that get "real time" priorities are "starving" other threads (& processes), where "starving" means that they get too little processor time. This is a safety measure to make sure that processes/threads that have access to real time priorities don't use up so much CPU time that other tasks get none anymore.

So apparently some thread(s) that got real-time priorities from rtkit is/are misbehaving, and trying to monopolize the CPU, rtkit detects this with its "canary thread", and thus rtkit takes away the real-time priorities.