How do I change, sort, add, remove graphs with Munin?

To keep an eye on what's happening with our production servers I recently discovered Munin. It was trivial to install but I'm having a lot of trouble working out how to add/remove graphs.

I found some documentation suggesting I can reorder them but very little discussion about how to manage plugin configuration. The bulk of the plugin documentation discusses how to create plugins; rather than use them.

The question is pretty basic: How do I use the vast library of plugins so I can see what's happening with, for example, nginx, mysql and memcache.

Enable and disable plugins on each node

Graphs are added and removed via symlinks in the /etc/munin/plugins/ directory of the node.

To remove a graph you must remove the symlink and restart the node:

rm /etc/munin/plugins/diskstats
service munin-node restart

To add a graph you must add a symlink in the plugins directory to an executable. eg:

ln -s /usr/share/munin/plugins/diskstats /etc/munin/plugins/diskstats
service munin-node restart

When you restart munin-node it runs immediately and any issues with the plugins appears in /var/log/munin/munin-node.log. If all is going well you'll see a CONNECT logged every cycle; this records the fact that the master connected to collect the latest data.

Process Backgrounded
2014/03/10-15:59:47 Munin::Node::Server (type Net::Server::Fork) starting! pid(32231)
Resolved [*]:4949 to [::]:4949, IPv6
Not including resolved host [0.0.0.0] IPv4 because it will be handled by [::] IPv6
Binding to TCP port 4949 on host :: with IPv6
2014/03/10-16:00:04 CONNECT TCP Peer: "[::ffff:203.28.51.227]:45965" Local: "[::ffff:50.23.111.122]:4949"
2014/03/10-16:05:04 CONNECT TCP Peer: "[::ffff:203.28.51.227]:46109" Local: "[::ffff:50.23.111.122]:4949"
2014/03/10-16:10:04 CONNECT TCP Peer: "[::ffff:203.28.51.227]:46109" Local: "[::ffff:50.23.111.122]:4949"

Configuring plugins on the node

Install prerequisite modules

But it may be that things don't go well. Errors when running the plugins are logged here as well and are critical clues to getting a plugin to work. Here we see a problem with the nginx_request plugin...

2014/03/10-11:25:05 CONNECT TCP Peer: "[::ffff:203.28.51.227]:38474" Local: "[::ffff:50.23.111.122]:4949"
2014/03/10-11:25:19 [22482] Error output from nginx_request:
2014/03/10-11:25:19 [22482]     Use of uninitialized value $LWP::VERSION in sprintf at /etc/munin/plugins/nginx_request line 109.
2014/03/10-11:25:19 [22482]     Can't locate object method "new" via package "LWP::UserAgent" at /etc/munin/plugins/nginx_request line 109.
2014/03/10-11:25:19 [22482] Service 'nginx_request' exited with status 2/0.

Plugins are executable programs - mostly written in Perl. There is a big collection installed at /usr/share/munin/plugins/ (Ubuntu). Being mostly user-contributed they appear to vary wildly in quality but all of them need to query the system for the data it needs then arrange it into a format munin can use.

Most of them rely on either a Perl library which you may or may not have installed already, or a diagnostic program for the service of interest. For example the plugin for varnish requires varnishstat. Similarly the plugin for memcached.

But back to the problem above... whereby, fortunately I've done a little bit of Perl in my time, and make an educated guess that the plugin requires a perl module LWP::Useragent. If you open the plugin that's throwing errors it may give you some clues to what it's requirements are. Some of them you can run from the commandline to see what happens/if they work.

To continue the example above this might squash the nginx_request error shown above:

apt-get install libwww-perl

Or it may not...

Enable data sources

By way of example, nginx, if you take a peek at the src of the plugin it will reveal that it collects it's data from http://localhost/nginx-status so you'll need to enable that. Or, as previously mentioned varnishstat is an example.

Or it may be right to go as is.

Provide environment variables

Additionally you might need to feed the plugin some environment/configuration variables. This is done by editing /etc/munin/plugin-conf.d/munin-node and adding whatever it needs. eg:

[mysql_innodb]
env.warning 0
env.critical 0

You can also do this by creating a new file with this kind of content as it will process any file in that directory; but I didn't try that.

To test the plugin such that it actually reads the environment variables there is a tool munin-run. Use it just by naming the file in the plugins dir

% munin-run nginx_request
request.value 10275
% 

What about these plugins with hanging underscore filenames?

Quite a lot of plugins in the /usr/share/munin/plugins/ directory have plugins which don't work if you just symlink to them. They are the ones with an underscore as their last character. eg: memcached_, diskstat_, if_. They leave the hanging underscore so they can be reused with varying parameters.

To use them symlink from /etc/munin/plugins/ but append a parameter which (a) may be a keyword eg memcached_bytes or, (b) may refer to a parameter eg: if_eth0. The best way to tell what it expects is to read the contents of the plugin. Some of the better constructed plugins may even help you. eg: /usr/share/munin/plugins/mysql_ suggest outputs a list of suitable suffixes.

lrwxrwxrwx 1 root root 36 Mar 10 11:45 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes
lrwxrwxrwx 1 root root 31 Mar 10 11:47 mysql_commands -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Mar 10 11:45 mysql_connections -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Mar 10 11:46 mysql_qcache -> /usr/share/munin/plugins/mysql_

Configure remote access

The master polls the node for data so, if you're using a remote master, don't forget to grant it access via /etc/munin/munin-node.conf on the node.

Configure the master

You can run master and node on the same host. Or you can put the master someplace else - which is advisable if you're watching a busy production server as generating the graphs every few minutes is quite a bit of work.

I got that going no problem using the standard documentation and adding nodes into /etc/munin/munin.conf.

To see how the polling is going keep an eye on /var/log/munin/munin-update.log. When data from a new plugin is first seen you should get a bunch of log entries recording the construction of a new rrd database:

2014/03/10 12:45:15 [INFO] creating rrd-file for mysql_innodb->free: '/var/lib/munin/caradvice.com.au/syd.caradvice.com.au-mysql_innodb-free-g.rrd'

This is also the place to find warnings and errors relating to the incoming data.

One final gotcha is that, by default, the node data collection only happens every two minutes, and the remote polling only happens every 5 minutes. Additionally it can take 60 seconds to turn the data into graphs, and finally the first entry is possibly a single data point which isn't visible. So, unless you fiddle with the cycle times, it might be 15 minutes between getting the plugin correctly configured and when data appears on your graph.

I'm out of steam. If this doesn't help someone else, it will at least help me remember what the heck i've done.

The final result is great infoporn...

Can I remove some graphs and add others, is it all fixed?

Remove a plugin from your munin-node configuration and that plugin's graphs will be removed.

How would I add plugins to show me what's happening with, eg, nginx, mysql and memcache.

You install the munin plugins for those products on the munin-node.