How do you delete a counter in graphite whisper?
The reason so many people are struggling with this issue (me included) is because statsd (depending on its settings) continues to write data to old stats (whisper files) even after you delete the whisper file (.wsp).
Check your statsd settings at /etc/statsd/config.js (on Ubuntu). Example settings at https://github.com/etsy/statsd/blob/master/exampleConfig.js. I would recommend setting deleteIdleStats to true so that statsd does not write stats when they are not received.
The downfall of deleteIdleStats is that Graphite may now have null values for certain stats. This can be handled on the visualization layer (I recommend Grafana).
You have two options to stop regeneration of deleted whisper files:
- simply restart statsd with
service statsd restart
or - update statsd settings to not send stats to backend if the stats are not received AND then
service statsd restart
To delete wsp files that you don't need anymore:
- List the files to be deleted with
find pathToUnwantedStatistic -print
- if that is printing only the files you want deleted, run the following line to delete the unwanted files
find pathToUnwantedStatistic -delete
I had a similar problem where entries still existed in Graphite although no data was available. I am using statsd and it didn't have metrics in there either. It turns out I had to delete these folders in addition to the wsp files.
My exact case is, I used the machine name as prefix for statsd. Carbon created folder for each of the machines in e.g /opt/graphite/storage/whisper/stats/ip-0A215A08, where it stored the wsp for my metrics. Because of that, once I deleted the wsp files, I still had a bunch of "folders" in Graphite w/o any data that still showed up in the UI.
Checks on Carbon/Graphite
- .wsp files
- directories for the stats you got rid of
On statsd (if you are using it), verify there are no more gauges, timers or counters for the metrics you want to get rid of. For this, you may also telnet on your instance (default port 8126) and check for metrics with proper commands of the Admin interface instead of killing the process.