How to find out the number of time series stored in Prometheus LevelDB

i'm responsible for maintaining the Prometheus servers in our company. The metrics however are provided by the teams.

Is there a way to find out the number of time series stored in the Prometheus datadase? We are using the default LevelDB data storage. I need these values to find out if i need to tweak the local storage settings of our Prometheus instances.

Thanks for your help.

Solution 1:

prometheus_tsdb_head_series

Just because I always forget and have to Google this, and this question is at the top. As per this answer there will likely be a variance between prometheus_tsdb_head_series & count({__name__=~".+"}) because of variations in what they consider "active", but unless you have a large variability in metric count, I'd recommend prometheus_tsdb_head_series because if you have a lot of metrics, it's a lot faster to query.

Solution 2:

How about count({__name__=~".+"})?

It does return number of time series in the database. I compared with amount of metrics currently exposed by each target by manual scrapping, and it matches it +/- 10%. I guess the difference is due to some targets that I had in the past and now they are offline.

Solution 3:

After some more research and thanks to the comments of David.B i found a solution that 'works for me' ™

To find the number of time series stored by Prometheus i use this command in the storage.local.path folder :
ls -l {{0..9},{a..f}}{{0..9},{a..f}} | grep -E "*.db$" | wc -l

In addition in found some metrics in the prometheus documentation that might help when dealing with memory problems/optimization.

This might not be the most sophisticated way but it gave me the numbers i was looking for.

Solution 4:

https://prometheusui.com:9090/status ---> Head Stats --> this will give you the entire status.