Spark Prometheus metrics name contain random generated data?

Metric names that Spark 3.2.0 generates are as follows:

metrics_spark_a929f03bcd084dd08ffe5a3c9ba71018_driver_DAGScheduler_stage_waitingStages_Number{type="gauges"} 0
metrics_spark_a929f03bcd084dd08ffe5a3c9ba71018_driver_DAGScheduler_messageProcessingTime_50thPercentile{type="timers"} 24749.0

I suspect that the part a929f03bcd084dd08ffe5a3c9ba71018 is random generated. How can one write proper Grafana queries to match these metrics? Is it possible to configure Spark not to add these hashes into the metric names? Whare is the purpose for these random hashes in the metrics naming?

(Checked Spark's example spark.metrics.conf for 3.2.0, but found no option to configure this.)

Solution 1:

In Grafana you can filter metrics with the internal __name__ label matcher, and you can write a regular expression to match the above metrics, regardless of the hash. These will match the above metrics:

{\_\_name\_\_=~"metrics_spark_.*_driver_DAGScheduler_stage_waitingStages_Number"}
{\_\_name\_\_=~"metrics_spark_.*_driver_DAGScheduler_messageProcessingTime_50thPercentile"}