Prometheus auto scrape metrics from multiple kube-state-metrics in kubernetes?

Solution 1:

As I already mentioned in my comment. What you need is the Prometheus Self Discovery on Kubernetes:

Kubernetes Self Discovery configurations allow retrieving scrape targets automatically, as and when new targets come up. The scraping is based on Kubernetes service names, so even if the IP address change (and they will), Prometheus can seamlessly scrape the targets.

Prometheus self discovery is based on Kubernetes labels and annotations explained here. This allows a great deal of granularity on choosing the applications to be scraped. It is also important here to understand the role field, since it defines the behavior of a scraping job. role defines the type of Kubernetes resource you want Prometheus to look for. Currently it can be endpoints, service, pod, node, or ingress. For example, if role is set to pod, Prometheus will discover a target for each pod and exposes their containers as targets.

With it you will be able to scrap metrics from new nodes when they come up. More details and examples can be found in the linked docs.