Micrometer - Prometheus: Some meters are showing, others are not

I have a rabbitmq message queue on which many other services report status updates for what's called a Point. Now, on a separate service (written with SpringBoot), I need to listen for those Point updates and convert that into a Prometheus-scrapable endpoint.

So my plan is to convert the incoming Point objects into Meters and register them in the MeterRegistry. And that works, but only for some of the points. I haven't figured out, yet, which exactly are visible and which aren't because it looks like that depends on the order in which they come in after a restart of the service. I couldn't figure out any pattern, yet, that would help troubleshooting.

From what I understood reading the micrometer documentation, the Meter is created once, we give it an object and a function that allows it to retrieve the double value from that object for the metric. Since, I have new instances of Point coming in every couple of seconds, that value won't just be updated as the Meter is referencing the old Point.

Assuming this is correct, I added a little wrapper around that Point (the PointWrapper) that I pass to the Meter and cache instance of PointWrapper myself. Now when a new Point comes in, I check if I already have a PointWrapper for that Point and if so, I replace the Point instance in the wrapper with the new one.

@Service
public class PointSubscriber {
    
    private final MetricsService metrics;

    public PointSubscriber(@Autowired MetricsService metrics) {
        this.metrics = metrics;
    }

    @Bean
    public Consumer<PointUpdate> processPoint() {
        return (update) -> {
            metrics.update(update.getPoint());
        };
    }

@Service
@RequiredArgsConstructor(onConstructor = @__(@Autowired))
public class MetricsService {

    private Logger logger = LoggerFactory.getLogger(getClass());

    private final MeterRegistry meterRegistry;

    private Map<String, PointWrapper> cache = new HashMap<>();

    public void update(Point point) {

        // Check if wrapper already in cache
        String pointId = point.getId();
        PointWrapper cached = cache.get(pointId);
        
        // Replace the point in the wrapper to update the value
        if (cached != null) {
            logger.debug("Updating value for {}", point.getId());
            cached.setPoint(point);
            
        // Create the wrapper, cache it and register Meter
        } else {
            PointWrapper pointMeter = PointWrapper.from(point.getId(), point);
            // Don't register Meters that will return null
            if (pointMeter.getMetricValue() == null) {
                logger.debug("Not going to register point with null value: {}", point.getId());
                return;
            }
            logger.debug("Registering point {}", point.getId());
            register(pointMeter, meterRegistry);
            cache.put(pointId, pointMeter);
        }
    }
    
    public Meter register(PointWrapper pointMeter, MeterRegistry registry) {
        Set<Tag> tags = new HashSet<>();
        tags.add(Tag.of("pointId", pointMeter.getPoint().getId()));
        tags.addAll(pointMeter.getPoint().getLabels().entrySet().stream()
            .map (e -> Tag.of(e.getKey(),e.getValue()))
            .collect(Collectors.toSet()));
        
        return Gauge.builder(pointMeter.getMetricName(), pointMeter, PointWrapper::getMetricValue)
            .tags(tags)
            .register(registry);
    }
    
}

@Data
@Builder
public class PointWrapper {
    
    public static PointWrapper from(String id, Point point) {       
        return PointWrapper.builder()
            .id(id)
            .metricName("symphony_point")
            .point(point)
            .build();
    }
        
    private String id;
    
    private String metricName;
     
    @EqualsAndHashCode.Exclude
    private Point point; 
    
    public Double getMetricValue() {
        if (point == null)
            return null;
        if (point instanceof QuantityPoint) {
            return ((QuantityPoint) point).getValue();
        } else if (point instanceof StatePoint<?>) {
            StatePoint<?> s = (StatePoint<?>) point;
            if (s.getState() == null)
                return null;
            return Double.valueOf(s.getState().asNumber());
        }
        return null;
    }
        
}

As I mentioned, this leads to a bunch of missing data points in the prometheus endpoint. I read that Meters are uniquely identified by their name and tags. The name is always symphony_point but I add the Point's ID as a tag called pointId. Just because of that, every Meter.Id is unique.

I can see logs like

Registering point outdoor_brightness_north

but that point is missing in the Prometheus endpoint.

Any ideas?

UPDATE @checketts pointed out that metrics with the same name must have the same labels set. I checked quickly can confirm, that's not the case with the data I am using:

symphony.point area pointId device property floor room
symphony.point area pointId device property floor room
symphony.point area pointId property room floor device
symphony.point area pointId property room floor device
symphony.point area room pointId device property floor
symphony.point pointId area room device property floor
symphony.point area room pointId device property floor
symphony.point area room property pointId floor device
symphony.point pointId area property device
symphony.point area device property pointId
symphony.point area room pointId floor device property
symphony.point area pointId device property floor room
symphony.point area pointId device property room floor
symphony.point area pointId property floor device room
symphony.point area room property pointId floor device
symphony.point area property room floor pointId device
symphony.point pointId area room property floor device
symphony.point area device pointId property
symphony.point area device property pointId floor room
symphony.point area pointId room device property floor
symphony.point area pointId room device property floor
symphony.point area room pointId device property floor
symphony.point area pointId room floor device property
symphony.point pointId area device property
symphony.point area property room floor device pointId
symphony.point area pointId device room property floor
symphony.point area room device property floor pointId
symphony.point area device pointId property floor room
symphony.point area pointId property floor device room
symphony.point pointId area device property
symphony.point area pointId device property floor room
symphony.point area pointId property room floor device
symphony.point area pointId room device property floor
symphony.point pointId property area device
symphony.point area property pointId floor device room
symphony.point area room property pointId floor device
symphony.point area room pointId property floor device
symphony.point area pointId floor device property room
symphony.point area room device pointId property floor
symphony.point pointId property area device
symphony.point area room device property pointId floor
symphony.point area device property floor pointId room
symphony.point area room pointId floor device property
symphony.point area pointId property room floor device
symphony.point area room device property floor pointId
symphony.point area room device pointId property floor
symphony.point pointId area device property
symphony.point area property floor pointId device room
symphony.point area pointId device property floor room
symphony.point area property pointId device
symphony.point pointId area property floor device room
symphony.point area pointId floor device property room
symphony.point area property pointId floor device room
symphony.point area room pointId floor device property
symphony.point pointId area device property
symphony.point area room pointId property floor device
symphony.point area room pointId floor device property
symphony.point area room device property pointId floor
symphony.point area pointId room property floor device
symphony.point area room device property floor pointId
symphony.point area pointId property room floor device
symphony.point pointId area property device
symphony.point area pointId device property floor room
symphony.point area device pointId property floor room
symphony.point area room pointId property floor device
symphony.point area pointId device property floor room
symphony.point area pointId device room property floor
symphony.point area room pointId device property floor
symphony.point area property room pointId floor device
symphony.point pointId area device property

That's a big bummer, since the labels that come via Points (that's what I build the Tags from) aren't well-defined. Still I need to be able to make queries based on them. I could add them all to the name but then things like "show all indoor temperatures" become a very unpleasant experience to query.

Anyway, I will try to validate this is the root cause of my problem.

Solution 1:

This line is suspicious:

tags.addAll(pointMeter.getPoint().getLabels().entrySet().stream()
            .map (e -> Tag.of(e.getKey(),e.getValue()))
            .collect(Collectors.toSet()));

Do all points have the same labels? With Prometheus, all meters with the same name need to have the same tag names (aka labels). The first point with its label names will become the default and all other will be rejected.

Micrometer - Prometheus: Some meters are showing, others are not

Solution 1:

Related

Recent Posts