Large custom GCP metric

What's the best way of creating large custom metrics in GCP?

We are running some compute instances, and want to track some custom values, e.g. the quality of an algorithm, and visualize the data into dashboards. Our current solution is to write log messages, and use Logs-based metrics. That works, but it requires that we write a log message for every single value, which sounds kinda inefficient (lots of meta infos, filtering log messages, ...) and amounts to a quite high logging bill.

Is there a better, more efficient way of managing custom metrics?

I've seen that you can create custom metrics, and add new points to them with the monitorin API (https://cloud.google.com/monitoring/custom-metrics/creating-metrics), however those are rate limited, and you can write just one point each 10 seconds, which is far too less, and you get errors like the following:

google.api_core.exceptions.InvalidArgument: 400 One or more TimeSeries could not be written: 
One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0]

Solution 1:

Here are some options for custom metrics in GCP you may want to check out:

1. Custom metrics with OpenCensus
2. Custom Metrics with Prometheus
3. Custom metrics with OpenTelemetry