how to send alert on the same incident using logs-based metrics
On GCP we have created various logs-based metric where we basically are looking for log entries with some keywords like 'error', 'warning' etc...
We also created alerts based on those metrics.
Most of the logs come from cron like applications.
However when GCP creates incident it does not allow to manually close it. According to documentation the incident will close itself after 7 days or so. GCP sends alert only on the first occurrence of the incident. Therefore subsequent incidents, generated by cron (for example every hour) are treated as existing incident and alert is not being sent.
Is there any way to configure GCP Monitoring and Alerting to sent alert on every occurrence of the same incident?
The Alerting tool has been thought to send emails only when the incident appears and when it finishes, it isn't able to send an email every time your internal cron reports the error, if the original incident wasn't closed first.
An alternative could be to use your custom log metrics, logs router and pub/sub.
Through logs router, create a sink with a filter (of whichever error you want to report), then export it to pub/sub and through pub/sub trigger a cloud function to send an email with whichever content you require.
You can find more information on exporting logs here, how to use pub/sub and cloud functions in here and in this link how to send an email.