Monitor the health of the SystemLink Enterprise Alarm Service using OpenTelemetry metrics and Prometheus metrics.

Introduced in June 2024

Refer to the following table of metrics emitted by the Alarm Service and related dependencies.
Note You can deploy and configure the OpenTelemetry collector to expose all OpenTelemetry metrics as Prometheus metrics. Exposing these metrics to Prometheus allows you to view the metrics in a tool such as Grafana.

Alarm Service

Table 56. Performance Metrics for the Alarm Service
KPI? Metric Type Description Labels
Yes ni.alarm.alarms.count Counter The number of active alarms and inactive alarms

Use this metric to understand how many alarms the deployment is managing.

The Alarm Service returns an error and performance degrades under the following conditions.
  • You have 10,000 or more active alarms.
  • You have 100,000 total alarms.

NI recommends that you configure a warning alert for when you are approaching the limit for total alarms. For example, you receive an alert when you have reached 7,500 active alarms and 75,000 total alarms.

NI recommends that you also configure an alert for when you have reached the limit for total alarms. For more information about alarm limits, refer to Related tasks.

 ni.alarm.alarms.active: ["true", "false"]
Yes ni.alarm.notification_sender.notifications.failed.count Counter The number of notifications that the service has failed to send.

To filter by the cause of the error, use the ni.alarm.notification_sender.failure_type label, with values server and client. Metrics with a server label failed to send due to issues with the Notification Service. Metrics with a client label failed to send due to client errors. For example, specifying an invalid or deleted notificationStrategyId in a request to trigger an alarm.

 ni.alarm.notification_sender.failure_type: ["client", "server"]
No ni.alarm.notification_sender.notifications.sent.count Counter The number of notifications that the service has sent successfully.

Use this metric to monitor how many notifications the service is sending. If the service is sending more notifications than expected, clients might be misusing the service.

 None
No ni.alarm.update_operations.retry.count Counter The number of update operations the service has retried. ni.alarm.update_operations.operation: ["CreateOrUpdate", "Acknowledge"]

Alarm Service Dependencies

To learn about other available performance metrics and how to use the metrics, refer to the documentation for the Alarm Service dependencies.

Table 57. References for Performance Metrics for Alarm Service Dependencies
Dependency Where to Find Information
ASP.NET For a list of the metrics provided by ASP.NET, refer to ASP.NET Core metrics and ASP.NET runtime metrics.
Kubernetes For a list of the metrics provided by Kubernetes, refer to Kubernetes Metrics Reference and the kube-state-metrics Documentation.
MongoDB For more information on MongoDB, refer to How to Monitor MongoDB.