Rule-based Alerts
Container Observability detects an anomaly when a predefined alert rule is violated and generates incident tickets where the user can analyze the alert details from the alert panels.
Note
Container Observability enables you to analyze RCA for rule-based alerts triggered for pods and deployments when anomalies are detected.
In Alerts, if the Stage column for an alert ticket lists R, then you can view RCA results for the alert ticket to determine the cause of the issue.
Container Observability triggers an alert based on the following:
Rule-based:
Metric
Log
Trace
SLO-based
Note
Note: Container Observability also allows you to define new custom Alert Rules for the Alert notifications.
Metric-based alert
Container Observability triggers an alert if the metric health is not as per the pre-defined metric rule conditions. It also allows users to define a new custom Metric-Rule for an entity within the Kubernetes cluster as per users' requirements.
Note
You can define new metric conditions for an entity from the cluster as per your requirement.
Container Observability triggers a metric-based alert based on pre-defined metric conditions.
Helps you to define the new metric condition for an entity, like SLO target for a metric on Containers, Pods, and Nodes.
If a metric is not behaving as per the defined metric-rule condition, Container Observability detects and pushes an alert to the user notification channels immediately.
You view the details of the metric-based alerts triggered by Container Observability in the Alert panels, and determine the cause of the anomaly in detail.
Log-based alert
Log-based alerts are triggered if an unexpected error is found in the logs list. Container Observability identifies such errors, and you can create new Log alert rule conditions that will take care of such system-based log errors in the future. If a similar log error appears, Container Observability creates a log-based alert notification to users' notification channels.
From log-based alert, you can:
Analyze the cause of log errors through various alert panels from Alerts.
Compare the current log error to previous logs.
Auto SLO-based alert
 |
Container Observability monitors services of each metric health, and automatically configures and suggests the Threshold SLO value for the metric based on the data captured during the last 24 hours. It automatically compares the current behavior of metric SLO with the SLO value captured for the last 24 hours.
For any deviation in the metric, Container Observability automatically detects and triggers an SLO-based Alert. For example, an increase in latency, an increase in SLO value, or an increase in response time.
The SLO-based Alerts allow you to perform the root cause analysis.
Note
As Container Observability automatically triggers SLO-based Alerts, and it also enables users to configure SLO targets manually based on the 24 hours of suggested data.
Event-based alert
Event alert is a type of rule identified by Container Observability, where the conditions are pre-defined concerning a state change that occurs in the Kubernetes cluster.
If the cluster exhibits an unexpected change or a new error that caused the creation of events, the user can identify such new errors. You can define a new rule for event-based alerts so that Container Observability detects and triggers an immediate alert for the errors related to repeated events occurring in the cluster.
Helps you to debug and resolve the tickets generated based on the event, and track them as required if the same error occurs.
You can identify and analyze the cause of the occurrence through various alert panels to ensure whether it is still ongoing or resolved.
Tracepath-based alert
As with containers, alerts are created when traces show anomalies. Typically, these are driven by breaches of Service Level Objectives (SLOs) on response time in the trace or when errors are detected.
Since detecting alerts on individual traces is challenging given the large number of traces an application can generate per monitoring period or scrape interval, Container Observability detects anomalies in traces by detecting them on Tracepaths, which are the common business flows that group traces sharing the same flow.
With Container Observability’s Tracepath-based alerting, you can
Automatically detect anomalous Tracepaths that contain traces that show response time anomalies or errors without searching for them.
Drill down to the individual traces from the anomalous Tracepaths.
To view the tracepath-based alerts in Global View, see Tracepath RCA alert.
Container Observability manages the entire lifecycle of a tracepath alert from data collection to root cause analysis.
Dynamic baseline and thresholding
Container Observability dynamically computes response-time SLOs from historical observations (using p99 latency baselines per trace and service operation). When a tracepath's latency deviates significantly from its baseline for longer than the configured duration (`breachOn`), an anomaly is detected.
Alert consolidation
Alerts are raised at the tracepath level rather than per individual trace or span. When subsequent slow or failing traces occur, CO links them as activity items and increments the repeat count on the existing parent tracepath alert, attaching the specific trace identifiers
Multi-layer RCA
When an anomaly is flagged, CO executes an automated RCA plan to isolate the underlying infrastructure or application issue:
Plan and collector orchestration: Selects the RCA plan based on the anomaly's
problemCategory(such as Performance) andproblemSubCategory(such as Latency).Multi-dimensional assessors: It evaluates each service hop across multiple layers, including metrics (such as, rate, errors, and duration) for service operations, pod and container resource utilization (CPU, memory, and disk), node and host pressure from noisy neighbors, logs of concern such as errors, exceptions, and timeouts, and Kubernetes events or recent changes such as restarts, scaling events, and deployment or configuration updates.
Causal chain and topology classification: Identifies the detected cause (the terminal or deepest failing service operation/infrastructure bottleneck). Identifies upstream callers as Symptoms (since cumulative caller latency includes downstream wait time).