Self-monitoring with Container Observability
Self-monitoring with CO provides a dedicated observability layer for the Virtana platform itself, so the infrastructure that monitors your applications is monitored too. By deploying an additional CO instance specifically for self-monitoring, you gain real-time insight into the health, performance, and logs of all Virtana components, whether you choose to install GV with IO, GV with CO, or GV with both IO and CO.
This is achieved by installing a dedicated CO instance on the same Kubernetes cluster as your existing Virtana deployment, configuring it with a separate tenant, and pointing its South modules at the cluster to collect metrics and logs from all running Virtana services.
Prerequisites
Before deploying CO Self-Monitoring, ensure the following components and services are present in your Kubernetes cluster.
Install Self-Monitoring CO Frontend and Backend
This section covers installing the CO frontend and backend services that power the self-monitoring dashboards and data pipelines. Follow each step in order.
Set up a dedicated tenant or organization
Use a dedicated tenant or organization for self-monitoring to isolate monitoring data from production workloads.
If you are deploying Global View only for Self-Monitoring, you can use the default tenant created during Global View deployment.
If Global View is already deployed and Self-Monitoring needs to be added, create an additional tenant by following the Create Additional Tenant.
Make a note of the OrgID from the Curl Output, as this will be the Tenant/Org ID for Self-Monitoring.
Deploy CO Frontend and Backend
Deploy a dedicated CO north instance by following the frontend and backend deployment guide. Choose the right sizing for your CO based on your environment. See Deploy CO Frontend and Backend for more details.
Choose the right sizing for your CO. See Prerequisites for more details.
Note
The CO frontend and backend must be deployed on the same K8s cluster where other Virtana components are installed.
Set the following value in the CO backend values.yaml to enable CO Self-Monitoring dashboards:
global: customer_name: "opscruise"
Deploy CO South modules
Deploy CO South modules in the same K8s cluster where all CO North instances are running. These modules handle data collection from the monitored environment. See Deploy CO South for more details.
By default, Loki and Promtail will be deployed alongside the South modules to collect logs. These logs are viewable in the CO UI for debugging purposes.
Post-Installation monitoring setup
After installation is complete, use this section to validate your deployment, verify data flow, and configure ongoing monitoring and alerting.
Validate data flow in CO
Confirm that the self-monitoring instance is receiving data correctly by checking the following in the CO interface:
Go to the CO Self-Monitoring cluster.
Check SystemStatus to ensure gateways are sending data properly.
Verify that namespaces, pods, and other objects appear in:
AppsMap page > Topology
NodeMap page > Application > NodeMap
Viewing and downloading logs
Use the CO dashboard to verify log collection and export logs for analysis or troubleshooting.
Navigate to the CO dashboard > Data Explorer > Logs to ensure logs are being collected.
Perform the steps to download logs:
Select the appropriate label under the Labels section to filter desired logs.
Specify the required time range.
Scroll through pages by clicking the >> button at the bottom center of the screen until you reach the last page.
Click Download to export the logs.
Configure notification policies
To receive alert notifications via Slack, Email, or other channels for alerts generated within CO Self-Monitoring, follow the Alert Notification Policy guide. This document provides instructions to enable and configure notification policies based on your requirements.
Changing log levels of Virtana services
By default, all Virtana services are configured to expose only ERROR level logs. You can adjust this to capture more detailed logs, for example, INFO, for debugging or troubleshooting.
Select log level | Command | Description |
|---|---|---|
All modules | global: log_level: INFO # or ERROR | The command |
A specific module | module: log_level: INFO # or ERROR | The command |
Note
Replace INFO or ERROR with the desired log level depending on your logging requirements.