Container Observability post installation

After you deploy Container Observability (CO), you can enable optional observability features to extend what CO collects. This section covers those post-installation tasks. Enable the features that match the data you want to bring into CO, such as distributed traces, external logs, or both.

CO tracing

CO tracing connects Container Observability to an application backend to capture distributed traces of requests as they move across your services. You trace a slow request across several microservices to find which service is adding the latency. To view both the standard and external methods to enable CO tracing, see Enable tracing for Container Observability.

CO Splunk log collection

Configures Container Observability to collect logs from an external Splunk instance instead of the default log source. You can correlate application logs already stored in your organization's Splunk alongside CO metrics in one place. See Configuring Splunk log collection for details.

Enable tracing for Container Observability

This guide explains how to enable distributed tracing for Container Observability (CO) in the Virtana Platform. Tracing allows you to track requests as they travel across your microservices, making it easier to identify performance bottlenecks and diagnose errors.

Virtana's tracing architecture consists of three components that work together:

Table 91.

Component	Role
OTEL Collector	Deployed in each South cluster as part of the CO South deployment. Collects trace spans from instrumented applications and forwards them to the Trace Router.
Trace Router	A central hub is deployed once in the virtana-trace-router namespace. It receives traces from all South clusters by OTEL collectors and routes them to the correct TraceGW based on a cluster tag.
TraceGW (Trace Gateway)	Deployed per-cluster in the North Backend namespace. Receives trace data and forwards it to your Tempo backend. Also calculates SLO (Service Level Objective) metrics from trace spans.
Tempo (external method only)	Your external distributed tracing backend (not deployed by Virtana). TraceGW forwards processed traces to it for storage and querying.

Virtana supports two methods for enabling tracing. Both methods share the same core pipeline. They differ in what happens after TraceGW processes the traces.

Standard method: TraceGW stores SLO metrics in-cluster and surfaces them in the Virtana CO UI. No external backend is required. Use this method for most on-premises customer deployments.
Data flow: OTEL Collector > Trace Router > TraceGW > Virtana CO UI
External method: TraceGW forwards processed traces to an external Tempo backend in addition to computing SLO metrics. Use this method when your organization already operates a centralized Tempo instance.
Data flow: OTEL Collector > Trace Router > TraceGW > Tempo

Prerequisites

Complete all prerequisites before you start any deployment steps.

CO South deployment

The OTEL Collector is deployed as part of the CO South deployment. Before enabling tracing, confirm that CO South is already running in every South cluster where you want to collect traces. If it is not, complete the Deploy CO South procedure first. See the CO South deployment for more details.

Namespace and quota requirements

Create the Trace Router namespace in Fuze before deploying. The table below lists the minimum required resource quotas for each namespace involved in the tracing pipeline.

Table 92.

Namespace	PVCs	NodePorts	CPU	Memory
North Backend namespace	3	2	2 cores	10 GB
virtana-trace-router	3	2	2 cores	10 GB

To create the Trace Router namespace, run the following command in Fuze:

kubectl create namespace virtana-trace-router

Secret synchronization

Sync the required secrets from Fuze to the following namespaces before deploying. These secrets supply image pull credentials and Keycloak authentication settings that TraceGW and the Trace Router need at startup.

Table 93.

Target namespace	Required secrets	Purpose
North Backend namespace (TraceGW)	`oc-ns-docker-creds`, `oc-kc-secret`	Image pull credentials and Keycloak authentication for TraceGW.
virtana-trace-router	`oc-ns-docker-creds`	Image pull credentials for the Trace Router.

Tempo backend details (external method only)

If you are using the external method, obtain the following values from your Tempo instance before you begin. If Virtana manages your Tempo instance, contact your Virtana administrator. You do not need these values for the standard method.

Table 94.

Parameter	Description
Trace_external_url	The URL of your external Tempo backend. TraceGW uses this address to forward processed trace data.
Trace_headers	Optional HTTP headers are included in every request to Tempo. Used for multi-tenant Tempo setups. Leave empty if your Tempo instance does not require custom headers.
Trace_auth_token	A non-expiring Bearer token for authenticating requests to Tempo. Ensure the token does not have an expiry date. An expired token will silently break trace ingestion.

Standard method

Use this method for most on-premises customer environments. It enables full tracing and SLO computation within the Virtana platform without requiring an external backend.

Step 1: Enable tracing in the North Backend values file

Update the global section of your North Backend Helm values file to activate tracing. In the standard method, tracing is handled entirely within the cluster, so trace_external is set to false and no external URL is needed.

global:
backend:
trace_enabled: true
trace_client: "Jaeger"
trace_external: false
trace_headers: ""

The following table describes each field in the configuration file.

Table 95.

Field	Description	Default value
`trace_enabled`	Activates the tracing feature in the CO backend. Set to `true` to enable tracing.	`true`
`trace_client`	Specifies the tracing client protocol.	`Jaeger`
`trace_external`	Set to false for the standard method. Traces are processed and stored in-cluster by TraceGW.	`false`
`trace_headers`	Not required for the standard method.	`""`

Step 2: Create or update the ArgoCD values files for TraceGW

Add the TraceGW application configuration to your ArgoCD-managed values.yaml file. This deploys one TraceGW instance per cluster. Replace the xxxx placeholder values with your actual cluster names and tenant IDs.

namespaces:
  argocd: argocd

apps:
  virtana_tracing_onprem:
    enabled: true
    version: 2024.12.4
    clusters:
    - name: xxxx
      tenantID: xxxx
    names:
      app: "virtana-tracegw"
    values:
      global:
        secret_source: "none"
        gatewayCreds:
          environment:
            OPSCRUISE_ENDPOINT: "kafka-service.{{ .clusterName }}.svc.cluster.local:9092"
            KEYCLOAK_ENABLED: "true"
            KEYCLOAK_URL: "https://wind-keycloak.oc.dev.cloud.virtana.com:443"
            KEYCLOAK_REALM: "{{ .tenantID }}"
            OPSCRUISE_ACCOUNT_ID: "{{ .clusterName }}"
        k8sgw:
          enabled: false
        promgw:
          enabled: false
        loggw-loki:
          enabled: false
        loki-stack:
          enabled: false
        prometheus:
          enabled: false
        opscruise-node-exporter-new:
          enabled: false
        kube-state-metrics:
          enabled: false
        tracegw:
          enabled: true
      tracegw:
        storageClassName: ""
        slo_storage_size: 20Gi
        config:
          tracegw:
            tracegwconf.trace-source: jaeger
            tracegwconf.slo-enabled: "true"
            tracegwconf.slo-training-minutes: "30"
            tracegwconf.write-slo-slow-traces-to-alogq: "true"
            tracegwconf.meters-exclude-all-tags: "true"
            tracegwconf.mode: listen
            tracegwconf.enable-trace-forwarding: "false"
            tracegwconf.filter-tag-key: "cluster"
            tracegwconf.find-cluster-tag-keys: "cluster"

Cluster top-level fields

Table 96.

Field	Description	Default value
`enabled`	Activates this ArgoCD application. Set to `false` to disable without removing the configuration.	`true`
`version`	The Helm chart version to deploy. Check the Virtana Helm chart repository for the current version.	`2024.12.4`
`clusters[].name`	The cluster name for this TraceGW instance. Also used as the Kafka topic identifier and the cluster tag key for routing.	`xxxx`
`clusters[].tenantID`	Your Virtana tenant ID is used as the Keycloak realm identifier. Contact your Virtana administrator if you are unsure of this value.	`xxxx`

Gateway Credentials fields

Table 97.

Field	Description	Default value
`OPSCRUISE_ENDPOINT`	Kafka broker endpoint for publishing processed trace metrics.	`kafka-service.{{.clusterName}}...:9092`
`KEYCLOAK_ENABLED`	Enables Keycloak authentication for TraceGW. Set to `true` in all standard deployments.	`true`
`KEYCLOAK_URL`	URL of your Keycloak instance. Replace with your actual endpoint.	`(your Keycloak URL)`
`KEYCLOAK_REALM`	Keycloak realm for TraceGW authentication. Automatically populated from tenantID.	`{{ .tenantID }}`
`OPSCRUISE_ACCOUNT_ID`	Account identifier used when publishing data to Virtana. Automatically populated from the cluster name.	`{{ .clusterName }}`

Tracegw configuration fields

Table 98.

Field	Description	Default value
`storageClassName`	Kubernetes storage class for TraceGW's PVCs. Leave empty to use the cluster default.	`""`
`slo_storage_size`	Size of the persistent volume for SLO training data. Increase for high-traffic environments.	`20Gi`
`tracegwconf.trace-source`	Trace backend type.	`jaeger`
`tracegwconf.slo-enabled`	Enables SLO calculation from trace spans. Set to `false` to disable SLO processing and reduce resource usage.	`true`
`tracegwconf.slo-training-minutes`	Minutes of trace data TraceGW observes before its SLO baseline model is trained.	`30`
`tracegwconf.write-slo-slow-traces-to-alogq`	When `true`, traces that exceed the SLO latency threshold are written to AlogQ for later review.	`true`
`tracegwconf.meters-exclude-all-tags`	When `true`, span tags are excluded from SLO metrics sent to Kafka, reducing cardinality.	`true`
`tracegwconf.mode`	Operating mode for TraceGW.	`listen`
`tracegwconf.enable-trace-forwarding`	Set to `false` for the standard method. Traces are not forwarded to an external backend. If set to `true`, TraceGW forwards processed traces to the external backend as specified in `trace_external_url`.	`false`
`tracegwconf.filter-tag-key`	Span tag key TraceGW uses to identify the originating cluster.	`cluster`
`tracegwconf.find-cluster-tag-keys`	Comma-separated list of tag keys TraceGW searches to identify the cluster. Typically matches filter-tag-key.	`cluster`

Step 3: Deploy Trace Router and TraceGW using ArgoCD

Apply the following ArgoCD Application manifest to deploy both TraceGW (one per cluster) and the Trace Router (one shared instance). Save the file as virtana-tracing.yaml and apply it to your ArgoCD namespace.

{{- $app := .Values.apps.virtana_tracing_onprem }}
{{- $namespace := .Values.namespaces.argocd }}
{{- if $app.enabled }}
{{- range $cluster := $app.clusters }}
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
    name: {{ $app.names.app }}-{{ $cluster.name }}
    namespace: {{ $namespace }}
    finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
    project: default
    destination:
        server: https://kubernetes.default.svc
        namespace: {{ $cluster.name }}
    source:
        chart: virtana-co
        repoURL: https://virtana.gitlab.io/helm-charts
        targetRevision: {{ $app.version }}
        helm:
            releaseName: virtana-tracegw
            valueFiles:
                - values.yaml
    syncPolicy:
        automated:
            prune: true
            selfHeal: true
        syncOptions:
        - CreateNamespace=true
{{- end }}
{{- end }}
# --- Trace Router (single shared instance) ---
{{- if $app.enabled }}
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
    name: virtana-trace-router
    namespace: {{ $namespace }}
    finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
    project: default
    destination:
        server: https://kubernetes.default.svc
        namespace: virtana-trace-router
    source:
        chart: virtana-co
        repoURL: https://virtana.gitlab.io/helm-charts
        targetRevision: {{ $app.version }}
        helm:
            releaseName: virtana-trace-router      
            valueFiles:
                - values.yaml
            values: |
                global:
          secret_source: "none"
          k8sgw:
            enabled: false
          promgw:
            enabled: false
          loggw-loki:
            enabled: false
          loki-stack:
            enabled: false
          prometheus:
            enabled: false
          opscruise-node-exporter-new:
            enabled: false
          kube-state-metrics:
            enabled: false
          tracegw:
            enabled: false
          trace-router:
            enabled: true
        trace-router:
          machine_type: large
          service:
            type: NodePort
          configMap:
            tracerouter:
              trace-gateway-endpoints:
              {{- range $cluster := $app.clusters }}
              - key: cluster
                value: {{ $cluster.name }}
                endpoint: tracegw-service.{{ $cluster.name }}.svc.cluster.local:9000
              {{- end }}
            envConfig:
              tracerouter.traces-in-queue: "0"
              tracerouter.initial-delay-secs: "30"
              tracerouter.scale-tracegw: false
              tracerouter.trace-completion-wait-time-secs: "120"
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
{{- end }}

Trace Router parameters

Table 99.

Field	Description	Default value
`machine_type`	Resource profile for the Trace Router pod. large is correct for production.	`large`
`service.type`	Exposes the Trace Router on a node port so OTEL collectors in South clusters can reach it without an ingress controller.	`NodePort`
`trace-gateway-endpoints[].key`	Span tag key the Trace Router checks to identify the destination cluster.	`cluster`
`trace-gateway-endpoints[].value`	The cluster name value the tag must equal. The Trace Router forwards the trace to the TraceGW endpoint whose value matches.	`{{ $cluster.name }}`
`trace-gateway-endpoints[].endpoint`	Internal DNS address of the TraceGW service for this cluster.	`tracegw-service.{{ $cluster.name }}...:9000`
`tracerouter.traces-in-queue`	Traces to buffer before forwarding. 0 means immediate forwarding without buffering.	`"0"`
`tracerouter.initial-delay-secs`	Seconds the Trace Router waits after startup before forwarding. Allows TraceGW to finish initializing first.	`"30"`
`tracerouter.scale-tracegw`	Reserved for future use. Leave set to false.	`false`
`tracerouter.trace-completion-wait-time-secs`	Seconds the Trace Router waits after the first span arrives before forwarding the complete trace.	`"120"`

Step 4: Configure OTEL collectors in South clusters

Update the OTEL collector configuration in each South cluster to export traces to the Trace Router. The collector is part of the CO South deployment. Locate its Helm values or configMap and update the exporters section.

exporters:
otlp:
    endpoint: trace-router-service.virtana-trace-router.svc.cluster.local:4317
    timeout: 60s
    retry_on_failure:
       enabled: true

Table 100.

Field	Description	Default value
`endpoint`	In-cluster DNS address and port of the Trace Router service. Update the namespace if you used a different name.	`trace-router-service.virtana-trace-router...:4317`
`timeout`	The maximum time the collector waits for the Trace Router to acknowledge a batch before retrying.	`60s`
`retry_on_failure.enabled`	When true, the collector retries failed export attempts automatically.	`true`

Note

Choose a port based on your OTEL collector's trace format.

GRPC Jaeger Port: 9001
GRPC OTLP Port: 4317

External method

Use this method when your organization operates a centralized Tempo backend, and you want TraceGW to forward processed traces to it for long-term storage and querying. This method extends the standard pipeline with additional Tempo connection settings.

Step 1: Enable tracing in the North Backend values files

Update the global section of your North Backend Helm values file to activate tracing and configure the Tempo connection. The key difference from the standard method is that trace_external is set to true, and the Tempo URL and headers are provided.

global:
backend:
trace_enabled: true
trace_client: "TEMPO"
trace_external: true
trace_external_url: "http://example.tempo.com"
## Optional. Format: "X-Scope-OrgID=my-org, Bearer=<token>"
trace_headers: ""

The following table describes each field in the configuration file.

Table 101.

Field	Description	Default value
`trace_enabled`	Activates the tracing feature in the CO backend.	`true`
`trace_client`	Specifies the tracing client protocol.	`TEMPO`
`trace_external`	Set to true for the external Tempo method. TraceGW uses `trace_external_url` to forward processed traces to your Tempo instance over the network.	`true`
`trace_external_url`	Full URL of your external Tempo backend. Replace the example value with your actual Tempo endpoint.	`http://example.tempo.com`
`trace_headers`	Optional HTTP headers are included in every request to Tempo. Use for multi-tenant Tempo setups that require an org ID header (for example, X-Scope-OrgID=my-org).	`""`

Step 2: Create or update the ArgoCD values files for TraceGW

The ArgoCD values configuration for the external Tempo method is the same as the standard method, with one important difference: set tracegwconf.enable-trace-forwarding to true to activate forwarding to the Tempo backend.

namespaces:
argocd: argocd
apps:
virtana_tracing_onprem:
enabled: true
version: 2024.12.4
clusters:
- name: xxxx 
tenantID: xxxx 
names:
app: "virtana-tracegw"
values:
global:
secret_source: "none"
gatewayCreds:
environment:
OPSCRUISE_ENDPOINT: "kafka-service.{{ .clusterName }}.svc.cluster.local:9092"
KEYCLOAK_ENABLED: "true"
KEYCLOAK_URL: "https://wind-keycloak.oc.dev.cloud.virtana.com:443"
KEYCLOAK_REALM: "{{ .tenantID }}"
OPSCRUISE_ACCOUNT_ID: "{{ .clusterName }}"
k8sgw:
enabled: false
promgw:
enabled: false
loggw-loki:
enabled: false
loki-stack:
enabled: false
prometheus:
enabled: false
opscruise-node-exporter-new:
enabled: false
kube-state-metrics:
enabled: false
tracegw:
enabled: true
tracegw:
storageClassName: ""
slo_storage_size: 20Gi
config:
tracegw:
tracegwconf.trace-source: tempo
tracegwconf.slo-enabled: "true"
tracegwconf.slo-training-minutes: "30"
tracegwconf.write-slo-slow-traces-to-alogq: "true"
tracegwconf.meters-exclude-all-tags: "true"
tracegwconf.mode: listen
tracegwconf.enable-trace-forwarding: "true" 
tracegwconf.filter-tag-key: "cluster"
tracegwconf.find-cluster-tag-keys: "cluster"

Refer to Step 2 of the Standard method for a detailed description of all fields.

Steps 3 and 4: Trace Router and OTEL Collector configuration

The Trace Router deployment (Step 3) and OTEL Collector configuration (Step 4) are identical for both methods.

Verify the deployment

After completing all deployment steps for either method, verify that the tracing components are running and data is flowing correctly.

Perform the following steps:

Run the command to check that TraceGW pods are running in the North Backend namespace:
```
kubectl get pods -n <your-backend-namespace> | grep tracegw
```
Check that the Trace Router pod is running:
```
kubectl get pods -n virtana-trace-router
```
Check that the OTEL Collector pods are running in each South cluster:
```
kubectl get pods -n <your-south-namespace> | grep otel
```
Send a test trace from a South cluster OTEL Collector and confirm it reaches TraceGW.
For the external method, also verify that the trace appears in your Tempo backend.
In the Virtana CO UI, navigate to the service map and confirm that trace-based latency and error metrics are visible for your instrumented services.

Configuring Splunk log collection

Splunk Log Collection is a logging integration that allows the Virtana Platform to forward application logs, traces, and metadata to an external Splunk instance. When enabled, logs generated by both the frontend and backend services are streamed to your Splunk environment, enabling centralized log management.

Enabling Splunk Log Collection aggregates logs from all Virtana Platform services into a single Splunk instance for unified visibility. It quickly searches and correlates logs across pods, namespaces, and containers to diagnose issues. It leads to setting up Splunk alerts and dashboards based on real-time log data from your clusters.

By default, the platform does not send logs to Splunk. You need to configure this if your organization uses Splunk as its log management solution and wants to integrate Virtana Platform logs into it.

To enable Splunk Log Collection, set the log_source parameter to splunk in your Helm chart values files and configure the required parameters below. Configuration is split across two files:

Frontend values.yaml: Contains connection details, headers, labels, SSL, and trace settings.
Backend values.yaml: Contains the cluster name and Splunk index definitions.

Configuration values file

global:
  log_config:
    log_source: splunk
     splunk_external_url: ""
     splunk_custom_headers: ""
     splunk_params: ""
     splunk_label_configs:
       cluster_name: "cluster_name"
       pod: "pod"
       namespace: "namespace"
       node: "node"
       container_name: "container_name"
       spanID: ""
       traceID: ""
     splunk_enable_ssl_context: false
     splunk_auth_token: ""
     splunk_trace_params: ""

    splunk_cluster_name: ""
    splunk_indexes:
    #   - main
    #   - prod

The following table describes each field in the configuration file.

Table 102.

Field	Description	Default value
General parameter
`log_source`	The logging backend to use. When set to `splunk`, all other Splunk-related parameters below become applicable.	`splunk`
Frontend parameters
`splunk_external_url`	The URL of your external Splunk HTTP Event Collector (HEC) endpoint.	`""`
`splunk_custom_headers`	Comma-separated custom HTTP headers to include in requests to Splunk.	`""`
`splunk_params`	Additional query parameters appended to Splunk log requests.	`""`
`splunk_label_configs`	A mapping of Kubernetes metadata labels to include with each log entry sent to Splunk.
`cluster_name`	Label key for the Kubernetes cluster name.	`cluster_name`
`pod`	Label key for the pod name that generated the log.	`pod`
`namespace`	Label key for the Kubernetes namespace.	`namespace`
`node`	Label key for the Kubernetes node where the pod is running.	`node`
`container_name`	Label key for the container name within the pod.	`container_name`
`spanID`	Label key for the distributed trace span ID.	`""`
`traceID`	Label key for the distributed trace ID.	`""`
`splunk_enable_ssl_context`	Set to `true` to enable SSL/TLS verification when connecting to the Splunk endpoint.	`false`
`splunk_auth_token`	The authentication token for the Splunk HEC endpoint.	`""`
`splunk_trace_params`	Filter parameters for trace data sent to Splunk.	`""`
Backend parameters
`splunk_cluster_name`	The name of the Kubernetes cluster as it should appear in Splunk.	""
`splunk_indexes`	A list of Splunk index names where logs should be stored.	[]

Getting started

Perform the following steps:

Obtain a Splunk HEC token and endpoint URL from your Splunk administrator.
Set log_source: splunk under global.log_config.
In your frontend values.yaml, provide splunk_external_url and splunk_auth_token at minimum.
In your backend values.yaml, set splunk_cluster_name and optionally define splunk_indexes.
Deploy or upgrade your Helm release with the updated values files.
Verify logs are arriving in Splunk by searching for entries tagged with your cluster name.

In this section:

Container Observability post installation

CO tracing

CO Splunk log collection

Enable tracing for Container Observability

Prerequisites

CO South deployment

Namespace and quota requirements

Secret synchronization

Tempo backend details (external method only)

Error tag list

Standard method

Step 1: Enable tracing in the North Backend values file

Step 2: Create or update the ArgoCD values files for TraceGW

Step 3: Deploy Trace Router and TraceGW using ArgoCD

Step 4: Configure OTEL collectors in South clusters

Note

External method

Step 1: Enable tracing in the North Backend values files

Step 2: Create or update the ArgoCD values files for TraceGW

Steps 3 and 4: Trace Router and OTEL Collector configuration

Verify the deployment

Configuring Splunk log collection

Configuration values file

Getting started

Search results