Skip to main content

VM Process - Collectors and Exporters deployment

This guide explains how to collect metrics and logs from virtual machines (VMs) that run outside Kubernetes and send that data to Virtana South in your Kubernetes cluster. It describes the data flow between the VM, Virtana South, and Virtana North, so you understand where each change applies.

The monitoring stack involves three layers:

  • On the VM: Docker containers run Process Exporter (process-level metrics), Node Exporter (host-level metrics), and Promtail (log collection and forwarding).

  • On Virtana South (Kubernetes namespace: opscruise): Prometheus or OpenTelemetry Collector (OTEL) scrapes metrics from the exporters; Loki ingests logs pushed by Promtail.

  • On Virtana North (optional): Enables an S3-backed Loki log store to support log-based alerting and long-term log retention.

Virtana South configuration

This section covers the changes you must make to Virtana South before the virtual machine (VM) can send data to it. You expose the Loki service so Promtail can reach it, configure the metric scraper to poll the VM exporters, and then redeploy. Complete sections 1.1 through 1.4 in order.

Expose the Loki service

Promtail on the VM pushes logs to Loki over HTTP. To make Loki reachable from outside the cluster, you must expose it through a LoadBalancer service. Add the following configuration to virtana-co-values.yaml and then redeploy Virtana South.

loki-stack:
  loki:
    service:
      type: LoadBalancer  

The following table describes each parameter in the snippet above.

Table 131.

Parameter

Description

loki-stack

Top-level chart values section for the Loki stack in Virtana South.

loki-stack.loki

Configuration block for the Loki service.

loki-stack.loki.service

Network exposure settings for the Loki pod inside the cluster.

loki-stack.loki.service.type: LoadBalancer

Provisions a cloud or native load balancer and assigns a stable external IP address. Promtail uses this IP to push logs from the VM.



After redeploying, confirm that Loki has been assigned an external IP address by running the following command:

kubectl get svc opscruise-bundle-loki -n opscruise    

Configure the metric collector

Virtana South can receive VM metrics using either Prometheus scraping or the OpenTelemetry Collector (OTEL). Choose the method that fits your environment and add the corresponding configuration to virtana-co-values.yaml.

Prometheus scraping is the most common method. Add the following scrape job to virtana-co-values.yaml. Replace each x.x.x.x placeholder with the VM's IP address.

global:
  metricScraper: "prometheus"

prometheus:
  nonIstioConfigMap:
    additionalScrapeConfigs:
      - job_name: monitor-process-vms
        static_configs:
          - targets:
              - "x.x.x.x:9256"  
              - "x.x.x.x:9100"  
              - "x.x.x.x:9080"  
        scheme: "http"
        tls_config:
          insecure_skip_verify: true

The following table describes each parameter in the snippet above.

Table 132.

Parameter

Description

global.metricScraper

Selects the metric pipeline. Set to "prometheus" to enable Prometheus scraping.

prometheus.nonIstioConfigMap

The Prometheus configuration block used when Istio is not managing scraping in your cluster.

prometheus.nonIstioConfigMap.additionalScrapeConfigs

Appends custom scrape jobs to the existing Prometheus configuration without overwriting the base config.

job_name: monitor-process-vms

A logical label for the VM scraping job. Use this name when filtering targets or creating alerts in Prometheus.

targets: x.x.x.x:9256

The Process Exporter metrics endpoint on the VM.

targets: x.x.x.x:9100

The Node Exporter metrics endpoint on the VM.

targets: x.x.x.x:9080

The Promtail internal metrics endpoint on the VM. This target is optional and is not required for log ingestion.

scheme: "http"

The protocol Prometheus uses to scrape the targets.

tls_config.insecure_skip_verify: true

Disables TLS certificate verification when scraping. This setting is safe when scraping over plain HTTP or when using self-signed certificates.



You can configure Virtana South to ingest VM metrics using the OpenTelemetry (OTEL) Collector instead of Prometheus. Add the following scrape job tovirtana-co-values.yaml. Replace each x.x.x.x placeholder with the VM's IP address.

global:
  metricScraper: "otel"

otel-metric-collector:
  additional_receivers_configs:
    prometheus:
      config:
        scrape_configs:
          - job_name: monitor-process-vms
            static_configs:
              - targets:
                  - "x.x.x.x:9256"  
                  - "x.x.x.x:9100"  
                  - "x.x.x.x:9080"  
            scheme: "http"
            tls_config:
              insecure_skip_verify: true

Deploy or update Virtana South

After editing virtana-co-values.yaml, apply your changes using Helm so that the Loki exposure and metric scraping configuration takes effect. Complete both steps below.

  1. Run your standard Helm upgrade or install command for the Virtana South bundle.

  2. Confirm that the opscruise namespace is targeted during the upgrade.

Verify the deployment

After the Helm upgrade completes, confirm that all components are healthy and your configuration is active. The following commands check the most common failure points.

Run the following command to check that all pods are running.

kubectl get pods -n opscruise

Run the following command to confirm Loki has an external IP.

kubectl get svc opscruise-bundle-loki -n opscruise

Run the following command to confirm the Prometheus config map is present.

kubectl get cm prometheus-configmap -n opscruise    

Note

If the monitor-process-vms job appears in Prometheus Targets with a status of UP, the metrics pipeline is ready.

VM configuration

This section walks you through deploying the three Docker containers that collect metrics and logs on the virtual machine (VM), using Docker-based exporters and Promtail.

Two deployment methods are available:

  • Docker Compose (recommended for most environments)

  • Docker CLI (when Compose is not available)

Prerequisites

Before deploying containers, confirm that the VM meets the following requirements.

  • Docker Engine is installed on the VM. See Install Docker Engine.

  • You can authenticate to the container registry:

    docker login -u 'xxxxx' -p 'xxxxx'        
  • You have created the promtail.yaml configuration file on the VM.

Promtail configuration

Promtail collects logs from the VM and forwards them to Loki in Virtana South. Before deploying Promtail, create a promtail.yaml file on the VM using the template below. This file defines the HTTP server settings, the Loki push endpoint, file position tracking, and the log scrape rules for syslog, /var/log, Docker containers, and systemd journals.

client:
  backoff_config:
    max_period: 5m
    max_retries: 10
    min_period: 500ms
  batchsize: 1048576
  batchwait: 1s
  timeout: 10s
  external_labels: {}
positions:
  filename: /run/promtail/positions.yaml
server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080
target_config:
  sync_period: 10s

scrape_configs:
  - job_name: syslog-debian
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          namespace: syslog
          logtype: syslog
          __path__: /var/log/syslog
    pipeline_stages:
      - regex:
          expression: '(?P<month>[^ ]*) (?P<day>[^ ]*) (\d+:\d+:\d+) (?P<host>[^ ]*) (?P<syslog_type>[^ ]*)\b(\[(?P<procid>[^ ]*)\])?: .*'
      - metrics:
          log_lines_total: 

The following table describes each parameter in promtail.yaml.

Parameter

Description

client.backoff_config.min_period

Minimum wait time between retry attempts when Loki is unavailable.

client.backoff_config.max_period

Maximum wait time between retry attempts. Retries back off up to this limit.

client.backoff_config.max_retries

Maximum number of retry attempts before Promtail stops trying to push a batch.

client.batchsize

Maximum uncompressed size of a log batch in bytes (1 MiB) before it is pushed to Loki.

client.batchwait

Maximum time Promtail waits before sending a batch, even if batchsize has not been reached.

client.timeout

HTTP timeout for push requests to Loki.

client.external_labels

Static labels added to all log streams pushed to Loki. Leave empty, or add labels such as environment or region for filtering.

positions.filename

Path where Promtail stores the last-read file offsets. Promtail uses this file to resume reading from the correct position after a restart.

server.http_listen_address

The IP address Promtail's internal HTTP server listens on. 0.0.0.0 binds to all network interfaces.

server.http_listen_port

The port for Promtail's HTTP server. This port also exposes the /metrics endpoint for Prometheus scraping.

target_config.sync_period

How often Promtail syncs target state and file positions.

scrape_configs[*].job_name

A logical name for each log scraping job. Used as a label in Loki for filtering.

static_configs.labels.__path__

A file path or glob pattern that tells Promtail which files to tail.

pipeline_stages.regex.expression

A regular expression that parses each log line into named fields (for example, month, host, syslog_type).

pipeline_stages.metrics

Emits internal Promtail counters (such as line count and byte count) as Prometheus metrics.

labels

Adds extracted or static key-value labels to log streams for filtering and querying in Loki.

docker_sd_configs

Discovers Docker containers via the Docker socket. Used to collect logs from running containers; you can filter to specific containers such as Virtana services.

relabel_configs

Renames service discovery metadata into meaningful Loki labels, such as container, pod, and namespace.

journal.max_age

Sets how far back Promtail reads from the systemd journal on startup. For example, 12h reads entries from the past 12 hours.

Docker Compose deploys Promtail, Node Exporter, and Process Exporter as a single managed stack. This method is easier to operate and upgrade than running individual docker run commands.

  1. Organize all configuration files in a single folder to keep the stack self-contained.

    mkdir virtana-docker-compose
    cd virtana-docker-compose       
  2. Create a .env file in the project directory to centralize image versions and the Loki endpoint. Update the values when you need to upgrade a component or change the Loki address.

    IMAGE_PROMTAIL=grafana/promtail:2.9.2
    
    IMAGE_NODE_EXPORTER=virtana/oc-node-exporter:core-1.1.0-opsc-2024.12.1
    
    IMAGE_PROCESS_EXPORTER=virtana/oc-process-exporter:rel.48.1.0
    
    PROMTAIL_CONFIG_PATH=./promtail.yaml
    
    LOKI_ENDPOINT=x.x.x.x:3100        

    The following table describes each variable.

    Variable

    Description

    IMAGE_PROMTAIL

    The Promtail container image and tag to deploy. Update the tag to change the Promtail version.

    IMAGE_NODE_EXPORTER

    The Node Exporter container image and tag to deploy.

    IMAGE_PROCESS_EXPORTER

    The Process Exporter container image and tag to deploy.

    PROMTAIL_CONFIG_PATH

    Local path to the promtail.yaml file on the VM. This file is mounted into the Promtail container at runtime.

    LOKI_ENDPOINT

    The external IP address and port of the Loki LoadBalancer provisioned in Expose the Loki service. For example, 203.0.113.10:3100.

  3. Create a docker-compose.yaml file in the project directory. The file defines the three services and their host-level volume and network settings.

    services:
      virtana-promtail:
        image: ${IMAGE_PROMTAIL}
        container_name: virtana-promtail
        restart: always
        ports:
          - "9080:9080"
        volumes:
          - /run/promtail:/run/promtail
          - /var/run/docker.sock:/var/run/docker.sock
          - /var/lib/docker/containers:/var/lib/docker/containers
          - ${PROMTAIL_CONFIG_PATH}:/etc/promtail/promtail.yaml
          - /var/log:/var/log
          - /var/log/journal:/var/log/journal
        command:
          - "--config.file=/etc/promtail/promtail.yaml"
          - "--config.expand-env=true"
          - "--client.url=http://${LOKI_ENDPOINT}/loki/api/v1/push"
    
      virtana-node-exporter:
        image: ${IMAGE_NODE_EXPORTER}
        container_name: virtana-node-exporter
        restart: always
        network_mode: host
        pid: host
        privileged: true
        cap_add:
          - SYS_ADMIN
          - SYS_TIME
          - NET_ADMIN
        volumes:
          - /etc/os-release:/host/etc/os-release:ro
          - /var:/host/var:ro
          - /dev:/host/dev:ro
          - /proc:/host/proc:ro
          - /sys:/host/sys:ro
          - /:/rootfs:ro
          - /boot:/boot:ro
          - /lib/modules:/lib/modules:ro
          - /usr/src:/usr/src:ro
        command:
          - '--path.procfs=/host/proc'
          - '--path.sysfs=/host/sys'
          - '--collector.conntrackext.forward-and-reverse-path-packets'
          - '--collector.ocflowbpfcollector.enable-all-metrics'
          - '--collector.ocflowbpfcollector.interface=^ens|^cali|^eth|^veth|^eni'
          - '--no-collector.rapl'
          - '--no-collector.pressure'
          - '--no-collector.ocprocessevent'
          - '--no-collector.ipvs_extended'
          - '--no-collector.ipvs'
          - '--no-collector.xfs'
          - '--no-collector.powersupplyclass'
          - '--no-collector.netstat'
          - '--no-collector.thermal_zone'
          - '--no-collector.entropy'
          - '--no-collector.schedstat'
          - '--no-collector.sockstat'
          - '--no-collector.softnet'
          - '--no-collector.timex'
          - '--no-collector.ocflowbpfcollector.drop-summary-metric-for-single-url'
    
      virtana-process-exporter:
        image: ${IMAGE_PROCESS_EXPORTER}
        container_name: virtana-process-exporter
        restart: always
        privileged: true
        network_mode: host
        volumes:
          - /proc:/host/proc:ro
        command:
          - "--procfs=/host/proc"
          - "-config.path=/opt/opscruise/config.yaml"        

    The following table describes the key settings in each service definition.

    Setting

    Description

    virtana-promtail.ports: 9080:9080

    Exposes Promtail's HTTP server on port 9080 on the host. Used for local health checks and optional Prometheus scraping of Promtail's own metrics.

    virtana-promtail.volumes

    Mounts the Docker socket, container log directories, the promtail.yaml config, host log files, and the systemd journal into the Promtail container so it can collect all log sources.

    virtana-promtail.command --client.url

    Sets the Loki push endpoint. The value is built from the LOKI_ENDPOINT environment variable.

    virtana-node-exporter.network_mode: host

    Runs the container in the host network namespace so Node Exporter can observe host-level network metrics.

    virtana-node-exporter.pid: host

    Shares the host PID namespace with the container, giving Node Exporter visibility into host processes and namespaces.

    virtana-node-exporter.cap_add: SYS_ADMIN, SYS_TIME, NET_ADMIN

    Grants the capabilities required for kernel, network, and time-related metrics collection without running as fully privileged.

    virtana-node-exporter.volumes

    Mounts host filesystems (/proc, /sys, /dev, and others) in read-only mode so Node Exporter can read kernel and hardware metrics without modifying the host.

    --collector.conntrackext.forward-and-reverse-path-packets

    Enables collection of conntrack packet metrics for both forward and reverse packet paths.

    --collector.ocflowbpfcollector.enable-all-metrics

    Enables all available BPF-based network flow metrics.

    --collector.ocflowbpfcollector.interface

    Regex pattern matching the network interfaces to monitor for flow metrics (for example, eth0, ens3, veth*).

    --no-collector.* flags

    Disables specific collectors (such as rapl, netstat, and timex) that are either not needed or produce noisy data in most VM environments.

    virtana-process-exporter.volumes: /proc:/host/proc:ro

    Mounts the host /proc filesystem in read-only mode so Process Exporter can enumerate all running processes.

    virtana-process-exporter.command --procfs

    Points Process Exporter to the mounted host /proc directory.

    virtana-process-exporter.command -config.path

    Path to the bundled process configuration file inside the container. This file maps each process by username, command, and PID.

  4. Copy your promtail.yaml file to the path specified in PROMTAIL_CONFIG_PATH. Confirm that the client.url value in the file points to the correct Loki LoadBalancer endpoint.

  5. From the project directory, start all three containers in detached mode:

    docker compose up -d        
  6. Check that all containers started successfully and review the logs for startup errors:

    docker compose ps
    docker compose logs virtana-promtail
    docker compose logs virtana-node-exporter
    docker compose logs virtana-process-exporter        
  7. Confirm that each exporter is serving metrics. Replace <VM_IP> with the VM's IP address.

    curl http://<VM_IP>:9080/metrics   
    curl http://<VM_IP>:9100/metrics   
    curl http://<VM_IP>:9256/metrics          

    A successful response returns a list of Prometheus-format metric lines.

Use this method if Docker Compose is not available on the VM. Each container is deployed individually using docker run. Complete the three sub-sections below in order: Process Exporter, Node Exporter, then Promtail.

Deploy Process Exporter

The following commands remove any existing Process Exporter container and deploy a fresh one. The container mounts the host /proc filesystem to collect process metrics.

docker rm -f virtana-process-exporter

IMAGE="virtana/oc-process-exporter:rel.48.1.0"

docker run --name virtana-process-exporter \
  --detach=true --restart always \
  -p 9256:9256 --privileged \
  -v /proc:/host/proc --net=host $IMAGE \
  --procfs /host/proc \
  -config.path /opt/opscruise/config.yaml      

The bundled configuration file inside the container (/opt/opscruise/config.yaml) maps each process by username, command name, and PID:

process_names:
  - name: "{{.Username}}^{{.Comm}}^{{.PID}}"
    cmdline:
      - '.+'      

The following table describes the key flags used in the docker run command above.

Flag

Description

--net=host

Runs the container in the host network namespace for complete and accurate process enumeration.

-v /proc:/host/proc

Mounts the host /proc filesystem so Process Exporter can read all running processes.

-config.path

Uses the built-in configuration file that identifies each process by username, command, and PID.

Verify that the container is running and serving metrics:

docker container ls | grep virtana
docker logs virtana-process-exporter -f
curl -s http://<VM_IP>:9256/metrics    

Deploy Node Exporter

The following commands remove any existing Node Exporter container and deploy a fresh one with the appropriate host-level access for kernel, network, and process metrics.

docker rm -f virtana-node-exporter

IMAGE="virtana/oc-node-exporter:core-1.1.0-opsc-2024.12.1"

docker run --name virtana-node-exporter --detach=true --restart always \
  -v /etc/os-release:/host/etc/os-release \
  -v /var:/host/var -v /dev:/host/dev \
  -v /proc:/host/proc -v /sys:/host/sys \
  -v /:/rootfs -v /boot:/boot \
  -v /lib/modules:/lib/modules -v /usr/src:/usr/src \
  -p 9100:9100 --net=host \
  --pid=host --privileged \
  --cap-add=SYS_ADMIN --cap-add=SYS_TIME --cap-add=NET_ADMIN \
  -u root "${IMAGE}" \
  --path.procfs "/host/proc" \
  --path.sysfs "/host/sys" \
  --collector.conntrackext.forward-and-reverse-path-packets \
  --collector.ocflowbpfcollector.enable-all-metrics \
  --collector.ocflowbpfcollector.interface "^ens|^cali|^eth|^veth|^eni" \
  --no-collector.rapl --no-collector.pressure \
  --no-collector.ocprocessevent \
  --no-collector.ipvs_extended --no-collector.ipvs \
  --no-collector.xfs --no-collector.powersupplyclass \
  --no-collector.netstat --no-collector.thermal_zone \
  --no-collector.entropy --no-collector.schedstat \
  --no-collector.sockstat --no-collector.softnet \
  --no-collector.timex \
  --no-collector.ocflowbpfcollector.drop-summary-metric-for-single-url     

The following table describes the key flags used in the docker run command above.

Flag

Description

--pid=host

Shares the host PID namespace so Node Exporter can accurately read process and namespace metrics.

--cap-add flags

Grants the capabilities (SYS_ADMIN, SYS_TIME, NET_ADMIN) needed for kernel, network, and time metrics collection.

--collector.conntrackext.*

Enables conntrack and BPF-based flow metrics for detailed network visibility.

--no-collector.* flags

Disables collectors not needed in most VM environments, reducing overhead and metric noise. See Create docker-compose.yaml for descriptions of individual flags.

Verify that the container is running and serving metrics:

docker container ls | grep virtana
docker logs virtana-node-exporter -f
curl -s http://<VM_IP>:9100/metrics     

Deploy Promtail

The following commands remove any existing Promtail container and deploy a fresh one. Promtail reads your promtail.yaml file and pushes logs to the Loki LoadBalancer endpoint in Virtana South. Replace x.x.x.x:3100 with your Loki LoadBalancer IP and port, and /home/xxxx/promtail.yaml with the full path to your promtail.yaml file on the VM.

docker rm -f virtana-promtail

IMAGE="grafana/promtail:2.9.2"
LOKI_ENDPOINT="x.x.x.x:3100"
PROMTAIL_CONFIG_PATH="/home/xxxx/promtail.yaml"

docker run --name virtana-promtail --detach=true --restart=always \
  --privileged -p 9080:9080 \
  -v /run/promtail:/run/promtail \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/containers:/var/lib/docker/containers \
  -v ${PROMTAIL_CONFIG_PATH}:/etc/promtail/promtail.yaml \
  -v /var/log:/var/log \
  -v /var/log/journal:/var/log/journal \
  $IMAGE \
  --config.file=/etc/promtail/promtail.yaml \
  --config.expand-env=true \
  --client.url=http://${LOKI_ENDPOINT}/loki/api/v1/push  

Verify that the container is running and serving metrics:

docker container ls | grep virtana
docker logs virtana-promtail -f
curl -s http://<VM_IP>:9080/metrics     

Virtana North configuration (optional)

This section applies only if you need log-based alerting or long-term log retention. Virtana North supports these features by backing Loki with an Amazon S3 log store. Complete this section after Virtana South and the VM exporters are fully deployed and verified.

Note

If you are using a hosted Virtana SaaS environment, contact Virtana Support to apply this configuration on your behalf. Do not perform the steps in this section directly on a SaaS-managed installation.

Update the configuration

Add the following S3 settings to opscmain-values.yaml. Replace each placeholder value with your AWS credentials and the region where your S3 bucket is located.

loki:
  s3Logstore:
    enabled: true
    aws_access_key_id: "xxxx"
    aws_secret_access_key: "xxxx"
    aws_default_region: "xxxx"    

Caution

aws_access_key_id and aws_secret_access_key are sensitive credentials. Do not commit these values to version control. Use a secrets manager or Helm secrets plugin to inject them at deploy time.

The following table describes each parameter in the snippet above.

Parameter

Description

loki.s3Logstore.enabled

Set to true to activate the S3 storage backend for Loki on Virtana North. When enabled, Loki writes all ingested logs to the configured S3 bucket instead of local storage.

loki.s3Logstore.aws_access_key_id

AWS access key ID for the IAM user or role that has read and write permissions on the target S3 bucket.

loki.s3Logstore.aws_secret_access_key

AWS secret access key paired with aws_access_key_id. Keep this value confidential and out of version control.

loki.s3Logstore.aws_default_region

AWS region where the S3 bucket is located. For example, us-west-2. The region must match the bucket's actual region to avoid request failures.

Apply the configuration

Run the required Helm install or upgrade to apply the updated Loki values to Virtana North. After the upgrade completes, confirm that all pods roll successfully before you begin using log-based alerting.

  1. Run your standard Helm upgrade or install command for the Virtana North bundle, targeting the opscruise namespace.

  2. Confirm that all pods in the opscruise namespace are running and none are stuck in a crash loop:

    kubectl get pods -n opscruise

Validation and troubleshooting

Use the following checks to verify end-to-end data flow from the VM through to Virtana South and Virtana North, and to diagnose common problems. Perform these checks after all three components, Virtana South, the VM exporters, and Virtana North (if applicable), are deployed.

Check

How to verify

Prometheus targets

In Virtana South, open the Prometheus Targets view and confirm that the monitor-process-vms job shows a status of UP for all three VM endpoints.

Loki log ingestion

In Grafana or the Virtana log UI, filter by the labels defined in promtail.yaml — for example, namespace=syslog or job=varlogs — and confirm that log lines appear.

Network connectivity

From the VM, confirm that you can reach the Loki LoadBalancer IP on port 3100:

curl http://<LOKI_IP>:3100/ready

If the request times out, check that firewall rules allow outbound traffic from the VM to the cluster on port 3100.

Time synchronization

Confirm that NTP is active on the VM:

timedatectl status

Clock skew between the VM and the cluster can cause log ordering problems and missed scrape windows.

Resource limits

If log volume is high and Loki is struggling to keep up, reduce the batchsize or batchwait values in promtail.yaml, or increase the Loki resource limits in Virtana South. See Promtail configuration for parameter descriptions.