VM Process - Collectors and Exporters deployment
This guide explains how to collect metrics and logs from virtual machines (VMs) that run outside Kubernetes and send that data to Virtana South in your Kubernetes cluster. It describes the data flow between the VM, Virtana South, and Virtana North, so you understand where each change applies.
The monitoring stack involves three layers:
On the VM: Docker containers run Process Exporter (process-level metrics), Node Exporter (host-level metrics), and Promtail (log collection and forwarding).
On Virtana South (Kubernetes namespace:
opscruise): Prometheus or OpenTelemetry Collector (OTEL) scrapes metrics from the exporters; Loki ingests logs pushed by Promtail.On Virtana North (optional): Enables an S3-backed Loki log store to support log-based alerting and long-term log retention.
Virtana South configuration
This section covers the changes you must make to Virtana South before the virtual machine (VM) can send data to it. You expose the Loki service so Promtail can reach it, configure the metric scraper to poll the VM exporters, and then redeploy. Complete sections 1.1 through 1.4 in order.
Expose the Loki service
Promtail on the VM pushes logs to Loki over HTTP. To make Loki reachable from outside the cluster, you must expose it through a LoadBalancer service. Add the following configuration to virtana-co-values.yaml and then redeploy Virtana South.
loki-stack:
loki:
service:
type: LoadBalancer The following table describes each parameter in the snippet above.
Parameter | Description |
|---|---|
| Top-level chart values section for the Loki stack in Virtana South. |
| Configuration block for the Loki service. |
| Network exposure settings for the Loki pod inside the cluster. |
| Provisions a cloud or native load balancer and assigns a stable external IP address. Promtail uses this IP to push logs from the VM. |
After redeploying, confirm that Loki has been assigned an external IP address by running the following command:
kubectl get svc opscruise-bundle-loki -n opscruise
Configure the metric collector
Virtana South can receive VM metrics using either Prometheus scraping or the OpenTelemetry Collector (OTEL). Choose the method that fits your environment and add the corresponding configuration to virtana-co-values.yaml.
Prometheus scraping is the most common method. Add the following scrape job to virtana-co-values.yaml. Replace each x.x.x.x placeholder with the VM's IP address.
global:
metricScraper: "prometheus"
prometheus:
nonIstioConfigMap:
additionalScrapeConfigs:
- job_name: monitor-process-vms
static_configs:
- targets:
- "x.x.x.x:9256"
- "x.x.x.x:9100"
- "x.x.x.x:9080"
scheme: "http"
tls_config:
insecure_skip_verify: true
The following table describes each parameter in the snippet above.
Parameter | Description |
|---|---|
| Selects the metric pipeline. Set to |
| The Prometheus configuration block used when Istio is not managing scraping in your cluster. |
| Appends custom scrape jobs to the existing Prometheus configuration without overwriting the base config. |
| A logical label for the VM scraping job. Use this name when filtering targets or creating alerts in Prometheus. |
| The Process Exporter metrics endpoint on the VM. |
| The Node Exporter metrics endpoint on the VM. |
| The Promtail internal metrics endpoint on the VM. This target is optional and is not required for log ingestion. |
| The protocol Prometheus uses to scrape the targets. |
| Disables TLS certificate verification when scraping. This setting is safe when scraping over plain HTTP or when using self-signed certificates. |
You can configure Virtana South to ingest VM metrics using the OpenTelemetry (OTEL) Collector instead of Prometheus. Add the following scrape job tovirtana-co-values.yaml. Replace each x.x.x.x placeholder with the VM's IP address.
global:
metricScraper: "otel"
otel-metric-collector:
additional_receivers_configs:
prometheus:
config:
scrape_configs:
- job_name: monitor-process-vms
static_configs:
- targets:
- "x.x.x.x:9256"
- "x.x.x.x:9100"
- "x.x.x.x:9080"
scheme: "http"
tls_config:
insecure_skip_verify: trueDeploy or update Virtana South
After editing virtana-co-values.yaml, apply your changes using Helm so that the Loki exposure and metric scraping configuration takes effect. Complete both steps below.
Run your standard Helm upgrade or install command for the Virtana South bundle.
Confirm that the
opscruisenamespace is targeted during the upgrade.
Verify the deployment
After the Helm upgrade completes, confirm that all components are healthy and your configuration is active. The following commands check the most common failure points.
Run the following command to check that all pods are running.
kubectl get pods -n opscruise
Run the following command to confirm Loki has an external IP.
kubectl get svc opscruise-bundle-loki -n opscruise
Run the following command to confirm the Prometheus config map is present.
kubectl get cm prometheus-configmap -n opscruise
Note
If the monitor-process-vms job appears in Prometheus Targets with a status of UP, the metrics pipeline is ready.
VM configuration
This section walks you through deploying the three Docker containers that collect metrics and logs on the virtual machine (VM), using Docker-based exporters and Promtail.
Two deployment methods are available:
Docker Compose (recommended for most environments)
Docker CLI (when Compose is not available)
Prerequisites
Before deploying containers, confirm that the VM meets the following requirements.
Docker Engine is installed on the VM. See Install Docker Engine.
You can authenticate to the container registry:
docker login -u 'xxxxx' -p 'xxxxx'
You have created the
promtail.yamlconfiguration file on the VM.
Promtail configuration
Promtail collects logs from the VM and forwards them to Loki in Virtana South. Before deploying Promtail, create a promtail.yaml file on the VM using the template below. This file defines the HTTP server settings, the Loki push endpoint, file position tracking, and the log scrape rules for syslog, /var/log, Docker containers, and systemd journals.
client:
backoff_config:
max_period: 5m
max_retries: 10
min_period: 500ms
batchsize: 1048576
batchwait: 1s
timeout: 10s
external_labels: {}
positions:
filename: /run/promtail/positions.yaml
server:
http_listen_address: 0.0.0.0
http_listen_port: 9080
target_config:
sync_period: 10s
scrape_configs:
- job_name: syslog-debian
static_configs:
- targets: [localhost]
labels:
job: syslog
namespace: syslog
logtype: syslog
__path__: /var/log/syslog
pipeline_stages:
- regex:
expression: '(?P<month>[^ ]*) (?P<day>[^ ]*) (\d+:\d+:\d+) (?P<host>[^ ]*) (?P<syslog_type>[^ ]*)\b(\[(?P<procid>[^ ]*)\])?: .*'
- metrics:
log_lines_total: The following table describes each parameter in promtail.yaml.
Parameter | Description |
|---|---|
| Minimum wait time between retry attempts when Loki is unavailable. |
| Maximum wait time between retry attempts. Retries back off up to this limit. |
| Maximum number of retry attempts before Promtail stops trying to push a batch. |
| Maximum uncompressed size of a log batch in bytes (1 MiB) before it is pushed to Loki. |
| Maximum time Promtail waits before sending a batch, even if |
| HTTP timeout for push requests to Loki. |
| Static labels added to all log streams pushed to Loki. Leave empty, or add labels such as |
| Path where Promtail stores the last-read file offsets. Promtail uses this file to resume reading from the correct position after a restart. |
| The IP address Promtail's internal HTTP server listens on. |
| The port for Promtail's HTTP server. This port also exposes the |
| How often Promtail syncs target state and file positions. |
| A logical name for each log scraping job. Used as a label in Loki for filtering. |
| A file path or glob pattern that tells Promtail which files to tail. |
| A regular expression that parses each log line into named fields (for example, |
| Emits internal Promtail counters (such as line count and byte count) as Prometheus metrics. |
| Adds extracted or static key-value labels to log streams for filtering and querying in Loki. |
| Discovers Docker containers via the Docker socket. Used to collect logs from running containers; you can filter to specific containers such as Virtana services. |
| Renames service discovery metadata into meaningful Loki labels, such as |
| Sets how far back Promtail reads from the systemd journal on startup. For example, |
Docker Compose deploys Promtail, Node Exporter, and Process Exporter as a single managed stack. This method is easier to operate and upgrade than running individual docker run commands.
Organize all configuration files in a single folder to keep the stack self-contained.
mkdir virtana-docker-compose cd virtana-docker-compose
Create a
.envfile in the project directory to centralize image versions and the Loki endpoint. Update the values when you need to upgrade a component or change the Loki address.IMAGE_PROMTAIL=grafana/promtail:2.9.2 IMAGE_NODE_EXPORTER=virtana/oc-node-exporter:core-1.1.0-opsc-2024.12.1 IMAGE_PROCESS_EXPORTER=virtana/oc-process-exporter:rel.48.1.0 PROMTAIL_CONFIG_PATH=./promtail.yaml LOKI_ENDPOINT=x.x.x.x:3100
The following table describes each variable.
Variable
Description
IMAGE_PROMTAILThe Promtail container image and tag to deploy. Update the tag to change the Promtail version.
IMAGE_NODE_EXPORTERThe Node Exporter container image and tag to deploy.
IMAGE_PROCESS_EXPORTERThe Process Exporter container image and tag to deploy.
PROMTAIL_CONFIG_PATHLocal path to the
promtail.yamlfile on the VM. This file is mounted into the Promtail container at runtime.LOKI_ENDPOINTThe external IP address and port of the Loki LoadBalancer provisioned in Expose the Loki service. For example,
203.0.113.10:3100.Create a
docker-compose.yamlfile in the project directory. The file defines the three services and their host-level volume and network settings.services: virtana-promtail: image: ${IMAGE_PROMTAIL} container_name: virtana-promtail restart: always ports: - "9080:9080" volumes: - /run/promtail:/run/promtail - /var/run/docker.sock:/var/run/docker.sock - /var/lib/docker/containers:/var/lib/docker/containers - ${PROMTAIL_CONFIG_PATH}:/etc/promtail/promtail.yaml - /var/log:/var/log - /var/log/journal:/var/log/journal command: - "--config.file=/etc/promtail/promtail.yaml" - "--config.expand-env=true" - "--client.url=http://${LOKI_ENDPOINT}/loki/api/v1/push" virtana-node-exporter: image: ${IMAGE_NODE_EXPORTER} container_name: virtana-node-exporter restart: always network_mode: host pid: host privileged: true cap_add: - SYS_ADMIN - SYS_TIME - NET_ADMIN volumes: - /etc/os-release:/host/etc/os-release:ro - /var:/host/var:ro - /dev:/host/dev:ro - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro - /boot:/boot:ro - /lib/modules:/lib/modules:ro - /usr/src:/usr/src:ro command: - '--path.procfs=/host/proc' - '--path.sysfs=/host/sys' - '--collector.conntrackext.forward-and-reverse-path-packets' - '--collector.ocflowbpfcollector.enable-all-metrics' - '--collector.ocflowbpfcollector.interface=^ens|^cali|^eth|^veth|^eni' - '--no-collector.rapl' - '--no-collector.pressure' - '--no-collector.ocprocessevent' - '--no-collector.ipvs_extended' - '--no-collector.ipvs' - '--no-collector.xfs' - '--no-collector.powersupplyclass' - '--no-collector.netstat' - '--no-collector.thermal_zone' - '--no-collector.entropy' - '--no-collector.schedstat' - '--no-collector.sockstat' - '--no-collector.softnet' - '--no-collector.timex' - '--no-collector.ocflowbpfcollector.drop-summary-metric-for-single-url' virtana-process-exporter: image: ${IMAGE_PROCESS_EXPORTER} container_name: virtana-process-exporter restart: always privileged: true network_mode: host volumes: - /proc:/host/proc:ro command: - "--procfs=/host/proc" - "-config.path=/opt/opscruise/config.yaml"The following table describes the key settings in each service definition.
Setting
Description
virtana-promtail.ports: 9080:9080Exposes Promtail's HTTP server on port 9080 on the host. Used for local health checks and optional Prometheus scraping of Promtail's own metrics.
virtana-promtail.volumesMounts the Docker socket, container log directories, the
promtail.yamlconfig, host log files, and the systemd journal into the Promtail container so it can collect all log sources.virtana-promtail.command --client.urlSets the Loki push endpoint. The value is built from the
LOKI_ENDPOINTenvironment variable.virtana-node-exporter.network_mode: hostRuns the container in the host network namespace so Node Exporter can observe host-level network metrics.
virtana-node-exporter.pid: hostShares the host PID namespace with the container, giving Node Exporter visibility into host processes and namespaces.
virtana-node-exporter.cap_add: SYS_ADMIN, SYS_TIME, NET_ADMINGrants the capabilities required for kernel, network, and time-related metrics collection without running as fully privileged.
virtana-node-exporter.volumesMounts host filesystems (
/proc,/sys,/dev, and others) in read-only mode so Node Exporter can read kernel and hardware metrics without modifying the host.--collector.conntrackext.forward-and-reverse-path-packetsEnables collection of conntrack packet metrics for both forward and reverse packet paths.
--collector.ocflowbpfcollector.enable-all-metricsEnables all available BPF-based network flow metrics.
--collector.ocflowbpfcollector.interfaceRegex pattern matching the network interfaces to monitor for flow metrics (for example,
eth0,ens3,veth*).--no-collector.*flagsDisables specific collectors (such as
rapl,netstat, andtimex) that are either not needed or produce noisy data in most VM environments.virtana-process-exporter.volumes: /proc:/host/proc:roMounts the host
/procfilesystem in read-only mode so Process Exporter can enumerate all running processes.virtana-process-exporter.command --procfsPoints Process Exporter to the mounted host
/procdirectory.virtana-process-exporter.command -config.pathPath to the bundled process configuration file inside the container. This file maps each process by username, command, and PID.
Copy your
promtail.yamlfile to the path specified inPROMTAIL_CONFIG_PATH. Confirm that theclient.urlvalue in the file points to the correct Loki LoadBalancer endpoint.From the project directory, start all three containers in detached mode:
docker compose up -d
Check that all containers started successfully and review the logs for startup errors:
docker compose ps docker compose logs virtana-promtail docker compose logs virtana-node-exporter docker compose logs virtana-process-exporter
Confirm that each exporter is serving metrics. Replace
<VM_IP>with the VM's IP address.curl http://<VM_IP>:9080/metrics curl http://<VM_IP>:9100/metrics curl http://<VM_IP>:9256/metrics
A successful response returns a list of Prometheus-format metric lines.
Use this method if Docker Compose is not available on the VM. Each container is deployed individually using docker run. Complete the three sub-sections below in order: Process Exporter, Node Exporter, then Promtail.
Deploy Process Exporter
The following commands remove any existing Process Exporter container and deploy a fresh one. The container mounts the host /proc filesystem to collect process metrics.
docker rm -f virtana-process-exporter IMAGE="virtana/oc-process-exporter:rel.48.1.0" docker run --name virtana-process-exporter \ --detach=true --restart always \ -p 9256:9256 --privileged \ -v /proc:/host/proc --net=host $IMAGE \ --procfs /host/proc \ -config.path /opt/opscruise/config.yaml
The bundled configuration file inside the container (/opt/opscruise/config.yaml) maps each process by username, command name, and PID:
process_names:
- name: "{{.Username}}^{{.Comm}}^{{.PID}}"
cmdline:
- '.+' The following table describes the key flags used in the docker run command above.
Flag | Description |
|---|---|
| Runs the container in the host network namespace for complete and accurate process enumeration. |
| Mounts the host |
| Uses the built-in configuration file that identifies each process by username, command, and PID. |
Verify that the container is running and serving metrics:
docker container ls | grep virtana docker logs virtana-process-exporter -f curl -s http://<VM_IP>:9256/metrics
Deploy Node Exporter
The following commands remove any existing Node Exporter container and deploy a fresh one with the appropriate host-level access for kernel, network, and process metrics.
docker rm -f virtana-node-exporter
IMAGE="virtana/oc-node-exporter:core-1.1.0-opsc-2024.12.1"
docker run --name virtana-node-exporter --detach=true --restart always \
-v /etc/os-release:/host/etc/os-release \
-v /var:/host/var -v /dev:/host/dev \
-v /proc:/host/proc -v /sys:/host/sys \
-v /:/rootfs -v /boot:/boot \
-v /lib/modules:/lib/modules -v /usr/src:/usr/src \
-p 9100:9100 --net=host \
--pid=host --privileged \
--cap-add=SYS_ADMIN --cap-add=SYS_TIME --cap-add=NET_ADMIN \
-u root "${IMAGE}" \
--path.procfs "/host/proc" \
--path.sysfs "/host/sys" \
--collector.conntrackext.forward-and-reverse-path-packets \
--collector.ocflowbpfcollector.enable-all-metrics \
--collector.ocflowbpfcollector.interface "^ens|^cali|^eth|^veth|^eni" \
--no-collector.rapl --no-collector.pressure \
--no-collector.ocprocessevent \
--no-collector.ipvs_extended --no-collector.ipvs \
--no-collector.xfs --no-collector.powersupplyclass \
--no-collector.netstat --no-collector.thermal_zone \
--no-collector.entropy --no-collector.schedstat \
--no-collector.sockstat --no-collector.softnet \
--no-collector.timex \
--no-collector.ocflowbpfcollector.drop-summary-metric-for-single-url The following table describes the key flags used in the docker run command above.
Flag | Description |
|---|---|
| Shares the host PID namespace so Node Exporter can accurately read process and namespace metrics. |
| Grants the capabilities ( |
| Enables conntrack and BPF-based flow metrics for detailed network visibility. |
| Disables collectors not needed in most VM environments, reducing overhead and metric noise. See Create docker-compose.yaml for descriptions of individual flags. |
Verify that the container is running and serving metrics:
docker container ls | grep virtana docker logs virtana-node-exporter -f curl -s http://<VM_IP>:9100/metrics
Deploy Promtail
The following commands remove any existing Promtail container and deploy a fresh one. Promtail reads your promtail.yaml file and pushes logs to the Loki LoadBalancer endpoint in Virtana South. Replace x.x.x.x:3100 with your Loki LoadBalancer IP and port, and /home/xxxx/promtail.yaml with the full path to your promtail.yaml file on the VM.
docker rm -f virtana-promtail
IMAGE="grafana/promtail:2.9.2"
LOKI_ENDPOINT="x.x.x.x:3100"
PROMTAIL_CONFIG_PATH="/home/xxxx/promtail.yaml"
docker run --name virtana-promtail --detach=true --restart=always \
--privileged -p 9080:9080 \
-v /run/promtail:/run/promtail \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /var/lib/docker/containers:/var/lib/docker/containers \
-v ${PROMTAIL_CONFIG_PATH}:/etc/promtail/promtail.yaml \
-v /var/log:/var/log \
-v /var/log/journal:/var/log/journal \
$IMAGE \
--config.file=/etc/promtail/promtail.yaml \
--config.expand-env=true \
--client.url=http://${LOKI_ENDPOINT}/loki/api/v1/push Verify that the container is running and serving metrics:
docker container ls | grep virtana docker logs virtana-promtail -f curl -s http://<VM_IP>:9080/metrics
Virtana North configuration (optional)
This section applies only if you need log-based alerting or long-term log retention. Virtana North supports these features by backing Loki with an Amazon S3 log store. Complete this section after Virtana South and the VM exporters are fully deployed and verified.
Note
If you are using a hosted Virtana SaaS environment, contact Virtana Support to apply this configuration on your behalf. Do not perform the steps in this section directly on a SaaS-managed installation.
Update the configuration
Add the following S3 settings to opscmain-values.yaml. Replace each placeholder value with your AWS credentials and the region where your S3 bucket is located.
loki:
s3Logstore:
enabled: true
aws_access_key_id: "xxxx"
aws_secret_access_key: "xxxx"
aws_default_region: "xxxx" Caution
aws_access_key_id and aws_secret_access_key are sensitive credentials. Do not commit these values to version control. Use a secrets manager or Helm secrets plugin to inject them at deploy time.
The following table describes each parameter in the snippet above.
Parameter | Description |
|---|---|
| Set to |
| AWS access key ID for the IAM user or role that has read and write permissions on the target S3 bucket. |
| AWS secret access key paired with |
| AWS region where the S3 bucket is located. For example, |
Apply the configuration
Run the required Helm install or upgrade to apply the updated Loki values to Virtana North. After the upgrade completes, confirm that all pods roll successfully before you begin using log-based alerting.
Run your standard Helm upgrade or install command for the Virtana North bundle, targeting the
opscruisenamespace.Confirm that all pods in the
opscruisenamespace are running and none are stuck in a crash loop:kubectl get pods -n opscruise
Validation and troubleshooting
Use the following checks to verify end-to-end data flow from the VM through to Virtana South and Virtana North, and to diagnose common problems. Perform these checks after all three components, Virtana South, the VM exporters, and Virtana North (if applicable), are deployed.
Check | How to verify |
|---|---|
Prometheus targets | In Virtana South, open the Prometheus Targets view and confirm that the |
Loki log ingestion | In Grafana or the Virtana log UI, filter by the labels defined in |
Network connectivity | From the VM, confirm that you can reach the Loki LoadBalancer IP on port 3100: curl http://<LOKI_IP>:3100/ready If the request times out, check that firewall rules allow outbound traffic from the VM to the cluster on port 3100. |
Time synchronization | Confirm that NTP is active on the VM: timedatectl status Clock skew between the VM and the cluster can cause log ordering problems and missed scrape windows. |
Resource limits | If log volume is high and Loki is struggling to keep up, reduce the |