Kafka

Kafka is a publish-subscribe message queuing system that’s designed like a distributed commit log. CloudWisdom can help monitor the performance and throughput of your Kafka server using our Kafka collector for the Linux agent. Kafka, Kafka Consumer Lag, and Zookeeper metrics are all collected using this collector.

Prerequisites

The Linux Agent is required. If you need to disable the Linux integration or view the unique API key assigned to your account, navigate to the Integrations page under the user account drop-down menu and click the integration designated as Infrastructure under the Integration column.

Configuration

  1. Download the Jolokia JVM JAR file.

  2. Move the downloaded file to the /opt/netuitive-agent/ directory.

  3. Run the following at the command line to set an environment variable:

    export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar"
    
  4. Restart Kafka, and confirm Jolokia is running by accessing http://localhost:8778/jolokia/

  5. Navigate to the collectors folder, /opt/netuitive-agent/conf/collectors.

  6. Open the KafkaJolokiaCollector.conf file.

  7. Change the enabled setting to True.

  8. Update version to be the version of Kafka running on your machine and uncomment the line.

    • Optionally, update host and port to the correct settings.

  9. Save the file and restart the Linux Agent.

This integration’s package will be automatically enabled and provisioned to your account as soon as CloudWisdom receives data from the integration. The PACKAGES button on the integration setup page will become active once data is received, so you’ll be able to disable and re-enable the package at will.

Collector Options

The path and jolokia_path settings should not be changed without consulting Virtana Support first.

Option

Default

Description

enabled

FALSE

Enable collecting Kafka metrics.

path

kafka

The metric prefix, e.g., how you want the metrics to show up in CloudWisdom.

jolokia_path

jolokia

Part of the URL path that points to where your application serves metrics. Typically jmx or jolokia.

metrics_blacklist

.*Percentile$

.FifteenMinuteRate.

host

localhost

Hostname/IP address by which the Kafka instance can be reached by the Linux agent.

port

8778

The port that the Jolokia JAR file is listening on.

version

0.8

Specifies the Kafka major release being used on the host. Currently, Virtana supports 0.8, 0.9, and 0.10.

zookeeper

localhost:2181

The hostname / IP address as well as the port by which the Zookeeper instance can be reached by the Linux agent.

bin

/opt/kafka/bin/kafka-run-class.sh

The location of the kafka-run-class.sh file. Certain metrics must be retrieved via command line calls, thus the location of kafka-run-class.sh is a necessity. The exception to this being if you’re running Kafka in a Docker container; see the argssetting below for more information.

args

exec {kafka-container-id} {kafka-run-class.sh}

The argument passed to Kafka running in a docker container. If you are running Kafka in a docker container, you’ll need to set the bin setting to the location of the docker binary. Then, you’ll need to replace {kafka-container-id} with the ID of your Kafka container and {kafka-run-class.sh} with the location of the kafka-run-class.sh file within the container.

consumer_groups

Group1, Group2, Group3

For Kafka 8 or earlier: This setting specifies a list of consumer groups that you want to receive consumer lag metrics. If you do not specify this, no consumer lag metrics will be collected. For Kafka 9 or later: This setting is ignored as consumer groups can be auto-discovered.

topics

Topic1, Topic2, Topic3

For Kafka 8 or earlier: This setting specifies a list of topics associated with the consumer groups. If you do not specify this, all topics are considered. For Kafka 9 or later: this setting is ignored.

byte_unit

Default numeric output(s).

measure_collector_time

Measure the collector’s run time in milliseconds.

metrics_whitelist

Regex list to match metrics to transmit. Mutually exclusive with metrics_blacklist option.