Kafka Monitoring User Guide

General information

Overview

Kafka monitoring is a Gateway configuration file that enables monitoring of Kafka Brokers through a set of samplers with customised JMX plug-in settings.

Kafka is a distributed streaming platform that allows you to:

  • Publish and subscribe to stream of records.
  • Store streams of records in a fault-tolerant way.
  • Process streams of records as they occur.

It is important to monitor Kafka because it carries crucial data that many applications rely on. Geneos provides a JMX server sampler configuration to monitor Kafka.

This guide discusses the steps to set up the Kafka integration on a Gateway. Once the integration is set up, the samplers providing the dataviews become available to that Gateway.

To view the sample metrics and dataviews, see Kafka Monitoring Technical Reference.

User requirements

This monitoring template is a Gateway configuration setup that can be included in the Gateway Setup Editor.

To use this template, your configuration must meet the following requirements:

  • A machine running the Netprobe must have access to the Kafka published HTTP address and port.
  • An Active Console 2 that is connected to the Gateway.
  • Kafka configuration files extracted to their Gateway set-up.
  • Specify two (2) Managed entities variables: KafkaJmxHost and KafkaJmxPort.
  • A JMX-enabled Kafka Cluster.

System requirements

The following requirements must be met prior to the installation and setup of the template:

  • Template package: geneos-integration-kafka-<version>.zip.
  • Managed entities utilising the samplers defined in the include/KafkaMonitoring.xml.
  • Netprobe version 4.5 or higher.

Audience

This document is a reference guide to introduce templates and scripts built using the Geneos development toolkit plug-ins.

The template allows you to integrate specific applications and services to collect metrics without having to create a new plug-in.

 

Install and set up

Ensure that you have read and can follow the system requirements prior to installation and setup of this integration template.

  1. Download the config file (geneos-integration-kafka-<version>.zip) from the ITRS Downloads site.
  2. Open Active Console 2.
  3. Extract the Kafka monitoring configuration file into the Gateway Setup directory.
  4. Click Includes to create a new file in the Navigation panel.
  5. Enter the location of the file to include in the Location field. In this example, use the include/KafkaMonitoring.xml:
  6. Note: The priority controls the importance of a file when merging. Sections in a higher priority file will take precedence over sections in a lower priority file. This priority setting affects the priority of configuration in the main setup file.

  7. Expand the file location in the Includes section.
  8. Select Click to load...
  9. Click Yes to load the new Kafka include file.
  10. Click Managed entities in the Navigation panel.
  11. Add the Kafka-Broker and Kafka-Cluster types to the Managed Entity section that you will use to monitor Kafka.
  12. Click the Validate button to check your configuration.

The Validate button allows you to check if there are any errors or warnings in your configuration set-up.

Once the Gateway configuration appears in the Includes section, you can add the samplers and other variables.

Set up the samplers

These are the pre-configured samplers available to use in KafkaMonitoring.xml.

Configure the required fields by referring to the table below:

Samplers
Kafka-HeapMemoryUsage
Kafka-Broker
Kafka-Topics
Kafka-Cluster
 

 

Set up the variables

The KafkaMonitoring.xml template provides the variables that are set in the Environments section:

Variable Description
KAFKA_MONITORING_GROUP_NAME Sampler group name.
KAFKA_JMX_HOST Host name (or IP address) of the machine hosting the broker.
KAFKA_JMX_PORT JMX port that has been configured for that broker.
   

After checking and saving the changes, the samplers you have set in the Gateway configuration display in Active Console 2.

Set up the rules

The KafkaMonitoring-SampleRules.xml template also provides a separate sample rules that you can use to configure the Gateway Setup Editor.

Your configuration rules must be set in the Includes section.

  1. Enter the location of the file to include in the Location field. In this example, use the include/KafkaMonitoring-SampleRules.xml:
  2. Note: The priority controls the importance of a file when merging. Sections in a higher priority file will take precedence over sections in a lower priority file. This priority setting affects the priority of configuration in the main setup file.

  3. Expand the file location in the Includes section.
  4. Select Click to load...
  5. Click Yes to load the new Kafka include rules file.
  6. Click Rules in the Navigation panel to create new rules.
Sample Rules Description
Broker - Kafka Status

Generates a human readable version of the numeric State value of the broker and has the following conditions:

  • Red - the broker State=0: Not Running
  • Amber - the broker State=2, 5, or 6
Cluster - Partition Underreplicated Applies to all cells in a Kafka-Cluster. The rule turns red if the value is > 0.
Broker - Active Controller Applies to all Active Controller Count Cells in the Kafka-Broker. The Active Controller count shows the number of active controllers across the cluster. A value of greater than 1 in this column indicates that there are more than one controllers with alerts.
Broker - Offline Partitions Applies to all offline partitions count in the Kafka-Broker. The offline partitions count shows the total number of under replicated partitions across the cluster. A value of greater than 0 indicates that there are under replicated partitions with alerts.