Infrastructure JMX Metrics

This page describes how to configure JMX metrics for Connections in Lenses.

All core services (Kafka, Schema Registry, Kafka Connect, Zookeeper) use the same set of properties for services’ monitoring.

The Agent will discover all the brokers by itself and will try to fetch metrics using metricsPort, metricsCustomUrlMappings and other properties (if specified).

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX

SSL

kafka:
  tags: []
  templateName: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: 
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
      value: true

Basic Auth

kafka:
  tags: []
  templateName: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: 
       value: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

For JOLOKIA each entry value in metricsCustomUrlMappings must contain protocol.

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
     value: false
    metricsHttpSuffix: 
     value: /jolokia/

Custom Http Request Timeout

JOLOKIA monitoring works on the top of HTTP protocol. To fetch metrics the Agent has to perform either GET or POST request. There is a way of configuring http request timeout using httpRequestTimeout property (ms value). Its default value is 20 seconds.

httpRequestTimeout: 
  value: 30000

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

metricsHttpSuffix: 
  value: /custom/

AWS

Before enabling collection of metrics within Agents provision configuration, make sure in your MSK Provisioned cluster you have enabled open monitoring with Prometheus.

AWS has predefined metrics configuration. The Agent hits the Prometheus endpoint using port 11001 for each broker. There is an option of customizing AWS metrics connection in Lenses by using metricsUsername, metricsPassword, httpRequestTimeout, metricsHttpSuffix, metricsCustomUrlMappings, metricsSsl properties, but most likely no one will need to do that - AWS has its own standard and most probably it won’t change. Customization can be achieved only by API or CLI - UI does not support it.

kafka:
  tags: [ "dev", "dev-2", "eu"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://my-broker-0:9098
        - SASL_SSL://my-broker-1:9098
        - SASL_SSL://my-broker-2:9098
    protocol:
      value: SASL_SSL
    saslMechanism:
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsType:
      value: AWS

Custom URL mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

• my-kafka-host-0:9582 - because of metricsCustomUrlMappings

• my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings

kafka:
  tags: ["optional-tag"]
  name: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsCustomUrlMappings:
      value:
        "my-kafka-host-0:9092": my-kafka-host-0:9582