Infrastructure JMX Metrics

This page describes how to configure JMX metrics for Connections in Lenses.

All core services (Kafka, Schema Registry, Kafka Connect, Zookeeper) use the same set of properties for services’ monitoring.

The Agent will discover all the brokers by itself and will try to fetch metrics using metricsPort, metricsCustomUrlMappings and other properties (if specified).

See JSON schema for support.

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  value: "${ENV_VAR_NAME}"

[connnection]
  configuration:
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  name: kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX

SSL

kafka:
  tags: []
  name: kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: 
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
      value: true

Basic Auth

kafka:
  tags: []
  name: kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: 
       value: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

For JOLOKIA each entry value in metricsCustomUrlMappings must contain protocol.

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  name: kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    # For GET method: JOLOKIAG
    # For POST method: JOLOKIAP 
    metricsType:
      value: JOLOKIAG
    metricsSsl: 
      value: false
    metricsHttpSuffix: 
      value: /jolokia/

Custom Http Request Timeout

JOLOKIA monitoring works on the top of HTTP protocol. To fetch metrics the Agent has to perform either GET or POST request. There is a way of configuring http request timeout using httpRequestTimeout property (ms value). Its default value is 20 seconds.

httpRequestTimeout: 
  value: 30000

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

metricsHttpSuffix: 
  value: /custom/

AWS

Before enabling collection of metrics within Agents provision configuration, make sure in your MSK Provisioned cluster you have enabled open monitoring with Prometheus.

AWS has a predefined metrics configuration.

The Agent hits the Prometheus endpoint using port 11001 for each broker.

There is an option of customizing AWS metrics connection in Lenses by using:

• metricsUsername

• metricsPassword

• metricsHttpTimeout

• metricsHttpSuffix

• metricsCustomUrlMappings and

• metricsSsl properties.

• However, except for metricsHttpTimeout, the other settings will seldom be needed - AWS has its standard, which is unlikely to change. Customization can be achieved only by API or CLI.

kafka:
  tags: [ "dev", "dev-2", "eu"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://my-broker-0:9098
        - SASL_SSL://my-broker-1:9098
        - SASL_SSL://my-broker-2:9098
    protocol:
      value: SASL_SSL
    saslMechanism:
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsType:
      value: AWS
    metricsHttpTimeout: # optional, milliseconds
      value: 20000

In some cases, the metricsHttpTimeout option may be required. Typically, this occurs when the OpenMetrics instance is undersized for the size of the MSK cluster, resulting in longer-than-usual metric retrieval times.

Each Kafka partition adds a large number of metrics, so the OpenMetrics instance should ideally be sized to accommodate the number of partitions that the MSK will host.

Another common pitfall with MSK OpenMetrics is that there exists a global rate limit for each instance. If more than one service hits the OpenMetrics endpoint, the rate limit may be triggered, and the clients will receive an HTTP error code 429. To overcome this, you can set the lenses.interval.metrics.refresh.broker option in Lenses Agent. As an example, to make Lenses request metrics every minute, set the value to 60000 (milliseconds).

Custom URL mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

• my-kafka-host-0:9582 - because of metricsCustomUrlMappings

• my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings

kafka:
  tags: ["optional-tag"]
  name: kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsCustomUrlMappings:
      value:
        "my-kafka-host-0:9092": my-kafka-host-0:9582