Monitoring

Monitor infrastructure and applications using:

For integration with Prometheus and Grafana Lenses provides templates and curated dashboards to monitoring applications and your Kafka cluster.

Lenses for Apache Kafka Monitoring Suite is a set of pre-defined templates, that use:

  • A Time Series database (Prometheus)
  • Custom JMX exporters
  • A Data Visualization application (Grafana)
  • Built-in domain intelligence about operating Kafka with confidence while in production

If you do not already have an existing installation of Grafana and Prometheus please visit their documentation for details and best practices.

Kafka monitoring with Grafana

Prometheus exporter 

To use the Lenses Grafana dashboard configure the Prometheus Java exporter with the configuration files located in your client area along with a packaged export jars. The following are available for each service:

  1. broker.yml
  2. client.yml
  3. connect-worker.yml
  4. kafka-rest.yml
  5. schema-registry.yml

Start the exporter in server mode:

java -jar /path/to/fastdata_server.jar [PORT] [CONF_FILE]

Where [PORT] is the listening port for scrape requests from Prometheus and [CONF_FILE] the configuration file.

To run as a Java agent, add the following to the environment variables controlling JMX for each service:

Kafka Broker

export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/path/to/fastdata_agent.jar=[PORT]:/path/to/broker.yml"

Kafka Connect

export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/path/to/fastdata_agent.jar=[PORT]:/path/to/connect-worker.yml"

Lenses

export LENSES_OPTS="$LENSES_OPTS -javaagent:/path/to/fastdata_agent.jar=[PORT]:/path/to/client.yml"

Schema Registry

export SCHEMA_REGISTRY_OPTS="$SCHEMA_REGISTRY_OPTS -javaagent:/path/to/fastdata_agent.jar=[PORT]:/path/to/schema-registry.yml"