View the latest documentation 5.5
Learn to manage you Kafka offsets in a secure way, how to skip a bad kafka message or replay a set of events from a specific offset or time
Consumer Groups or Consumers are …
Access Management & permissions
Lenses continuously monitors Kafka consumers, applications consuming data from your cluster. It uses the native Kafka client to calculate in real-time metrics around the Lag per partition and status information. Consumer Groups represent Kafka applications that consume data from one or more Kafka topics. The consumers typically commit their offsets to a system topic called __consumer_offsets to persist their state (until which point they have consumed).
__consumer_offsets
You can view all the consumers by navigating to Consumers
Inactive Consumers
Inactive consumers are not enabled by default so if you want to access them select the Show Inactive option
The Consumer Lag represents the lag between the data in topic ( data produced ) and the data that have been processed by the consumer application. Every Consumer, consumes data at their own pace and when the rate of produced data exceeds the rate which is getting consumed, this creates a lag. So the lag, is the difference between the latest offset in the topic partition and the latest committed offset by the consumer.
The red bar under the consumer lag column, indicates the lag range (minimum and maximum lag per partition). When a single partition is involved the lag is a single number.
To view the exact lag per partition navigate to the consumer details and see how to add consumer lag alerts.
Navigate to each Consumer to view the details for its instances and how the topic partitions have been distributed to each instance as well as the offset details and lag for each topic-partition.
Coordinator ID
Latest Offset
Current Offset
Consumer Groups may consume from multiple topics at the same time and the load is automatically distributed across each application instance in the group. Your applications, Kafka Connectors and Streaming SQL have a consumer group when they are consuming events from Kafka.
You can add custom threshold alerts for the consumer lag. For each consumer group and topic combination you can set an alert trigger which will occur when the threshold exceeds.
Click on the View Alerts button to see the associated alerts with this consumer group or add a new one.
If you want to track in a time-series system, such as Prometheus / Grafana the historic values of the Consumer Lag, just add Lenses as a target. Lenses calculates and makes available consumer lag info in a prometheus compatible format under the Lenses API:
<your-lenses-host>/metrics
You can then hook this to your Grafana:
With Lenses you can manage the Consumer offsets. You can replay or skip messages for a partition or go back in time for multiple partitions. To edit the offsets your Consumer must be inactive, so that is not committing offsets while reading the topic. If your Consumers are active and committing their offsets back to Kafka, Lenses has the options for changing offsets disabled.
Find inactive consumers:
To change the offset for the inactive Consumer navigate to the Consumer details. For each topic-partition you can change the offset:
From the Actions menu, select Change multiple offsets
And select the one of the options:
If your Consumer consumes data from multiple topics you can select to apply the change to all or conditionally to specific topics
To replay records for a consumer group, the consumer must be stopped in order not to commit offsets while consuming. Therefore the option in Lenses is disabled when the consumer is up and running.
How can I see my Consumers in Topology
Any registered application can be viewed as part of the overall topology. For applications registered via the JVM we can also view the collected metrics.
How can I integrate Consumer Lag metrics to my monitoring system?
Consumer lag is exposed via the /metrics API
/metrics
How can I delete a Consumer?
At the moment Lenses is not giving you the ability to delete a Consumer group from its UI or API. You can use the command line tools of Kafka
kafka-consumer-groups.sh --bootstrap-server {Kafka instance connection address} --delete --group {consumer group name}
On this page