Kafka


Lenses integrates with Kafka to give observability and management capabilities over your topics, the data, schemas and metrics. Granular permissions apply for the different feature so that self-service capabilities can be customized to your needs.

If you integrate with Schema Registry (SR), Lenses will read those schemas to enhance the experience. If an SR service isn’t available, Lenses will maintain types and schemas for topics at its own internal store to enable the different features at a field level.

As a user, you can get full control of your topics, query the data, update the configuration, review metrics and add alerts.

Required permissions 

The level of permissions in a Kafka topic is fully controlled via Groups Namespaces. From there you can tailor to which topics a group of users has access and what operations they can perform. Review the permission matrix to see all the available operations.

Access Management and permissions

Kafka topics concepts 

A topic is a named stream of events (or messages). The topic data is partitioned to one or more partitions. The messages on each partition are ordered by a number the offset.

Partition is the unit of scale in Kafka. If a topic has multiple partitions, the clients can produce data in parallel to increase throughput by distributing the partitions across the cluster. The number of partitions also influences the balancing of workload among consumers.

To improve availability, each topic can be replicated onto multiple brokers. For each partition, one of the brokers is the leader, and the other brokers are the followers. Replication works by the followers repeatedly fetching messages from the leader.

The events are key-value pairs with additional metadata can be added in headers. To read/write the data you need to deserialize/serialize in a given format and/or schema.

Event

Topic headers 

Lenses has primitive support for the headers and they can be displayed to your results. There is also support for SQL functions to work best with the header data.

Topic metadata indicators 

The indicators next to the topic name display additional information:

  • C, highlights when a topic is compacted.
  • S, appears when a topic is a system topic.
  • d, appears when a topic is marked for deletion.
Topics list compacted

Lenses system topics 

Systems topics (datasets) are used to indicate which topics have been automatically created by the system (ie. __consumer_offsets ). By default they are filtered out, but you can choose to view them by ticking the box on the filter list.

The default setting includes Lenses system topics, SQL processors’ Kstreams topics, consumer offsets, schemas, and transactions. You can add topics of your own as well, but it is advised to keep the default ones too, so they will not be transferred to your user topics. The setting takes prefixes so, for example, the lsql_ item matches all topics starting with lsql_.

Below is the list of built-in configuration to identify them.

  • _schemas
  • __consumer_offsets
  • _kafka_lenses_
  • lsql_*
  • lsql-*
  • __transaction_state
  • __topology
  • __topology__metrics
  • _confluent*
  • *-KSTREAM-*
  • *-TableSource-*
  • *-changelog
  • __amazon_msk*

TODO - Move to self service