Managing Topics

With Lenses, you can create, delete and change the configuration of topics by navigating to the Topics view on the side menu.

This view displays:

  1. A chart of the data flowing across the Kafka cluster being monitored.
  2. A topic list and details for each topic.
  3. Functionality to create and delete topics.
../../_images/topic-view.png

Create a topic

To create a topic select the New Topic button and enter the required details to create the topic. The minimum requirements are:

  1. name,
  2. partitions
  3. replication.

Additionally, you can supply extra configuration to override the broker default values by selecting the Add Config link. This also allows setting common overrides such as retention time, retention size and clean up policy.

../../_images/create_topic.png

Delete a topic

To delete a topic, select the trash can button for the topic and confirm that you wish Lenses to remove that topic.

../../_images/delete_topic.png

Note

A topic will be marked for deletion and Kafka will remove it once the consumers have stopped.

Modify a topic

To alter the configuration of an existing topic, select the topic and then the Config tab.

../../_images/modify_topic.png

Important

You might want to have control over who makes topic changes. By default topic creation, deletion and editing is only allowed to users with WRITE access, such as admin and write.

View topic partitions

Detailed information and metrics about a topic can be seen in the topic Partitions tab. It displays the distribution of messages across all the topic partitions, provides detailed information for each partition, but also monitors and alerts on in-sync replicas.

../../_images/topic_partitions.png

Consumers

Lenses tracks consumers in real time; to view the consumer groups for a given topic select the Consumers tab once you have navigated to the desired topic. This will display:

  1. Consumer groups active on this topic
  2. The clients in the consumer group
  3. The lead broker
  4. The consumer lag across the group updated in real time
  5. The stability of the consumer group
../../_images/topic-consumers.png

Viewing live data

Lenses provides the ability to visualize live data in topics by utilizing the Lenses SQL engine. Simply select the topic and you are dropped into the Live Stream tab. This view provides:

  1. A SQL window to select which data you want to see from the topic.
  2. Ability to inspect the data.
  3. Pause and restart the stream.

The SQL window allows you to select and filter fields from both the key and value of the records. For a full description of the capabilities please refer to the Lenses SQL section.

Hovering over the records will automatically pause the stream to allow you to inspect records.

../../_images/topic-live.png

Viewing historical data

Lenses provides the ability to visualize and download historical data in topics by utilizing the Lenses SQL engine. Simply select the topic and select the Data tab. This view provides:

  1. A SQL window to select which data you want to see from the topic.
  2. An offset slider to scroll through data.
  3. A tree view of the data.
  4. A grid-like view of the data.
  5. A raw JSON view of the data.

The SQL window allows you to select and filter fields from both the key and value of the records. For a full description of the capabilities please refer to the Lenses SQL section.

../../_images/topic-historical-sql.png
../../_images/topic-historical-filter-grid.png

Inserting new data

Lenses allows you to insert new data into a topic, provided your account has either write or admin permissions to that topic.

../../_images/lenses-insert-data.png

You can insert new AVRO or JSON payload by dropping new records onto the UI. Additionally, you can even sample or auto generate messages.

This is usually helpful for your QA team when testing micro-services architectures or for pushing new scheduling messages.

You can insert null as the value on an AVRO topic (i.e. when using compacted topics).

Deleting data

With Lenses you can to delete data from a partition of a topic up to a specific offset.

../../_images/lenses-delete-messages.png

The action will be audited in the audit-log, the physical data will be removed, and notifications will be pushed via the SSE & Alert APIs for the particular action.