1 of 12

Topics

Allow users to create and manage their own topics and apply topic settings as guard rails.

For automation use the .

Creating topics

To create a topic go to Workspace->Explore->New Topic. Enter the name, partitions and replication factor.

If apply you will not be able to create the topic unless the rules have been met.

Viewing topic details

The Explore screen lists high-level details of the topics

Selecting a topic allows you to drill into more details.

Topics marked for deletion will be highlighted with a D.

Compacted topics will be highlighted with a C.

Increasing the number of partitions

To increase the number of partitions, select the topic, then select Increase Partitions from the actions menu. Increasing the number of partitions does not automatically rebalance the topic.

Overriding a topic configurations

Topics inherit their configurations from the broker defaults. To override a configuration, select the topic, then the Configuration tab. Search for the desired configuration and edit its value.

Deleting topics

To delete a topic, click the trash can icon.

Topics can only be deleted if all clients reading or writing to the topic have been stopped. The topic will be marked for deletion with a D until the clients have stopped.

Empty or compacted topics

To quickly find compacted or empty topics use quick filter checkboxes, for example, you can find all empty topics and perform a bulk delete action on them.

Finding topics & fields

This page describes how to use Lenses to search for topics and fields across Kafka, Postgres and Elasticsearch.

Searching

In the Explore screen, you can use the search bar to filter for both fields and topic names in your Kafka cluster, additionally, it will search across any Elasticsearch or Postgres instances that have been connected.

Searching for messages

This page describes how to use Lenses to use the Explore screen to explore, search and debug messages in a topic.

Examining a message

After selecting a topic you will be shown more details of the topic. The SQL Snapshot engine will return to you 200 of the latest messages for partition 0. Both the key and value of the message are displayed in a tree format which is expandable.

At the top of each message, the Kafka metadata (partition, timestamp, offset) is displayed.

Hovering to the right allows you a message the clipboard.

{ "key": { "MMSI": 219347000 }, "value": { "Type": 1, "Repeat": 0, "MMSI": 219347000, "Speed": 0, "Accuracy": true, "Longitude": "9.747901666666667", "Latitude": 59.006915, "location": "59.006915,9.747902", "Course": 141.1, "Heading": 511, "Second": 14, "RAIM": true, "Radio": 23096, "Status": 0, "Turn": -128, "Maneuver": 0, "Timestamp": "1491318149612948547" } }

To download all messages to JSON or CSV see here.

Flattening the data view

The SQL Snapshot engine deserializes the data on the backend of Lenses and sends it over the WebSocket to the client. By default, the data is presented in a tree format but it's also possible to flatten the data into a grid view. Select the grid icon.

Searching by partition

Use the partition drop-down to change the partition to return messages you are interested in.

Searching by timestamp

Use the timestamp picker to search for messages from a timestamp.

Searching by offset

Use the offset select to search for messages from an offset.

Live sample

The SQL Snapshot engine has a live mode. In this mode the engine will return a sample of messages matching the query. To enable this, select the Live Sample button. The data view will now update with live records as the are written to the topic. You can also edit the query if required.

This is sample data, not the full set to avoid overloading the browser

Changing the SQL deserializer format

For the SQL Snapshot engine to return data it needs to understand the format of the data in a topic. If a topic is backed by a Schema registry it is automatically set to AVRO. For other types, such as JSON or Strings the engine tries to determine the format.

If you wish to override or correct the format used select either Reset Types or Change Types from the action menu.

Inserting & deleting messages

This page describes how to use Lenses to insert or delete messages in Kafka.

Inserting a message

To insert a message, select Insert Message from the Action menu. Either enter a message, according to the topic schema or have a message auto-generated for you.

Deleting messages

Deleting messages deletes messages based on an offset range. Select Delete Messages from the Action menu.

Viewing topic metrics

This page describes how to use Lenses to view metrics for a topic.

To view a live snapshot of the metrics for a topic, select the metrics tab for the topic.

This will show you metric information over the last 30 days, alert rules on the topic and low JMX metrics.

Viewing topic partitions

This page describes how to use Lenses to view topic partition metrics and configuration.

To view topic partitions select the Partition tab. Here you can see a heat map of messages in the topic and their distribution across the partitions.

Is the map evenly distributed? If not you might have partition skew.

Further information about the partitions and replicas is displayed, for example, whether the replicas are in-sync or not.

If the replicas are not in-sync an infrastructure alert will be raised.

Topic Settings

This page describes how to use Lenses topic settings to provide governance when creating topics in your Kafka cluster.

Topic settings and naming rules allow for the enforcement of best practices when onboarding new teams and topics into your data platform.

Topic configuration rules

Topic configuration rules can be used to enforce partition sizing, replication, and retention configuration during topic creation. Go to Admin->Topic Settings->Edit.

Naming conventions

By setting naming conventions you can control how topics are named. To define a naming convention, go to Admin->Topic Settings->Edit. Naming rules allow you to select from predefined regex or apply your own.

Adding metadata & tags to topics

This page describes how to use Lenses to add metadata and tags to topics in Kafka.

To add descriptions or tags to datasets, click the edit icon in the Summary panel.

Managing topic configurations

This page describes how to use Lenses to view and manage topic configurations in Kafka.

To view a configuration for a topic select the Configuration tab. Here you will see the current configurations inherited (default) from the brokers and if they have been overridden (current value).

To edit a configuration click the Edit icon and enter your value.

Approval requests

This page describes how to use Lenses approval requests.

To enable Approval Requests for a group, grant the group Create Topic Request permission. When a user belonging to this group creates a topic it will be sent for approval first.

Enabling approval requests

To enable approval requests, create a group with, or add to a group, the Create Topic Request permission to the data namespace.

Viewing, approving, or rejecting approval requests

Go to Admin->Audits->Requests, select the request, and click view.

Approve or reject the request. If you Approve the topic will be created.

Downloading messages

This page describes how to use Lenses to download messages to CSV or JSON from a Kafka topic.

Only the data returned to the frontend is downloaded.

Data can be downloaded, optionally including headers, as JSON or as CSV with a choice of delimiters.

Backup & Restore

This page describes how to use Lenses to back and restore data in a Kafka topic to AWS S3.

To initiate either a topic backup to S3 or topic restoration from S3, follow these steps:

Navigate to the Actions menu within the Kafka topic details screen.
Choose your desired action: “Backup Topic to S3” or “Restore Topic from S3.”
A modal window will open, providing step-by-step guidance to configure your backup or restoration entity.

A single topic can be backed up or restored to/from multiple locations.

Identifying if a topic is being backed up

If a topic is being backed up it will be displayed on the topology.

Additional information on the location of the backup can be found by navigating to the topic in the Explore screen where the information is available in the Summary section.

Backing up a topic

To back up a topic, navigate to the topic you wish to back up and select Backup Topic to S3 from the Actions menu.

Enter the S3 bucket ARN and select the Connect Cluster that has the Lenses S3 connector installed.

Click Backup Topic, an S3 sink connector instance will now be deployed and configured automatically to back up data from the topic to the specified bucket.

Restoring a topic

To restore a topic, navigate to the topic you wish to restore and select Restore Topic from S3 from the Actions menu.

Enter the S3 bucket ARN and select the Connect Cluster that has the Lenses S3 connector installed. Click Restore Topic, an S3 source connector instance will now be deployed and configured automatically to restore data to the topic from the specified bucket.