Topics can be accessed via the Global Data Catalogue or at an environment level.

Creating topics

To create a topic go to Environments->[Your Environment]->Workspace->Explore->New Topic. Enter the name, partitions and replication factor.

If topic settings apply, you will not be able to create the topic unless the rules have been met.

Viewing topic details

The Explore screen lists high-level details of the topics

Selecting a topic allows you to drill into more details.

Topics marked for deletion will be highlighted with a D.

Compacted topics will be highlighted with a C.

Increasing the number of partitions

To increase the number of partitions, select the topic, then select Increase Partitions from the actions menu. Increasing the number of partitions does not automatically rebalance the topic.

Overriding a topic configurations

Topics inherit their configurations from the broker defaults. To override a configuration, select the topic, then the Configuration tab. Search for the desired configuration and edit its value.

Deleting topics

To delete a topic, click the trash can icon.

Topics can only be deleted if all clients reading or writing to the topic have been stopped. The topic will be marked for deletion with a D until the clients have stopped.

Empty or compacted topics

To quickly find compacted or empty topics use quick filter checkboxes, for example, you can find all empty topics and perform a bulk delete action on them.

Searching

In the Explore screen, you can use the search bar to filter for both fields and topic names in your Kafka cluster, additionally, it will search across any Elasticsearch or Postgres instances that have been connected.

Inserting a message

To insert a message, select Insert Message from the Action menu. Either enter a message, according to the topic schema or have a message auto-generated for you.

Deleting messages

Deleting messages deletes messages based on an offset range. Select Delete Messages from the Action menu.

To view a live snapshot of the metrics for a topic, select the metrics tab for the topic.

This will show you metric information over the last 30 days, alert rules on the topic and low JMX metrics.

To view topic partitions select the Partition tab. Here you can see a heat map of messages in the topic and their distribution across the partitions.

Further information about the partitions and replicas is displayed, for example, whether the replicas are in-sync or not.

To add descriptions or tags to datasets, click the edit icon in the Summary panel.

Enabling approval requests

To enable approval requests, create a group with, or add to a group, the Create Topic Request permission to the data namespace.

Viewing, approving, or rejecting approval requests

Go to Environments->[Your Environment]->Admin->Audits->Requests, select the request, and click view.

Approve or reject the request. If you Approve the topic will be created.

To view a configuration for a topic select the Configuration tab. Here you will see the current configurations inherited (default) from the brokers and if they have been overridden (current value).

To edit a configuration click the Edit icon and enter your value.

Topic settings and naming rules allow for the enforcement of best practices when onboarding new teams and topics into your data platform.

Topic configuration rules

Topic configuration rules can be used to enforce partition sizing, replication, and retention configuration during topic creation. Go to Environments->[Your Environment]->Admin->Topic Settings->Edit.

Naming conventions

By setting naming conventions you can control how topics are named. To define a naming convention, go to Environments->[Your Environment]->Admin->Topic Settings->Edit. Naming rules allow you to select from predefined regex or apply your own.

Examining a message

After selecting a topic you will be shown more details of the topic. The SQL Snapshot engine will return to you 200 of the latest messages for partition 0. Both the key and value of the message are displayed in a tree format which is expandable.

At the top of each message, the Kafka metadata (partition, timestamp, offset) is displayed.

Hovering to the right allows you a message the clipboard.

{ "key": { "MMSI": 219347000 }, "value": { "Type": 1, "Repeat": 0, "MMSI": 219347000, "Speed": 0, "Accuracy": true, "Longitude": "9.747901666666667", "Latitude": 59.006915, "location": "59.006915,9.747902", "Course": 141.1, "Heading": 511, "Second": 14, "RAIM": true, "Radio": 23096, "Status": 0, "Turn": -128, "Maneuver": 0, "Timestamp": "1491318149612948547" } }

To download all messages to JSON or CSV see here.

Flattening the data view

The SQL Snapshot engine deserializes the data on the backend of Lenses and sends it over the WebSocket to the client. By default, the data is presented in a tree format but it's also possible to flatten the data into a grid view. Select the grid icon.

Searching by partition

Use the partition drop-down to change the partition to return messages you are interested in.

Searching by timestamp

Use the timestamp picker to search for messages from a timestamp.

Searching by offset

Use the offset select to search for messages from an offset.

Live sample

The SQL Snapshot engine has a live mode. In this mode the engine will return a sample of messages matching the query. To enable this, select the Live Sample button. The data view will now update with live records as the are written to the topic. You can also edit the query if required.

Changing the SQL deserializer format

For the SQL Snapshot engine to return data it needs to understand the format of the data in a topic. If a topic is backed by a Schema registry it is automatically set to AVRO. For other types, such as JSON or Strings the engine tries to determine the format.

If you wish to override or correct the format used select either Reset Types or Change Types from the action menu.

To initiate either a topic backup to S3 or topic restoration from S3, follow these steps:

• Navigate to the Actions menu within the Kafka topic details screen.

• Choose your desired action: “Backup Topic to S3” or “Restore Topic from S3.”

• A modal window will open, providing step-by-step guidance to configure your backup or restoration entity.

Identifying if a topic is being backed up

If a topic is being backed up it will be displayed on the topology.

Additional information on the location of the backup can be found by navigating to the topic in the Explore screen where the information is available in the Summary section.

Backing up a topic

To back up a topic, navigate to the topic you wish to back up and select Backup Topic to S3 from the Actions menu.

Enter the S3 bucket ARN and select the Connect Cluster that has the Lenses S3 connector installed.

Click Backup Topic, an S3 sink connector instance will now be deployed and configured automatically to back up data from the topic to the specified bucket.

Restoring a topic

To restore a topic, navigate to the topic you wish to restore and select Restore Topic from S3 from the Actions menu.

Enter the S3 bucket ARN and select the Connect Cluster that has the Lenses S3 connector installed. Click Restore Topic, an S3 source connector instance will now be deployed and configured automatically to restore data to the topic from the specified bucket.

Data can be downloaded, optionally including headers, as JSON or as CSV with a choice of delimiters.