HQ
Change log for Lenses HQ 6.1.
6.1.3
Kafka to Kafka Replication
The latest release of Lenses K2K integration now supports bi-directional. This feature allows data to be copied from Cluster A's Topic One to Cluster B's Topic Two and vice versa. The bi-directional support ensures that the data does not replicate indefinitely, preventing continuous loops. Two K2K instances should be deployed one copying from cluster A to cluster B, the second one from cluster B to cluster A, and both should have the bi-directional flag on.
SQL Studio
A bunch of great improvements to make the experience of building and troubleshooting streaming applications better.
Topic and Schema listing support
In addition to the global topic listing page, in Studio you can now list all topics and schemas per environment. The listing, as a tab, in Studio, provides all the same functionality as the main topic listing, but allows you to split the view for IDE style work, list all your topics, while you work on schema or other resources in one view.
Click the "open listing in a new tab" on the topic or schema nodes in the navigation tree.
Preview Panel
We added a preview panel to help you quickly identify key information about topics, such as replication factor, message counts, size and partitions.
Improved search
A new search has been added, its more intuitive and lists per resource type what is found, and giving you access to the context menu actions available.
Bookmark Schemas
You can now bookmark schemas in addition to environments and topics.
IAM Role Policy creation & templates
The IAM role policy creation flow now lets you directly choose from a default template and select groups to add the role to, previously this had to be performed in separate steps.
Additionally the Policy editor has been enhanced with more JSON schema support, a problem panel to see issues and a diff editor to see what you have changed in the policy.
Alerts
Alert events are now available in HQ, no need to drill down to the Legacy Agent UX to:
View events
Manage alert channels
Create alert rules
Audits
Audit events are now available in HQ, no need to drill down to the Legacy Agent UX to:
View audit events
Manage audit channels
SQL Processors Avro alias support
Avro schema support field alias. With this release, SQL processors support writing queries referencing the alias field. For example given this schema:
{
"type": "record",
"name": "InsuranceContribution",
"namespace": "com.example.insurance",
"doc": "Insurance contribution payment record",
"fields": [
{
"name": "id",
"type": "string"
},
{
"name": "paymentDuration",
"aliases": [
"contributionDuration"
],
"type": [
"null",
"int"
],
"default": null,
"doc": "Payment duration in months"
},
{
"name": "amount",
"type": [
"null",
"double"
],
"default": null
},
{
"name": "timestamp",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
}
]
}The following SQL processor projection is valid:
INSERT INTO alias_avro_copied
SELECT STREAM
contributionDuration as a,
paymentDuration as b
FROM alias_avroCLI SQL Studio Support
The lenses-cli now supports SQL Studio operations
Improvements
Environment collapsible configuration panelTopic details page on the Topics screen has been simplified and improved to share the basic information.
Kafka ACL, Quotas and Data policies now support bulk deletion allowing you to delete multiple in one go.
Added quick access to legacy agent UI from HQ environments
Excel-style cell selection in data grids
Better diff visualization when editing environments
Fixed
Fixed duplicate alert rule creation for consumer and producer alerts
Fixed tag visibility in topic drawer
Fixed topic marker sizing and letter centering
Fixed reset button not clearing grid filters
Fixed Monaco tooltip display in SQL studio
Fixed YAML worker disposal and file upload issues
Improved IAM policy examples and core roles
6.1.2
Kafka to Kafka Replication
We're making significant strides in integrating Lenses K2K into the Lenses ecosystem, which is set to deliver enhanced governance and control over the existing standalone solution. In this release, you'll find:
Evolving Environment Management: Initial roll-out of new capabilities in environment management, serviceability, and ACL management within HQ.
Continued Studio Enhancements: Studio now enables topic creation, editing, and viewing configurations, progressively reducing the need for the legacy Agent UX (still accessible if needed).
Ongoing Styling Improvements: Gradual updates to HQ's styling for a more refined user experience.
We're eager for your feedback as we continue this integration journey.
The latest release of the Lenses K2K integration introduces enhanced Kafka connection support, extending beyond PlainText and MSK with IAM Roles for Service Accounts (IRSA). This update now includes support for mTLS and SASL/SSL, along with PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 authentication mechanisms.
Adding a New Kafka Connection
To add a new Kafka connection:
Navigate to the Apps menu.
Choose the Kafka Connections tab.
Click on New Kafka Connection.
A guided wizard will assist you through the necessary configuration steps.
In our ecosystem, Kafka connections are utilized by K2K and future applications provided by Lenses. These connections are managed by HQ. The Lenses agent does not directly use these connections to interact with the Kafka cluster.
Creating a K2K App
Creating a K2K app now offers enhanced control, including:
Header replication : copies the Kafka message headers to the target
Error handling
Tracing headers for source record offset, partition, topic, and pipeline
Managing underlying topic names required by K2K for offsets tracking or exactly-once semantics
Adjusting K2K configurations such as batch size, commit size, interval, and exactly-once fencing parameters
This release introduces compatibility with various Kafka providers, including:
Confluent
Google
Azure Event Hubs
IBM Event Streams
Azure HDInsight Kafka
Warpstream
Instaclustr
Aiven
Oracle
RedPanda
Topic management & exploration in Studio
To create topics in SQL Studio:
Navigate to the desired environment node.
Click the ellipsis (three dots) and select Create topic.
Choose a creation method:
Use the default modal, or
Switch to Advanced for the native YAML editor.
Use breadcrumbs at the top to navigate across views.
Each topic offers accessible options for summary, data, schema, and configuration via the navigation tree.
Use the split view to get side-by-side views.
Insert messages on Studio
No need to navigate to the agent to insert messages into a topic, its now available in the HQ Studio.
Select "Insert Message" on the context menu in the Studio navigation tree and;
manual add messages
auto generate message
insert multiple messages
Support is also included for message headers.
Regenerate environment agent key
You can now regenerate an environment agent key from HQ and either apply it directly to the agent or update the provisioning file manually.
To apply the new key via the API, the agent must be connected. The agent will disconnect and reconnect once the key is successfully applied.
Access: Environment context menu → Regenerate agent key
Kafka ACLs management in HQ
Manage Kafka Access Control Lists (ACLs) directly from the HQ UI. View, create, and delete ACLs for your environments with full filtering and search capabilities.
Key capabilities
View all ACLs with sortable columns (Principal, Resource Type, Resource Name, Operation, Permission, Host)
Create new ACLs with form validation and topic autocomplete
Delete ACLs with a confirmation dialog
Filter by resource type, permission type, and operation
Search across principal, resource name, and host
Download ACLs as JSON
Paginate through large ACL lists
Access: Environment sidebar → Kafka ACLs or environment context menu → View Kafka ACLs
Deleting or modifying ACLs can impact client access to Kafka resources. Review changes carefully before applying them in production environments.
Kafka Quotas management in HQ
Manage Kafka quotas directly from the HQ UI. View, create, and delete quotas for your environments to control producer/consumer throughput and request rates.
Key capabilities
View all quotas with sorting on columns (Entity Type, User, Client ID, Producer Rate, Consumer Rate, Request %)
Create quotas for specific users, client IDs, user+client combinations, or default quotas
Delete quotas with granular property selection (remove individual rate limits)
Filter by entity type (User, Client ID, User + Client ID)
Search across user and client ID names
Download quotas as JSON
Paginate through large quota lists
Quota types supported
User
Quota for a specific authenticated user principal
Client ID
Quota for a specific client application identifier
User + Client
Quota for a specific user and client ID combination
Default User
Baseline quota for all users without a specific quota
Default Client
Baseline quota for all client IDs without a specific quota
Access
Environment sidebar → Kafka Quotas or environment context menu → View Kafka Quotas
Environment health dashboard
Monitor the health status of your environment's core services from a single dashboard.
Key capabilities
Overview tab with key metrics (brokers, topics, partitions, consumer groups)
Brokers tab with detailed broker status and configuration
Connect tab for Kafka Connect cluster health
Schema Registries tab for schema registry status
Real-time status indicators (healthy, warning, error)
Health summary in the environment listing side drawer
Access: Environment context menu → View Health
Agent logs viewer in HQ
View and filter agent log entries directly in the HQ UI for debugging and monitoring.
Key capabilities
Real-time log streaming
Filter by log level (DEBUG, INFO, WARN, ERROR)
Search within log messages
Filter by time range using timestamps
Auto-refresh capability
Access: Environment context menu → View Logs
Agent advanced configuration
You can now access and manage advanced agent configuration settings directly from HQ. This enables you to adjust more detailed options without editing configuration files manually.
6.1.1
Schema support in SQL Studio
Schema Registry entries can now be listed and the schema details viewed from the Studio.
Improvements
K2K
When a new K2K app is created, it automatically sets the Docker image to K2K Kafka-to-Kafka replicator version 1.1.0.
K2K IAM
The IAM checks for the K2K application have been enhanced to include dependencies such as source and target environment Kafka connections, Kubernetes cluster, and namespace. This update ensures comprehensive permission validation across multiple axes.
New actions introduced:
ManageOffsets: Allows management of K2K application resources.
GetKafkaConnectionDetails: Retrieves details for Kafka connection resources.
Studio
Global Topic page was not accessible in case K2K license for HQ was not issued
Left panel with explorer toggle had been removed.
Context menus are now available in both explorer and breadcrumbs.
Various fixes have been applied throughout the user interface to address glitches and inconsistencies, enhancing the in-app user experience.
6.1.0
Kafka Connections
Kafka Connections allow administrators to define connections to Kafka as Kubernetes secrets or service accounts, that reference the credentials to connect with.
Most organisations already use secret providers such as AWS Secret Manager or Vault, and sync these to Kubernetes secrets automatically. This ensures Lenses or users deploying applications never need to deal with the credentials themselves.
See the documentation to get started.
Kafka to Kafka Replication
Lenses Kafka to Kafka Replicator is now integrated into Lenses. You can configure and deploy Lenses, with predefined Kafka Connections to move data between AWS MSK IAM clusters.
More cluster authentication methods and providers coming soon!
See the documentation to get started.
Configure Agent Provisioning from HQ
You can now create an environment and configure the Agent provisioning directly from HQ. JSON schema support is added, providing syntax highlighting, auto completion and error reporting.
You can also view and edit the existing provisioning files of agents already connected.
SQL Studio
SQL Studio is moving forward again, and now brings a more IDE style experience. An improved tree navigation provides:
Improved search functionality
Expanded topics nodes to browse schemas and consumer groups associated with topics
Context menu support to allow actions on topics
Bookmarking of favourite topics.
Improvements
Enhanced the performance of the environments screen, making it more responsive and capable of handling larger datasets.
Last updated
Was this helpful?