Release Notes¶
Note
London, UK - September 26, 2019, Lenses 3.0 is now generally available
If you are upgrading from an older version, make sure to check the Upgrade Notes.
What’s New¶
- Data Privacy and User Management introduces the concept of data ownership and enables seamless multi-tenant capabilities on Apache Kafka. A data namespace identifies one of more topics (or wildcards) and allows assigning permissions, such as create, query, update topics or schemas.
- Pluggable Alerts Lenses provides data and application performance monitoring and alerting for real time data. With this new release, the alerting mechanism is now pluggable. What if I want an infrastructure alert to automatically generate a ticket or notify my in-house incident management solution?
- Revamped User Experience This new major release revamped the user experience, grouping all administrative tasks under “Admin” menu option at the top of the page and simplified and improved the overall experience.
- Enhanced Kafka Cloud support Using Lenses to a managed service such as AWS MSK or Azure HDInsight and other is now easier than ever. Auto-configuration and auto-detection and out of the box support of TLS setups.
- Data Federation via a re-worked security module, that enables multiple authentication providers. One can now use LDAP, BASIC, Kerberos and/or CUSTOM (pluggable) authentication and for example authenticate internal users based on KERBEROS and external partners via LDAP.
- Learning center to easier get productive with real-time data and an improved open-source CLI for automation and CI/CD.
- Auditing improvement. With this new release Lenses further extends and improves auditing information, better tracking SQL data commands.
Improvements¶
- Performance improvements for alerts notification.
- SQL table engine support for help command, for example,
HELP lengthwill describe the function namedlength. - Better experience for querying audits and alerts events logs. When searching the records there’s much more flexibility on the search and faster performance.
- The JDBC driver for Kafka is now using our generation 2 engine. This means the same queries you are running in Lenses SQL studio can be used with the JDBC driver.
- SQL Streaming allows to use
SET schema.registry.*='???'to overwrite your schema registry settings at the SQL processor level. - SQL Streaming does not clean the topics and schemas when the SQL processor is deleted. This is to allow recreating the SQL processor with the code amendments and continue from where it left off.
- SQL statements for DROP/DELETE/TRUNCATE now can accept Kafka topics containing
-or they start with numbers. Therefore you do not need to escape the topic. - SQL Table engine error handling has been improved to avoid rendering to the screen code related exceptions.
- SQL Intellisense has been improved to display
- all the supported keys for a SET statement
- views and virtual tables (__table, __fields, __queries, __dual)
- the exists function
- the key fields in joins
- _meta.__keysize which allows to address the Kafka record key raw bytes size
- _meta.__valsize which allows to address the Kafka record value raw bytes size
- SQL Table engine introduces support for creating a table if it does not exists
CREATE IF NOT EXISTS TableA - SQL Table engine supports Windowed Keys resulted from Stream processing aggregation. For those payloads with WSTRING, WLONG, WAVRO, WJSON the returned key also displays the window start timestamp.
- SQL Table engine reports a user-friendly error when trying to use Avro but your setup does not have Schema Registry set up.
- SQL Table engine does not consume the entire topic data when using upper bounds offset/timestamp limits
SELECT * FROM A WHERE _meta.offset <=70. - SQL Table engine queries for equality offset or timestamp complete as fast as Apache Kafka returns the record, and does not read the entire topic anymore.
- SQL Table engine support offset and timestamp filter in queries like this
SELECT * FROM A WHERE 1235 < _meta.offsetorSELECT * FROM A WHERE '2019-08-01 12:30:19' <= _meta.timestamp. - SQL Table engine returns a friendly error when the partition number is incorrect.
- SQL date math supports now plurals (day-> days, minute->minutes, etc).
- SQL Table engine time parsing supports now these formats:
yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss.SSS,yyyy-MM-dd'T'HH:mm:ss.SSS+nnn.
Fixes¶
- SQL multiple INSERT statements will guarantee the order of the data as it appears in the SQL code.
- SQL Table engine does not throw an exception when using an ISO date filter like
>'2019-08-01 12:30:00'. - SQL Statements inserting LONG/INT values does not throw an exception anymore.
- SQL Statements inserting FLOAT values does not throw an exception anymore.
- SQL Statements inserting DECIMAL values respect the table column decimal scale and precision
CREATE TABLE abc (n decimal(18,38)); INSERT INTO abc(n) VALUES(3). - SQL Table engines multi joins statements filters are now working correctly.
- SQL Table engine introduced a regression on corrupted records it does not provide the offset and partition details on the error (You need to set skip.bad.records=false).
Deprecated API¶
With this version, some HTTP endpoints have been removed but an equivalent has been provided. A full list of public endpoints exposed by Lenses can be found here <http://api.lenses.io>
| Description | Old | New | Documentation |
|---|---|---|---|
| Kafka Topics | GET /api/topics | GET /api/v1/kafka/topics | check here |
| Kafka Topic Data | GET /api/sql/data?sql=… | GET /api/ws/v2/sql/execute | check here |
| SQL Validation | GET /api/sql/validation | removed | |
| Alerts | GET /api/alerts | GET /api/alerts | Return payload changed check it here |
| Audit | GET /api/audit | GET /api/audit | Return payload changed check it here |
Other Changes¶
- Using service accounts need to pass to Lenses this composite token
${AccountName}:${ServiceAccountToken}
3.0.1 Patch Release¶
Note
London, UK - October, 2019 Lenses 3.0.1 is now generally available
Fixes¶
- Consumers’ screen occasionally not updating
- DESCRIBE TABLE fails on some occasions
- Issues with WJSON
- Broken toggle for some alert rules
- Issue with scoped application permissions
3.0.2 Patch Release¶
Note
London, UK - November, 2019 Lenses 3.0.2 is now generally available
Fixes¶
- Incorrect number of workers in Connect page
- Custom SERDE schemas are registered with the wrong name
- Zookeeper connection issues when using Kerberos
- Wrong state in dashboard when JMX is configured but not accessible
- Issue with decommission broker dialog
- Issue with SQL studio screen when a query is already running
- Hide empty menus in Schema Registry page
3.0.3 Patch Release¶
Note
London, UK - December, 2019 Lenses 3.0.3 is now generally available
New Features¶
- Support for TLS Authentication to Lenses
- Support for Prometheus-based metrics for Amazon’s Managed Kafka Service (MSK)
Fixes¶
- Fix issues with nested data in JSON tree view
- Display properly BigInts in UI
- Fix download link for updates
- Support different JSON structures in SQL Studio grid view
- Fix table layout issues when different JSON schemas are used concurrently
- Update k8s library
- Data policies changes should be effective immediately
3.0.4 Patch Release¶
Note
London, UK - December, 2019 Lenses 3.0.4 is now generally available
New Features¶
- Data policies can be applied to nested fields for example payment.credit_card_num
- Data policies can use now wildcards. For example a*c.*.*xyz will match any field path in a 3 level nested structure
Fixes¶
- Handling Avro payload with Fixed field types in SQL Engine
- SQL Streaming Engine passes the topic settings to the underlying change log topics created by Kafka Streams
3.0.5 Patch Release¶
Note
London, UK - December, 2019 Lenses 3.0.5 is now generally available
Fixes¶
- Handling Avro payload with Fixed field types in SQL Engine
3.0.6 Patch Release¶
Note
London, UK - February, 2020 Lenses 3.0.6 is now generally available
Improvements¶
- SQL Processors can be deployed on Kubernetes clusters where namespaces listing is disabled
Fixes¶
- Inserting data on AVRO topics now correctly handles enums
- SQL Processor being deleted when deployed to Kubernetes does not delete all the services and ingresses in the namespace
- Custom Serdes schema is not correctly displayed
- Login issue on Lenses restart for a user who is part of at least two groups has been fixed.
3.0.7 Patch Release¶
Note
London, UK - February, 2020 Lenses 3.0.7 is now generally available
Improvements¶
- SQL queries (via SQL Studio) tuning can be disabled via configuration using lenses.sql.tuning.allowed=false
Fixes¶
- Avro recursive schemas does not causes Lenses to fail. Support for recursive schemas is planned for the future.
- Connectors screen won’t list entries for which the User Group permission does not allow access to
- Updating a Kafka Connect config will reflect correctly in the application landscape topology
3.0.8 Patch Release¶
Note
London, UK - February, 2020 Lenses 3.0.8 is now generally available
Improvements¶
- The Kafka ACL management screen now displays pattern types and allows to set this property to either LITERAL or PREFIXED
- SQL processors deployed in Kubernetes now allows to set pod CPU limits via lenses.kubernetes.processor.cpu.request and
lenses.kubernetes.processor.cpu.limit
- The Kafka ACL management screen now displays pattern types and allows to set this property to either LITTERAL or PREFIXED
3.0.9 Patch Release¶
Note
London, UK - April, 2020 Lenses 3.0.9 is now generally available
Improvements¶
- SQL performance increase
Fixes¶
- Fix Alertmanager plugin not passing Source and Generator URL settings
- Fix UI AVRO validation in schema creation failing for some valid schemas
3.0.10 Patch Release¶
Note
London, UK - April, 2020 Lenses 3.0.10 is now generally available
Improvements¶
- SQL performance increase
- SQL CAST function now allows to change Timestamp to Long CAST(_meta.timestamp as LONG)
- SQL CAST function now allows to change INT/LONG to Timestamp CAST(fieldA as TIMESTAMP)
Fixes¶
- Display the retention bytes or milliseconds as
infinitewhen the topic config is -1 (Infinite) - Do not ignore a
retention.msvalue of-1when creating a topic - Do not override brokers metrics settings in
lenses.confwith the autodetected values from Zookeeper - Improve logs output when we the configuration files exist but we cannot read them (e.g permissions issue)