Lenses for Kafka

Lenses is an enterprise-grade product enabling you to deliver streaming applications faster and to manage your data flows natively over Apache Kafka.

Lenses enriches your Kafka eco-system with a rich user interface, vital enterprise capabilities which enable engineering teams to query real-time data, create and monitor Kafka streaming topologies and integrate with other systems.

Lenses also integrates with your Kubernetes cluster and enables you to operate and manage your modern scalable microservices and streaming data flow architectures.

Enterprise Grade

Data Visibility and Control

View, Query, Subscribe to Kafka Topics, Manage Schemas and monitor the Data Consumers.

Connect Data Store

Lenses supports every Kafka Connector. There are more than 100 available open-source connectors.

Process Data with SQL

Run and operate streaming SQL executed as KStreams consuming and producing data. View you topologies, with built-in monitoring. Works with your Kubernetes and Microservices for your end to end pipelines.

Data Visibility and Control

View, Query, Subscribe to Kafka Topics, Manage Schemas and monitor the Data Consumers.

Rich Ecosystem

Use the open-source Lenses libraries with Python (and Data Science Notebooks), or JDBC driver (connect your BI tools, Spark Jobs) and Javascript (build web apps).

Monitoring & Alerting

Monitor end-to-end pipelines and low level operations in Kafka. Lenses integrates with Prometheus, Grafana and Alert Manager to set up custom Alerts for Healthchecks, Applications, Consumers etc.

Data Governance

Lenses provides auditing on user activity, access and provides full and visible lineage of your cluster. Compliance settings, Anonymise fields, Manage Schemas support.

Admin Operations & Security

Fine-grained Access controls support via LDAP, Kerberos, Active Directory, TLS Certificates, ACLs, Quotas. Whitelisting and Blacklisting capabilities.

Fits your CI/CD

Use the Lenses APIs or CLI tool to extract configurations as code, and also drive your CI/CD pipelines.

Development Friendly, Cloud Ready

Lenses box contains all the required services to enable you to start developing Kafka applications instantly, whereas Lenses itself integrates natively with Kubernetes, and Cloud discovery helps you setup and operate cloud ready data pipelines.

Features Overview

Monitor Kafka

Lenses Kafka Monitoring Suite is a set of pre-defined templates, that use

  • A Time Series database (Prometheus)
  • Custom JMX exporters
  • A Data Visualization application (Grafana)
  • Built-in domain intelligence about operating Kafka with confidence in production

Lenses is great at continuously monitoring your Kafka cluster and can raise alerts for important metrics degradation, such as consumer lag and offline or under-replicated partitions. However, it does not strive to become a time series database because there are well established solutions the likes of Prometheus.

See kafka monitoring suite setup for details.

../_images/kafka-monitoring.png

Kafka Development

../_images/Lenses-Box-Red.png

Lenses is the right way for you to get started with Apache Kafka:

You start with a single docker command and have a full development and debugging experience. Lenses opens up Kafka by providing an intuitive user interface, out of the box monitoring capabilities and assists you move to production faster. Data inspection via SQL is one of the most powerful features of Lenses enabling you, to easier develop and debug your applications.

To move data in and out of Kafka, Lenses provides out of the box and easy to use integration with more than 100 Kafka Connect connectors. At Landoop we have developed 30 plus of those connectors.

To ensure your application and infrastructure behaves as expected, Lenses monitors all the aspects required to make sure you do not lose data. Lenses can provide continuous monitoring of all you Consumers, Producers, and other Streaming applications and depict it in a global topology chart, so you can follow you data-pipelines and how data moves around.

Connectors and SQL can cover 65% to 75% of your needs in terms of data processing when you need filtering, transformations, joins, counts and your custom applications can cover the real business value.

Discover Lenses Box and Lenses Ecosystem

Message Payloads

Lenses & Lenses SQL Engine supports multiple data types out of the box but also allows you to register custom serializers and deserializers. It is the only SQL engine for Apache Kafka which is opened to any data format.

You can browse, query and register processors for AVRO, JSON, Protobuf, XML, CSV, Array payloads, query plain text with Regex. For AVRO payload, Lenses integrates with Schema Registry (Confluent and Hortonworks) but adds lineage and role-based access on top to meet the demands of an enterprise environment.

../_images/topic_serdes.png

Customize

Lenses works with your existing Kafka installation and allows multiple connection protocols and security authorizers. Visual topologies and monitoring features become available for your topics, connectors, and applications. You can spin up custom connectors and hook custom applications into the topology in order to leverage monitoring and alerting. You can allow and deny access to topics and even register your custom serializers for data browsing and processing.

../_images/topology-apps1.png