View the latest documentation 5.5
The Topology is a rich UI that enables the visualization of detailed Data Flow Graphs, ETL Pipelines Stream Processing and Analytics workflows and their relation to native Lenses/Kafka resources, such as Kafka Topics, SQL Processors, and Kafka Connectors.
Topology
Data Flow Graphs
ETL Pipelines
Stream Processing
Analytics
Kafka Topics
SQL Processors
Kafka Connectors
Lenses Topology is comprised of several different Nodes Types. From Topics, SQL Processors, and Connectors, to External Application. Each Node on the UI is clickable. That way, we can take a close view and look for more information regarding each Node.
Topics
Connectors
External Application
Topology screen allows for Search in the Nodes by Node name which helps navigation in bigger application landscapes. We categorized the Nodes, based on their type(i.e: SQL Processors, Connectors etc.) for easier identification.
Kafka Topics, are “stores” on information. In the Topology, they are depicted as folders, and by clicking them we can see more information, about their usages as described in the image below:
folders
Lenses SQL Processors, allow you to use Streaming SQL to process real-time data. In the Topology, they are depected with a Thunder icon. By clicking them, we can see information such as:
Thunder
Connectors, are used to either import or export Data to and from your Kafka Cluster. They are represented in Lenses Topology as Puzzle icon. By clicking them, we can see information such as:
Puzzle
Applications deployed outside of Lenses can be registered Lenses’ Topology. They are presented with a Lenses App icon. By clicking them, we can see information such as:
Lenses App
If you are unfamiliar with our RBAC Security, you can find more on Application Permissions. You can use either the Lenses HTTP API or the JVM Topology Client Library to instrument your custom applications.
Lenses HTTP API
JVM Topology Client Library
Lenses Topology has built-in support for overlaying metrics for each entity. You can now monitor “Input” and “Output” Rates for Records and Data, as well as the “Status” of the Application in question. You can configure the frequency of the metrics received via the configuration
How can I add new nodes to the topology?
All of your processors, connectors and their associated topics are visible in the Topology screen. Any other node must be added by registering the application by following this guide.
Can I see my consumers/producers to the topology?
Any registered application can be viewed as part of the overall topology and see where it fits in the streaming pipeline.
My kafka connector is not automatically showing in the topology. Why?
This might be happening for two reasons:
Performance issue in many nodes
When your application landscape becomes too heavy (over 5000 nodes), you might experience performance issues in the topology screen.
On this page