Topology Configs

Lenses provides you with an interactive graph containing your streaming pipelines over Kafka in the Topology view. A streaming topology consists of topics, connectors, SQL processors and custom application nodes.

Lenses works out the nodes and generates the flows which end up in the global topology graph. This provides you with the go-to place for monitoring all your data streaming pipelines. It gives you the high-level view of how your data moves in and out of Kafka.

Connector Nodes

How Lenses identifies connectors

Lenses fully supports Kafka Connect. Therefore you are able to manage and monitor your connectors to external systems. It follows the standard Connect API, and identifies the connector plugins that you have provided to the cluster.

However, when it comes to the topology view, the platform needs to have some extra information about the connectors. Lenses already supports over 45 Kafka Connect metadata, and through configuration any new ones can be added.

Configure custom connectors

If you have a custom connector that you want to enable it for the topology view, or a connector not supported out of the box, you will need to set the lenses.connectors.info configuration entry.

Here is how you can configure connectors in order to appear in the topology graph:

lenses {

  ...

  connectors.info = [
      {
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"
      }
      ...
  ]
}

Source connectors require an extra configuration in order to extract the topics information from the connector configuration. The extractor class should be: io.lenses.core.kafka.connect.SimpleTopicsExtractor. When using this extractor it is also expected to provide a property configuration which specifies the field in the connector runtime configuration listing the topics to publish to. Below you can find an example of the file stream source:

class.name = "org.apache.kafka.connect.file.FileStreamSource"
name = "File"
instance = "file"
sink = false
property = "topic"
extractor.class = "io.lenses.core.kafka.connect.SimpleTopicsExtractor"

Here are some of the entries the default Lenses configuration provides automatically - of course, all Landoop connectors are already covered.

lenses {

  ...

  connectors.info = [
    {
      class.name = "com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector"
      name = "Cassandra"
      instance = "connect.cassandra.contact.points"
      sink = true
      icon = "cassandra.jpg"
      description = "Store Kafka data into Cassandra"
      docs = "//docs.lenses.io/connectors/sink/cassandra.html"
      author = "Lenses.io"
    },
    {
      class.name = "com.datamountaineer.streamreactor.connect.cassandra.source.CassandraSourceConnector"
      name = "Cassandra"
      instance = "connect.cassandra.contact.points"
      sink = false
      property = "connect.cassandra.kcql"
      extractor.class = "io.lenses.config.kafka.connect.KcqlInsertTopicsExtractor"
      icon = "cassandra.jpg"
      description = "Extract Cassandra data using the CQL driver into Kafka"
      docs = "//docs.lenses.io/connectors/source/cassandra.html"
      author = "Lenses.io"
    },
    {
      class.name = "com.datamountaineer.streamreactor.connect.ftp.source.FtpSourceConnector"
      name = "Ftp"
      instance = "connect.ftp.address"
      sink = false
      property = "connect.ftp.monitor.tail,connect.ftp.monitor.update"
      extractor.class = "io.lenses.config.kafka.connect.FtpTopicsExtractor"
      icon = "ftp.png"
      description = "Tail remote FTP folders and bring messages in Kafka"
      docs = "//docs.lenses.io/connectors/source/ftp.html"
      author = "Lenses.io"
    },
    {
      class.name = "com.datamountaineer.streamreactor.connect.jms.source.JMSSourceConnector"
      name = "Jms"
      instance = "connect.jms.url"
      sink = false
      property = "connect.jms.kcql"
      extractor.class = "io.lenses.config.kafka.connect.KcqlInsertTopicsExtractor"
      icon = "jms.png"
      description = "Get data from JMS into Kafka"
      docs = "//docs.lenses.io/connectors/source/jms.html"
      author = "Lenses.io"
    },
    {
      class.name = "org.apache.kafka.connect.file.FileStreamSinkConnector"
      name = "File"
      instance = "file"
      sink = true
      extractor.class = "io.lenses.core.kafka.connect.SimpleTopicsExtractor"
      icon = "file.png"
      description = "Store Kafka data into files"
      author = "Apache Kafka"
    },
    {
      class.name = "org.apache.kafka.connect.file.FileStreamSourceConnector"
      name = "File"
      instance = "file"
      sink = false
      property = "topic"
      extractor.class = "io.lenses.core.kafka.connect.SimpleTopicsExtractor"
      icon = "file.png"
      description = "Tail files or folders and stream data into Kafka"
      author = "Apache Kafka"
    },
    {
      class.name = "io.debezium​.connector.mysql.MySqlConnector"
      name = "CDC MySQL",
      instance = "database.hostname"
      sink = false,
      property = "database.history.kafka.topic"
      extractor.class = "io.lenses.core.kafka.connect.SimpleTopicsExtractor"
      icon = "debezium.png",
      description = "CDC data from RDBMS into Kafka"
      docs="//debezium.io/docs/connectors/mysql/"
      author = "Debezium"
    }
    ...
  ]
}

Note

If the connector configuration is not present in the configuration, Lenses will ignore the connector and therefore it won’t show up in the topology view.

Key	Description	Optional	Type	Default
class.name	The connector full class name. i.e. org.apache.kafka.connect.file.FileStreamSourceConnector	No	String	N/A
name	The name as it will appear in the topology view. For example: File	No	String	N/A
instance	Contains the connector configuration key(-s) the extractor instance will use to get the information. Consider the FTP source the value is set to connect.ftp.monitor.tail,connect.ftp.monitor.update. The extractor will get the result of both those fields and provide a list of involved topics	No	String	N/A
extractor.class	Used for Connect sources only to extract the target topics from the connector runtime configuration.	No	String	N/A
icon	The path to an icon file the UI will use to display |the connector.	Yes	string	N/A
description	A short sentence to say what the connector does	Yes	string	N/A
author	Who is providing the connector	Yes	string	N/A

Lenses provides these classes to extract the information from a Kafka Connect connector configuration:

Class	Description
io.lenses.config.kafka.connect.KcqlInsertTopicsExtractor	A class responsible for extracting the target Kafka topics defined by a KCQL statement. This targets our own source connectors providing an easy way to describe their Action via SQL like syntax. The class can extract from syntax like INSERT INTO $TOPICA SELECT ...
io.lenses.config.kafka.connect.FtpTopicsExtractor	A class responsible for extracting the target topics for the FTP source
io.lenses.core.kafka.connect.SimpleTopicsExtractor	A class responsible for extracting the information out of a connector configuration entry. For example, topic:topicA, topicB

Lenses SQL Processor Nodes

Lenses SQL Processors are continuously running queries that translate to a Kafka Stream application behind the scenes. It’s an easy way to generate scalable apps by leveraging SQL semantics. Lenses handles the processor nodes, and you don’t need any added configuration.

Custom Nodes

The Topology view is a powerful feature of Lenses that is not only limited to Connectors and Processors. Users may supply their own custom applications, visualize and monitor them in Lenses. The developers need to use our open source library Topology-Client.

Read more on the Developer Guide for Topology.