Installation

Note: Lenses v1.1 supports Kafka versions 0.10.2 or later

Download

The current version of Lenses is v1.1. See release notes .

For those upgrading from 1.0, please see the upgrade path .

Kafka Development Environment

If you are looking for the best Kafka Development Tool download and use it for free.

System Requirements

Hardware and software requirements may vary depending on the machine and operating system.

The minimum hardware specifications to install and effectively operate Lenses are:

64bit Operating System
4GB of free RAM
300MB of disk space
Lenses is not CPU heavy, unless in-memory Streaming SQL is used (IN_PROC).

Note: You can reduce the RAM resources below the above-specified minimum for development and testing purposes, however performance may be affected.

JVM

JVM specific options can be set via the LENSES_HEAP_OPTS environment variable. For example to increase the heap space you will have to do the following:

  export LENSES_HEAP_OPTS="-Xmx6g -Xms512m"

It is recommended that Lenses is deployed in the same data center as the Kafka cluster linked to it.

Open File Descriptors

If you are running on a Linux machine, Lenses requires a slightly larger open file descriptor limit than typical desktop applications. You can check your current limit with the ulimit command:

  $ ulimit -n -S # soft limit
  $ ulimit -n -H # hard limit

Typically on Linux, the soft limit is 1024 and can be set up to the hard limit by the user. In order to change the limit, superuser access is needed. A soft limit of 4096 is recommended for Lenses, though 2048 may be enough. To change the soft limit run:

  $ ulimit -S -n 4096

Kafka Support

Versions

Lenses has been tested with the Apache Kafka releases from version 0.10.2 upwards.

Distributions

Distributions from Apache Kafka and Confluent are both endorsed.

Connectivity

Your Apache Kafka cluster must be accessible from the host running Lenses. Think of it as a typical producer or consumer which should have access to all the brokers in the cluster and the Zookeeper ensemble. Apache Kafka Connect and Confluent’s, Apache 2 licensed, Schema Registry are fully supported however their presence is optional.

Message Payloads

Lenses supports multiple types of Kafka message payload. JSON and AVRO support are first class citizens, but Lenses also works with String, Long, Int or binary content. If Schema Registry is used, Lenses provides easy schema management and more.

JMX

In order to make the most out of the monitoring capabilities, Lenses requires JMX metrics to be enabled for the Kafka Brokers, Zookeepers, Schema Registry, Connect workers. To enable JMX the environment variable needs to be exported for each of the above components:

JMX_PORT=[PORT], where [PORT] is the port that should be used for the JMX access.

Quick Setup

Once the archive has been downloaded, it should be extracted to a location of your choice. We recommend avoiding white spaces for the installation path.

The directory structure is:

    lenses
    ├── LICENSE.txt
    ├── NOTICE.txt
    ├── bin/
    ├── lenses.conf
    ├── license.json
    ├── jre8u131/
    ├── lib/
    ├── licenses/
    └── logs/

lenses.conf file contains the application specific settings. You will need to edit the connection details for the Kafka cluster before using the software. Other configuration options include the user accounts and roles, port, logs directory and more. Find here, a complete overview of the configurations options .

Example Configuration

# Set the ip:port for Lenses to bind to
lenses.ip = 0.0.0.0
lenses.port = 9991
#lenses.jmx.port = 9992

# License file allowing connecting to up to N brokers
lenses.license.file = "license.json"

# topics created on start-up that Lenses uses to store state
lenses.topics.audits = "_kafka_lenses_audits"
lenses.topics.cluster = "_kafka_lenses_cluster"
lenses.topics.metrics = "_kafka_lenses_metrics"
lenses.topics.profiles = "_kafka_lenses_profiles"
lenses.topics.processors = "_kafka_lenses_processors"

# Set up infrastructure end-points
lenses.kafka.brokers        = "PLAINTEXT://host1:9092,PLAINTEXT://host2:9092,PLAINTEXT://host3:9092"
lenses.zookeeper.hosts      = "zkhost1:2181,zkhost2:2181,zkhost3:2181"
lenses.schema.registry.urls = "http://schema-host:8081"
lenses.connect.clusters     = [{name: "dev", url: "http://kafka-connect-host-or-ip:8083", statuses: "connect-statuses", configs: "connect-configs", offsets: "connect-offsets" }]

# Set up monitoring end-points
lenses.jmx.schema.registry = "schema-host:jmx-port"
lenses.jmx.zookeepers = "zkhost1:jmx-port,zkhost2:jmx-port,zkhost3:jmx-port"
lenses.jmx.connect = [ {dev: "connect1:jmx-port,connect2:jxm-port"} ]

Run & Deploy

Once the configuration in lenses.conf is complete, Lenses is ready to be run. The bin folder contains the start scripts. Depending on your operating system you have the choice between lenses and lenses.bat. Lenses application requires a startup argument which should contain the path to the configuration file. Here is an example of how to launch Lenses:

To run on Linux or macOS ::

  $ bin/lenses

Or to explicitly set the configuration file:

  $ bin/lenses lenses.conf

To run on Windows:

  $ bin/lenses.bat lenses.conf

Once the application is running, open your browser and navigate to http://address:port based on the configuration file. The default is http://localhost:9991.

Login using the credentials you have provided, otherwise the default values are admin:admin. We strongly recommend changing the default credentials!

To stop Lenses, press CTRL+C.

Lenses Instances

We recommend running 1 instance of Lenses per Kafka cluster. Running multiple instances against the same infrastructure, (if not configured appropriately) could result in Lenses SQL processors duplicating data.

Manual Deployment

Some best practice advice can be offered for installation on a server:

Install Lenses under /opt
Make sure the directory is owned by the root user
Create a user specially for Lenses
Lenses configuration may be placed under /opt/lenses/lenses.conf, /etc/lenses/lenses.conf or passed in at the command line when staring Lenses.

      bin/lenses myconf.conf

The configuration file contains information about the Lenses users’ credentials. Please make sure it is only readable by the Lenses user:

chmod 0600 /path/to/lenses.conf
chown [lenses-user]:root /path/to/lenses.conf

Please adjust the logs configuration to a path for which the running Lenses process has write access to or, to stdout/stderr in case the log management should happen by the process supervisor. See the logging configuration section for more information.

If your server is on a current Linux distribution and it uses the systemd service manager, the script below can be used to start and stop Lenses with the management and supervision facilities of systemd.

[Unit]
Description=Run Lenses.io DataOps Service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses
#Environment=LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback..xml"
ExecStart=/opt/lenses/bin/lenses /etc/lenses/lenses.conf

[Install]
WantedBy=multi-user.target