Installation

Note

Lenses supports Kafka versions 0.10.2+. Kafka 0.11.0+ is strongly recommended.

Download

The current version of Lenses is v2.0. See Release Notes. For those upgrading from 1.1, please see the Upgrade Notes.

Archives

Lenses for Apache Kafka is available to download as a platform-specific archive.

The supported platforms are:

• Linux (64bit),
• macOS and
• Windows (64bit)

We provide tar.gz archives for Linux (64bit) and macOS, and a zip archive for Windows (64bit).

Request your free trial here. The download link for the software will be sent via email to the individuals who have registered for an enterprise setup.

Containers

For Docker, Kubernetes or Openshift environments a docker container for Lenses is available under landoop/lenses. Find additional information here: https://github.com/lensesio/lenses-docker .

Development Environment

If you are looking for a Lenses Development Environment (free) just download it from here .

System Requirements

Hardware and software requirements may vary depending on the machine and operating system.

The minimum hardware specifications to install and effectively operate Lenses are:

• 64bit Operating System
• 4GB of free RAM
• 500MB of disk space

JVM

JVM specific options can be set via the LENSES_HEAP_OPTS environment variable. For example to increase the heap space you will have to do the following:

export LENSES_HEAP_OPTS="-Xmx6g -Xms512m"

It is recommended that Lenses is deployed in the same data center as the Apache Kafka cluster it links to. See more on supported Java Options in the configuration section.

Open File Descriptors

If you are running on a Linux machine, Lenses requires a slightly larger open file descriptor limit than typical desktop applications. You can check your current limit with the ulimit command:

$ ulimit -n -S # soft limit
$ ulimit -n -H # hard limit

Typically, on Linux, the soft limit is 1024 and can be set up to the hard limit by the user. In order to change the limit, superuser access is needed. A soft limit of 4096 is recommended for Lenses, though 2048 may be enough. To change the soft limit run:

$ ulimit -S -n 4096

Kafka Support

Versions

Lenses for Apache Kafka has been tested with the Apache Kafka releases version 0.10.2 upwards. We strongly recommend 0.11.0 or greater.

Distributions

Distributions from i) Apache Kafka ii) Cloudera iii) Confluent and iv) HortonWorks are endorsed. Other distributions and custom builds should work as well.

Connectivity

Your Apache Kafka cluster must be accessible from the host running Lenses. Think of it as a typical producer or consumer which should have access to all the brokers in the cluster and the Zookeeper ensemble. Apache Kafka Connect and Confluent’s, Apache 2 licensed, Schema Registry are fully supported however their presence is optional.

Message Payloads

Lenses supports multiple types of Kafka message payload with Json and Avro being first class citizens. Alongside these two the following payload types are also covered: String, Long, Int and binary content. If Schema Registry is used, the platform provides easy schema management and more.

JMX

In order to make the most out of the monitoring capabilities, JMX metrics need to be enabled for the Kafka Brokers, Zookeepers, Schema Registry, Connect workers. To enable JMX, the environment variable needs to be set for each of the above components before they are started:

JMX_PORT=[PORT], where [PORT] is the port that should be used for the JMX access.

See JMX section for more details on enabling JMX.

Quick Setup

Once the archive has been downloaded, it should be extracted to a location of your choice. We recommend avoiding white spaces for the installation path.

The directory structure is:

lenses
├── LICENSE.txt
├── NOTICE.txt
├── bin/
├── lenses.conf
├── license.json
├── security.json
├── jre8u131/
├── lib/
├── licenses/
└── logs/

lenses.conf file contains the application specific settings. You will need to edit the connection details for the Kafka cluster before using the software. Other configuration options include the user accounts and roles, port, logs directory and more. A complete overview of the configurations options can be found here.

Example Configuration

# Set the ip:port for Lenses to bind to
lenses.ip = 0.0.0.0
lenses.port = 9991
#lenses.jmx.port = 9992

# License file allowing connecting to up to N brokers
lenses.license.file = "license.json"

# topics created on start-up that Lenses uses to store state
lenses.topics.audits = "_kafka_lenses_audits"
lenses.topics.metrics = "_kafka_lenses_metrics"
lenses.topics.cluster = "_kafka_lenses_cluster"
lenses.topics.profiles = "_kafka_lenses_profiles"
lenses.topics.processors = "_kafka_lenses_processors"
lenses.topics.alerts.settings= "_kafka_lenses_settings"
lenses.topics.alerts.storage= "_kafka_lenses_alerts" # Persists all alerts
lenses.topics.lsql.storage= "_kafka_lenses_lsql_storage" #Stores all LSQL run

# Set up infrastructure end-points
lenses.kafka.brokers        = "PLAINTEXT://host1:9092,PLAINTEXT://host2:9092,PLAINTEXT://host3:9092"
lenses.zookeeper.hosts      = [
  { url:"host1:2181", jmx:"host1:9010" },
  { url:"host2:2181", jmx:"host2:9010" },
  { url:"host3:2181", jmx:"host3:9010" }
]
lenses.zookeeper.chroot     = ""         # Optional in case a ZK chroot path is in use
lenses.schema.registry.urls = [
  { url:"http://host1:18081", jmx:"host1:19395" },
  { url:"http://host2:18081", jmx:"host2:19395" }
]
lenses.connect.clusters     = [
  { name: "connect_cluster_X", statuses: "connect-statuses", configs: "connect-configs", offsets: "connect-offsets",
    urls: [
     { url: "http://host1:8083", jmx: "host1:18083" },
     { url: "http://host2:8083", jmx: "host2:1880"  }
    ]
  }
]

security.conf file contains the security configuration. For more information, read Security.

Note

We recommend going through Lenses Configuration to understand how to fully configure your installation

Deploy & Run

Once the configuration is complete, Lenses is ready to run. We recommend running 1 instance of Lenses per Kafka cluster. Running multiple instances against the same infrastructure, (if not configured appropriately) could result in Lenses SQL processors duplicating data if the processors are set to run in the same process as Lenses.

The bin folder contains the start scripts and depending on your operating system you have the choice between lenses and lenses.bat. Lenses application requires a startup argument which should be the path to the configuration file. Here is an example of how to launch it:

To run on Linux or macOS

$ bin/lenses

Or to explicitly set the configuration file:

$ bin/lenses lenses.conf

To run on Windows:

$ bin/lenses.bat lenses.conf

Once the application is running, open your browser and navigate to http://address:port based on the configuration file. The default is http://localhost:9991. Log in using the credentials you have provided, otherwise the default values are admin:admin. We strongly recommend changing the default credentials!

To stop Lenses, press CTRL+C.

The configuration file is using the HOCON format.

JMX for Cluster Services

JMX is used to gain better operational monitoring of your infrastructure and it should be enabled to benefit from the best experience provided by the platform. Enable JMX on all:

Service	Description
Kafka Brokers	Update bin/kafka-server-start.sh and add export JMX_PORT=${JMX_PORT:-9581} to enable JMX on port 9581
Schema-Registries	Update JMX_PORT=9582
Kafka-REST	Update JMX_PORT=9583
Kafka-Connect	Update the worker properties, connect-distributed.conf or connect-standalone.conf for Confluent and include JMX_PORT=9584
Zookeepers	Update bin/zkServer.sh or `` and follow the details to enable JMX to i.e. port 9585

Note

The above ports are only for reference only. After enabling JMX on each service you will have to restart the service. Ensure the ports are reflected correctly in the lenses.conf file.

JMX for Lenses

A JMX agent starts automatically for Lenses. This behavior and which endpoints to poll can be configured in the JMX section of the configuration file.

Manual Deployment

Some best practice advice can be offered for installation on a server:

1. Install under /opt

2. Make sure the directory is owned by the root user

3. Create a user specially for Lenses

4. The configuration file may be placed under /opt/lenses/lenses.conf, /etc/lenses/lenses.conf or passed in at the command line when staring Lenses.

   bin/lenses myconf.conf
   

5. The security configuration file contains information about the users and groups. Please make sure it is only readable by the Lenses user:

   chmod 0600 /path/to/security.conf
   chown [lenses-user]:root /path/to/security.conf
   

6. Please adjust the logs configuration to a path for which the running Lenses process has write access to or, to stdout/stderr in case the log management should happen by the process supervisor. See the logging configuration section for more information.

If your server is on a current Linux distribution and it uses the systemd service manager, the script below can be used to start and stop Lenses with the management and supervision facilities of systemd.

[Unit]
Description=Run Landoop Lenses Service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses
#Environment=LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback..xml"
ExecStart=/opt/lenses/bin/lenses /etc/lenses/lenses.conf

[Install]
WantedBy=multi-user.target

Proxy and SSL Termination

Lenses does not support SSL termination but rather relies on a proxy set in front of it (such as Apache, nginx and Caddy) to provide SSL to clients. Attention should be given to set the proxy to support websocket connections under the path /api/kafka/ws.

Sample configuration for nginx:

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen 80;
    server_name example.lenses.url;

    # websocket paths
    location /api/ws {
        proxy_pass http://lenses.url:9991;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_redirect off;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  Host $http_host;
    }
    location /api/kafka/ws {
        proxy_pass http://lenses.url:9991;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_redirect off;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  Host $http_host;
    }

    # all other paths
    location / {
        proxy_pass http://lenses.url:9991;
        proxy_http_version 1.1;

        proxy_redirect off;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  Host $http_host;
    }
}

Sample caddy configuration:

proxy /api/kafka/ws http://lenses.url:9991 {
    websocket
}
proxy /api/ws http://lenses.url:9991 {
    websocket
}
proxy / http://lenses.url:9991