1.0
Installation
Lenses for Apache Kafka is available to download as a platform specific archive. Linux (64bit), macOS and Windows (64bit) platforms are supported.
Requirements
Lenses 1.0 has a direct dependency on a deployment of Apache Kafka deployment from version 0.10.2 onwards. Whilst earlier versions of Kafka may work, there will be no official support.
3GB of free RAM and about 300MB of disk space is recommended. Lenses will work with less memory but performance may suffer. There is no minimum processor requirement although the embedded JRE runs only on 64bit Operating Systems. More memory and a better CPU might be needed especially for multiple Lenses SQL queries running concurrently.
JVM options can be set via the LENSES_HEAP_OPTS
environment variable. For example to increase the heap space:
export LENSES_HEAP_OPTS="-Xmx3g -Xms512m"
It is recommended that Lenses is deployed in the same data center as the Kafka cluster it is monitoring.
Apache Kafka
Lenses has been tested with the latest Kafka releases (i.e. 0.11.0
and 0.10.2
).
Releases from Apache Kafka and Confluent are both endorsed.
The configured Kafka cluster must be accessible from the host running Lenses. Think of it as a typical producer or consumer, which should have access to all the brokers in the cluster and the Zookeeper ensemble.
Confluent’s, Apache licensed, Schema Registry and Kafka Connect are first class citizens of Lenses, although their presence is optional.
Payload Formats
Lenses supports multiple type of Kafka message payloads. Json and Avro support are first class citizens, but Lenses can also work with String, Long, Int or binary content. If Schema Registry is used, Lenses provides easy schema management and more.
Kafka JMX
To leverage Lenses full monitoring capabilities we advise you to enable JMX for the Kafka components and Zookeeper.
Export the environment variable JMX_PORT=[PORT]
for each software instance (broker, zookeeper node, schema registry,
connect worker), where [PORT]
is the port that should be used for JMX access.
Lenses Setup
Lenses is available as a platform specific archive. There are tar.gz
archives for Linux (64bit) and macOS, and
a zip archive for Windows (64bit).
Once the archive is downloaded, it should be uncompressed and extracted to a location of your choice. We recommend to avoid white spaces for the installation path.
The installation directory structure will be:
lenses ├── LICENSE.txt ├── NOTICE.txt ├── bin/ ├── lenses.conf ├── license.json ├── jre8u131/ ├── lib/ ├── licenses/ └── logs/
The file lenses.conf
contains the configuration. Edit the connection details for the Kafka cluster before using
the software. Other configuration options include the user accounts and roles, port, logs directory and more.
A complete overview of the configurations options can be found here.
If you are on Linux, Lenses requires a slightly larger open file descriptor limit than typical desktop applications. You can check you current limit with the ulimit command:
$ ulimit -n -S # soft limit $ ulimit -n -H # hard limit
Typically on Linux the soft limit is 1024
and can be set up to the hard limit by the user. In order to further
increase the limit, superuser access is needed. A soft limit of 4096
is recommended for Lenses,
though 2048 may be enough. To change the soft limit run:
$ ulimit -S -n 4096
Once the configuration in lenses.conf
is complete and the open file descriptor limit has been set in case of
Linux platforms, Lenses is ready to run.
To run on Linux or macOS:
$ bin/lenses
Or to explicitly set the configuration file:
$ bin/lenses lenses.conf
To run on Windows:
$ bin/lenses.bat lenses.conf
Once Lenses is running, visit the address:port
set in the configuration. The default is http://localhost:9991 and
login using the credentials. The default Lenses credentials are admin:admin
.
Always remember to change the default credentials!
To stop Lenses, press CTRL+C
.
Quick Start
To quick start with Lenses just edit the lenses.conf
file and set the following properties to your infrastructure end-points:
lenses {
# Set the ip:port for Lenses to bind to
ip = 0.0.0.0
port = 9991
# Set up infrastructure end-points
kafka.brokers = "PLAINTEXT://host1:9092,PLAINTEXT://host2:9092"
zookeeper.hosts = "zkhost1:2181,zkhost2:2181,zkhost3:2181"
schema.registry.urls = "http://schema-host:8081"
connect = [
{dev: "http://kafka-connect-dev-host-or-ip:8082"}
#, {uat: "http://connect1:8082,http://connect2:8082"}
]
# Set up monitoring end-points
jmx {
brokers = "host1:jmx-port,host2:jmx-port,host3:jmx-port"
schema.registry = "schema-host:jmx-port"
zookeepers = "zkhost1:jmx-port,zkhost2:jmx-port,zkhost3:jmx-port"
connect = [ {dev: "kafka-connect-dev-host-or-ip:jmx-port"} ]
}
}
Then execute bin/lenses and you should be able to loging with admin/admin within moments at http://host-or-ip:9991
Deployment
Lenses currently supports only manual deployment. Future releases will be enhanced with package repositories, containers, etc. Some best practice advice can be offered for installation on a server:
- Install Lenses under
/opt
- Make sure the directory is owned by the root user
- Create a user specially for Lenses.
- Lenses configuration may be placed under
/opt/lenses/lenses.conf
,/etc/lenses/lenses.conf
or passed in at the command line when staring Lenses.
bin/lenses myconf.conf
- The configuration file contains information about the Lenses users’ credentials. Please make sure it is only readable by the Lenses user:
chmod 0600 /path/to/lenses.conf
chown [lenses-user]:root /path/to/lenses.conf
- Please adjust the
logs
configuration to a path for which the running Lenses process has write access to or, to stdout/stderr in case the log management should happen by the process supervisor.
If your server is on a current Linux distribution and it uses the systemd
service manager, the script below can be
used to start and stop Lenses with the management and supervision facilities of systemd.
[Unit]
Description=Run Lenses.io DataOps Service
[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses
#Environment=JAVA_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback..xml"
ExecStart=/opt/lenses/bin/lenses /etc/lenses/lenses.conf
[Install]
WantedBy=multi-user.target
Logging
The logback.xml
file can be used to control logging. By default Lenses will log into logs directory. See the
logging configuration section for more information.