Lenses.io Help Center
Latest 5.0 4.3 4.2 4.1 4.0 3.2 3.1 3.0 2.3 2.2 2.1 2.0 1.1 1.0
Latest 5.0 4.3 4.2 4.1 4.0 3.2 3.1 3.0 2.3 2.2 2.1 2.0 1.1 1.0

1.0

You are viewing documentation for an older version of Lenses.io View latest documentation here

Development Environment

All in one Docker 

The easiest way to start with Lenses for Apache Kafka is to get the Development Environment, a docker that contains Kafka, Zookeeper, Schema Registry, Kafka Connect, Lenses and a collection of 25+ Kafka Connectors and example streaming data.

Download the best Kafka Tool for development here

You will immediately receive an email with your Access Key, and instructions to quick-start. The docker command looks like this:

    docker run --rm -it -p 3030:3030 --net=host -e LICENSE_URL="https://milou.landoop.com/download/lensesdl/?id=CHECK_YOUR_EMAIL_FOR_PERSONAL_ID" landoop/kafka-lenses-dev:latest

This container is useful for developers that have at least a 8GB RAM laptop, and want to have a full environment with an efficient memory footprint. It takes 30-45 seconds for the services to become available.

Visit http://localhost:3030 and login with admin/admin and enjoy!

Kafka Development docker tool

Don’t have docker? Check installation instructions for docker here

Note: Kafka Connect will require a few more minutes to start up, as it iterates and loads all the available connectors in the background.

Running Kafka SQL processors 

We have prepared example SQL processor queries, ready to be executed with the synthetic data the container contains:

INSERT INTO position_reports_Accurate
SELECT * FROM `sea_vessel_position_reports`
WHERE _vtype ='AVRO'
AND _ktype ='AVRO'
AND Accuracy IS true

INSERT INTO position_reports_latitude_filter
SELECT Speed, Heading, Latitude, Longitude, Radio
FROM `sea_vessel_position_reports`
WHERE _vtype = 'AVRO'
AND _ktype = 'AVRO'
AND Latitude > 58

INSERT INTO position_reports_MMSI_large
SELECT *
FROM `sea_vessel_position_reports`
WHERE _vtype = 'AVRO'
AND _ktype = 'AVRO'
AND MMSI > 100000

Of course, there is support for JSON messages as well:

INSERT INTO backblaze_smart_result
SELECT (smart_1_normalized + smart_3_normalized) AS sum1_3_normalized, serial_number
FROM `backblaze_smart`
WHERE _vtype = 'JSON'
AND _ktype = 'JSON'
AND _key.serial_number LIKE 'Z%'

Lenses SQL, supports setting consumer, producer and streaming configs as well as JOINS and aggregations

SET `autocreate`=true;
SET `auto.offset.reset`='earliest';
SET `commit.interval.ms`='120000';
SET `compression.type`='snappy';

INSERT INTO `cc_payments_fraud`
WITH tableCards AS (
  SELECT *
  FROM `cc_data`
  WHERE _ktype='STRING' AND _vtype='AVRO' )

SELECT STREAM
  p.currency,
  sum(p.amount) as total,
  count(*) usage
FROM `cc_payments` AS p LEFT JOIN tableCards AS c ON p._key = c._key
WHERE p._ktype='STRING' AND p._vtype='AVRO' and c.blocked is true
GROUP BY tumble(1,m), p.currency

By clicking on any node of the topology, you can review the node configuration, or view topic data. For the above query also see this Kafka fraud detection blog

Kafka Streaming SQL

Frequently Asked Questions 

1. How can i run this docker with a Custom/Advertised hostname ?

If you are using docker-machine or setting this up in a Cloud or DOCKER_HOST is a custom IP address such as 192.168.99.100 you will need to additionally use the parameters --net=host -e ADV_HOST=192.168.99.100

docker run --rm -it -p 3030:3030 --net=host -e ADV_HOST=192.168.99.100 \
       -e LICENSE_URL="https://milou.landoop.com/download/lensesdl/?id=CHECK_YOUR_EMAIL_FOR_PERSONAL_ID" landoop/kafka-lenses-dev

2. How can i run the Development Environment offline ?

To run the docker offline, save the download access token to a file when downloading from https://lenses.io/apache-kafka-docker/ and do the equivalent of the below command for your operating system

LFILE=`cat license.json`
docker run --rm -it -p 3030:3030 -e LICENSE="$LFILE" landoop/kafka-lenses-dev:latest

3. How can i persist data ?

If you want your Development environment to persist data between multiple executions, provide a name i.e.

docker run -p 3030:3030 -e LICENSE_URL="CHECK_YOUR_EMAIL_FOR_PERSONAL_ID" \
       --name=lenses-dev landoop/kafka-lenses-dev

Once you want to free up resources, just press Control-C

Now you have two options: either remove the Development Environment:

    docker rm lenses-dev

Or use it at a later time, continue from where you left of:

    docker start -a lenses-dev

4. How much memory does Lenses require ?

Lenses has been built with “mechanical sympathy” in mind. Before releasing it we make sure it can operate with 4GB RAM on a Production cluster of 30 Kafka Brokers, with > 10K Avro Schemas, a 1000 topics and tens of connectors.

The Development Environment is running multiple services in a container: Kafka, ZK, SR, Kafka Connect, synthetic data generators and of course Lenses. That means that the recommendation is 5GB of RAM, although it can operate with even less than 4GB (your mileage might vary).

Setting docker memory for Kafka

5. How can i connect to a large kafka cluster ?

To run Lenses on a large Kafka cluster, contact-us , to receive the Enterprise Edition that also contains Kubernetes support and additional operational lenses.

6. Why does my Developer Environment have topics ?

To make your experience easier, we have pre-configured a set of synthetic data generators for streaming data. By default we have the data generators running. You may turn them off by setting the environment variable -e SAMPLEDATA=0

7. How can i access Kafka from other Clients ?

Due to the way docker and the kafka broker works, accessing kafka from your own consumer or producer may be tricky. The Kafka Broker advertises a —usually autodetected— address that must be accessible from your client. To complicate things docker, when run on macOS or Windows, runs inside a virtual machine, adding an extra networking layer.

If you run docker on macOS or Windows, you may need to find the address of the VM running docker. On macOS it usually is 192.168.99.100 and export it as the advertised address for the broker. At the same time you should give the kafka-lenses-dev image access to the VM’s network:

docker run -e LICENSE_URL="CHECK_YOUR_EMAIL_FOR_PERSONAL_ID" \
           -e ADV_HOST="192.168.99.100" \
           --net=host --name=lenses-dev \
           landoop/kafka-lenses-dev

If you run on Linux you don’t need the ADV_HOST but you can do something cool with it. If you set as the ADV_HOST your machine’s IP address you will be able to access Kafka from all clients in the network. If you decide to run kafka-lenses-dev in the cloud, you could access Kafka from your local machine. Just remember to provide the public IP of your server!

8. Is the Development Environment free ?

Developer licenses are free. You can get one —or more— from our website . The license will expire in six months unless you renew it —for free. You may start your Lenses instance with a different license file. Your setup will not be affected.

There are three ways to provide the license file. The first we already saw, provide the license url via LICENSE_URL:

    -e LICENSE_URL="[LICENSE_URL]"

If you instead choose to save the license file locally, you can either provide it as a file:

    -v /path/to/license.json:/license.conf

Or as an environment variable:

    -e LICENSE="$(cat /path/to/license.json)"

9. How can i access the command line ?

You can access the various Kafka command-line tools, such as the console producer and consumer from a terminal in the container:

    docker exec -it lenses-dev bash

Or directly:

    docker exec -it lenses-dev kafka-topics --zookeeper localhost:2181 --list

If you enter the container, you will discover that we even provide bash auto-completion for some of the tools.