GCP

Lenses can be deployed in GCP with your own Apache Kafka cluster with a Deployment manager template. The fields which you need to fill in are:

Values Configuration

Field Description Type Default Required
machineType
The type of the machine which lenses run.
Recommended >= n1-standard-2
string n1-standard-2 no
network
The network in which Lenses.io will be deployed
string default no
lensesLicense
The Lenses license.
You can download here https://lenses.io/downloads/
string   yes
lensesPort
The port which Lenses will run and be available
int 9991 no
lensesPassword
The admin password of Lenses
string admin no
zone The GCP zone to run the VM string   yes
sourceRanges
Source ranges for Firewall configuration
string 0.0.0.0/0 yes
kafkaBrokers
A list of host/port pairs to
use for establishing the initial connection to the
Kafka cluster. Add just a few broker addresses
here and Lenses will bootstrap and discover the
full cluster membership (which may change dynamically).
This list should be in the form
"host1:port1,host2:port2,host3:port3"
string   yes
zookeeper
A list of all the zookeeper nodes
array   no
connect Defines the Kafka connect clusters array   no
schemaRegistry
A list of Schema Registry nodes
array   no

Deployment Manager Templates

There is one template which you can use to deploy Lenses in GCP. Based on Hardware & OS requirements for Lenses, we recommend to start with n1-standard-2 instances or any other instance type with >= 7GB.

Container Optimized Deployment

Container-Optimized OS is an operating system image for your Compute Engine VMs that is optimized for running Docker containers. With Container-Optimized VM, you can bring up Lenses running with Docker container on Google Cloud Platform quickly, efficiently, and securely.

The Container Optimized Deployment for Lenses supports also service discovery of Kafka Brokers, Zookeeper, Schema Registry and Connect based on the provided zone. Use the config.yaml to override the values for your configuration.

imports:
- path: lenses.jinja

resources:
- name: lenses
  type: lenses.jinja
  properties:
    zone: europe-west3-a
    sourceRanges: "0.0.0.0/0"
    machineType: n1-standard-2
    kafkaBrokers: "PLAINTEXT://<YOUR-KAFKA-BROKER-IP>:9092"
    lensesLicense: |
    zookeeper: |
      [
        {url:"zookeeper.1.url:2181", jmx:"zookeeper.1.url:9585"},
        {url:"zookeeper.2.url:2181", jmx:"zookeeper.2.url:9585"}
      ]
    schemaRegistry: |
      [
        {url:"http://schema.registry.1.url:8081",jmx:"schema.registry.1.url:9582"},
        {url:"http://schema.registry.2.url:8081",jmx:"schema.registry.2.url:9582"}
      ]
    connect: |
      [
        {
          name:"data_science",
          urls: [
            {url:"http://connect.worker.1.url:8083",jmx:"connect.worker.1.url:9584"},
            {url:"http://connect.worker.2.url:8083",jmx:"connect.worker.2.url:9584"}
          ],
          statuses:"connect-statuses-cluster-a",
          configs:"connect-configs-cluster-a",
          offsets:"connect-offsets-cluster-a"
        }
      ]

You can find in Github Repo the available template. You can deploy it with the following command:

gcloud deployment-manager deployments create lenses --config config.yaml

Service Discovery

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring and default ports. left at default values. Lenses VM should have the scope https://www.googleapis.com/auth/compute.readonly.

SD_CONFIG=provider=gce zone_pattern=europe-west1.*

SD_BROKER_FILTER=tag_value=broker

SD_ZOOKEEPER_FILTER=tag_value=zookeeper
SD_ZOOKEEPER_JMX_PORT=9585

SD_REGISTRY_FILTER=tag_value=schema-registry
SD_REGISTRY_JMX_PORT=9582

SD_CONNECT_FILTERS=tag_value=connect-worker-testing,tag_value=connect-worker-production
SD_CONNECT_NAMES=testing,production
SD_CONNECT_STATUSES=connect-statuses-testing,connect-statuses-production
SD_CONNECT_CONFIGS=connect-configs-testing,connect-configs-production
SD_CONNECT_OFFSETS=connect-offsets-testing,connect-offsets-production
SD_CONNECT_JMX_PORTS=9584