1 of 25

CLI

This page describes using the Lenses CLI.

Lenses offers a powerful CLI (command-line tool) built-in Go that utilizes the REST and WebSocket APIs of Lenses to communicate with Apache Kafka. It provides a straightforward way to perform common data engineering and site reliability engineering tasks.

Quick start

To use the CLI you must first configure the workspace:

lenses-cli configure

And fill in the details:

? Host: https://lenseshost:9991
? Auth token or username:
? Password: ******
? Save workspace name: dataopsDev

The configuration will be stored in ~/.lenses/config.yml

Controlling the output format

The —output flag, controls the format of the output.

The value of output can be Table, YAML or JSON. By default, the results of a command are printed as a table.

Additionally, when using the –output flag with JSON, two more optional flags are available: –pretty and –query. These flags can be passed to all commands that fetch and return JSON-formatted results.

—pretty - Enable the pretty format for JSON output of commands (default false).
—query - A jmespath query expression. This allows for querying the JSON output of commands. For more details view the JMESPATH documentation.

Usage with a service account

To use the CLI with a service account for CI/CD you need to pass these options:

lenses-cli topics \
  --token=<service-account-name>:<service-account-token> \
  --host=<lenses-url-host>

# Real Example
lenses-cli topics \
  --token=ci:58d86476-bcc6-47e2-a57e-0c6bbd9c88b9 \
  --host=http://<your-lenses-url>:9991

Data

Datasets

This page describes commands to update various metadata associated with datasets in Lenses via the CLI.

Update the description of a dataset

To set a dataset description, use:

lenses-cli dataset update-description \
    --connection="connectionName" \
    --name="topicName" \
    --description="A Description"

It is also possible to remove the dataset description by supplying the --remove-description flag.

lenses-cli dataset remove-description \
    --connection="connectionName" \
    --name="topicName" \

Update the dataset tags

To set one or multiple dataset tags, use:

lenses-cli dataset update-tags \
    --connection="connectionName" \
    --name="topicName" \
    --tag="tag-1" \ 
    --tag="tag-2" \
    --tag="tag-3"

Notice that the --tags parameter is required and should contain a non-empty, space separated list of tags (a tag max length is 255 characters). To delete all the tags associated to a dataset, you can use the remove-tags subcommand:

lenses-cli dataset remove-tags \
    --connection="connectionName" \
    --name="topicName" \

Data Policies

This page describes command to manage data policies in Lenses via the CLI.

View data policies

View a specific policy

Create a policy

Fields are set by specifying a comma-separated set of fields.

From a file.

Example file:

Update a policy

Fields are set by specifying a comma-separated string to the field flag:

Delete a policy

Elasticsearch

This page describes commands to access the Elasticsearch indexes in Lenses via the CLI.

List all the available indexes

# list all elasticsearch indexes
lenses-cli elasticsearch-indexes

# or filter by connection and include system indexes
lenses-cli elasticsearch-indexes \
    --connection es-default \
    --include-system-indexes

View details for a supplied index and connection

lenses-cli elasticsearch-index \
    --connection es-default \
    --name my-index

Kafka topics

This page describes the commands to manage Kafka topics via the Lenses CLI.

View all topics

The optional names flag wil display only the names of the topics. If you use the –unwrap flag, each topic name will be displayed on a new line as text:

View a specific topic

Create topics

Update from a file:

Example file:

Update a topic’s configuration

Update from a file:

Example file:

Update a topic’s metadata configuration

Update from a file:

Example file:

Delete topics

Querying data

This page describes how to query data in Lenses via the CLI.

The CLI allows you to browse and subscribe to topic data with SQL. Please refer to the Lenses SQL documentation for more details.

Use these flags to control the data that’s returned:

keys print message keys
keys-only print the keys only, not the value of the message
meta print the message metadata, partition number and offsets
stats print query statistics as the last message

Browsing

Browse data via the query command, optionally validate and output stats:

Execute with stats and show offsets

Live

Live continuous queries update according to the query and never stop until terminated:

Interactive shell

The interactive shell allows you to have an interactive experience and issue Lenses SQL queries from the command line.

To start the interactive shell, issue the following command:

The available options are the following:

keys print message keys
keys-only print the keys only, not the value of the message
meta print the message metadata, partition number and offsets
stats print query statistics as the last message
pretty pretty print the JSON output
live-stream run as a continuous query
!options prints the current options

Options are set using the ! prefix
Queries are executed by ;

Multiline queries are possible but not available in the history inside the shell. After a restart, the multiline queries are concatenated into one line and can be selected.

Query history is stored in .lenses/history in the home directory of the user.

Admin

Alerts

This page describes the commands to manage Lenses Alerts via the CLI.

View all raised alert notifications

Copy

View all alert settings

Copy

View an alert setting

Enable or disable an alert setting

Enabling

Disabling

View all conditions that belong to a specific alert setting

Create or update an existing condition for a specific alert setting

Update an existing condition using flags

Create or update a condition from a file

Command line flags are ignored when loading from files.

Delete an existing condition for a specific alert

The value of the `--alert` flag is the Alert ID that the alert setting belongs to whereas the value of the `--condition` flag is the UUID of the condition to delete.

Alert Channels

This page describes managing alert channels in Lenses via the CLI.

List alert channels

View details of all alert channels

List alert channels with possible flags

Create a new alert channel

Create a new alert channel using a YAML file:

Update an alert channel

Update the settings of an alert channel from a file:

Delete an alert channel

Audit Channels

This page describes the commands to manage audit channels in Lenses via the CLI.

List audit channels

Basic list

Detailed view

List audit channels with possible flags

List audit channel templates

When creating audit channels, one of the existing predefined channel templates needs to be used.

Such a command lists all available audit channel templates:

To get all the details, use JSON format:

Create a new audit channel

Create a new audit channel using a YAML file:

Save the file as audit_chann.yml and execute it like this:Copy

Update an audit channel

Delete an audit channel

Connections

This page describes the commands to manage connections in Lenses via the CLI.

Viewing connections

lenses-cli connections

Viewing connection details

lenses-cli connections get --name connection-name

Creating a connection

lenses-cli connections create \
    --name SlackConnection \
    --template-name Slack \
    --tag slack \ --tag \connection --tag slack-connection \
    --connection-config '[{"key":"webhookUrl","value":"https://hooks.slack.com/"}]'

Consider using the provision for managing dynamic connections.

Update a connection

lenses-cli connections update --name SlackConnection \
    --tag slack --tag connection --tag slack-connection \
    --connection-config '[{"key":"webhookUrl","value":"https://hooks.slack.com/"}]'

Delete a connection

lenses-cli connections delete --name SlackConnection

Groups

This page describes the commands to manage Groups in Lenses via the CLI.

View groups

List the current user groups.

lenses-cli groups

View group

Get the details of a user group:

lenses-cli groups get --name Group1

Get the assigned data namespaces of a user group:

lenses-cli groups get --name Group1 --dataNamespaces

Create

Create a user group via the command line flags or file:

lenses-cli groups create \
    --name MyGroup \
    --description "My test group" \
    --applicationPermissions ViewKafkaConsumers \
    --applicationPermissions ManageKafkaConsumers \
    --applicationPermissions ViewConnectors \
    --connectClustersPermissions dev \
    --dataNamespaces '[{"wildcards":["*"],"permissions":["CreateTopic","DropTopic","ConfigureTopic","QueryTopic","ShowTopic","ViewSchema","InsertData","DeleteData","UpdateSchema"],"system":"Kafka","instance":"Dev"}]'

# from a file:
lenses-cli groups create ./group.yaml

Example file

name: MyGroup
description: "My test group"
applicationPermissions:
  - ViewKafkaConsumers
  - ManageKafkaConsumers
  - ViewConnectors
connectClustersPermissions:
  - dev
dataNamespaces:
  - wildcards:
      - "*"
    permissions:
      - CreateTopic
      - DropTopic
      - ConfigureTopic
      - QueryTopic
      - ShowTopic
      - ViewSchema
      - InsertData
      - DeleteData
      - UpdateSchema
    system: Kafka
    instance: Dev

Update

Update a user group:

lenses-cli groups update \
    --name MyGroup \
    --description "My test group" \
    --applicationPermissions ViewKafkaConsumers \
    --applicationPermissions ManageKafkaConsumers \
    --applicationPermissions ViewConnectors \
    --connectClustersPermissions dev \
    --dataNamespaces '[{"wildcards":["*"],"permissions":["CreateTopic","RequestTopicCreation", "DropTopic","ConfigureTopic","QueryTopic","ShowTopic","InsertData","DeleteData","UpdateSchema","ViewSchema"],"system":"Kafka","instance":"Dev"}]'

Delete

Delete a user group:

lenses-cli groups delete --name Group1

Clone

Clone a user group:

lenses-cli groups clone --name Group1 --cloneName Group1Cloned

Schemas

This page describes the commands to manage schemas in Lenses via the CLI.

View all schemas

View the current schemas registered:

lenses-cli schemas --unwrap [--output json/table/yaml [--query] [--no-pretty]]

The --unwrap flag is optional, prints only the names separated by new line

View a schema

Schemas can be filtered by name:

lenses-cli schema --name="sea_vessel_position_reports-value"

View schema versions

Schemas versions can be viewed filtered by name:

lenses-cli schema versions --name="sea_vessel_position_reports-value"
# view by version
lenses-cli schema --name="sea_vessel_position_reports-value" --version="latest"

View schema compatibility levels

Schema compatibility levels can be viewed and filtered by name:

lenses-cli schema compatibility --name="register1

Delete schemas

Delete all versions of a schema by name:

lenses-cli schema delete --name="coyote_test_02"

Delete schemas by version

Delete a specific version of a schema:

# (defaults to "latest" if not passed)
lenses-cli schema delete-version --name="register1" --version=2

Change the compatibility of a Schema

Change the compatibility level of a schema:

lenses-cli schema compatibility  set FULL --name="register1"

Register a schema

lenses-cli schema register \
  --name="schemaName" \
  --avro="{ \"type\": \"string\" }"

lenses-cli schema register ./schema.yaml

schema.yaml:

name: schemaName
avroSchema: |-
  {
    "type": "record",
    "name": "evolution",
    "namespace": "io.lenses",
    "doc": "This is a sample AVRO schema to get you started. Please edit",
    "fields": [
      {
       "name": "name",
       "type": "string"
      },
      {
       "name": "number1",
       "type": "int"
      },
      {
       "name": "number2",
       "type": "float"
      }
    ]
   }

Service Accounts

This page describes the commands to manage Service Accounts in Lenses via the CLI.

The CLI can create, update, delete and revoke Lenses service accounts.

Create a service account

# with flags (autogenerated password (token))
lenses-cli serviceaccounts create \
    --name MySvcAcc \
    --owner admin \
    --groups MyGroup

# with user-specified password (token) 
lenses-cli serviceaccounts create \
    --name MySvcAcc \
    --owner admin \
    --groups MyGroup \
    --token 1234-foo-bar

From a file:

lenses-cli serviceaccounts create ./serviceaccount.yaml

Example file:

name: MySvcAcc
owner: admin
groups:
- MyGroup

Update a service account

Update a service account via the command line flags or file:

# with flags
lenses-cli serviceaccounts update \
    --name MySvcAcc \
    --owner admin \
    --groups MyGroup

Delete a service account

lenses-cli serviceaccounts delete --name MySvcAcc

Revoke

lenses-cli serviceaccounts revoke --name MySvcAcc

Revoke a service account token and use your own custom token

lenses-cli serviceaccounts revoke \
    --name MySvcAcc \
    --token my-custom-token

License

This page describes the commands to manage a license in Lenses via the CLI.

View license info

lenses-cli license get

Update the license of a Lenses instance at runtime

lenses-cli license update --file <license.json>

Users

This page describes the commands to manage Users in Lenses via the CLI.

Create a user

lenses-cli users create \
    --username john \
    --password secretpass \
    --security basic \
    --groups MyGroup

# with a file:
lenses-cli users create ./user.yaml

Example file:

username: john
password: admin
security: basic
groups:
- MyGroup

Update a user

lenses-cli users update \
    --username john \
    --groups MyGroup

Delete a user

lenses-cli users delete --username john

Change the password of a user

lenses-cli users password \
    --username john \
    --secret mynewpassword

Find users by one group or many

lenses-cli users --groups MyGroup

Applications

Kafka Connectors

This page describes the commands to manage Connectors in Lenses via the CLI.

The CLI can create, modify, pause, restart and remove Apache Kafka Connect connectors.

Kafka Connect cluster permissions will be applied to the operation.

View connectors

List the currently deployed connectors, optionally filter by name, cluster and namespace:

lenses-cli connectors --cluster-name="dev" 

#[[--names [--unwrap]]

The --names flag displays only the names of the connectors. When used with --unwrap, it will print each connector name on a separate line in the output.

List all the supported connectors

lenses-cli connectors --supported

View the available plugins per cluster

lenses-cli connectors plugins --cluster-name="dev"

Create connectors

lenses-cli connector create \
    --cluster-name="dev" \
    --name="connectorName" \
    --configs="{\"key\": \"value\"}"

From a file:

lenses-cli connector create ./connector.yaml

Example file:

clusterName: development
config:
  connector.class: "org.apache.kafka.connect.file.FileStreamSinkConnector"
  file: "/dev/null"
  tasks.max: "1"
  topics: "reddit_posts"
  name: "nullsink"

Update a connector

# Inline
lenses-cli connector update --name=nullsink --cluster-name=dev \
  --configs="{\"connector.class\":\"org.apache.kafka.connect.file.FileStreamSinkConnector\",\"file\":\"/dev/null\",\"name\":\"nullsink\",\"tasks.max\":\"8\",\"topics\":\"reddit_posts\"}"

# or from file
lenses-cli connector update ./connector.yaml

View the configuration of a connector

lenses-cli connector config \
    --cluster-name="dev" \
    --name="connectorName"

View the status of a connector

lenses-cli connector status \
    --cluster-name="dev" \
    --name="connectorName"

Pause a connector

lenses-cli connector pause \
    --cluster-name="dev" \
    --name="connectorName"

Resume a connector

lenses-cli connector resume \
    --cluster-name="dev" \
    --name="connectorName"

SQL Processors

This page describes the commands to manage SQL Processors in Lenses via the CLI.

The CLI can create, modify, pause, restart and remove Lenses SQL Processors.

View processors

View all processors registered. Optionally filter by name, cluster and namespace.

Create processors

For IN_PROC you only need the name, for KUBERNETES all three. Creating an IN_PROC processor will start it automatically. This is not the case for Kubernetes deployments, a start command (see below) is required once created.

--id (the processor id, optional)
--name (optional)
--cluster-name (should be passed if KUBERNETES)
--namespace (should be passed if KUBERNETES)
--runners (number of times to deploy while in KUBERNETES)

From a file:

Stop a processor

For IN_PROC you only need the name, for KUBERNETES all three:

Start a processor

For IN_PROC you only need the name, for KUBERNETES all three:

Update the number of runners

Update a processor from the command line flags or files. Only for KUBERNETES:

From a file.

Delete a processor

Delete a processor by name, cluster and namespace. For IN_PROC you only need the name, and for KUBERNETES all 3 (name, cluster, namespace).

View processors logs

Available only for KUBERNETES.

Manage Kafka

ACLs

This page describes to commands to manage Kafka ACLs in Lenses via the CLI.

View all ACLs

Update ACLs

ACLs can also be created from a file:

Example file:

Delete ACLs

ACLs can also be deleted from a file:

Consumers

This page describes the commands for managing Kafka consumers in Lenses via the CLI.

Commands for updating consumer groups offsets.

Flags --group and --topic are mandatory and specify the consumer group ID and topic name respectively.

To affect all topics use the —all-topics flag.

Flag

Description

update-single-partition accepts only one --topic flag while update-multiple-partitions may accept multiple.

Update a single topic’s partition offset to the specified value

Update to the earliest offset available

Update to the latest offset available

Update given topics for all their partition offsets to the specified date time

Reset given topics for all their partition offsets to the earliest offset possible

Reset given topics for all their partition offsets to the latest offset possible

Reset offsets for all topics of this consumer group

Quotas

This page describes the commands for mamaging Kafka quotas in Lenses via the CLI.

View quotas

Create and update quotas for users

From a file.

File example:

Create and update quotas for clients

From a file.

Example file.

Remove user quota config’s specific properties

Delete the default user quota

Delete for a specific user quota

Delete for a specific user and client

Remove client quota config’s specific properties

If empty then all properties will be passed on automatically and the client quota will be removed entirely.

Delete the default client quota

Delete for a specific client quota

Delete for a specific client quota’s property

Golang client

This page describes using the Lenses Golang client.

Installation

Getting started

Configuration

Usage

All lenses-go#Client methods return a typed value based on the call and an error as second output to catch any errors coming from backend or client, forget panics.