1 of 25

CLI

This page describes using the Lenses CLI.

Lenses offers a powerful CLI (command-line tool) built-in Go that utilizes the REST and WebSocket APIs of Lenses to communicate with Apache Kafka. It provides a straightforward way to perform common data engineering and site reliability engineering tasks.

Quick start

To use the CLI you must first configure the workspace:

lenses-cli configure

And fill in the details:

? Host: https://lenseshost:9991
? Auth token or username:
? Password: ******
? Save workspace name: dataopsDev

The configuration will be stored in ~/.lenses/config.yml

Controlling the output format

The —output flag, controls the format of the output.

The value of output can be Table, YAML or JSON. By default, the results of a command are printed as a table.

Additionally, when using the –output flag with JSON, two more optional flags are available: –pretty and –query. These flags can be passed to all commands that fetch and return JSON-formatted results.

—pretty - Enable the pretty format for JSON output of commands (default false).
—query - A jmespath query expression. This allows for querying the JSON output of commands. For more details view the JMESPATH documentation.

Usage with a service account

To use the CLI with a service account for CI/CD you need to pass these options:

lenses-cli topics \
  --token=<service-account-name>:<service-account-token> \
  --host=<lenses-url-host>

# Real Example
lenses-cli topics \
  --token=ci:58d86476-bcc6-47e2-a57e-0c6bbd9c88b9 \
  --host=http://<your-lenses-url>:9991

Data

Datasets

This page describes commands to update various metadata associated with datasets in Lenses via the CLI.

Update the description of a dataset

To set a dataset description, use:

lenses-cli dataset update-description \
    --connection="connectionName" \
    --name="topicName" \
    --description="A Description"

It is also possible to remove the dataset description by supplying the --remove-description flag.

lenses-cli dataset remove-description \
    --connection="connectionName" \
    --name="topicName" \

Update the dataset tags

To set one or multiple dataset tags, use:

lenses-cli dataset update-tags \
    --connection="connectionName" \
    --name="topicName" \
    --tag="tag-1" \ 
    --tag="tag-2" \
    --tag="tag-3"

Notice that the --tags parameter is required and should contain a non-empty, space separated list of tags (a tag max length is 255 characters). To delete all the tags associated to a dataset, you can use the remove-tags subcommand:

lenses-cli dataset remove-tags \
    --connection="connectionName" \
    --name="topicName" \

Data Policies

This page describes command to manage data policies in Lenses via the CLI.

View data policies

lenses-cli policies

View a specific policy

lenses-cli policy view --name my-policy

Create a policy

Fields are set by specifying a comma-separated set of fields.

lenses-cli policy create \
    --name my-policy \
    --category my-category \
    --impact HIGH \
    --redaction First-1 \
    --fields f1,f2,f3

From a file.

lenses-cli policy create --field f1,f2,f3 ./policy.yaml

Example file:

name: my-policy
category: my-category
impactType: HIGH
obfuscation: First-1

Update a policy

Fields are set by specifying a comma-separated string to the field flag:

lenses-cli policy update \
    --id my-policy-id \
    --name my-policy \
    --category my-category \
    --impact HIGH \
    --redaction First-1 \
    --fields f1,f2,f3

Delete a policy

lenses-cli policy delete --id my-policy-id

Elasticsearch

This page describes commands to access the Elasticsearch indexes in Lenses via the CLI.

List all the available indexes

# list all elasticsearch indexes
lenses-cli elasticsearch-indexes

# or filter by connection and include system indexes
lenses-cli elasticsearch-indexes \
    --connection es-default \
    --include-system-indexes

View details for a supplied index and connection

lenses-cli elasticsearch-index \
    --connection es-default \
    --name my-index

Kafka topics

This page describes the commands to manage Kafka topics via the Lenses CLI.

View all topics

The optional names flag wil display only the names of the topics. If you use the –unwrap flag, each topic name will be displayed on a new line as text:

lenses-cli topics [--names [--unwrap]]

View a specific topic

lenses-cli topic --name=reddit_posts

Create topics

lenses-cli topic create \
    --name="topicName" \
    --replication=1 \
    --partitions=1 \
    --configs="{\"max.message.bytes\": \"1000010\"}"

Update from a file:

lenses-cli topic create ./topic.yml

Example file:

name: topicName
replication: 1
partitions: 1
configs:
  max.message.bytes: "1000010"

Update a topic’s configuration

lenses-cli topic update \
    --name="topic2" \
    --configs="{\"max.message.bytes\": \"1000020\"}"

Update from a file:

lenses-cli topic update ./topic.yaml

Example file:

name: topicName
replication: 1
partitions: 1
configs:
  max.message.bytes: "1000010"

Update a topic’s metadata configuration

# first change the key/value type
lenses-cli topics metadata set \
    --name=topicName \
    --key-type=bytes \
    --value-type=xml

# now you may change its schema
lenses-cli topics metadata set \
    --name=topicName \
    --key-type=bytes \
    --key-schema="{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"keyField\",\"type\":\"string\"}]}" \
    --value-type=xml \
    --value-schema="{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"valueField\",\"type\":\"string\"}]}"

Update from a file:

lenses-cli topic update ./topic.yaml

Example file:

topicName: topicName
keyType: bytes
valueType: xml
valueSchema: "{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"heading3\",\"type\":\"string\"}]}"

Delete topics

lenses-cli topic delete --name="topic2"

Querying data

This page describes how to query data in Lenses via the CLI.

The CLI allows you to browse and subscribe to topic data with SQL. Please refer to the Lenses SQL documentation for more details.

Use these flags to control the data that’s returned:

keys print message keys
keys-only print the keys only, not the value of the message
meta print the message metadata, partition number and offsets
stats print query statistics as the last message

Browsing

Browse data via the query command, optionally validate and output stats:

lenses-cli query \
    --keys \
    --meta "SELECT score, author FROM reddit_posts LIMIT 50"

Execute with stats and show offsets

lenses-cli query \
    --stats "SELECT score, author FROM reddit_posts LIMIT 50"

Live

Live continuous queries update according to the query and never stop until terminated:

lenses-cli \
    query \
    --live-stream "SELECT score, author FROM reddit_posts WHERE _sample=2 AND _sampleWindow=200"

Interactive shell

The interactive shell allows you to have an interactive experience and issue Lenses SQL queries from the command line.

To start the interactive shell, issue the following command:

lenses-cli shell

The available options are the following:

keys print message keys
keys-only print the keys only, not the value of the message
meta print the message metadata, partition number and offsets
stats print query statistics as the last message
pretty pretty print the JSON output
live-stream run as a continuous query
!options prints the current options

Options are set using the ! prefix
Queries are executed by ;

Multiline queries are possible but not available in the history inside the shell. After a restart, the multiline queries are concatenated into one line and can be selected.

Query history is stored in .lenses/history in the home directory of the user.

    __                                 ________    ____
   / /   ___  ____  ________  _____   / ____/ /   /  _/
  / /   / _ \/ __ \/ ___/ _ \/ ___/  / /   / /    / /
 / /___/  __/ / / (__  )  __(__  )  / /___/ /____/ /
/_____/\___/_/ /_/____/\___/____/   \____/_____/___/
Docs at https://docs.lenses.io
Connected to [https://master.lenses.io:443] as [andrew], context [master]
Use "!" to set output options [!keys|!keysOnly|!stats|!meta|!pretty]
Crtl+D to exit

lenses-sql>

Admin

Alerts

This page describes the commands to manage Lenses Alerts via the CLI.

View all raised alert notifications

Copy

lenses-cli alerts

View all alert settings

Copy

lenses-cli alert settings

View an alert setting

lenses-cli alert setting --id=2000

Enable or disable an alert setting

Enabling

lenses-cli alert setting --id=1000 --enable

Disabling

lenses-cli alert setting --id=1000 --enable=false

View all conditions that belong to a specific alert setting

lenses-cli alert setting conditions --alert=2000

Create or update an existing condition for a specific alert setting

lenses-cli alert setting condition set \
--alert=2000 \
--condition="lag >= 200000 on group groupA and topic topicA" \
--channels="f9d6140c-4823-4157-a93c-5147aa0e2f29"

# For "Data Produced" type of alert rules
lenses-cli alert setting condition set \
    --alert=5000 \
    --topic=my-topic \
    --duration=PT6H \
    --more-than=5

# optional channels can be used more than once
# You can get the ID of a channel with `lenses-cli alertchannels`
lenses-cli alert setting condition set \
    --alert=5000 \
    --topic=my-topic \
    --duration=PT6H \
    --more-than=5 \
    --channels="9176c428-be0e-4c36-aa1e-8b8a66782232" \
    --channels="7b184b13-f37c-4eee-9c0a-17dec2fd7bf5"

Update an existing condition using flags

lenses-cli alert setting condition set --alert 2000 \
    --condition="lag >= 95 on group group-1 and topic topic-1" \
    --conditionID="5a6c3951-dfde-478e-8952-8193e62a9a8a" \
    --channels="143315dd-80bf-4833-a13a-394be06dda87" \
    --channels="f9d6140c-4823-4157-a93c-5147aa0e2f29"

# For "Data Produced" type of alert rules
# update - identify rule to update by its condition ID
lenses-cli alert settings --query="categories.producers" | jq .

lenses-cli alert setting condition set \
    --conditionID=f4a623e3-5287-4ca3-9d05-8f0ab68a9b4b \
    --alert=5000 \
    --topic=my-topic \
    --duration=PT6H \
    --more-than=5

Create or update a condition from a file

cat > alert_cond.yaml <<- "EOF"
alert: 2000
conditionID: "5a6c3951-dfde-478e-8952-8193e62a9a8a"
condition:
  "lag >= 30 on group schema-registry and topic backblaze_smart"
channels:
  - "143315dd-80bf-4833-a13a-394be06dda87"
  - "f9d6140c-4823-4157-a93c-5147aa0e2f29"
EOF

lenses-cli alert setting condition set alert_cond.yaml

# For "Data Produced" type of alert rules
cat > data_produced_alert.yaml <<- "EOF"
topic: "my-yaml-topic"
alert: 5000
more-than: 6990
duration: "PT6H"
EOF

lenses-cli alert setting condition set data_produced_alert.yml

Command line flags are ignored when loading from files.

Delete an existing condition for a specific alert

lenses-cli alert setting condition delete \
    --alert=2000 \
    --condition="7643d1ec-73f9-4913-95d1-87e62f1ec172"

# For "Data Produced" type of alert rules
alert setting condition delete \
    --alert 5000 \
    --condition=19e7791d-9fbf-4b6b-b198-8490e5e30d29

The value of the `--alert` flag is the Alert ID that the alert setting belongs to whereas the value of the `--condition` flag is the UUID of the condition to delete.

Alert Channels

This page describes managing alert channels in Lenses via the CLI.

List alert channels

lenses-cli alertchannels

View details of all alert channels

lenses-cli alertchannels --details

List alert channels with possible flags

lenses-cli alertchannels \
--page=1 \
--pageSize=10 \
--sortField="name" \
--sortOrder="asc" \
--templateName="test" \
--channelName="slack"

Create a new alert channel

lenses-cli alertchannels create \
--name "my-slack-channel" \
--connectionName="my-slack-connection" \
--templateName="Slack" \
--properties="[{\"key\":\"username\",\"value\":\"@johndoe\"},{\"key\":\"channel\",\"value\":\"company-slack-channel\"}]"

Create a new alert channel using a YAML file:

name: "My alert channel"
templateName: "Slack"
connectionName: "slackconn"
   properties:
   - key: "username"
     value: "George"
   - key: "channel"
     value: "#dev-ops"

lenses-cli alertchannels create alert_chann.yml

Update an alert channel

lenses-cli alert setting set \
    --id 1000 \
    --enable=false \
    --channels="143315dd-80bf-4833-a13a-394be06dd
    --channels="f9d6140c-4823-4157-a93c-5147aa0e2f29"

Update the settings of an alert channel from a file:

id: 1000
enable: false
channels:
- "f9d6140c-4823-4157-a93c-5147aa0e2f29"
- "143315dd-80bf-4833-a13a-394be06dda87"

lenses-cli alert setting set alert_sett.yaml

Delete an alert channel

lenses-cli alertchannels \
    delete channelID="f9d6140c-4823-4157-a93c-5147aa0e2f29"

Audit Channels

This page describes the commands to manage audit channels in Lenses via the CLI.

List audit channels

Basic list

lenses-cli auditchannels

Detailed view

lenses-cli auditchannels --details

List audit channels with possible flags

lenses-cli auditchannels \
--page=1 \
--pageSize=10 \
--sortField="name" \
--sortOrder="asc" \
--templateName="test" \
--channelName="slack"

List audit channel templates

When creating audit channels, one of the existing predefined channel templates needs to be used.

Such a command lists all available audit channel templates:

lenses-cli auditchannel-templates

To get all the details, use JSON format:

lenses-cli auditchannel-templates --output JSON

Create a new audit channel

lenses-cli auditchannels create \
  --name "my-audit-channel-1" \
  --connectionName="w1" \
  --templateName="Webhook" \
  --properties="[{\"key\":\"body\",\"value\":\"payload1-created\"},{\"key\":\"insecure\",\"value\":true},{\"key\":\"method\",\"value\":\"GET\"}]"

Create a new audit channel using a YAML file:

name: "my-audit-channel-2"
templateName: "Webhook"
connectionName: "w1"
properties:
  - key: "body"
    value: "body-payload"
  - key: "insecure"
    value: true
  - key: "method"
    value: "GET"

Save the file as audit_chann.yml and execute it like this:Copy

lenses-cli auditchannels create audit_chann.yml

Update an audit channel

lenses-cli auditchannels update \
  --id "ID_OF_THE_CHANNEL_TO_BE_UPDATED" \
  --name "my-audit-channel-2-updated" \
  --connectionName="w1" \
  --templateName="Webhook" \
  --properties="[{\"key\":\"body\",\"value\":\"payload1-created\"},{\"key\":\"insecure\",\"value\":true},{\"key\":\"method\",\"value\":\"GET\"}]"

Delete an audit channel

lenses-cli auditchannels delete \
    --channelID="701ff62c-8d14-4ef9-8dd0-fe2cd3c73779"

Connections

This page describes the commands to manage connections in Lenses via the CLI.

Viewing connections

lenses-cli connections

Viewing connection details

lenses-cli connections get --name connection-name

Creating a connection

lenses-cli connections create \
    --name SlackConnection \
    --template-name Slack \
    --tag slack \ --tag \connection --tag slack-connection \
    --connection-config '[{"key":"webhookUrl","value":"https://hooks.slack.com/"}]'

Consider using the provision for managing dynamic connections.

Update a connection

lenses-cli connections update --name SlackConnection \
    --tag slack --tag connection --tag slack-connection \
    --connection-config '[{"key":"webhookUrl","value":"https://hooks.slack.com/"}]'

Delete a connection

lenses-cli connections delete --name SlackConnection

Groups

This page describes the commands to manage Groups in Lenses via the CLI.

View groups

List the current user groups.

lenses-cli groups

View group

Get the details of a user group:

lenses-cli groups get --name Group1

Get the assigned data namespaces of a user group:

lenses-cli groups get --name Group1 --dataNamespaces

Create

Create a user group via the command line flags or file:

lenses-cli groups create \
    --name MyGroup \
    --description "My test group" \
    --applicationPermissions ViewKafkaConsumers \
    --applicationPermissions ManageKafkaConsumers \
    --applicationPermissions ViewConnectors \
    --connectClustersPermissions dev \
    --dataNamespaces '[{"wildcards":["*"],"permissions":["CreateTopic","DropTopic","ConfigureTopic","QueryTopic","ShowTopic","ViewSchema","InsertData","DeleteData","UpdateSchema"],"system":"Kafka","instance":"Dev"}]'

# from a file:
lenses-cli groups create ./group.yaml

Example file

name: MyGroup
description: "My test group"
applicationPermissions:
  - ViewKafkaConsumers
  - ManageKafkaConsumers
  - ViewConnectors
connectClustersPermissions:
  - dev
dataNamespaces:
  - wildcards:
      - "*"
    permissions:
      - CreateTopic
      - DropTopic
      - ConfigureTopic
      - QueryTopic
      - ShowTopic
      - ViewSchema
      - InsertData
      - DeleteData
      - UpdateSchema
    system: Kafka
    instance: Dev

Update

Update a user group:

lenses-cli groups update \
    --name MyGroup \
    --description "My test group" \
    --applicationPermissions ViewKafkaConsumers \
    --applicationPermissions ManageKafkaConsumers \
    --applicationPermissions ViewConnectors \
    --connectClustersPermissions dev \
    --dataNamespaces '[{"wildcards":["*"],"permissions":["CreateTopic","RequestTopicCreation", "DropTopic","ConfigureTopic","QueryTopic","ShowTopic","InsertData","DeleteData","UpdateSchema","ViewSchema"],"system":"Kafka","instance":"Dev"}]'

Delete

Delete a user group:

lenses-cli groups delete --name Group1

Clone

Clone a user group:

lenses-cli groups clone --name Group1 --cloneName Group1Cloned

Schemas

This page describes the commands to manage schemas in Lenses via the CLI.

View all schemas

View the current schemas registered:

lenses-cli schemas --unwrap [--output json/table/yaml [--query] [--no-pretty]]

The --unwrap flag is optional, prints only the names separated by new line

View a schema

Schemas can be filtered by name:

lenses-cli schema --name="sea_vessel_position_reports-value"

View schema versions

Schemas versions can be viewed filtered by name:

lenses-cli schema versions --name="sea_vessel_position_reports-value"
# view by version
lenses-cli schema --name="sea_vessel_position_reports-value" --version="latest"

View schema compatibility levels

Schema compatibility levels can be viewed and filtered by name:

lenses-cli schema compatibility --name="register1

Delete schemas

Delete all versions of a schema by name:

lenses-cli schema delete --name="coyote_test_02"

Delete schemas by version

Delete a specific version of a schema:

# (defaults to "latest" if not passed)
lenses-cli schema delete-version --name="register1" --version=2

Change the compatibility of a Schema

Change the compatibility level of a schema:

lenses-cli schema compatibility  set FULL --name="register1"

Register a schema

lenses-cli schema register \
  --name="schemaName" \
  --avro="{ \"type\": \"string\" }"

lenses-cli schema register ./schema.yaml

schema.yaml:

name: schemaName
avroSchema: |-
  {
    "type": "record",
    "name": "evolution",
    "namespace": "io.lenses",
    "doc": "This is a sample AVRO schema to get you started. Please edit",
    "fields": [
      {
       "name": "name",
       "type": "string"
      },
      {
       "name": "number1",
       "type": "int"
      },
      {
       "name": "number2",
       "type": "float"
      }
    ]
   }

Service Accounts

This page describes the commands to manage Service Accounts in Lenses via the CLI.

The CLI can create, update, delete and revoke Lenses service accounts.

Create a service account

# with flags (autogenerated password (token))
lenses-cli serviceaccounts create \
    --name MySvcAcc \
    --owner admin \
    --groups MyGroup

# with user-specified password (token) 
lenses-cli serviceaccounts create \
    --name MySvcAcc \
    --owner admin \
    --groups MyGroup \
    --token 1234-foo-bar

From a file:

lenses-cli serviceaccounts create ./serviceaccount.yaml

Example file:

name: MySvcAcc
owner: admin
groups:
- MyGroup

Update a service account

Update a service account via the command line flags or file:

# with flags
lenses-cli serviceaccounts update \
    --name MySvcAcc \
    --owner admin \
    --groups MyGroup

Delete a service account

lenses-cli serviceaccounts delete --name MySvcAcc

Revoke

lenses-cli serviceaccounts revoke --name MySvcAcc

Revoke a service account token and use your own custom token

lenses-cli serviceaccounts revoke \
    --name MySvcAcc \
    --token my-custom-token

License

This page describes the commands to manage a license in Lenses via the CLI.

View license info

lenses-cli license get

Update the license of a Lenses instance at runtime

lenses-cli license update --file <license.json>

Users

This page describes the commands to manage Users in Lenses via the CLI.

Create a user

lenses-cli users create \
    --username john \
    --password secretpass \
    --security basic \
    --groups MyGroup

# with a file:
lenses-cli users create ./user.yaml

Example file:

username: john
password: admin
security: basic
groups:
- MyGroup

Update a user

lenses-cli users update \
    --username john \
    --groups MyGroup

Delete a user

lenses-cli users delete --username john

Change the password of a user

lenses-cli users password \
    --username john \
    --secret mynewpassword

Find users by one group or many

lenses-cli users --groups MyGroup

Applications

Kafka Connectors

This page describes the commands to manage Connectors in Lenses via the CLI.

The CLI can create, modify, pause, restart and remove Apache Kafka Connect connectors.

Kafka Connect cluster permissions will be applied to the operation.

View connectors

List the currently deployed connectors, optionally filter by name, cluster and namespace:

lenses-cli connectors --cluster-name="dev" 

#[[--names [--unwrap]]

The --names flag displays only the names of the connectors. When used with --unwrap, it will print each connector name on a separate line in the output.

List all the supported connectors

lenses-cli connectors --supported

View the available plugins per cluster

lenses-cli connectors plugins --cluster-name="dev"

Create connectors

lenses-cli connector create \
    --cluster-name="dev" \
    --name="connectorName" \
    --configs="{\"key\": \"value\"}"

From a file:

lenses-cli connector create ./connector.yaml

Example file:

clusterName: development
config:
  connector.class: "org.apache.kafka.connect.file.FileStreamSinkConnector"
  file: "/dev/null"
  tasks.max: "1"
  topics: "reddit_posts"
  name: "nullsink"

Update a connector

# Inline
lenses-cli connector update --name=nullsink --cluster-name=dev \
  --configs="{\"connector.class\":\"org.apache.kafka.connect.file.FileStreamSinkConnector\",\"file\":\"/dev/null\",\"name\":\"nullsink\",\"tasks.max\":\"8\",\"topics\":\"reddit_posts\"}"

# or from file
lenses-cli connector update ./connector.yaml

View the configuration of a connector

lenses-cli connector config \
    --cluster-name="dev" \
    --name="connectorName"

View the status of a connector

lenses-cli connector status \
    --cluster-name="dev" \
    --name="connectorName"

Pause a connector

lenses-cli connector pause \
    --cluster-name="dev" \
    --name="connectorName"

Resume a connector

lenses-cli connector resume \
    --cluster-name="dev" \
    --name="connectorName"

SQL Processors

This page describes the commands to manage SQL Processors in Lenses via the CLI.

The CLI can create, modify, pause, restart and remove Lenses SQL Processors.

View processors

View all processors registered. Optionally filter by name, cluster and namespace.

lenses-cli processors
# filter by namespace
lenses-cli processors --namespace lenses-sql

# Please note the following columns in the table: 
# * 'ID' which should be used as the 'id' required for the commands below.
# * 'PROCESSORID' , which the user may have configured in creating the processor, and will be used as the consumer group of the processor.

Create processors

For IN_PROC you only need the name, for KUBERNETES all three. Creating an IN_PROC processor will start it automatically. This is not the case for Kubernetes deployments, a start command (see below) is required once created.

--id (the processor id, optional)
--name (optional)
--cluster-name (should be passed if KUBERNETES)
--namespace (should be passed if KUBERNETES)
--runners (number of times to deploy while in KUBERNETES)

lenses-cli processor create \
    --name="processorName" \
    --runners=1 \
    --cluster-name="clusterName" \
    --namespace="ns" \
    --pipeline="prod" \
    --sql="SET defaults.topic.autocreate=true;INSERT INTO topicB SELECT STREAM * FROM topic1"

From a file:

lenses-cli processor create ./processor.yaml

name: processorName
runners: 1
sql: |-
    SET defaults.topic.autocreate=true;INSERT INTO topicB SELECT STREAM * FROM topic1

Stop a processor

For IN_PROC you only need the name, for KUBERNETES all three:

lenses-cli processor stop \
    --cluster-name="clusterName?" \
    --namespace="namespace?" \
    --name="processorName"

Start a processor

For IN_PROC you only need the name, for KUBERNETES all three:

lenses-cli processor start \
    --cluster-name="clusterName?" \
    --namespace="namespace?" \
    --name="processorName"

Update the number of runners

Update a processor from the command line flags or files. Only for KUBERNETES:

lenses-cli processor update \
    --cluster-name="clusterName?" \
    --namespace="namespace" \
    --name="processorName" \
    --runners=2

From a file.

lenses-cli processor update ./processor.yaml

name: processorName
runners: 1
sql: |-
    SET defaults.topic.autocreate=true;INSERT INTO topicB SELECT STREAM * FROM topic1

Delete a processor

Delete a processor by name, cluster and namespace. For IN_PROC you only need the name, and for KUBERNETES all 3 (name, cluster, namespace).

lenses-cli processor delete \
    --namespace="namespace" \
    --cluster-name="clusterName" \
    --name="processorName

View processors logs

Available only for KUBERNETES.

lenses-cli processors logs \
    --cluster-name=clusterName \
    --namespace=nameSpace \
    --podName=runnerStateID [--follow --lines]

Manage Kafka

ACLs

This page describes to commands to manage Kafka ACLs in Lenses via the CLI.

View all ACLs

lenses-cli acls

Update ACLs

lenses-cli acl set \
    --resource-type="Topic" \
    --resource-name="transactions" \
    --principal="principalType:principalName" \
    --permission-type="Allow" \
    --acl-host="*" \
    --operation="Read" \
    --pattern-type="literal"

ACLs can also be created from a file:

lenses-cli acl create ./file.yaml

Example file:

resourceType: Topic
resourceName: transactions
principal: principalType:principalName
permissionType: Allow
host: "*"
operation: Read
patternType: literal

Delete ACLs

lenses-cli acl delete \
    --resource-type="Topic" \
    --resource-name="transactions" \
    --principal="principalType:principalName" \
    --permission-type="Allow" \
    --acl-host="*" \
    --operation="Read" \
    --pattern-type="literal"

ACLs can also be deleted from a file:

lenses-cli acl delete ./file.yaml

Consumers

This page describes the commands for managing Kafka consumers in Lenses via the CLI.

Commands for updating consumer groups offsets.

Flags --group and --topic are mandatory and specify the consumer group ID and topic name respectively.

To affect all topics use the —all-topics flag.

Flag

Description

-g, --group

Set the Consumer Group ID

-t, --topic

Set the topic name

--all-topics

Select implicitly all the topics

--topics

The topic to reset offsets for

--partition

Set the partition ID

--to-earliest

Reset partition offset to earliest offset

--to-latest

Reset partition offset to latest offset

--to-offset

Reset the partition offset to the provided value

update-single-partition accepts only one --topic flag while update-multiple-partitions may accept multiple.

Update a single topic’s partition offset to the specified value

lenses-cli consumers \
    offsets \
    update-single-partition \
    --group <group_name> \
    --topic <topic_name> \
    --partition <partition_id> \
    --to-offset <offset_id>

Update to the earliest offset available

lenses-cli consumers \
    offsets \
    update-single-partition \
    --group <group_name> \
    --topic <topic_name> \
    --partition <partition_id> \
    --to-earliest

Update to the latest offset available

lenses-cli consumers \
    offsets \
    update-single-partition \
    --group <group_name> \
    --topic <topic_name> \
    --partition <partition_id> \
    --to-latest

Update given topics for all their partition offsets to the specified date time

lenses-cli consumers \
    offsets \
    update-multiple-partitions \
    --group <group_name> \
    --topic <topic_name> \
    --to-datetime <datetime>

Reset given topics for all their partition offsets to the earliest offset possible

lenses-cli consumers \
    offsets \
    update-multiple-partitions \
    --group <group_name> \
    --topic <topic_name> \
    --to-earliest

Reset given topics for all their partition offsets to the latest offset possible

lenses-cli consumers \
    offsets \
    update-multiple-partitions \
    --group <group_name> \
    --topic <topic_name> \
    --to-latest

Reset offsets for all topics of this consumer group

lenses-cli consumers \
    offsets \
    update-multiple-partitions \
    --group <group_name> \
    --all-topics \
    --to-latest

Quotas

This page describes the commands for mamaging Kafka quotas in Lenses via the CLI.

View quotas

lenses-cli quotas [--output json/table/yaml]

Create and update quotas for users

lenses-cli quota users set [--quota-user="user"] [--quota-client=""] \
    --quota-config="{\"producer_byte_rate\": \"100000\",\"consumer_byte_rate\": \"200000\",\"request_percentage\": \"75\"}"

From a file.

lenses-cli quota users set ./quota.yaml

File example:

user: user
clientID: "*"
config:
  ProducerByteRate: "100000"
  ConsumerByteRate: "200000"
  RequestPercentage: "75"

Create and update quotas for clients

lenses-cli quota clients set [--quota-client=""] \
    --quota-config="{\"producer_byte_rate\": \"100000\",\"consumer_byte_rate\": \"200000\",\"request_percentage\": \"75\""

From a file.

lenses-cli quota clients set ./quota-clients.yaml

Example file.

clientID: "*"
config:
  ProducerByteRate: "100000"
  ConsumerByteRate: "200000"
  RequestPercentage: "75"

Remove user quota config’s specific properties

lenses-cli quota users delete [--quota-client=""] [--quota-user=""] \
    producer_byte_rate consumer_byte_rate request_percentage

Delete the default user quota

lenses-cli quota users delete

Delete for a specific user quota

lenses-cli quota users delete \
    --quota-user="user"

Delete for a specific user and client

lenses-cli quota users delete \
    --quota-user="user" \
    --quota-client="clientID"

lenses-cli quota users delete \
    --quota-user="user" \
    request_percentage

Remove client quota config’s specific properties

If empty then all properties will be passed on automatically and the client quota will be removed entirely.

lenses-cli quota clients delete [--quota-client=""] \
    producer_byte_rate consumer_byte_rate request_percentage

Delete the default client quota

lenses-cli quota clients delete

Delete for a specific client quota

lenses-cli quota clients delete \
    --quota-client="clientID"

Delete for a specific client quota’s property

lenses-cli quota clients delete \
    --quota-client="clientID" \
    request_percentage

Golang client

This page describes using the Lenses Golang client.

Installation

go get -u github.com/lensesio/lenses-go

Getting started

import "github.com/lensesio/lenses-go"

// Prepare authentication using raw Username and Password.
auth := lenses.BasicAuthentication{Username: "user", Password: "pass"}


#auth := lenses.KerberosAuthentication{
#    ConfFile: "/etc/krb5.conf",
#    Method:   lenses.KerberosWithPassword{
#        Realm: "my.realm or default if empty",
#        Username: "user",
#        Password: "pass",
#    },
#}

#auth := lenses.KerberosAuthentication{
#    ConfFile: "/etc/krb5.conf",
#    Method:   lenses.KerberosWithKeytab{KeytabFile: "/home/me/krb5_my_keytab.txt"},
#}

#auth := lenses.KerberosAuthentication{
#    ConfFile: "/etc/krb5.conf",
#    Method:   lenses.KerberosFromCCache{CCacheFile: "/tmp/krb5_my_cache_file.conf"},
#}

Configuration

// Prepare the client's configuration based on the host and the authentication above.
currentConfig := lenses.ClientConfig{Host: "domain.com", Authentication: auth, Timeout: "15s", Debug: true}

// Creating the client using the configuration.
client, err := lenses.OpenConnection(currentConfig)
if err != nil {
    // handle error.
}

Usage

All lenses-go#Client methods return a typed value based on the call and an error as second output to catch any errors coming from backend or client, forget panics.

topics, err := client.GetTopics()
if err != nil {
    // handle error.
}

// Print the length of the topics we've just received from our Lenses Box.
print(len(topics))