1 of 6

Data

Datasets

This page describes commands to update various metadata associated with datasets in Lenses via the CLI.

Update the description of a dataset

To set a dataset description, use:

It is also possible to remove the dataset description by supplying the --remove-description flag.

Update the dataset tags

To set one or multiple dataset tags, use:

Notice that the --tags parameter is required and should contain a non-empty, space separated list of tags (a tag max length is 255 characters). To delete all the tags associated to a dataset, you can use the remove-tags subcommand:

Data Policies

This page describes command to manage data policies in Lenses via the CLI.

View data policies

lenses-cli policies

View a specific policy

lenses-cli policy view --name my-policy

Create a policy

Fields are set by specifying a comma-separated set of fields.

lenses-cli policy create \
    --name my-policy \
    --category my-category \
    --impact HIGH \
    --redaction First-1 \
    --fields f1,f2,f3

From a file.

lenses-cli policy create --field f1,f2,f3 ./policy.yaml

Example file:

name: my-policy
category: my-category
impactType: HIGH
obfuscation: First-1

Update a policy

Fields are set by specifying a comma-separated string to the field flag:

lenses-cli policy update \
    --id my-policy-id \
    --name my-policy \
    --category my-category \
    --impact HIGH \
    --redaction First-1 \
    --fields f1,f2,f3

Delete a policy

lenses-cli policy delete --id my-policy-id

Elasticsearch

This page describes commands to access the Elasticsearch indexes in Lenses via the CLI.

List all the available indexes

# list all elasticsearch indexes
lenses-cli elasticsearch-indexes

# or filter by connection and include system indexes
lenses-cli elasticsearch-indexes \
    --connection es-default \
    --include-system-indexes

View details for a supplied index and connection

lenses-cli elasticsearch-index \
    --connection es-default \
    --name my-index

Kafka topics

This page describes the commands to manage Kafka topics via the Lenses CLI.

View all topics

The optional names flag wil display only the names of the topics. If you use the –unwrap flag, each topic name will be displayed on a new line as text:

lenses-cli topics [--names [--unwrap]]

View a specific topic

lenses-cli topic --name=reddit_posts

Create topics

lenses-cli topic create \
    --name="topicName" \
    --replication=1 \
    --partitions=1 \
    --configs="{\"max.message.bytes\": \"1000010\"}"

Update from a file:

lenses-cli topic create ./topic.yml

Example file:

name: topicName
replication: 1
partitions: 1
configs:
  max.message.bytes: "1000010"

Update a topic’s configuration

lenses-cli topic update \
    --name="topic2" \
    --configs="{\"max.message.bytes\": \"1000020\"}"

Update from a file:

lenses-cli topic update ./topic.yaml

Example file:

name: topicName
replication: 1
partitions: 1
configs:
  max.message.bytes: "1000010"

Update a topic’s metadata configuration

# first change the key/value type
lenses-cli topics metadata set \
    --name=topicName \
    --key-type=bytes \
    --value-type=xml

# now you may change its schema
lenses-cli topics metadata set \
    --name=topicName \
    --key-type=bytes \
    --key-schema="{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"keyField\",\"type\":\"string\"}]}" \
    --value-type=xml \
    --value-schema="{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"valueField\",\"type\":\"string\"}]}"

Update from a file:

lenses-cli topic update ./topic.yaml

Example file:

topicName: topicName
keyType: bytes
valueType: xml
valueSchema: "{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"heading3\",\"type\":\"string\"}]}"

Delete topics

lenses-cli topic delete --name="topic2"

Querying data

This page describes how to query data in Lenses via the CLI.

The CLI allows you to browse and subscribe to topic data with SQL. Please refer to the Lenses SQL documentation for more details.

Use these flags to control the data that’s returned:

keys print message keys
keys-only print the keys only, not the value of the message
meta print the message metadata, partition number and offsets
stats print query statistics as the last message

Browsing

Browse data via the query command, optionally validate and output stats:

lenses-cli query \
    --keys \
    --meta "SELECT score, author FROM reddit_posts LIMIT 50"

Execute with stats and show offsets

lenses-cli query \
    --stats "SELECT score, author FROM reddit_posts LIMIT 50"

Live

Live continuous queries update according to the query and never stop until terminated:

lenses-cli \
    query \
    --live-stream "SELECT score, author FROM reddit_posts WHERE _sample=2 AND _sampleWindow=200"

Interactive shell

The interactive shell allows you to have an interactive experience and issue Lenses SQL queries from the command line.

To start the interactive shell, issue the following command:

lenses-cli shell

The available options are the following:

keys print message keys
keys-only print the keys only, not the value of the message
meta print the message metadata, partition number and offsets
stats print query statistics as the last message
pretty pretty print the JSON output
live-stream run as a continuous query
!options prints the current options

Options are set using the ! prefix
Queries are executed by ;

Multiline queries are possible but not available in the history inside the shell. After a restart, the multiline queries are concatenated into one line and can be selected.

Query history is stored in .lenses/history in the home directory of the user.

    __                                 ________    ____
   / /   ___  ____  ________  _____   / ____/ /   /  _/
  / /   / _ \/ __ \/ ___/ _ \/ ___/  / /   / /    / /
 / /___/  __/ / / (__  )  __(__  )  / /___/ /____/ /
/_____/\___/_/ /_/____/\___/____/   \____/_____/___/
Docs at https://docs.lenses.io
Connected to [https://master.lenses.io:443] as [andrew], context [master]
Use "!" to set output options [!keys|!keysOnly|!stats|!meta|!pretty]
Crtl+D to exit

lenses-sql>

Kafka topics

This page describes the commands to manage Kafka topics via the Lenses CLI.

View all topics

The optional names flag wil display only the names of the topics. If you use the –unwrap flag, each topic name will be displayed on a new line as text:

lenses-cli topics [--names [--unwrap]]

View a specific topic

lenses-cli topic --name=reddit_posts

Create topics

lenses-cli topic create \
    --name="topicName" \
    --replication=1 \
    --partitions=1 \
    --configs="{\"max.message.bytes\": \"1000010\"}"

Update from a file:

lenses-cli topic create ./topic.yml

Example file:

name: topicName
replication: 1
partitions: 1
configs:
  max.message.bytes: "1000010"

Update a topic’s configuration

lenses-cli topic update \
    --name="topic2" \
    --configs="{\"max.message.bytes\": \"1000020\"}"

Update from a file:

lenses-cli topic update ./topic.yaml

Example file:

name: topicName
replication: 1
partitions: 1
configs:
  max.message.bytes: "1000010"

Update a topic’s metadata configuration

# first change the key/value type
lenses-cli topics metadata set \
    --name=topicName \
    --key-type=bytes \
    --value-type=xml

# now you may change its schema
lenses-cli topics metadata set \
    --name=topicName \
    --key-type=bytes \
    --key-schema="{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"keyField\",\"type\":\"string\"}]}" \
    --value-type=xml \
    --value-schema="{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"valueField\",\"type\":\"string\"}]}"

Update from a file:

lenses-cli topic update ./topic.yaml

Example file:

topicName: topicName
keyType: bytes
valueType: xml
valueSchema: "{\"type\":\"record\",\"name\":\"lenses_record\",\"namespace\":\"lenses\",\"fields\":[{\"name\":\"heading3\",\"type\":\"string\"}]}"

Delete topics

lenses-cli topic delete --name="topic2"

Querying data

This page describes how to query data in Lenses via the CLI.

The CLI allows you to browse and subscribe to topic data with SQL. Please refer to the Lenses SQL documentation for more details.

Use these flags to control the data that’s returned:

keys print message keys
keys-only print the keys only, not the value of the message
meta print the message metadata, partition number and offsets
stats print query statistics as the last message

Browsing

Browse data via the query command, optionally validate and output stats:

lenses-cli query \
    --keys \
    --meta "SELECT score, author FROM reddit_posts LIMIT 50"

Execute with stats and show offsets

lenses-cli query \
    --stats "SELECT score, author FROM reddit_posts LIMIT 50"

Live

Live continuous queries update according to the query and never stop until terminated:

lenses-cli \
    query \
    --live-stream "SELECT score, author FROM reddit_posts WHERE _sample=2 AND _sampleWindow=200"

Interactive shell

The interactive shell allows you to have an interactive experience and issue Lenses SQL queries from the command line.

To start the interactive shell, issue the following command:

lenses-cli shell

The available options are the following:

keys print message keys
keys-only print the keys only, not the value of the message
meta print the message metadata, partition number and offsets
stats print query statistics as the last message
pretty pretty print the JSON output
live-stream run as a continuous query
!options prints the current options

Options are set using the ! prefix
Queries are executed by ;

Multiline queries are possible but not available in the history inside the shell. After a restart, the multiline queries are concatenated into one line and can be selected.

Query history is stored in .lenses/history in the home directory of the user.

    __                                 ________    ____
   / /   ___  ____  ________  _____   / ____/ /   /  _/
  / /   / _ \/ __ \/ ___/ _ \/ ___/  / /   / /    / /
 / /___/  __/ / / (__  )  __(__  )  / /___/ /____/ /
/_____/\___/_/ /_/____/\___/____/   \____/_____/___/
Docs at https://docs.lenses.io
Connected to [https://master.lenses.io:443] as [andrew], context [master]
Use "!" to set output options [!keys|!keysOnly|!stats|!meta|!pretty]
Crtl+D to exit

lenses-sql>