Importing


The CLI allows you to import resources from files in Lenses into another environment.

Import is done on a per resource basis, the directory structure defined by the CLI whereas a base directory can be provided by the –dir flag.

Processors, connectors, topics and schemas have an additional prefix flag to restrict resources to export. The commands to import the configurations are shown in the table below.

CommandsDescription
aclsImoprt the ACL configuration
alert-channelsImoprt the alert channels configuration
alert-settingsImoprt the alert setting configuration
connectionsImoprt the connections configuration
connectorsImoprt the connectors configuration
processorsImoprt the processors created
quotasImoprt the quotas configuration in Kafka
schemasImoprt the schamas configuration
topicsImoprt the topics registered
policiesImoprt the policies configuration
groupsImoprt the groups created
serviceaccountsImoprt the service accounts created
topic-settingsImoprt the topics settings

Exporting Configurations 

Commands showing how to import the configurations:

lenses-cli import acls --dir my-dir
lenses-cli import alert-channels --dir my-dir
lenses-cli import alert-settings --dir my-dir
lenses-cli import connections --dir my-dir
lenses-cli import connectors --dir my-dir
lenses-cli import processors --dir my-dir
lenses-cli import quotas --dir my-dir
lenses-cli import schemas --dir my-dir
lenses-cli import topics --dir my-dir
lenses-cli import policies --dir my-dir
lenses-cli import groups --dir my-dir
lenses-cli import serviceaccounts --dir my-dir
lenses-cli import topic-settings --dir my-dir

The directory structure is:

my-dir
├── alert-settings
│   └── alert-setting.yaml
├── apps
│   ├── connectors
│   │   ├── connector-1.yaml
│   │   └── connector-2.yaml
│   └── sql
├── groups
│   └── groups.yaml
├── kafka
│   ├── quotas
│   │   └── quotas.yaml
│   └── topics
│       ├── topic-1.yaml
│       └── topic-2.yaml
├── policies
│   └── policies-city.yaml
├── service-accounts
│   └── servic-eaccounts.yaml
└── schemas
    ├── schema-1.yaml
    └── schema-2.yaml

AS-Code 

Lenses is on the journey to GitOps. The initial phase is to standardise the import and export of resources and to provide a way to manage resources in a declarative way. This is where AS-Code comes in. Since Lenses 5.5 the As-code feature is available for Kafka Connectors, the other resource will follow in the upcoming releases.

The general syntax for the AS-Code is:

lenses-cli apply resource.yaml

The resource.yaml file is a YAML file containing the resource definition. Specifically, for Kafka Connectors, the resource definition follows this structure:

apiVersion: lenses.io/v0beta
kind: KafkaConnector
spec:
  status: RUNNING
  cluster: a-cluster-name
  name: hdfs-sink-connector
  config:
    "connector.class": "io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector"
    "tasks.max": "10"
    "topics": "my-topic"
    "connect.s3.kcql": "insert into lensesio:demo select * from demo PARTITIONBY _value.metadata_id, _value.customer_id, _header.ts, _header.wallclock STOREAS `JSON` PROPERTIES ('flush.size'=1000000, 'flush.interval'=30, 'flush.count'=5000)"

In this structure:

The status field is optional and can be one of: RUNNING, STOPPED, or PAUSED. The cluster field specifies the name of the Connect cluster where the connector will be deployed. The name field denotes the name of the connector. The config field represents a map of the connector’s configuration.

--
Last modified: June 18, 2024