Construct provision.yaml

To provision Lenses you need to construct provision.yaml, a file in YAML format describing the resources you want Lenses to connect to. This file will be provided to Lenses CLI to communicate the appropriate state to Lenses API.

Currently provision.yaml support license/connections and uses the following high-level structure:

license:
  fileRef:
    # one of the following options
    # inline: ...
    # filePath: ...
    # URL: ...

connections:
  [CONNECTION NAME]:
    templateName: [CONNECTION TEMPLATE]
    configurationObject:
      ...
  [CONNECTION NAME]:
    templateName: [CONNECTION TEMPLATE]
    configurationObject:
      ...

License 

To set Lenses license use this format:

license:
  fileRef:
    # Read more for files below
    inline: '[LICENSE FILE CONTENTS]'

A few notes:

Connections 

Under this root key you can describe the connection details towards multiple core services. For more information see The whole list of services

To set a connection use this format:

connections:
  [CONNECTION NAME]:
    templateName: [CONNECTION TEMPLATE]
    tags: []  # Comma separated list of strings, is required even as an empty array `[]`
    configurationObject:
      ...

A few notes:

  • CONNECTION NAME:
    • should be unique
    • is bound to specific value on some connection types (use Constant connection name if specified in connection’s page)
    • should follow domain name naming conventions
  • configurationObject is specific to the connection type, actual values can be found in Dynamic configuration pages.
  • Learn more about fileRef.
  • To provision Lenses successfully the only required connection is towards the Kafka brokers.

Here’s an example with three connections:

connections:
  kerberos: # Constant connection name
    templateName: Kerberos
    tags: [ "production",  "cluster-1" ]
    configurationObject:
      kerberosKrb5:
        fileRef:
          inline: ...
  kafka: # Constant connection name
    templateName: Kafka
    tags: [ "production",  "cluster-1" ]
    configurationObject:
      protocol: SASL_SSL
      kafkaBootstrapServers:
      - SASL_SSL://my-cluster-1:9096
      - SASL_SSL://my-cluster-2:9096
      - SASL_SSL://my-cluster-3:9096
      keytab:
        fileRef:
          inline: ...
      saslJaasConfig: |
        com.sun.security.auth.module.Krb5LoginModule required
        ...        
      saslMechanism: GSSAPI
      sslTruststore:
        fileRef:
          inline: ...
      sslTruststorePassword: ...
  custom-connect-cluster: #
    templateName: KafkaConnect
    tags: [ "production",  "cluster-1", "lenses-processors" ]
    configurationObject:
      workers:
      - https://my-custom-connect-cluster:8083
      metricsPort: 8084
      metricsSsl: true
      metricsType: JMX

Connections with Dependencies 

Some connections depend on others. One example is the AWS Glue Schema Registry connection, which depends on an AWS connection. This is an example to provision Lenses with an AWS connection named my-aws-connection and an AWS Glue Schema Registry that references it:

connections:
  schema-registry:
    templateName: AWSGlueSchemaRegistry
    tags: []
    configurationObject:
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:123123123:registry/my-registry
  my-aws-connection:
    templateName: AWS
    tags: []
    configurationObject:
      accessKeyId: ExampleAccessKey
      secretAccessKey: ExampleSecretAccessKey
      region: my-region

Files 

fileRef is a specific key used to reference a file using one of these supported options:

  • inline: the raw file contents either raw or in base64 (lenses-cli will decode)
  • URL: url accessible at runtime
  • filepath: a filesystem reference pointing to a file
--
Last modified: April 24, 2024