View the latest documentation 5.5
The Provisioning Feature empowers users to define and manage data connections using two distinct input mechanisms: API and Filesystem (FS). Both mechanisms are centered around the provisioning.yaml file descriptor, which represents the desired connections state. This document outlines the nuances of each method and provides guidance on their usage.
provisioning.yaml
All manual adjustments to both the connection settings and the product license will be overridden by whichever provisioning method is in use, be it the API or the FS approach. It’s crucial to keep your provisioning descriptor, license information, and related files up-to-date to avoid unexpected changes.
The API input mechanism is continuously available, offering a direct route to configure data connections and manage the product license.
Availability: Always ON. Cannot be disabled.
Data Input: Data is provisioned through multipart API requests. The multipart request differentiates between attached files and the provisioning.yaml descriptor using the part’s content type.
text/plain; charset=utf-8
application/octet-stream
Documentation: Detailed API endpoints, multipart request formats, and examples are provided on our API Documentation Site.
Endpoints:
Connections: PUT "api/v1/state/connections"
PUT "api/v1/state/connections"
validateOnly
false
true
validateConnectivity
License: PUT "api/v1/state/license"
PUT "api/v1/state/license"
FS provisioning is a method where data is ingested directly from specified directories in the filesystem. It provides a consistent, automated way of updating connections and the product license.
license.json
To activate the FS provisioning approach, ensure you configure the following:
lenses.provisioning.path:
lenses.provisioning.interval:
java.time.FiniteDuration
- configured-provisioning-folder │ ├── files/ │ ├── file1.txt │ ├── file2.txt ├── provisioning.yaml (To define desired state of connections) │ └── license.json (To manage the product's license data)
Key Points:
provisioning.yaml: Outlines the desired state of data connections, detailing services, protocols, and configurations.
license.json: Captures the product’s licensing information.
files/: This subfolder should house any necessary certificates or materialized files. Connection properties within the provisioning.yaml pointing to these files must accurately reference the appropriate file within this directory.
Both the API and FS mechanisms employ the provisioning.yaml descriptor to outline the desired state of data connections.
componentName: - name: uniqueName version: versionNumber tags: [ 'tag1', 'tag2', ... ] configuration: directValueProperty: value: propertyValue fileReferenceProperty: file: fileName referenceProperty: reference: anotherUniqueName ...
componentName: Represents the type or category of the connection or service being described ( e.g., kafka, confluentSchemaRegistry, connect). This serves as a general classifier.
kafka
confluentSchemaRegistry
connect
name: A unique identifier for the connection or service instance within its category. This ensures each connection is distinguishable from the others.
version: Represents the version number of the connection descriptor. This can help track changes or upgrades to the connection setup.
tags: An array of strings that serve as labels or markers for the connection. These tags can be used for categorization, filtering, or simply adding context (e.g., ‘dev’, ‘production’, ‘finance-team’).
configuration: This is where the specific settings and parameters for the connection are defined. It contains a mixture of direct values, file references, and references to other configurations:
propertyName: The name of the specific setting or parameter. Its structure can be one of three:
value: Directly provides the setting’s value. This can be a string, number, boolean, or even a list, depending on the requirement. Example: Setting a protocol (protocol: SSL) or providing a list of bootstrap servers.
protocol: SSL
file: Points to a specific file that contains data or information related to the connection. Typically, it refers to certificates, keys, or any other file needed to model the connection.
FS Input. The value here should match the file name stored under the files subdirectory of the provisioning folder. For instance, if the system requires a keystore for an SSL connection, this might point to keystore.jks in the files subdirectory.
files
keystore.jks
API Input, The value should match the part name in the multipart request. For instance, if the system requires a keystore for an SSL connection, the value for such property should be the name of the part in the multipart request that contains the keystore file.
kafka: configuration: sslKeystore: file: keystoreFileName
curl --location --request PUT "${LENSES_ENV}/api/v1/state/connections" \ --header "X-Kafka-Lenses-Token: ${LENSES_SESSION_TOKEN}" \ --header 'Content-Type: multipart/form-data' \ --header 'Content-Disposition: form-data;' \ --form "keystoreFileName=@${PATH_TO_KEYSTORE_FILE};type=application/octet-stream" \ --form 'provisioning=@"resources/provisioning.yaml";type=text/plain(utf-8)'
In the above example, we can see that the part name is keystoreFileName, which is the value of the kafka.configuration.sslKeystore.file yaml property.
keystoreFileName
kafka.configuration.sslKeystore.file
reference: This is a reference to another connection’s property. Instead of providing a direct value or file, it denotes that the value for this property should be fetched from another previously defined connection. The value here should be the name of the connection from which the property value will be inherited.
name
This structure allows for a clear and organized definition of connections and their configurations, making implementation by systems more streamlined. Moreover, by utilizing direct values, file references, and inter-connection references, it offers flexibility in how configurations are set up and managed.
For a comprehensive list of connection properties tailored to each supported connection type, we encourage users to consult our Connections API documentation. This resource provides in-depth details, ensuring you have all the information needed to optimally configure your connections. You can access the documentation directly via Connection Templates API Documentation.
kafka: - name: kafka-dev version: 1 tags: [ 'kafka', 'dev' ] configuration: kafkaBootstrapServers: value: - SSL://broker1:port protocol: value: SSL sslKeystore: file: keystore confluentSchemaRegistry: - name: schema-reg-dev version: 1 tags: [ 'dev' ] configuration: schemaRegistryUrls: value: - host1:port metricsPort: value: 9582 connect: - name: connect-cluster-dev version: 1 tags: [ 'dev' ] configuration: aes256Key: value: customPassword workers: value: - host:13001
Overall, this provisioning.yaml file is meant to set up and define configurations for a Kafka cluster, a Schema Registry (Confluent compatible API, AWS Glue), and a Kafka Connect cluster. The specific configurations like brokers, registry URLs, encryption keys, and others are specified in each component’s configuration.
kafka-dev
1
[ 'kafka', 'dev' ]
SSL://broker1:port
keystore
schema-reg-dev
host1:port
9582
connect-cluster-dev
host:13001
Always ensure the provisioning.yaml descriptor accurately represents the desired system state. Any connections not present in this file but present in the system will be dropped.
On this page