Add the plugin to the worker classloader isolation via the plugin.path option:

Secret Rotation

To allow for secret rotation add config.action.reload to your Connect workers properties files.

This property accepts one of two options:

1. none - No action happens at a connector failure (e.g. it can no longer connect to an external system)

2. restart - The work will schedule a restart of the connector

Secure secrets in AWS Secret Manager and use them in Kafka Connect.

Authentication

Two authentication methods are supported:

1. credentails. When using this configuration the access-key and secret-key are used.

2. default. This method uses the default credential provider chain from AWS. The default credential first checks environment variables for configuration. If the environment configuration is incomplete, Java props, then the profile file and finally it will try managed identity.

Configuring the plugin

Example Worker Properties

config.providers=aws
config.providers.aws.class=io.lenses.connect.secrets.providers.AWSSecretProvider
config.providers.aws.param.aws.auth.method=credentials
config.providers.aws.param.aws.access.key=your-client-key
config.providers.aws.param.aws.secret.key=your-secret-key
config.providers.aws.param.aws.region=your-region
config.providers.aws.param.file.dir=/connector-files/aws

Usage

To use this provider in a connector, reference the SecretManager containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

• provider is the name of the provider in the worker property file set above

• path is the name of the secret

• key is the name of the secret key in secret to retrieve. AWS can store multiple keys under a path.

For example, if we store two secrets as keys:

• my_username_key with the value lenses and

• my_password_key with the value my-secret-password

in a secret called my-aws-secret we would set:

name=my-sink
class=my-class
topics=mytopic
username=${aws:my-aws-secret:my_username_key}
password=${aws:my-aws-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data encoding

AWS SecretManager BinaryString (API only), is not supported. The secrets must be stored under the secret name in key, value pair format. The provider checks the SecretString API and expects a JSON string to be returned.

For example for an RDS Postgre secret, the following is returned by AWS Secret Manager:

{
  "username": "xxx",
  "password": "xxx",
  "engine": "postgres",
  "host": "xxx",
  "port": 5432,
  "dbname": "xxx",
  "dbInstanceIdentifier": "xxxx"
}

The provider handles the following types:

• utf_8

• base64

The provider will look for keys prefixed with:

• UTF8

• UTF_FILE

• BASE64

• BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk, set the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.

Secure secrets in Azure KeyVault and use them in Kafka Connect.

Authentication

Two authentication methods are supported:

1. credentials. When using this configuration the client-id, tenant-id and secret-id for an Azure service principal that has access to key vaults must be provided

2. default. This method uses the default credential provider chain from Azure. The default credential first checks environment variables for configuration. If the environment configuration is incomplete, it will try to use managed identities.

Configuring the plugin

Example worker properties file:

config.providers=azure
config.providers.azure.class=io.lenses.connect.secrets.providers.AzureSecretProvider
config.providers.azure.param.azure.auth.method=credentials
config.providers.azure.param.azure.client.id=your-client-id
config.providers.azure.param.azure.secret.id=your-secret-id
config.providers.azure.param.azure.tenant.id=your-tenant-id
config.providers.azure.param.file.dir=/connector-files/azure

Usage

To use this provider in a connector, reference the keyvault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

• provider is the name of the provider in the worker property file set above

• path is the URL of the Azure KeyVault. DO NOT provide the https:// protocol for the in the keyvault name as the Connect worker will not parse it correctly

• key is the name of the secret key in the Azure KeyVault

For example, if we store two secrets:

• my_username with the value lenses and

• my_password with the value my-secret-password

in a Keyvault called my-azure-key-vault we would set:

name=my-sink
class=my-class
topics=mytopic
username=${azure:my-azure-key-vault.vault.azure.net:my_username}
password=${azure:my-azure-key-vault.vault.azure.net:my_password}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data encoding

The provider handles the following types:

• utf_8

• base64

The provider will look for a tag attached to the secret called file-encoding. The value for this tag can be:

• UTF8

• UTF_FILE

• BASE64

• BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents as are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk, set the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If no tag is found the contents of the secret string are returned.

Add the plugin to the worker classloader isolation via the plugin.path option:

The provider gets AES-256 encrypted value as a key and simply decrypts it to get the value (instead of e.g. looking up for the value somewhere).

The AES-256 encryption used for the value needs to be prefixed with base64 encoded initialisation vector and a space character, the encrypted value is also base64 encoded. So to corretly encrypt value1 I need to follow following steps:

• encrypted-bytes = aes-256 encrypted value1

• encrypted-base64 = base64 encrypted-bytes

• initialisation-vector = random bytes

• iv-base64 = base64 initialisation-vector

• encrypted-value = iv-base64 + + encrypted-base64

Configuring the plugin

The plugin needs to be configured with secret key that will be used for decoding. The key is a string and needs to have size of 32 bytes (UTF-8 encoded).

Example worker properties file:

Usage

To use this provider in a connector, reference the keyvault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

• provider is the name of the provider in the worker property file set above

• path used to provide encoding of the value: utf8, utf8_file, base64, base64_file

• key is the AES-256 encrypted value to be decrypted by the plugin

For example, if hello aes-256 encrypted using some key equals to xyxyxy - then if I configure connector to use ${aes256::xyxyxy} for a parameter value, the value should be substituted with “hello” string:

This would resolve at runtime to:

path belonging to key reference is used to specify encoding used to pass the value. The provider supports following encodings:

• base64: base-64 encoding of the textual value

• base64_file: base-64 encoding of the value that when decrypted should be stored in the file

• utf8_file: utf-8 encoding of the value that when decrypted should be stored in the file

• utf8: utf-8 encoding of textual value

The UTF8 means the value returned is the decrypted value of the encrypted value (key). The BASE64 means the value returned is the base64 decoded decrypted value of the encrypted value (key).

If the value for the encoding is UTF8_FILE the string contents are written to a file. The name of the file will be randomply generated. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the value for the encoding is BASE64_FILE the string contents are based64 decoded and written to a file. The name of the file will be randomply generated. For example, if a connector needs a PEM file on disk, set this as the path as BASE64_FILE. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the key reference path is not set or is set to unknown value - utf8 encoding is used as default.

For example, if we want to save hi there ! to the file, and aes-256 encrypted content equals xyxyxy - then if I configure connector to use ${aes256:utf8_file:xyxyxy} for a parameter value, the provider will create new file with random name (abc-def-ghi) and store hi there ! to the file. If configured store directory is /store-root, he value will be substituted with /store-root/secrets/abc-def-ghi string:

resolves to

Use Environment variables to hold secrets and use them in Kafka Connect.

Configuration

Example Worker Properties:

config.providers=env
config.providers.env.class=io.lenses.connect.secrets.providers.ENVSecretProvider
config.providers.env.param.file.dir=my-secret-dir

Usage

To use this provider in a connector, reference the ENVSecretProvider environment variable providing the value of the connector property.

The indirect reference is in the form ${provider::key} where:

• provider is the name of the provider in the worker property file set above

• key is the name of the environment variable holding the secret.

For example, if we store two secrets as environment variables:

• MY_ENV_VAR_USERNAME with the value lenses and

• MY_ENV_VAR_PASSWORD with the value my-secret-password

we would set:

name=my-sink
class=my-class
topics=mytopic
username=${env::MY_ENV_VAR_USERNAME}
password=${env::MY_ENV_VAR_PASSWORD}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data encoding

This provider inspects the value of the environment to determine how to process the value. The value can optionally provide value metadata to support base64 decoding and writing values to files.

To provide metadata the following patterns are expected:

where value is the actual payload and metadata can be one of the following:

• ENV-base64 - the provider will attempt to base64 decode the value string

• ENV-mounted-base64 - the provider will attempt to base64 decode the value string and write to a file

• ENV-mounted - the provider will write the value to a file

if no metadata is found the value of the environment variable is returned.

Secure secrets in Hashicorp Vault and use them in Kafka Connect.

Authentication

Multiple authentication methods are supported:

• approle

• userpass

• kubernetes

• cert

• token

• ldap

• gcp

• awsiam

• jwt

• github

Configuring the plugin

Example Worker Properties

config.providers=vault
config.providers.vault.class=io.lenses.connect.secrets.providers.VaultSecretProvider
config.providers.vault.param.vault.addr=http://localhost:8200
config.providers.vault.param.vault.auth.method=token
config.providers.vault.param.vault.token=my-token
config.providers.vault.param.file.dir=/connector-files/vault

Usage

To use this provider in a connector, reference the Hashicorp Vault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

• provider is the name of the provider in the worker property file set above

• path is the path of the secret in Hashicorp Vault

• key is the name of the secret key in secret to retrieve. Vault can store multiple keys under a path.

For example, if we store two secrets as keys:

• my_username_key with the value lenses and

• my_password_key with the value my-secret-password

in a secret called secret/my-vault-secret we would set:

name=my-sink
class=my-class
topics=mytopic
username=${vault:secret/my-vault-secret:my_username_key}
password=${vault:secret/my-vault-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data encoding

The provider handles the following types:

• utf_8

• base64

The provider will look for keys prefixed with:

• UTF8

• UTF_FILE

• BASE64

• BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk set the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.