Secret Provider¶

Secret provider for Kafka to provide indirect look up of configuration values.

Adding to your project¶

<dependency>
    <groupId>io.lenses</groupId>
    <artifactId>secret-provider</artifactId>
    <version>LATEST</version>
</dependency>

libraryDependencies += "io.lenses" % "secret-provider" % "latest"

compile 'io.lenses:secret-provider:latest'

Overview¶

External secret providers allow for indirect references to be placed in an applications configuration, so for example, that secrets are not exposed in the Worker API endpoints of Kafka Connect.

For Connect, the providers are configured in the properties files of the worker and loaded at startup. Each provider has different configuration properties.

The config provider jars, for Vault and AWS SecretManager must be placed in the plugin path for the worker as set in the worker.properties file.

Note

Azure must be added to the classpath of the Connect Worker.

plugin.path=/usr/share/connectors,/opt/stream-reactor

To configure the provider update the worker.properties file to include required configurations. The layout of the properties is:

# Properties specified in the Worker config
config.providers=vault   # can have multiple comma-separated values
config.providers.vault.class=io.lenses.connect.secrets.providers.VaultSecretProvider
config.providers.vault.param.addr=https://localhost
config.providers.vault.param.auth.method=token
config.providers.vault.param.token=my-token

# Properties specified in the Connector config
mysql.db.password=${vault:vault_path:vault_db_password_key}

Name	Description
config.providers	A comma-separated list of names for providers .e.g. config.providers=file,vault
config.providers.{name}.class	The Java class name for a provider.
config.providers.{name}.param.{param-name}	A parameter to be passed to the above Java class on initialization.
config.reload.action	none or restart

Multiple providers can be specified via the config.providers options, for either all connectors or different instances of the same type.

Outside Connect¶

Other applications, outside of Kafka Connect can also make use of this provider. The providers can be passed a properties file at start up to retrieve secrets.

val props = Map(
    AWSProviderConfig.AUTH_METHOD -> AuthMode.CREDENTIALS.toString,
    AWSProviderConfig.AWS_ACCESS_KEY -> "somekey",
    AWSProviderConfig.AWS_SECRET_KEY -> "secretkey",
    AWSProviderConfig.AWS_REGION -> "someregion"
).asJava

val config = AWSProviderConfig(props)
val settings = AWSProviderSettings(config)

val data = provider.get(secretName, Set("my-app-secret-key").asJava)

Azure KeyVault¶

This provider allows for fetching secret values from Azure KeyVault. Two authentication methods are support:

credentails. When using this mode the client-id, tenant-id and secret-id for an Azure service principal that has access to keyvaults must be provided
default. This method uses the default credential provider chain from Azure. The default credential first checks environment variables for configuration. If environment configuration is incomplete, it will try managed identity.

Configuration Options¶

Name	Description	Default
azure.auth.method	Azure authenticate method. ‘credentials’ to use the provided credentials or ‘default’ for the standard Azure provider chain	credentials
azure.client.id	Azure client id for the service principal. Valid is auth.method is ‘credentials’
azure.tenant.id	Azure tenant id for the service principal. Valid is auth.method is ‘credentials’
azure.secret.id	Azure secret id for the service principal. Valid is auth.method is ‘credentials’
file.dir	The base location for any files to stored

Example Worker Properties

config.providers=azure
config.providers.azure.class=io.lenses.connect.secrets.providers.AzureSecretProvider
config.providers.azure.param.azure.auth.method=credentials
config.providers.azure.param.azure.client.id=your-client-id
config.providers.azure.param.azure.secret.id=your-secret-id
config.providers.azure.param.azure.tenant.id=your-tenant-id
config.providers.azure.param.file.dir=/connector-files/azure

Install

Warning

Azure KeyVault SDK uses a service loader to find the HTTP Client. This uses the default system class loader. Connect loads the plugins with classloader isolation. This means the service loader for Azure cannot find the HTTP class. If you use the standard connect plugins directory for this connector you will hit this error:

java.lang.IllegalStateException: Cannot find any HttpClient provider on the classpath - unable to create a default HttpClient instance
at com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:35)

To install the AzureSecretProvider you must be set it in the classpath of the Connect Worker before you start the worker.

export CLASSPATH=/some-dir/secret-provider-X.X.X-all.jar

Usage¶

To use this provider in a connector, reference the keyvault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

provider is the name of the provider in the worker property file set above
path is the URL of the Azure KeyVault. DO NOT provide the https:// protocol for the in the keyvault name as the Connect worker will not parse it correctly
key is the name of the secret key in the Azure KeyVault

For example, if we store two secrets:

my_username with the value lenses and
my_password with the value my-secret-password

in a Keyvault called my-azure-key-vault we would set

name=my-sink
class=my-class
topics=mytopic
username=${azure:my-azure-key-vault.vault.azure.net:my_username}
password=${azure:my-azure-key-vault.vault.azure.net:my_password}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data Encoding

The provider handles the following types:

utf_8
base64

The provider will look for a tag attached to the secret called file-encoding. The value for this tag can be:

UTF8
UTF_FILE
BASE64
BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents as are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk, set this as the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If no tag is found the contents of the secret string are returned.

AWS Secret Manager¶

This provider allows for fetching secret values from AWS SecretManager. Two authentication methods are support:

credentails. When using this mode the access-key and secret-key are used.
default. This method uses the default credential provider chain from AWS. The default credential first checks environment variables for configuration. If environment configuration is incomplete, Java props, then profile file and finally it will try managed identity.

Configuration Options¶

Name	Description	Default
aws.auth.method	AWS authenticate method. ‘credentials’ to use the provided credentials or ‘default’ for the standard AWS provider chain	credentials
aws.client.key	AWS client key. Valid is auth.method is ‘credentials’
aws.secret.key	AWS secret key. Valid is auth.method is ‘credentials’
aws.region	AWS region the for the Secrets manager
file.dir	The base location for any files to stored

Example Worker Properties

config.providers=aws
config.providers.aws.class=io.lenses.connect.secrets.providers.AWSSecretProvider
config.providers.aws.param.aws.auth.method=credentials
config.providers.aws.param.aws.client.key=your-client-key
config.providers.aws.param.aws.secret.key=your-secret-key
config.providers.aws.param.aws.region=your-region
config.providers.aws.param.file.dir=/connector-files/aws

Usage¶

To use this provider in a connector, reference the SecretManager containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

provider is the name of the provider in the worker property file set above
path is the name of the secret
key is the name of the secret key in secret to retrieve. AWS can store multiple keys under a path.

For example, if we store two secrets as keys:

my_username_key with the value lenses and
my_password_key with the value my-secret-password

in a secret called my-aws-secret we would set

name=my-sink
class=my-class
topics=mytopic
username=${aws:my-aws-secret:my_username_key}
password=${aws:my-aws-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data Encoding

AWS SecretManager BinaryString (API only), is not supported. The secrets must be stored under the secret name in key, value pair format. The provider checks the SecretString API and expects a json string to be returned.

For example for an RDS Postgre secret, the following is returned by AWS Secret Manager:

{
"username": "xxx",
"password": "xxx",
"engine": "postgres",
"host": "xxx",
"port": 5432,
"dbname": "xxx",
"dbInstanceIdentifier": "xxxx"
}

The provider handles the following types:

utf_8
base64

The provider will look for keys prefixed with:

UTF8
UTF_FILE
BASE64
BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk, set this as the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.

Hashicorp Vault¶

This provider allows for fetching secret values from Vault. Multiple authentication methods are support:

approle
userpass
kubernetes
cert
token
ldap
gcp
awsiam
jwt
github

Configuration Options¶

Name	Description	Default
file.dir	The base location for any files to stored
vault.auth.method	Available values are approle , userpass, kubernetes, cert, token, ldap, gcp, awsiam, jwt, github, token
vault.addr	Address of the Vault server	localhost
vault.token	Vault app role token. vault.auth.method must be ‘token’
vault.namespace	Sets a global namespace to the Vault server instance. Required Vault Enterprize Pro
vault.pem	File containing the Vault Server certificate string contents
vault.client.pem	File containing the Client certificate string contents
vault.engine.version	KV Secrets Engine version of the Vault server instance. Defaults to 2	2
vault.ssl.truststore.location	The location of the trust store file
vault.ssl.keystore.location	The location of the key store file
vault.ssl.keystore.password	The password the key store file
app.role.id	Vault App role id. vault.auth.method must be ‘approle’ or ‘kubernetes’
app.role.secret.id	Vault App role name secret id. vault.auth.method must be ‘approle’
username	Username to connect to Vault with. vault.auth.method must be ‘userpass’
password	Password to connect to Vault with. vault.auth.method must be ‘userpass’
mount	The mount name of the userpass authentication back end. vault.auth.method must be userpas’	userpass
kubernetes.role	The kubernetes role used for authentication. vault.auth.method must be ‘kubernetes’
kubernetes.token.path	Path to the service account token. vault.auth.method must be kubernetes’	/var/run/secrets/kubernetes.io/serviceaccount/token
aws.role	Name of the role against which the login is being attempted. If role is not specified, then the login endpoint looks for a role bearing the name of the AMI ID of the EC2 instance that is trying to login if using the ec2 auth method, or the friendly name (i.e., role name or username) of the IAM principal authenticated. vault.auth.method must be awsiam
aws.request.url	PKCS7 signature of the identity document with all n characters removed.Base64-encoded HTTP URL used in the signed request. (base64-encoding of https://sts.amazonaws.com/) as most requests will probably use POST with an empty URI. vault.auth.method must be awsiam
aws.request.headers	Request headers. vault.auth.method must be ‘awsiam’
aws.request.body	Base64-encoded body of the signed request. The base64 encoding of Action=GetCallerIdentity&Version=2011-06-15. vault.auth.method must be awsiam
aws.mount	AWS auth mount. vault.auth.method must be awsiam	aws
ldap.username	LDAP username to connect to Vault with. vault.auth.method must be ‘ldap’
ldap.password	LDAP password to connect to Vault with. vault.auth.method must be ‘ldap’
mount	The mount name of the ldap authentication back end. vault.auth.method must be ldap	ldap
jwt.role	Role the JWT token belongs to. vault.auth.method must be jwt
jwt.provider	Provider of JWT token. vault.auth.method must be jwt
jwt	JWT token. vault.auth.method must be jwt
gcp.role	The gcp role used for authentication. vault.auth.method must be gcp
gcp.jwt	JWT token. vault.auth.method must be gcp
cert.mount	The mount name of the cert authentication back end. \|vault.auth.method must be cert	cert
github.token	The github app-id used for authentication. vault.auth.method must be github
github.mount	The mount name of the github authentication back end. vault.auth.method must be github	github

Example Worker Properties

config.providers.vault.class=io.lenses.connect.secrets.providers.VaultSecretProvider
config.providers.vault.param.addr=https://localhost
config.providers.vault.param.auth.method=token
config.providers.vault.param.token=my-token
config.providers.vault.param.file.dir=/connector-files/vault

Usage¶

To use this provider in a connector, reference the Hashicorp Vault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

provider is the name of the provider in the worker property file set above
path is the path of the secret in Hashicorp Vault
key is the name of the secret key in secret to retrieve. Vault can store multiple keys under a path.

For example, if we store two secrets as keys:

my_username_key with the value lenses and
my_password_key with the value my-secret-password

in a secret called secret/my-vault-secret we would set

name=my-sink
class=my-class
topics=mytopic
username=${vault:secret/my-vault-secret:my_username_key}
password=${vault:secret/my-vault-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

The provider handles the following types:

utf_8
base64

The provider will look for keys prefixed with:

UTF8
UTF_FILE
BASE64
BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk set, this as the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.

Environment Provider¶

This provider allows for fetching secret values from the environment and is intended for kubernetes where the value for the environment variables is provider by Kubernetes Secrets and not exposed in the Kubernetes manifests.

Example Worker Properties

config.providers=env
config.providers.env.class=io.lenses.connect.secrets.providers.ENVSecretProvider
config.providers.env.file.dir=my-secret-dir

Usage¶

To use this provider in a connector, reference the ENVSecretProvider environment variable providing the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

provider is the name of the provider in the worker property file set above
path is ignore and should be a blank string
key is the name of the environment variable holding the secret.

For example, if we store two secrets as environment variables:

MY_ENV_VAR_USERNAME with the value lenses and
MY_ENV_VAR_PASSWORD with the value my-secret-password

we would set

name=my-sink
class=my-class
topics=mytopic
username=${env::MY_ENV_VAR_USERNAME}
password=${env::MY_ENV_VAR_PASSWORD}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data Encoding

This provider inspects the value of the environment to determine how to process the value. The value can optionally provider value metadata to support base64 decoding and writing values to files.

To provide metadata the following patterns are expected:

where value is the actual payload and metadata can be one of the following:

ENV-base64 - the provider will attempt to base64 decode the value string
ENV-mounted-base64 - the provider will attempt to base64 decode the value string and write to a file
ENV-mounted - the provider will write the value to a file

if no metadata is found the value of the environment variable is return.

examples: