Secret Provider

Secret provider for Kafka to provide indirect look up of configuration values.

Secret Provider 0.0.1

Adding to your project

<dependency>
    <groupId>io.lenses</groupId>
    <artifactId>secret-provider</artifactId>
    <version>0.0.1</version>
</dependency>
libraryDependencies += "io.lenses" % "secret-provider" % "0.0.1"
compile 'io.lenses:secret-provider:0.0.1'

Overview

External secret providers allow for indirect references to be placed in an applications configuration, so for example, that secrets are not exposed in the Worker API endpoints of Kafka Connect.

For Connect, the providers are configured in the properties files of the worker and loaded at startup. Each provider has different configuration properties.

The config provider jars, for Vault and AWS SecretManager must be placed in the plugin path for the worker as set in the worker.properties file.

Note

Azure must be added to the classpath of the Connect Worker.

plugin.path=/usr/share/connectors,/opt/stream-reactor

To configure the provider update the worker.properties file to include required configurations. The layout of the properties is:

# Properties specified in the Worker config
config.providers=vault   # can have multiple comma-separated values
config.providers.vault.class=io.lenses.connect.secrets.providers.VaultSecretProvider
config.providers.vault.param.addr=https://localhost
config.providers.vault.param.auth.method=token
config.providers.vault.param.token=my-token

# Properties specified in the Connector config
mysql.db.password=${vault:vault_path:vault_db_password_key}
Name Description
config.providers A comma-separated list of names for providers .e.g. config.providers=file,vault
config.providers.{name}.class The Java class name for a provider.
config.providers.{name}.param.{param-name} A parameter to be passed to the above Java class on initialization.
config.reload.action none or restart

Multiple providers can be specified via the config.providers options, for either all connectors or different instances of the same type.

Outside Connect

Other applications, outside of Kafka Connect can also make use of this provider. The providers can be passed a properties file at start up to retrieve secrets.

val props = Map(
    AWSProviderConfig.AUTH_METHOD -> AuthMode.CREDENTIALS.toString,
    AWSProviderConfig.AWS_ACCESS_KEY -> "somekey",
    AWSProviderConfig.AWS_SECRET_KEY -> "secretkey",
    AWSProviderConfig.AWS_REGION -> "someregion"
).asJava

val config = AWSProviderConfig(props)
val settings = AWSProviderSettings(config)

val data = provider.get(secretName, Set("my-app-secret-key").asJava)

Azure KeyVault

This provider allows for fetching secret values from Azure KeyVault. Two authentication methods are support:

  1. credentails. When using this mode the client-id, tenant-id and secret-id for an Azure service principal that has access to keyvaults must be provided
  2. default. This method uses the default credential provider chain from Azure. The default credential first checks environment variables for configuration. If environment configuration is incomplete, it will try managed identity.

Configuration Options

Name Description Default
azure.auth.method Azure authenticate method. ‘credentials’ to use the provided credentials or ‘default’ for the standard Azure provider chain credentials
azure.client.id Azure client id for the service principal. Valid is auth.method is ‘credentials’  
azure.tenant.id Azure tenant id for the service principal. Valid is auth.method is ‘credentials’  
azure.secret.id Azure secret id for the service principal. Valid is auth.method is ‘credentials’  
file.dir The base location for any files to stored  

Example Worker Properties

config.providers=azure
config.providers.azure.class=io.lenses.connect.secrets.providers.AzureSecretProvider
config.providers.azure.param.azure.auth.method=credentials
config.providers.azure.param.azure.client.id=your-client-id
config.providers.azure.param.azure.secret.id=your-secret-id
config.providers.azure.param.azure.tenant.id=your-tenant-id
config.providers.azure.param.file.dir=/connector-files/azure

Install

Warning

Azure KeyVault SDK uses a service loader to find the HTTP Client. This uses the default system class loader. Connect loads the plugins with classloader isolation. This means the service loader for Azure cannot find the HTTP class. If you use the standard connect plugins directory for this connector you will hit this error:

java.lang.IllegalStateException: Cannot find any HttpClient provider on the classpath - unable to create a default HttpClient instance
at com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:35)

To install the AzureSecretProvider you must be set it in the classpath of the Connect Worker before you start the worker.

export CLASSPATH=/some-dir/secret-provider-X.X.X-all.jar

Usage

To use this provider in a connector, reference the keyvault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

  • provider is the name of the provider in the worker property file set above
  • path is the URL of the Azure KeyVault. DO NOT provide the https:// protocol for the in the keyvault name as the Connect worker will not parse it correctly
  • key is the name of the secret key in the Azure KeyVault

For example, if we store two secrets:

  1. my_username with the value lenses and
  2. my_password with the value my-secret-password

in a Keyvault called my-azure-key-vault we would set

name=my-sink
class=my-class
topics=mytopic
username=${azure:my-azure-key-vault.vault.azure.net:my_username}
password=${azure:my-azure-key-vault.vault.azure.net:my_password}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data Encoding

The provider handles the following types:

  • utf_8
  • base64

The provider will look for a tag attached to the secret called file-encoding. The value for this tag can be:

  • UTF8
  • UTF_FILE
  • BASE64
  • BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents as are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk, set this as the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If no tag is found the contents of the secret string are returned.

AWS Secret Manager

This provider allows for fetching secret values from AWS SecretManager. Two authentication methods are support:

  1. credentails. When using this mode the access-key and secret-key are used.
  2. default. This method uses the default credential provider chain from AWS. The default credential first checks environment variables for configuration. If environment configuration is incomplete, Java props, then profile file and finally it will try managed identity.

Configuration Options

Name Description Default
aws.auth.method AWS authenticate method. ‘credentials’ to use the provided credentials or ‘default’ for the standard AWS provider chain credentials
aws.client.key AWS client key. Valid is auth.method is ‘credentials’  
aws.secret.key AWS secret key. Valid is auth.method is ‘credentials’  
aws.region AWS region the for the Secrets manager  
file.dir The base location for any files to stored  

Example Worker Properties

config.providers=aws
config.providers.aws.class=io.lenses.connect.secrets.providers.AWSSecretProvider
config.providers.aws.param.aws.auth.method=credentials
config.providers.aws.param.aws.client.key=your-client-key
config.providers.aws.param.aws.secret.key=your-secret-key
config.providers.aws.param.aws.region=your-region
config.providers.aws.param.file.dir=/connector-files/aws

Usage

To use this provider in a connector, reference the SecretManager containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

  • provider is the name of the provider in the worker property file set above
  • path is the name of the secret
  • key is the name of the secret key in secret to retrieve. AWS can store multiple keys under a path.

For example, if we store two secrets as keys:

  1. my_username_key with the value lenses and
  2. my_password_key with the value my-secret-password

in a secret called my-aws-secret we would set

name=my-sink
class=my-class
topics=mytopic
username=${aws:my-aws-secret:my_username_key}
password=${aws:my-aws-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data Encoding

AWS SecretManager BinaryString (API only), is not supported. The secrets must be stored under the secret name in key, value pair format. The provider checks the SecretString API and expects a json string to be returned.

For example for an RDS Postgre secret, the following is returned by AWS Secret Manager:

{
"username": "xxx",
"password": "xxx",
"engine": "postgres",
"host": "xxx",
"port": 5432,
"dbname": "xxx",
"dbInstanceIdentifier": "xxxx"
}

The provider handles the following types:

  • utf_8
  • base64

The provider will look for keys prefixed with:

  • UTF8
  • UTF_FILE
  • BASE64
  • BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk, set this as the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.

Hashicorp Vault

This provider allows for fetching secret values from Vault. Multiple authentication methods are support:

  • approle
  • userpass
  • kubernetes
  • cert
  • token
  • ldap
  • gcp
  • awsiam
  • jwt
  • github

Configuration Options

Name Description Default
file.dir The base location for any files to stored  
vault.auth.method
Available values are approle
, userpass, kubernetes, cert,
token, ldap, gcp, awsiam, jwt,
github, token
 
vault.addr Address of the Vault server localhost
vault.token Vault app role token. vault.auth.method must be ‘token’  
vault.namespace
Sets a global namespace to
the Vault server instance.
Required Vault Enterprize Pro
 
vault.pem File containing the Vault Server certificate string contents  
vault.client.pem File containing the Client certificate string contents  
vault.engine.version KV Secrets Engine version of the Vault server instance. Defaults to 2 2
vault.ssl.truststore.location The location of the trust store file  
vault.ssl.keystore.location The location of the key store file  
vault.ssl.keystore.password The password the key store file  
app.role.id Vault App role id. vault.auth.method must be ‘approle’ or ‘kubernetes’  
app.role.secret.id Vault App role name secret id. vault.auth.method must be ‘approle’  
username Username to connect to Vault with. vault.auth.method must be ‘userpass’  
password Password to connect to Vault with. vault.auth.method must be ‘userpass’  
mount
The mount name of the userpass authentication back end.
vault.auth.method must be userpas’
userpass
kubernetes.role
The kubernetes role used
for authentication.
vault.auth.method must be ‘kubernetes’
 
kubernetes.token.path
Path to the service account token.
vault.auth.method must be kubernetes’
/var/run/secrets/kubernetes.io/serviceaccount/token
aws.role
Name of the role against which the login
is being attempted. If role is not specified,
then the login endpoint looks for a role bearing
the name of the AMI ID of the EC2 instance that
is trying to login if using the ec2 auth method,
or the friendly name (i.e., role name or username)
of the IAM principal authenticated.
vault.auth.method must be awsiam
 
aws.request.url
PKCS7 signature of the identity document
with all n characters removed.Base64-encoded
HTTP URL used in the signed request.
(base64-encoding of https://sts.amazonaws.com/)
as most requests will probably use POST with
an empty URI. vault.auth.method must be awsiam
 
aws.request.headers Request headers. vault.auth.method must be ‘awsiam’  
aws.request.body
Base64-encoded body of the signed request.
The base64 encoding of Action=GetCallerIdentity&amp;Version=2011-06-15.
vault.auth.method must be awsiam
 
aws.mount AWS auth mount. vault.auth.method must be awsiam aws
ldap.username LDAP username to connect to Vault with. vault.auth.method must be ‘ldap’  
ldap.password LDAP password to connect to Vault with. vault.auth.method must be ‘ldap’  
mount
The mount name of the ldap authentication back end.
vault.auth.method must be ldap
ldap
jwt.role Role the JWT token belongs to. vault.auth.method must be jwt  
jwt.provider Provider of JWT token. vault.auth.method must be jwt  
jwt JWT token. vault.auth.method must be jwt  
gcp.role The gcp role used for authentication. vault.auth.method must be gcp  
gcp.jwt JWT token. vault.auth.method must be gcp  
cert.mount
The mount name of the cert authentication back end.

|vault.auth.method must be cert

cert
github.token
The github app-id used for authentication.
vault.auth.method must be github
 
github.mount
The mount name of the github authentication back end.
vault.auth.method must be github
github

Example Worker Properties

config.providers.vault.class=io.lenses.connect.secrets.providers.VaultSecretProvider
config.providers.vault.param.addr=https://localhost
config.providers.vault.param.auth.method=token
config.providers.vault.param.token=my-token
config.providers.vault.param.file.dir=/connector-files/vault

Usage

To use this provider in a connector, reference the Hashicorp Vault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

  • provider is the name of the provider in the worker property file set above
  • path is the path of the secret in Hashicorp Vault
  • key is the name of the secret key in secret to retrieve. Vault can store multiple keys under a path.

For example, if we store two secrets as keys:

  1. my_username_key with the value lenses and
  2. my_password_key with the value my-secret-password

in a secret called secret/my-vault-secret we would set

name=my-sink
class=my-class
topics=mytopic
username=${vault:secret/my-vault-secret:my_username_key}
password=${vault:secret/my-vault-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

The provider handles the following types:

  • utf_8
  • base64

The provider will look for keys prefixed with:

  • UTF8
  • UTF_FILE
  • BASE64
  • BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk set, this as the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.

Environment Provider

This provider allows for fetching secret values from the environment and is intended for kubernetes where the value for the environment variables is provider by Kubernetes Secrets and not exposed in the Kubernetes manifests.

Example Worker Properties

config.providers=env
config.providers.env.class=io.lenses.connect.secrets.providers.ENVSecretProvider
config.providers.env.file.dir=my-secret-dir

Usage

To use this provider in a connector, reference the ENVSecretProvider environment variable providing the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

  • provider is the name of the provider in the worker property file set above
  • path is ignore and should be a blank string
  • key is the name of the environment variable holding the secret.

For example, if we store two secrets as environment variables:

  1. MY_ENV_VAR_USERNAME with the value lenses and
  2. MY_ENV_VAR_PASSWORD with the value my-secret-password

we would set

name=my-sink
class=my-class
topics=mytopic
username=${env::MY_ENV_VAR_USERNAME}
password=${env::MY_ENV_VAR_PASSWORD}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data Encoding

This provider inspects the value of the environment to determine how to process the value. The value can optionally provider value metadata to support base64 decoding and writing values to files.

To provide metadata the following patterns are expected:

where value is the actual payload and metadata can be one of the following:

  • ENV-base64 - the provider will attempt to base64 decode the value string
  • ENV-mounted-base64 - the provider will attempt to base64 decode the value string and write to a file
  • ENV-mounted - the provider will write the value to a file

if no metadata is found the value of the environment variable is return.

examples: