Hashicorp Vault

This page describes how to retrieve secrets from Hashicorp Vault for use in Kafka Connect.

Secure secrets in Hashicorp Vault and use them in Kafka Connect.

Secrets will only be reloaded if the Connector restarts.

From Version 2.2.0, the secret provider does not write secrets to files by default. If you require this behaviour (for trust stores, key stores or certs) you can enable this by adding the property file.write=true.

Authentication

Multiple authentication methods are supported:

• approle

• userpass

• kubernetes

• cert

• token

• ldap

• gcp

• awsiam

• jwt

• github

Configuring the plugin

Name	Description
file.dir	The base location for any files to be stored
file.write	Writes secrets to file on path. Required for Java trust stores, key stores, certs that need to be loaded from file. For ease of use for the secret provider, this is disabled by default.
vault.auth.method	Available values are approle, userpass, kubernetes, cert, token, ldap, gcp, awsiam, jwt, github, token
vault.addr	Address of the Vault server
vault.token	Use when ‘vault.auth.method’ is ‘token’ to specify the token value
vault.namespace	Set a global namespace to the Vault server instance. Requires Vault Enterprise Pro
vault.pem	File containing the Vault Server certificate content as string
vault.client.pem	File containing the Client certificate string content as string
vault.engine.version	KV Secrets Engine version of the Vault server instance. Default is 2
vault.ssl.truststore.location	The location of the trust store file
vault.ssl.keystore.location	The location of the key store file
vault.ssl.keystore.password	The password for the key store file
secret.default.ttl	If no TTL is configured in Vault, apply a default TTL.
app.role.id	Use when vault.auth.method is approle or kubernetes to specify the Vault App role id
app.role.secret.id	Use when vault.auth.method is approle tp specify the Vault App role name secret id
app.role.path	Use when vault.auth.method is approle to specify the Vault App role path
username	Use when vault.auth.method is userpass to specify the username to connect to Vault
password	Use when vault.auth.method is userpass to specify the password to connect to Vault
mount	Use when vault.auth.method is userpass to specify the mount name of the userpass authentication back end
ldap.username	Use when vault.auth.method is ldap to specify the LDAP username to connect to Vault with
ldap.password	Use when vault.auth.method is ldap to specify the LDAP password to connect to Vault with
mount	Use when vault.auth.method is ldap to specify the mount name of the ldap authentication back end
jwt.role	Use when vault.auth.method is jwt to specify the role the JWT token belongs to
jwt.provider	Use when vault.auth.method is jwt to specify the provider of the JWT token
jwt	Use when vault.auth.method is jwt to specify the JWT token
gcp.role	Use when vault.auth.method is gcp to specify the gcp role used for authentication
gcp.jwt	Use when vault.auth.method is gcp to specify the JWT token
cert.mount	Use when vault.auth.method is cert to specify the mount name of the cert authentication back end
github.token	Use when vault.auth.method is github to specify the github app-id to use for authentication
github.mount	Use when vault.auth.method is github to specify the mount name of the github authentication back end
kubernetes.role	Use when vault.auth.method is kubernetes to specify the kubernetes role for authentication
kubernetes.token.path	Use when vault.auth.method is kubernetes to specify the path to the service account token . Default is /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes.auth.path	Use when vault.auth.method is kubernetes to specify a custom mount path
aws.role	Use when vault.auth.method is awsiam. Name of the role to login. If role is not specified, the login endpoint uses the role bearing the name of the AMI ID of the EC2 instance or if using the ec2 auth method the friendly name (i.e., role name or username) of the IAM authenticated principal
aws.request.url	Use when vault.auth.method is awsiam. PKCS7 signature of the identity document with all n characters removed. Base64-encoded HTTP URL used in the signed request (i.e. base64-encoding of https://sts.amazonaws.com) as most requests will probably use POST with an empty URI
aws.request.body	Use when vault.auth.method is awsiam. Base64-encoded body of the signed request i.e. base64 of Action=GetCallerIdentity&Version=2011-06-15
aws.request.headers	Use when vault.auth.method is awsiam to specify any request headers
aws.mount	Use when vault.auth.method is awsiam. The AWS auth mount. Default is “aws”

Example Worker Properties

worker.props

config.providers=vault
config.providers.vault.class=io.lenses.connect.secrets.providers.VaultSecretProvider
config.providers.vault.param.vault.addr=http://localhost:8200
config.providers.vault.param.vault.auth.method=token
config.providers.vault.param.vault.token=my-token
config.providers.vault.param.file.dir=/connector-files/vault

Usage

To use this provider in a connector, reference the Hashicorp Vault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

• provider is the name of the provider in the worker property file set above

• path is the path of the secret in Hashicorp Vault

• key is the name of the secret key in secret to retrieve. Vault can store multiple keys under a path.

For example, if we store two secrets as keys:

• my_username_key with the value lenses and

• my_password_key with the value my-secret-password

in a secret called secret/my-vault-secret we would set:

connector.props

name=my-sink
class=my-class
topics=mytopic
username=${vault:secret/my-vault-secret:my_username_key}
password=${vault:secret/my-vault-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data encoding

The provider handles the following types:

• utf_8

• base64

The provider will look for keys prefixed with:

• UTF8

• UTF_FILE

• BASE64

• BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk set the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.