1 of 1

Hashicorp Vault

This page describes how to retrieve secrets from Hashicorp Vault for use in Kafka Connect.

Secure secrets in Hashicorp Vault and use them in Kafka Connect.

Secrets will only be reloaded if the Connector restarts.

From Version 2.2.0, the secret provider does not write secrets to files by default. If you require this behaviour (for trust stores, key stores or certs) you can enable this by adding the property file.write=true.

Authentication

Multiple authentication methods are supported:

approle
userpass
kubernetes
cert
token
ldap
gcp
awsiam
jwt
github

Configuring the plugin

Example Worker Properties

worker.props

config.providers=vault
config.providers.vault.class=io.lenses.connect.secrets.providers.VaultSecretProvider
config.providers.vault.param.vault.addr=http://localhost:8200
config.providers.vault.param.vault.auth.method=token
config.providers.vault.param.vault.token=my-token
config.providers.vault.param.file.dir=/connector-files/vault

Usage

To use this provider in a connector, reference the Hashicorp Vault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

provider is the name of the provider in the worker property file set above
path is the path of the secret in Hashicorp Vault
key is the name of the secret key in secret to retrieve. Vault can store multiple keys under a path.

For example, if we store two secrets as keys:

my_username_key with the value lenses and
my_password_key with the value my-secret-password

in a secret called secret/my-vault-secret we would set:

connector.props

name=my-sink
class=my-class
topics=mytopic
username=${vault:secret/my-vault-secret:my_username_key}
password=${vault:secret/my-vault-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data encoding

The provider handles the following types:

utf_8
base64

The provider will look for keys prefixed with:

UTF8
UTF_FILE
BASE64
BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk set the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option is given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.