Hashicorp Vault

Secure secrets in Hashicorp Vault and use them in Kafka Connect.

Add the plugin to the worker classloader isolation via the plugin.path option:

plugin.path=/usr/share/connectors,/opt/secret-providers

Multiple authentication methods are support:

  • approle
  • userpass
  • kubernetes
  • cert
  • token
  • ldap
  • gcp
  • awsiam
  • jwt
  • github

Configuring the plugin 


NameDescriptionDefault
file.dirThe base location for any files to stored
vault.auth.methodAvailable values are approle, userpass, kubernetes, cert,
token, ldap, gcp, awsiam, jwt,github, token
token
vault.addrAddress of the Vault serverlocalhost
vault.tokenVault app role token. vault.auth.method must be ‘token’
vault.namespaceSets a global namespace to
the Vault server instance.
Required Vault Enterprize Pro
vault.pemFile containing the Vault Server certificate string contents
vault.client.pemFile containing the Client certificate string contents
vault.engine.versionKV Secrets Engine version of the Vault server instance. Defaults to 22
vault.ssl.truststore.locationThe location of the trust store file
vault.ssl.keystore.locationThe location of the key store file
vault.ssl.keystore.passwordThe password the key store file
app.role.idVault App role id. vault.auth.method must be ‘approle’ or ‘kubernetes’
app.role.secret.idVault App role name secret id. vault.auth.method must be ‘approle’
usernameUsername to connect to Vault with. vault.auth.method must be ‘userpass’
passwordPassword to connect to Vault with. vault.auth.method must be ‘userpass’
mountThe mount name of the userpass authentication back end.
vault.auth.method must be userpass
userpass
kubernetes.roleThe kubernetes role used for authentication.
vault.auth.method must be ‘kubernetes’
kubernetes.token.pathPath to the service account token.
vault.auth.method must be kubernetes
/var/run/secrets/kubernetes.io/serviceaccount/token
aws.roleName of the role against which the login
is being attempted. If role is not specified,
then the login endpoint looks for a role bearing the name of the AMI ID of the EC2 instance that
is trying to login if using the ec2 auth method, or the friendly name (i.e., role name or username)
of the IAM principal authenticated.
vault.auth.method must be awsiam
aws.request.urlPKCS7 signature of the identity document
with all n characters removed.Base64-encoded
HTTP URL used in the signed request.
(base64-encoding of https://sts.amazonaws.com/)
as most requests will
probably use POST with
an empty URI. vault.auth.method must be awsiam
aws.request.headers Request headers. vault.auth.method must be ‘awsiam’
aws.request.bodyBase64-encoded body of the signed request.
The base64 encoding of Action=GetCallerIdentity&Version=2011-06-15.
vault.auth.method must be awsiam
aws.mountAWS auth mount. vault.auth.method must be awsiamaws
ldap.usernameLDAP username to connect to Vault with. vault.auth.method must be ‘ldap’
ldap.passwordLDAP password to connect to Vault with. vault.auth.method must be ‘ldap’
mountThe mount name of the ldap authentication back end.vault.auth.method must be ldap
jwt.roleRole the JWT token belongs to. vault.auth.method must be jwt
jwt.providerProvider of JWT token. vault.auth.method must be jwt
jwtJWT token. vault.auth.method must be jwt
gcp.roleThe gcp role used for authentication. vault.auth.method must be gcp
gcp.jwtJWT token. vault.auth.method must be gcp
cert.mountThe mount name of the cert authentication back end.
vault.auth.method must be cert
cert
github.tokenThe github app-id used for authentication.
vault.auth.method must be github
github.mountThe mount name of the github authentication back end.
vault.auth.method must be github
github

Example Worker Properties

config.providers.vault.class=io.lenses.connect.secrets.providers.VaultSecretProvider
config.providers.vault.param.addr=https://localhost
config.providers.vault.param.auth.method=token
config.providers.vault.param.token=my-token
config.providers.vault.param.file.dir=/connector-files/vault

Usage 

To use this provider in a connector, reference the Hashicorp Vault containing the secret and the key name for the value of the connector property.

The indirect reference is in the form ${provider:path:key} where:

  • provider is the name of the provider in the worker property file set above
  • path is the path of the secret in Hashicorp Vault
  • key is the name of the secret key in secret to retrieve. Vault can store multiple keys under a path.

For example, if we store two secrets as keys:

  • my_username_key with the value lenses and
  • my_password_key with the value my-secret-password

in a secret called secret/my-vault-secret we would set:

name=my-sink
class=my-class
topics=mytopic
username=${vault:secret/my-vault-secret:my_username_key}
password=${vault:secret/my-vault-secret:my_password_key}

This would resolve at runtime to:

name=my-sink
class=my-class
topics=mytopic
username=lenses
password=my-secret-password

Data encoding 

The provider handles the following types:

  • utf_8
  • base64

The provider will look for keys prefixed with:

  • UTF8
  • UTF_FILE
  • BASE64
  • BASE64_FILE

The UTF8 means the value returned is the string retrieved for the secret key. The BASE64 means the value returned is the base64 decoded string retrieved for the secret key.

If the value for the tag is UTF8_FILE the string contents are written to a file. The returned value from the connector configuration key will be the location of the file. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If the value for the tag is BASE64_FILE the string contents are based64 decoded and are written to a file. The returned value from the connector configuration key will be the location of the file. For example, if a connector needs a PEM file on disk set, this as the prefix as BASE64_FILE. The file location is determined by the file.dir configuration option given to the provider via the Connect worker.properties file.

If no prefix is found the contents of the secret string are returned.