AWS MSK

This page describes connection the Lenses Agent to a AWS MSK cluster.

Only one Kafka connection is allowed.

The name must be kafka.

See JSON schema for support.

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  # In case environment variable has single or double quotes escape it with 
  ## "\${ENV_VAR_NAME}
  value: "${ENV_VAR_NAME}"

It is recommended to install the Agent on an EC2 instance or with EKS in the same VPC as your MSK cluster. The Agent can be installed and preconfigured via the AWS Marketplace.

Open network connectivity

Edit the AWS MSK security group in the AWS Console and add the IP address of your Agent installation.

MSK Security group

Enable Open Monitoring

If you want to have the Agent collect JMX metrics you have to enable Open Monitoring on your MSK cluster. Follow the AWS guide here.

Select your MSK endpoint

Depending on your MSK cluster, select the endpoint and protocol you want to connect with.

It is not recommended to use Plaintext for secure environments. For these environments use TLS or IAM.

When the Agent is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

Configure Provisioning

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsType:
      value: AWS