From the Aiven, locate your Service URI.

In the Lenses bootstrap UI:

1. Set the Service URI as bootstrap servers

2. Set SASL SSL as the security protocol

3. Set SCRAM-SHA-SHA-256 as the security mechanism

4. Set the**jaas.conf** as the following:

org.apache.kafka.common.security.plain.PlainLoginModule required
username="[Your_Connection_Details_Username]"
password="[Your_Connection_Details_Password]"

Create a Data integration API key

1. From Data integration API keys, select Create Key.

2. For this guide select Global access

Creating a Connection

In the Lenses bootstrap UI, Select:

1. Security Protocol SASL SSL

2. SASL Mechanism PLAIN

In the JAAS Configuration update the username and password from the respective fields Key and Secret of the API key created above:

org.apache.kafka.common.security.plain.PlainLoginModule required 
username="[Your_API_KEY]" 
password="[Your_API_KEY_SECRET]"

1. Add your bootstrap brokers including ports

2. Optionally, security protocol e.g. SASL_SCRAM, SASL_SSL

3. Optionally, SASL Mechanism, e.g. SCRAM-SHA-256

SSL/TLS Configuration

If your Kafka connection requires TLS, set the following

• Truststore: The SSL/TLS trust store to use as the global JVM trust store. Available formats are .jks, .p12, .pfx.

• Keystore: The SSL/TLS keystore to use for the TLS listener for Lenses. Available format is .jks.

JMX Metrics

Lenses allows you to connect to brokers JMX. Supported formats are:

1. Simple with and without SSL

2. Jolokia (JOLOKIAG and JOLOKIAP)

   1. With and without SSL

   2. With Basic Auth

   3. Custom http requests and suffix

3. AWS Open Monitoring

In our Azure Portal, go to Dashboards > Ambari home.

1. Kafka endpoints: Go to Kafka > Configs > Kafka Broker > Kafka Broker hosts

2. Optionally get the Zookeeper endpoints: Go to Zookeeper > Configs > Zookeeper Server > Zookeeper Server hosts.

In the Lenses bootstrap UI:

1. Set the Kafka endpoints as bootstrap servers

2. Set the security protocol, mechanism and Jaas config according to your setup. For information on configuring clients (Lenses) for your HDInsight cluster see here for unauthenticated and here for authenticated.

TLS without Authentication

Set the following:

1. security.protocol to SSL

2. Set the password for your trust store

3. Upload your trust store

TLS with Authentication

Perform the additional steps to above

1. Set the password for your key store

2. Upload your key store

3. Set your key password

Lenses can connect to any Kafka cluster or service exposing the Apache Kafka APIs and supporting the authentication methods offered by Apache Kafka.

Follow the guide for your distribution to obtain the credentials and bootstrap broker to provide to Lenses.

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK Serverless cluster. Lenses can be installed and preconfigured via the AWS Marketplace.

Edit the relevant Security Group

Enable communications between Lenses & the Amazon MSK Serverless cluster by opening the Amazon MSK Serverless cluster's security group in the AWS Console and add the IP address of your Lenses installation.

Configure IAM Policies

To authenticate Lenses & access resources within our MSK Serverless cluster, we'll need to create an IAM policy and apply that to the resource (EC2, EKS cluster, etc) running the Lenses service. here is an example IAM policy with sufficient permissions which you can associate with the relevant IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:topic/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:group/[cluster_name]/[cluster_uuid]/*"
        }
    ]
}

MSK Serverless IAM to be used after cluster creation. Update this IAM policy with the relevant ARN.

Select your MSK endpoint

Click your MSK Serverless Cluster in the MSK console and select View Client Information page to check the bootstrap server endpoint.

Creating the Connection in Lenses

In the Lenses bootstrap UI, Select:

1. For the bootsrap server configuration, use the MSK Serverless endpoint

2. For the Security Protocol, set it to SASL_SSL

3. Customize the Sasl Mechanism and set it to AWS_MSK_IAM

4. Add software.amazon.msk.auth.iam.IAMLoginModule required; to the Sasl Jaas Config section

5. Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler in the Advances Kafka Properties section.

1. During the broker metrics export step, keep it disabled, as AWS Serverless does not export the metrics to Lenses. Click Next

2. Copy your license and add it to Lenses, validate your license, and click Next

3. Click on Save & Boot Lenses. Lenses will finish the setup on its own

Additional Configurations

To enable the creation of SQL Processors that create consumer groups, you need to add the following statement in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Topic*",
    "kafka-cluster:WriteData",
    "kafka-cluster:ReadData"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to add the following statement for the registries and schemas in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Group*"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to modify the security policy for the registry and schemas, which results in additional functions within it:

{
  "Action": [
    "glue:DeregisterDataPreview",
    "glue:ListRegistries",
    "glue:CreateRegistry",
    "glue:RegisterSchemaVersion",
    "glue:GetRegistry",
    "glue:UpdateRegistry",
    "glue:ListSchemas",
    "glue:DeleteRegistry",
    "glue:GetSchema",
    "glue:CreateSchema",
    "glue:ListSchemaVersions",
    "glue:GetSchemaVersion",
    "glue:UpdateSchema",
    "glue:DeleteSchemaVersions"
  ],
  "Resource": [
    "arn:aws:glue:[region]:[aws_account_id]:registry/*",
    "arn:aws:glue:[region]:[aws_account_id]:schema/*"
  ]
}

More details about how IAM works with MSK Serverless can be found in the documentation: MSK Serverless

Limitations

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK cluster. Lenses can be installed and preconfigured via the AWS Marketplace.

Open network connectivity

Edit the AWS MSK security group in the AWS Console and add the IP address of your Lenses installation.

Enable Open Monitoring

If you want to have Lenses collect JMX metrics you have to enable Open Monitoring on your MSK cluster. Follow the AWS guide here.

Select your MSK endpoint

Depending on your MSK cluster, select the endpoint and protocol you want to connect with.

Creating a Connection

In the Lenses bootstrap UI, Select:

1. Security Protocol and set the protocol you want to use

2. SASL Mechanism and set the mechanism you want to use.

Connecting with AWS IAM

In the Lenses bootstrap UI, Select:

1. Security Protocol and set it to SASL_SSL

2. Sasl Mechanism and set it to AWS_MSK_IAM

3. Add software.amazon.msk.auth.iam.IAMLoginModule required; to the Sasl Jaas Config section

4. Optionally upload your trust store

5. Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler in the Advances Kafka Properties section.

From the IBM Cloud console, locate your bootstrap_endpoints. for the service credentials you want to connect with.

In the Lenses bootstrap UI:

1. Set the bootstrap_endpoints as bootstrap servers

2. Set SASL SSL as the security protocol

3. Set PLAIN as the security mechanism

4. Set the**jaas.conf** as the following, using the apiKey value as the password.