AWS MSK Serverless
This page describes how to connect Lenses to an Amazon MSK Serverless cluster.
Last updated
This page describes how to connect Lenses to an Amazon MSK Serverless cluster.
Last updated
Lenses will not start without a valid Kafka Connection. You can either add the connection via the bootstrap wizard or use provisioning for automated deployments.
It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK Serverless cluster. Lenses can be installed and preconfigured via the AWS Marketplace.
Enable communications between Lenses & the Amazon MSK Serverless cluster by opening the Amazon MSK Serverless cluster's security group in the AWS Console and add the IP address of your Lenses installation.
To authenticate Lenses & access resources within our MSK Serverless cluster, we'll need to create an IAM policy and apply that to the resource (EC2, EKS cluster, etc) running the Lenses service. here is an example IAM policy with sufficient permissions which you can associate with the relevant IAM role:
MSK Serverless IAM to be used after cluster creation. Update this IAM policy with the relevant ARN.
Click your MSK Serverless Cluster in the MSK console and select View Client Information page to check the bootstrap server endpoint.
In the Lenses bootstrap UI, Select:
For the bootsrap server configuration, use the MSK Serverless endpoint
For the Security Protocol, set it to SASL_SSL
Customize the Sasl Mechanism and set it to AWS_MSK_IAM
Add software.amazon.msk.auth.iam.IAMLoginModule required;
to the Sasl Jaas Config section
Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
in the Advances Kafka Properties section.
During the broker metrics export step, keep it disabled, as AWS Serverless does not export the metrics to Lenses. Click Next
Copy your license and add it to Lenses, validate your license, and click Next
Click on Save & Boot Lenses. Lenses will finish the setup on its own
To enable the creation of SQL Processors that create consumer groups, you need to add the following statement in your IAM policy:
Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.
To integrate with the AWS Glue Schema Registry, you also need to add the following statement for the registries and schemas in your IAM policy:
Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.
To integrate with the AWS Glue Schema Registry, you also need to modify the security policy for the registry and schemas, which results in additional functions within it:
More details about how IAM works with MSK Serverless can be found in the documentation: MSK Serverless
When using Lenses with MSK Serverless:
Lenses does not receive Prometheus-compatible metrics from the brokers because they are not exported outside of CloudWatch.
Lenses does not configure quotas and ACLs because MSK Serveless does not allow this.