Create a Data integration API key

1. From Data integration API keys, select Create Key.

2. For this guide select Global access

Creating a Connection

In the Lenses bootstrap UI, Select:

1. Security Protocol SASL SSL

2. SASL Mechanism PLAIN

In the JAAS Configuration update the username and password from the respective fields Key and Secret of the API key created above:

org.apache.kafka.common.security.plain.PlainLoginModule required 
username="[Your_API_KEY]" 
password="[Your_API_KEY_SECRET]"

Kafka versions

Any version of Apache Kafka (2.0 or newer) on-premise and on-cloud.

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

JMX connectivity

Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are also supported, as well as JOLOKIA and OpenMetrics (MSK).

For more enable JMX for Lenses itself see here.

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Use 6GB RAM/4 CPUs and 500MB disk space.

Memory & CPU

This is the default configuration = Request 1 CPU & Memory 3Gi, Limit 2 CPU & Memory 5Gi

Browser

All recent versions of major browsers are fully supported.

APIs and Websockets

Every action in Lenses is backed by an API or websocket, documented at https://api.lenses.io. A Golang client is available and CLI (command line interface).

Lenses state store

Lenses can use an embedded H2 database or a Postgres database. Postgres is not supplied by Lenses.

TLS termination

By default, Lenses does not provide TLS termination but can be enabled via a configuration option. TLS termination is recommended for enhanced security and a prerequisite for integrating with SSO (Single Sign On) via SAML2.0.

TLS termination can be configured directly within Lenses or by using a TLS proxy or load balancer. Refer to the TLS documentation for additional information.

1. Add your bootstrap brokers including ports

2. Optionally, security protocol e.g. SASL_SCRAM, SASL_SSL

3. Optionally, SASL Mechanism, e.g. SCRAM-SHA-256

SSL/TLS Configuration

If your Kafka connection requires TLS, set the following

• Truststore: The SSL/TLS trust store to use as the global JVM trust store. Available formats are .jks, .p12, .pfx.

• Keystore: The SSL/TLS keystore to use for the TLS listener for Lenses. Available format is .jks.

JMX Metrics

Lenses allows you to connect to brokers JMX. Supported formats are:

1. Simple with and without SSL

2. Jolokia (JOLOKIAG and JOLOKIAP)

   1. With and without SSL

   2. With Basic Auth

   3. Custom http requests and suffix

3. AWS Open Monitoring

Lenses can connect to any Kafka cluster or service exposing the Apache Kafka APIs and supporting the authentication methods offered by Apache Kafka.

Follow the guide for your distribution to obtain the credentials and bootstrap broker to provide to Lenses.

From the Aiven, locate your Service URI.

In the Lenses bootstrap UI:

1. Set the Service URI as bootstrap servers

2. Set SASL SSL as the security protocol

3. Set SCRAM-SHA-SHA-256 as the security mechanism

4. Set the**jaas.conf** as the following:

org.apache.kafka.common.security.plain.PlainLoginModule required
username="[Your_Connection_Details_Username]"
password="[Your_Connection_Details_Password]"

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports

2. Enable basic auth if required and set the user name and password

3. Enable SSL if required and upload the keystore

4. Optionally upload a trust store

5. Set any additional properties

6. Optional enable metrics

To connect Lenses to your real environment you can:

1. Install Lenses (not the Box) and manually configure the connections to Kafka, Zookeepers, Schema Registries and Connect, or

2. Install Lenses and configure the connections in one go using provisioning.

How to connect to Kafka depends on your Kafka provider.

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK cluster. Lenses can be installed and preconfigured via the AWS Marketplace.

Open network connectivity

Edit the AWS MSK security group in the AWS Console and add the IP address of your Lenses installation.

Enable Open Monitoring

If you want to have Lenses collect JMX metrics you have to enable Open Monitoring on your MSK cluster. Follow the AWS guide here.

Select your MSK endpoint

Depending on your MSK cluster, select the endpoint and protocol you want to connect with.

Creating a Connection

In the Lenses bootstrap UI, Select:

1. Security Protocol and set the protocol you want to use

2. SASL Mechanism and set the mechanism you want to use.

Connecting with AWS IAM

In the Lenses bootstrap UI, Select:

1. Security Protocol and set it to SASL_SSL

2. Sasl Mechanism and set it to AWS_MSK_IAM

3. Add software.amazon.msk.auth.iam.IAMLoginModule required; to the Sasl Jaas Config section

4. Optionally upload your trust store

5. Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler in the Advances Kafka Properties section.

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK Serverless cluster. Lenses can be installed and preconfigured via the AWS Marketplace.

Edit the relevant Security Group

Enable communications between Lenses & the Amazon MSK Serverless cluster by opening the Amazon MSK Serverless cluster's security group in the AWS Console and add the IP address of your Lenses installation.

Configure IAM Policies

To authenticate Lenses & access resources within our MSK Serverless cluster, we'll need to create an IAM policy and apply that to the resource (EC2, EKS cluster, etc) running the Lenses service. here is an example IAM policy with sufficient permissions which you can associate with the relevant IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:topic/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:group/[cluster_name]/[cluster_uuid]/*"
        }
    ]
}

MSK Serverless IAM to be used after cluster creation. Update this IAM policy with the relevant ARN.

Select your MSK endpoint

Click your MSK Serverless Cluster in the MSK console and select View Client Information page to check the bootstrap server endpoint.

Creating the Connection in Lenses

In the Lenses bootstrap UI, Select:

1. For the bootsrap server configuration, use the MSK Serverless endpoint

2. For the Security Protocol, set it to SASL_SSL

3. Customize the Sasl Mechanism and set it to AWS_MSK_IAM

4. Add software.amazon.msk.auth.iam.IAMLoginModule required; to the Sasl Jaas Config section

5. Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler in the Advances Kafka Properties section.

1. During the broker metrics export step, keep it disabled, as AWS Serverless does not export the metrics to Lenses. Click Next

2. Copy your license and add it to Lenses, validate your license, and click Next

3. Click on Save & Boot Lenses. Lenses will finish the setup on its own

Additional Configurations

To enable the creation of SQL Processors that create consumer groups, you need to add the following statement in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Topic*",
    "kafka-cluster:WriteData",
    "kafka-cluster:ReadData"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to add the following statement for the registries and schemas in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Group*"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to modify the security policy for the registry and schemas, which results in additional functions within it:

{
  "Action": [
    "glue:DeregisterDataPreview",
    "glue:ListRegistries",
    "glue:CreateRegistry",
    "glue:RegisterSchemaVersion",
    "glue:GetRegistry",
    "glue:UpdateRegistry",
    "glue:ListSchemas",
    "glue:DeleteRegistry",
    "glue:GetSchema",
    "glue:CreateSchema",
    "glue:ListSchemaVersions",
    "glue:GetSchemaVersion",
    "glue:UpdateSchema",
    "glue:DeleteSchemaVersions"
  ],
  "Resource": [
    "arn:aws:glue:[region]:[aws_account_id]:registry/*",
    "arn:aws:glue:[region]:[aws_account_id]:schema/*"
  ]
}

More details about how IAM works with MSK Serverless can be found in the documentation: MSK Serverless

Limitations

Lenses requires a valid license to start. The license can be added via the UI when in bootstrap mode or at deployment time via the.

License Expired

If at any point the license becomes invalid (it expired / too many brokers were added to the cluster) only the license page will be available.

License Management

See .

In our Azure Portal, go to Dashboards > Ambari home.

1. Kafka endpoints: Go to Kafka > Configs > Kafka Broker > Kafka Broker hosts

2. Optionally get the Zookeeper endpoints: Go to Zookeeper > Configs > Zookeeper Server > Zookeeper Server hosts.

In the Lenses bootstrap UI:

1. Set the Kafka endpoints as bootstrap servers

2. Set the security protocol, mechanism and Jaas config according to your setup. For information on configuring clients (Lenses) for your HDInsight cluster see for unauthenticated and for authenticated.

TLS without Authentication

Set the following:

1. security.protocol to SSL

2. Set the password for your trust store

3. Upload your trust store

TLS with Authentication

Perform the additional steps to above

1. Set the password for your key store

2. Upload your key store

3. Set your key password

Apicuro supports the following versions of Confluent's API:

• Confluent Schema Registry API v6

• Confluent Schema Registry API v7

Set the schema registry URLs to include the compatibility endpoints, for example:

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports and compatibility endpoint path

2. Enable basic auth if required and set the user name and password

Lenses can work with the following schema registry implementations which can be added via the Connections page in Lenses.

Go to Admin->Connections->New Connections->Schema Registry and follow the guide for your registry provider.

Authentication

TLS and basic authentication are supported for connections to Schema Registries.

JMX Metrics

Lenses can collect Schema registry metrics via:

1. JMX

2. Jolokia

Supported formats

• AVRO

• PROTOBUF

To connect your Schema Registry with Lenses, select Schema Registry -> Create Connection.

Schema deletion

To enable the deletion of schemas in the UI, set the following in the lenses.conf file.

From the IBM Cloud console, locate your bootstrap_endpoints. for the service credentials you want to connect with.

In the Lenses bootstrap UI:

1. Set the bootstrap_endpoints as bootstrap servers

2. Set SASL SSL as the security protocol

3. Set PLAIN as the security mechanism

4. Set the**jaas.conf** as the following, using the apiKey value as the password.

Lenses provides support for AWS Glue to manage schema and also explore and process data linked to it via the Lenses

To connect to Glue, first create a AWS Connection. Go to Admin->Connections->AWS and enter your AWS credentials or select the IAM support if Lenses is running on an AWS host, e.g. EC2 instance or it has the AWS default credentials toolchain provider in place.

Next, Select New Connection->Schema Registry->AWS Glue. Select your AWS Connection with access to Glue, and enter the Glue ARN.

Connectivity to Zookeeper is optional for Lenses. Zookeeper is used by Lenses for such purposes:

1. To provide quotas management (until quotas can be managed via the Brokers API)

2. To autodetect the JMX connectivity settings to Kafka brokers (if metrics are not defined directly for Kafka connection).

To add a Zookeeper connection go to Admin->Connections->New Connection->Zookeeper.

1. Add a comma-separated list of Zookeepers, including port

2. Optionally set a session timeout

3. Optionally set a Chroot path

4. Optionally set a connection timeout

5. Optionally enable the collection of JMX metrics (Simple or Jolokia with SSL and Basic auth support)

To configure an application to use this compatibility API, specify the Schema Registry endpoint in the following format:

https://token:{$APIKEY}@{$HOST}/{confluent}

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports, adding the confluent path at the end. Use the value from the kafka_http_url field in the IBM Console Service Credentials tab

2. Enable basic auth if required

3. Set the user name "token"

4. Set the password as the value from API key in the IBM Console Service Credentials tab

Lenses can send out alerts and audits events, the following integrations are supported:

Alerts

1. DataDog

2. AWS CloudWatch

3. PagerDuty

4. Slack

5. Alert Manager

6. Webook (Email, SMS, HTTP and MS Teams)

Audits

1. Webhook

2. Splunk

Once you have configure alert and audit connections, you can create alert and audit channels to route events to them. See Monitoring & Alerting or Auditing for more information.

Lenses integrates with Kafka Connect Clusters to manage connectors.

Adding a connection

To add a connection, go to Admin->Connections->New Connection->Kafka Connect.

1. Provide a name for the Connect cluster

2. Add a comma-separated list of the workers in the Connector cluster, including ports

3. Optionally enable Basic Auth and set the username and password

4. Optionally enable SSL and upload the key-store file

5. Optionally upload a trust store

6. Optionally enable the collection of JMX metrics (Simple or Jolokia with SSL and Basic auth support)

Adding 3rd Party Connector to the Topology

If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info parameter in the lenses.conf file.

connectors.info = [
      {
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"
      }

  ]

Lenses uses an AWS in two places:

1. AWS IAM connection to MSK for Lenses itself

2. Connecting to AWS Glue

3. Alert channels to Cloud Watch.