5.0

AWS MSK

Lenses.io is an advanced Amazon MSK integration offering data observability, productivity, monitoring, security and governance for Apache Kafka and event/streaming applications.

MSK ARN

If you have an AWS MSK cluster, you can quickly install using

Learn more 

Read bellow to learn more about

AWS Marketplace 

Note down the Cluster ARN and Security group:

MSK ARN

1) Select CloudFormation Template, Lenses EC2 MSK and your region

MSK CloudFormation template

2) Choose Launch CloudFormation

MSK launch Lenses action

3) Continue with the default options for creating the stack in the AWS wizard.

At the Specify stack details, enter the following:

  • Name for your Lenses deployment at the top
  • InstanceType - Recommended t2.large
  • LensesLocation - 0.0.0.0/0
  • Lenses license - the contents of the JSON license file
  • AWS MSK ARN - From AWS MSK Console
  • AWS Cluster Security Group - From AWS MSK Console
  • Enable CloudWatch metrics

Optionally set the Connect and Schema Registry configurations if you have it installed.

Note: Ensure you select the subnets.

MSK CloudFormation details

4) Review the stack:

MSK Lenses installation review

5) Accept the terms and conditions and create the stack:

MSK Lenses ack

6) Once the stack has deployed, go to the Output tab and click on the FQDN link:

MSK stack FQDN link

7) Login to Lenses with admin and the password value you have submitted on paramater LensesAdminPassword.

Template policies 

The template enables the following policies:

  • logs:CreateLogGroup
  • logs:CreateLogStream
  • logs:PutLogEvents
  • kafka:Describe*
  • kafka:List*
  • logs:PutLogEvents
  • acm-pca:IssueCertificate
  • acm-pca:GetCertificate

Template options 

FieldDescriptionTypeRequired
LensesAdminPasswordThe admin user passwordstringyes
MSKClusterARNThe generated MSK Cluster ARNstringyes
MSKSecurityGroupThe MSK security group in order to allow all traffic as inbound traffic to Lensesstringno
SchemaRegistryURLsA list of Schema Registry nodesarrayno
ConnectURLsA list of all the Kafka Connect clustersarrayno

Persistence Storage options 

FieldDescriptionTypeRequired
LensesStorageThe Lenses Storage Mode - either local or postgres default localstringyes
StoragePostgresHostname(Required if storage is postgres) Postgres Hostnamestringno
StoragePostgresUsername(Required if storage is postgres) Postgres Usernamestringno
StoragePostgresPasswordRequired if storage is postgres) Postgres Passwordstringno
StoragePostgresDatabase(Required if storage is postgres) Postgres Database Namestringno

In-order to do in-place Lenses version upgrades, by downloading Linux binary from Lenses Archive , and you are using storage mode local it is suggested first to do an H2 database backup as in the following article: https://help.lenses.io/using-lenses/basics/backup

Connect with SSH to EC2 Instance 

In version 5.0.0 and onwards port 22 is no longer allowed as an Inbound rule, on the Lenses security group. This is in sync with AWS Marketplace security recommendations, since most users would leave the default allow rule of 0.0.0.0/0, which is a known security risk.

However, to be able to connect with SSH to the instance, you can apply the following workaround step-by-step guide, that utilizes EC2 Instance Connect browser feature:

1) Find your cloudformation stack and open the stack resources menu

Cloudformation Stack
Stack Resources

2) From there you can find the Lenses Instance details and on the Security tab, go to the Lenses Security group to Edit inbound rules.

EC2 Details
Lenses Security Group

3) Add a temporary SSH allow rule with type SSH, for All Ipv4 addresses and Save. This is a pre-requisite for EC2 Instance Connect feature.

Add 22 Rule All
Rule Applied

4) Go back to instance details and hit the Connect button. It will take you to the EC2 Instance Connect screen, to SSH via the browser. Leave parameters user root as is and hit Connect

Use the browser terminal to edit ~/.ssh/authorized_keys file.

EC2 Instance Connect
EC2 Instance Connect Terminal

5) Assuming you have an SSH keypair, copy the public key content to ~/.ssh/authorized_keys. EC2 Instance Connect terminal, uses Ctrl+V to paste copied content from outside the browser screen (such as clipboard). Save the file when done.

You can generate a new secure keypair using ssh-keygen -t rsa -b 4096 command.

Public Key setup

6) Now, after adding a public key, you no longer need EC2 Instance Connect so you can restrict SSH either to a specific IP address, e.g. a VPN server address or only your own public IP, using option My IP on the previous SSH rule.

Restrict SSH Rule
Restrict SSH Applied

7) You can now connect from your local environment to the Lenses Instance, using the private key as root user. Remember to revoke SSH access when done.

SSH Login

Summary 

Lenses adds to Amazon MSK a secure User Interface with DataOps capabilities for:

  • Data discovery
  • Data security
  • Data governance
  • Data monitoring
  • Data observability
  • Data alerts

Installation methods 

1. Secure AWS installation
2. AWS marketplace (Hourly usage)
3. AWS marketplace (BYOL - Bring Your own license) - get a trial license
4. AWS EDP Private Offer contact us for an AWS EDP offer.