Lenses.io is an advanced Amazon MSK integration offering data observability, productivity, monitoring, security and governance for Apache Kafka and event/streaming applications.
Read bellow to learn more about
Familiarity with AWS MSK (Managed Apache Kafka) is assumed.
The AWS Marketplace offering requires AWS MSK (Managed Apache Kafka) to be available. Optionally, AWS RDS (or any other PostgreSQL compatible database) can be configured for Lenses to store its state.
The following AWS resources are created:
Deployment takes approximately three minutes.
Select CloudFormation Template, Lenses EC2 and your region
2) Choose Launch CloudFormation
3) Continue with the default options for creating the stack in the AWS wizard.
Fill in the parameters at Specify stack details.
4) Review the stack:
5) Accept the terms and conditions and create the stack:
6) Once the stack has deployed, go to the Output tab and click on the FQDN link. If there are no outputs listed you might need to press the refresh button.
7) Login to Lenses with admin and the password value you have submitted on parameter LensesAdminPassword.
LensesAdminPassword
The template creates policies that allow the EC2 instance to:
logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents
cloudwatch:PutMetricData
There are two options for Lenses to persist its state: local or PostgreSQL. With the local option, Lenses creates a database locally on the EC2 instance' filesystem. With the PostgreSQL option, an external database is used.
In-order to do in-place Lenses version upgrades, by downloading Linux binary from Lenses Archive, and you are using storage mode local it is suggested first to do an H2 database backup as in the following article.
local
H2
Note that the storage may contain sensitive data, treat it accordingly.
When using PostgreSQL-compatible storage (recommended over local storage), use the backup and recovery mechanisms suitable for your database. When using AWS RDS, follow the instructions here
Find the instructions for upgrading to the latest version of Lenses here.
In version 5.0.0 and onwards port 22 is no longer allowed as an Inbound rule, on the Lenses security group. This is in sync with AWS Marketplace security recommendations, since most users would leave the default allow rule of 0.0.0.0/0, which is a known security risk.
5.0.0
22
0.0.0.0/0
However, to be able to connect with SSH to the instance, you can apply the following workaround step-by-step guide, that utilizes EC2 Instance Connect browser feature:
EC2 Instance Connect
1) Find your cloudformation stack and open the stack resources menu
2) From there you can find the Lenses Instance details and on the Security tab, go to the Lenses Security group to Edit inbound rules.
3) Add a temporary SSH allow rule with type SSH, for All Ipv4 addresses and Save. This is a pre-requisite for EC2 Instance Connect feature.
4) Go back to instance details and hit the Connect button. It will take you to the EC2 Instance Connect screen, to SSH via the browser. Leave parameters user root as is and hit Connect
root
Use the browser terminal to edit ~/.ssh/authorized_keys file.
~/.ssh/authorized_keys
5) Assuming you have an SSH keypair, copy the public key content to ~/.ssh/authorized_keys. EC2 Instance Connect terminal, uses Ctrl+V to paste copied content from outside the browser screen (such as clipboard). Save the file when done.
Ctrl+V
You can generate a new secure keypair using ssh-keygen -t rsa -b 4096 command.
ssh-keygen -t rsa -b 4096
6) Now, after adding a public key, you no longer need EC2 Instance Connect so you can restrict SSH either to a specific IP address, e.g. a VPN server address or only your own public IP, using option My IP on the previous SSH rule.
My IP
7) You can now connect from your local environment to the Lenses Instance, using the private key as root user. Remember to revoke SSH access when done.
Lenses supports connection to MSK brokers via IAM. If Lenses is deployed on an EC2 instance it will use the default credential chain loader to authenticate and connect to MSK.
The following Regions are supported:
us-east-1
us-east-2
us-west-1
us-west-2
ca-central-1
eu-central-1
eu-west-1
eu-west-2
eu-west-3
ap-southeast-1
ap-southeast-2
ap-south-1
ap-northeast-1
ap-northeast-2
sa-east-1
Please:
AWS billing applies for the EC2 instance, CloudWatch logs and optionally CloudWatch metrics.
For the hourly billed version additional hourly charges apply, which depend on the instance size. For the Bring Your Own License (BYOL) you can get a free trial license here.
In case you run into problems, e.g. you cannot connect to Lenses, then the logs could provide more information. The easiest route to do this is to go to CloudWatch in the AWS console. Here, find the log group corresponding to your deployment (it has the same name as the deployment) and pick a log stream. The stream with the /lenses.log suffix contains all log lines regardless of the log level; the stream with the /lenses-warn.log suffix only contains warning level logs.
/lenses.log
/lenses-warn.log
If the above fails, for example because the logs integration is broken, you can SSH into the EC2 instance. First, set up an ssh connection. Lenses is installed into /opt/lenses, the logs can be found under /opt/lenses/logs for further inspection.
/opt/lenses
/opt/lenses/logs
Lenses adds to Amazon MSK a secure User Interface with DataOps capabilities for:
1. Secure AWS installation2. AWS marketplace (Hourly usage)3. AWS marketplace (BYOL - Bring Your own license) - get a trial license4. AWS EDP Private Offer contact us for an AWS EDP offer.
On this page
