AWS

Lenses can be deployed in AWS with your own Apache Kafka or AWS MSK cluster with a CloudFormation template and it is available in AWS Marketplace. Based on Hardware & OS requirements for Lenses, we recommend to start with t2.large instances or any other instance type with >= 7GB.

When the deployment is done you can login to Lenses using the following credentials:

username: admin
password: <ec2-instance-ID>

Marketplace

AWS MSK

Lenses will be installed in the EC2 instance from the .tar.gz Lenses Linux distribution and will expose inbound traffic only for the port which you provided during the template deployment.

This template creates its own IAM profile and installs the AWS Log agent in EC2 instance. It is used to enable CloudWatch logging to be able to check all the available logs for the AWS Stack created for Lenses. Apart from this Lenses IAM profile has the ability to get MSK connection details and get and issue certificate for AWS Certificate Manager Private Certificate Authority. We need these permissions in order to be able to autodetect the MSK boostrap brokers and MSK Zookeeper nodes and in parallel support TLS encrypted traffic and configure Lenses in advanced. More specifically the template enables these policies:

  • logs:CreateLogGroup
  • logs:CreateLogStream
  • logs:PutLogEvents
  • kafka:Describe*
  • kafka:List*
  • logs:PutLogEvents
  • acm-pca:IssueCertificate
  • acm-pca:GetCertificate

Required fields

Field Description Type Required
LensesLicense The Lenses license string yes
MSKClusterARN
The generated MSK Cluster ARN
string yes
MSKSecurityGroup
The MSK security group in order to
allow all traffic as an inbound traffic to Lenses
string yes

Optional fields

Field Description Type Required
SchemaRegistryURLs A list of Schema Registry nodes array no
ConnectURLs A list of all the Kafka Connect clusters array no
Archive

Lenses will be installed in the EC2 instance from the .tar.gz Lenses Linux distribution and will expose inbound traffic only for the port which you provided during the template deployment.

This template creates its own IAM profile and installs the AWS Log agent in EC2 instance. It is used to enable CloudWatch logging to be able to check all the available logs for the AWS Stack created for Lenses. More specifically the template enables these policies:

  • logs:CreateLogGroup
  • logs:CreateLogStream
  • logs:PutLogEvents

Required fields

Field Description Type Required
Lenses License The Lenses license string yes
BrokerURLs
The connection string for Kafka brokers
eg. PLAINTEXT://broker.1.url:9092,PLAINTEXT://broker.2.url:9092.
string yes

Optional fields

Field Description Type Required
ZookeeperURLs A list of all the zookeeper nodes array no
SchemaRegistryURLs A list of Schema Registry nodes array no
ConnectURLs A list of all the Kafka Connect clusters array no

Deployment

AWS MSK

You can deploy Lenses using AWS MSK with one of the marketplace templates. In order Lenses and AWS MSK be able to communicate you need manually to do the following:

MSK VPC Security Group

  • Copy the Security Group ID of Lenses during CloudFormation deployment
  • Open the Amazon VPC console
  • In the navigation pane, choose Security Groups. In the table of security groups, find the security group which belongs to the VPC of your deployed AWS MSK. Choose the group by selecting the checkbox in the first column.
  • In the Inbound Rules tab, choose Edit rules and Choose Add Rule.
  • In the new rule, choose All traffic in the Type column. In the second field in the Source column, enter the ID of the security group of Lenses which you copied at first step. Choose Save rules.

This enables Lenses EC2 instance to communicate back and forth with your MSK cluster as this is the recommended configuration for AWS MSK here.

Common VPC
You can deploy Lenses in the same VPC Apache Kafka infrastructure and you can use the provided IPs for Brokers, Zookeeper, Schema Registry and Connect.
VPC-to-VPC Peering
You can deploy Lenses in a different VPC by your Apache Kafka infrastructure and with peering the two different VPCs, Lenses can communicate with Brokers, Zookeeper, Schema Registry and Connect.

Service Discovery

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster without JMX and everything (ports, connect topics, protocol) left at default values. Lenses VM should have the IAM permission ec2:DescribeInstances. The Schema Registry runs in the same instances as Connect. This example would work if you used Confluent’s AWS templates to deploy your cluster.

SD_CONFIG=provider=aws region=eu-central-1 addr_type=public_v4
SD_BROKER_FILTER=tag_key=Name tag_value=*broker*
SD_ZOOKEEPER_FILTER=tag_key=Name tag_value=*zookeeper*
SD_REGISTRY_FILTER=tag_key=Name tag_value=*worker*
SD_CONNECT_FILTERS=tag_key=Name tag_value=*worker*

If you use our Docker or Reference Architecture CloudFormation there will be fields which you can fill for your specific named tags.

Kubernetes with Helm

Nginx Controller

Lenses can be deployed with Nginx Ingress Controller in AWS with the following commands and additions in provided values for Lenses helm chart.

# Create Policy for IAM Role
curl https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/iam-policy.json -O
aws iam create-policy --policy-name <YOUR-POLICY-NAME> --policy-document file://iam-policy.json
aws iam attach-user-policy --user-name <YOUR-USER-NAME> --policy-arn `CREATED-POLICY-ARN`

# Install NGINX Ingress controller
helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
helm install stable/nginx-ingress --name <RELEASE-NAME>

# This will return the generated URL for Nginx Load Balancer
kubectl --namespace <THE-NAMESPACE-YOU-USED-TO-DEPLOY> get services -o wide -w <RELEASE-NAME>

When you run the above commands the NGINX ingress controller will be created.

restPort: 3030
servicePort: 3030

service:
  enabled: true
  type: ClusterIP
  annotations: {}

ingress:
  enabled: true
  host: <GENERATED-LB-NGINX-URL>

  annotations:
    kubernetes.io/ingress.class: nginx
Traefik Controller

Lenses can be deployed with Traefik Ingress Controller in AWS with the following commands and additions in provided values for Lenses helm chart.

# Create Policy for IAM Role
curl https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/iam-policy.json -O
aws iam create-policy --policy-name <YOUR-POLICY-NAME> --policy-document file://iam-policy.json
aws iam attach-user-policy --user-name <YOUR-USER-NAME> --policy-arn `CREATED-POLICY-ARN`

# Install NGINX Ingress controller
helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
helm install stable/traefik --name <RELEASE-NAME>

# Traefik's load balancer state
kubectl get svc <RELEASE-NAME>-traefik -w

# Once 'EXTERNAL-IP' is no longer '<pending>':
kubectl describe svc <RELEASE-NAME>-traefik | grep Ingress | awk '{print $3}'
restPort: 3030
servicePort: 3030

service:
  enabled: true
  type: ClusterIP
  annotations: {}

ingress:
  enabled: true
  host: <GENERATED-LB-TRAEFIK-URL>

  annotations:
    kubernetes.io/ingress.class: traefik

Note

If you receive an error with the service account which you will use (eg. default) about the followings:
  • Failed to list v1.Endpoints: endpoints is forbidden: User
  • Failed to list v1.Service: services is forbidden: User

Then you need to bind your service account with role cluster-admin with the following RBAC YAML:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
Application Load Balancer

Lenses can be deployed with AWS Application Load Balancer with the following commands and additions in provided values for Lenses helm chart. First, you need to attach the following IAM Policies to the EKS Node Instance IAM Role you will use to deploy ALB ingress controller.

# Create Policy for IAM Role
curl https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/iam-policy.json -O
aws iam create-policy --policy-name <YOUR-POLICY-NAME> --policy-document file://iam-policy.json
aws iam attach-user-policy --user-name <YOUR-USER-NAME> --policy-arn `CREATED-POLICY-ARN`

# Install ALB Ingress controller
helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
helm install incubator/aws-alb-ingress-controller --set clusterName=<EKS-CLUSTER-NAME> --set awsRegion=<YOUR-REGION> --set awsVpcID=<YOUR-VPC-ID> --name <RELEASE-NAME>

When you run the above commands the ALB ingress controller will not be created till you deploy Lenses which includes an ingress configuration for ALB. You need to add the following options:

restPort: 3030
servicePort: 3030

service:
  enabled: true
  type: ClusterIP
  annotations: {}

ingress:
  enabled: true
  host:

  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/subnets: <SUBNETS-VPC-OF-DEPLOYED-ALB>
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip

Then check the Load balancer DNS FQDN with:

kubectl get ingress -o wide -w

If you specify a host for ingress, you need to add ALB address to Route53 to be able to access it externally. Or, deploy external DNS to manage Route53 records automatically, which is also recommended.

ECS Fargate

Lenses will be installed in Elastic Container Service (ECS) with AWS Fargate compute engine that allows you to run containers without having to manage servers or clusters.
Lenses AWS ECS reference architecture