IAM (Identity and Access Management) in Lenses is controlled by Groups. Users and service accounts belong to groups. Permissions are assigned to groups and apply to the users and service accounts in those groups.

Authentication of users is determined by the configured authentication mechanism.

This section guides you through understanding what is required to utilize Lenses efficiently and securely.

Two files control Lenses configuration:

• lenses.conf - contains most of the configuration

• security.conf - sensitive configuration options such as passwords for authentication

When you first log in to Lenses, use the default credentials admin/admin

The default account is a super user and can be used to create groups and other accounts with appropriate permissions.

The default account username and password may be adjusted as below.

# Lenses Administrator settings
lenses.security.user = "admin"

## For the password you can either use the plaintext
#lenses.security.password = "admin"
## Or you may use the SHA256 checksum (advised)
lenses.security.password = "sha256:8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918"

To create a SHA256 checksum for your password you can use the command line tools available in your Linux server or macOS.

unset HISTFILE # Disable history for the current terminal
echo -n "password" | sha256sum

Disabling the Admin Account

To disable the Lenses Administrator user, set an adequately long random password. You can achieve this by using the snippet below:

dd if=/dev/urandom count=1 bs=1024 | sha256sum

1. Enable TLS (SSL) for Lenses HTTPS.

2. Create a keystore for SAML.

3. Choose your identity provider (IdP):

Set the following in the security.conf

lenses.security.saml.keystore.location = "/path/to/lenses.p12"
lenses.security.saml.keystore.password = "my_password"
lenses.security.saml.key.password = "my_password"

With Basic Auth, user accounts are managed by Lenses and a unique username and a password are used to log in.

Account locking

For BASIC and LDAP authentication type, there is the option to set a policy to temporarily lock the account when successive login attempts fail. Once the lock time window has passed the user can log in again.

.

Accounts storage

The internal database that stores user/group information is stored on disk, under the lenses.storage.directory or an external Postgres database.

If using the embedded H2 database keep this directory intact between updates and upgrades.

Password rules

To enforce specific password rules the following configurations need to be set:

Password history

To not allow previous passwords to be reused, use the following configuration:

With custom authentication, you can plug in your own authentication system by using HTTP headers.

In this approach, your own authentication proxy/code sitting in front of Lenses takes care of the authentication and injects appropriate Headers in all HTTP requests for verified users.

Setup a custom authentication layer

Set up a custom authentication layer by introducing in security.conf:

Lenses connects similarly to any other application to the infrastructure You can implement a plugin in a few hours in Java/Scala or other JVM technology by implementing one interface:

The returned object UserAndGroups will contain the username and the groups a person authentication belongs to (or raise an exception if no such user exists).

Example

The best way to get started is to look into a sample open-source implementation of such a plugin in .

Authentication is configured in the security configuration file. Lenses Administrator and Basic Auth do not require any configuration.

The following authentication methods are available. Users, regardless of the method need to be mapped to groups.

Account Locking

For BASIC and LDAP authentication types, there is the option to set a policy to temporarily lock the account when successive login attempts fail. Once the lock time window has passed the user can log in again.

These two configuration entries enable the functionality (both of them have to be provided to take effect):

Group Mapping

A Group is a collection of permissions that defines the level of access for users belonging to it. Groups consist of:

• Namespaces

• Application permissions

• Administration permissions

LDAP & Active Directory

When working with LDAP or Active Directory, user and group management is done in LDAP.

Lenses provides fine-grained role-based access (RBAC) for your existing groups of users over data and applications. Create a group in Lenses with the same name (case-sensitive) as in LDAP/AD.

SSO & SAML

When using an SSO solution such as Azure AD, Google, Okta, OneLogin or an open source like KeyCloak user and group management is done in the Identity Provider.

Lenses provides fine-grained role-based access (RBAC) for your existing groups of users over data and applications. Create a group in Lenses with the same name (case-sensitive) as in your SSO group.

Basic Auth

With Basic Authentication, create groups of users and add users to those groups. Authentication and authorization are fully managed, and users can change their passwords.

Groups are case-sensitive and mapped by UUID with Azure

Integrate your user-groups with Lenses using the Azure group IDs. Create a group in Lenses using the UUID as the name.

For example, if the Engineers group has the UUID ae3f363d-f0f1-43e6-8122-afed65147ef8, create a group with the same name.

Set up Microsoft Azure SSO

Learn more about

Add from Azure app-gallery

1. Go to Enterprise applications > + New Application

2. Search for Lenses.io in the gallery directory

3. Choose a name for Lenses e.g. Lenses.io and click Add

4. Select Set up single sign on > SAML

5. Configure the SAML details

1. Download the Federation Metadata XML file with the Azure IdP details. You will reference this file’s path in the Lenses security.conf configuration file.

Google doesn't expose the groups, or organization unit, of a user to a SAML app. This means we must set up a custom attribute for the Lenses groups that each user belongs to.

Create a custom attribute for Lenses groups

1. Open the from an administrator account.

2. Click the Users button

3. Select the More dropdown and choose Manage custom attributes

4. Click the Add custom attribute button

5. Fill the form to add a Text, Multi-value field for Lenses Groups, then click Add

Assign Lenses groups attributes to Google users

1. Open the from an administrator account.

2. Click the Users button

3. Select the user to update

4. Click User information

5. Click the Lenses Groups attribute

6. Enter one or more groups and click Save

Add Google custom SAML app

2. Click the Apps button

3. Click the SAML apps button

4. Select the Add App dropdown and choose Add custom SAML app

App Details

1. Enter a descriptive name for the Lenses installation

Download IdPXML file

Configure in security.conf.

Integrate your user groups with Lenses using the Keycloak group names. Create a group in Lenses using the same case-sensitive group name as in Keycloak.

For example, if the Engineers group is available in Keycloak, with Lenses assigned to it, create a group with the same name.

Create a new SAML application client in Keycloak

1. Go to Clients

2. Click Create

3. Fill in the details: see the table below.

4. Click Save

Change the settings on client you just created to:

Map user groups

Configure Keycloak to communicate groups to Lenses. Head to the Mappers section.

1. Click Create

2. Fill in the details: see table below.

3. Click Save

Download IdP XML file

Configure in the security.conf file.

Azure AD supports the LDAP protocol. You can use it as an authentication provider with users, passwords, and groups stored in Azure AD. When a user is authenticated successfully, Lenses queries Azure AD to get the user’s groups and authorizes the user with the selected permissions.

Here is a sample Lenses configuration:

Create Azure AD Domain Instance

1. In the Azure portal create a resource. Search for Domain service and select Azure AD Domain Services from the options.

2. Set the DNS Domain Name as the same one you have with for your existing Azure AD tenant

1. In the Administration tab, you can manage the group membership for the AAD DC Administrator and control the members with access rights on Azure AD.

Azure AD Domain Services provides one-way synchronization from Azure Active Directory to the managed domain. Only certain attributes are synchronized to the managed domain, along with groups, group memberships and passwords.

The Synchronization tab provides two options. The first one is All, where everything will be synchronized to Azure AD DS managed domain. The second one is Scoped, which allows the selection of specific groups to be synced.

Configure DNS server settings

Once the managed domain is ready to be used, configure the DNS server settings for the Azure Virtual Network. Click the button configure:

For the DNS changes to be applied, all the VMs are required to be restarted.

Azure AD DS needs password hashes in a format that’s suitable for NT LAN Manager (NTLM) and Kerberos authentication. Azure AD does not generate or store password hashes in the format that’s required for NTLM or Kerberos authentication until you enable Azure AD DS for your tenant.

For security reasons, Azure AD doesn’t store any password credentials in clear-text form. Therefore, Azure AD can’t automatically generate these NTLM or Kerberos password hashes based on users’ existing credentials.

Virtual network peering

The Virtual Network to deploy Lenses, requires enabling Virtual Network Peering. This allows it to communicate with Azure AD DS. You should add the IPs that have been generated in the previous step as DNS Servers.

Enable Secure LDAP

To enable the LDAP(S) protocol on Azure AD DS, use the following PowerShell to generate the self-signed certificate:

In case PowerShell is not available, you can use the openssl command. This following script generates a certificate for Azure AD DS.

Under Secure LDAP, upload the PFX certificate and make sure the options Allow secure LDAP and access over the Internet are enabled.

After the secure LDAP is enabled to allow secure LDAP access, use the Azure AD DS properties to review the external IP address that is used to expose the LDAP service.

Finally, you need to allow inbound traffic to the Azure AD DS network security group for the LDAPS port 636 and limit the access only to the the virtual machine or the range of the IPs to which they should have inbound access.

Lenses can be configured via LDAP handle the user authentication.

All the user’s groups are then matched by name (case sensitive) with the groups stored in Lenses. All the matching groups' permissions are combined. If a user has been assigned manually a set of Lenses groups, then the groups coming from LDAP are ignored.

Due to the LDAP standard ambiguity, it is impossible to support all the configurations in the wild. The most common pain point is LDAP group mapping. If the default class that extracts and maps LDAP groups to Lenses groups does not work, it is possible to implement your own.

Before setting up an LDAP connection, we advise you to familiarize yourself with LDAP and/or have access to your LDAP and/or Active Directory administrators.

An LDAP setup example with LDAP group mapping is shown below:

In the example above you can distinguish three key sections for LDAP:

• the connection settings,

• the user search settings,

• and the group search settings.

Lenses uses the connection settings to connect to your LDAP server. The provided account should be able to list users under the base path and their groups. The default group plugin only needs access to the memberOf attributes for each user, but your custom implementation may need different permissions.

When a user tries to log in, a query is sent to the LDAP server for all accounts that are under the lenses.security.ldap.base and match the lenses.security.ldap.filter. The result needs to be unique; a distinguished name (DN) —the user that will log in to Lenses.

In the example, the application would query the LDAP server for all entities under ou=Users,dc=example,dc=com that satisfy the LDAP filter (&(objectClass=person)(sAMAccountName=)) which would be replaced by the username that tries to login to Lenses. A more simple filter could be cn=, which for user Mark would return the DN cn=Mark,ou=Users,dc=example,dc=com.

Once the user has been verified, Lenses queries the user groups and maps them to Lenses groups. For every LDAP group that matches a Lenses group, the user is granted the selected permissions.

Depending on the LDAP setup, only one of the users or the Lenses service user may be able to retrieve the group memberships. This can be controlled by the option lenses.security.ldap.use.service.user.search.

The default value (false) uses the user itself to query for groups. Groups be can be created in the admin section of the web interface, or in the command line via the lenses-cli application.

Set lenses.security.ldap.use.service.user.search to true to use lenses.security.ldap.user account to list a logged user groups when your LDAP setup restricts most of the user's action to list their groups.

Group mapping

When working with LDAP or Active Directory, user and group management is done in LDAP.

Lenses provides fine-grained role-based access (RBAC) for your existing groups of users over data and applications.

If mapping LDAP groups to Lenses groups is not desired. Manually map LDAP users to Lenses groups, using the web interface, or the lenses-cli.

LDAP still provides the authentication, but all LDAP groups for this user are ignored.

When you create an LDAP user in Lenses, the username will be used in the search expression set in lenses.security.ldap.filter to authenticate them. If no user should be allowed to use the groups coming from LDAP, then this functionality should be disabled.

Set lenses.security.ldap.plugin.memberof.key or lenses.security.ldap.plugin.group.extract.regex to a bogus entry, rendering it unusable.

An example would be:

Group extract plugin

The group extract plugin is a class that implements an LDAP query that retrieves a user’s groups and makes any necessary transformation to match the LDAP group to a Lenses group name.

The default class implementation that comes with Lenses is io.lenses.security.ldap.LdapMemberOfUserGroupPlugin.

If your LDAP server supports the memberOf functionality, where each user has his/her group memberships added as attributes to his/her entity, you can use it by setting the lenses.security.ldap.plugin.class option to this class:

Below you will see a brief example of its setup.

As an example, the memberOf search may return two attributes for user Mark:

The regular expression (?i)cn=(\w+),ou=Groups.* will return these two regex group matches:

If any of these groups exist in Lenses, Mark will be granted the permissions of the matching groups.

The lenses.security.ldap.plugin.group.extract.regex should contain exactly one regular expression capturing group.

If you need to apply more groups for your matching purposes, you should use non-capturing groups (e.g (?:groupRegex).

As an example, the regular expression (?i)cn=((?:Kafka|Apps)Admin),ou=Groups,dc=example,dc=com applied to memberOf attributes:

will return these two regex group matches:

Custom LDAP plugin

Do not forget to grant to the account any permissions it may need for your plugin to work.

LDAP Configuration Options

The following configuration entries are specific to the default group plugin. A custom LDAP plugin might require different entries under lenses.security.ldap.plugin:

Groups are case-sensitive and mapped by name with Okta

Integrate your user-groups with Lenses using the Okta group names. Create a group in Lenses using the same case-sensitive group name as in Okta.

For example, if the Engineers group is available in Okta, create a group with the same name.

Set up Okta IdP

Lenses is available directly in Okta’s Application catalog.

Add application in the Catalog

1. Go to Applications > Applications

2. Click Add Application

3. Search for Lenses

4. Select by pressing Add

Set General Settings

1. App label: Lenses

2. Set the base url of your lenses installation e.g. https://lenses-dev.example.com

3. Click Done

Download IdP XML file

Download the Metadata XML file with the Okta IdP details.

1. Go to Sign On > Settings > SIGN ON METHODS

2. Click on Identity Provider metadata and download the XML data to a file.

3. You will reference this file’s path in the security.conf configuration file.

lenses.security.saml.idp.metadata.file="/path/to/OktaIDPMetadata.xml"

A Group is a collection of permissions that defines the level of access for users belonging to it. Groups consist of:

• Namespaces

• Application permissions

• Administration permissions

Groups must be pre-created, and the group's names in Lenses must match (case sensitive) those in the SSO provider.

Creating groups

To create a new Group, go to Admin->Groups->New Group.

For every Group, you must set the data namespaces for Kafka or other available connections to data sources.

Groups must be given names, optionally a description.

Adding namespaces to groups

Namespace permissions define the access to datasets. and

Each group must have a namespace. A namespace is a set of permissions that apply to topics or a set of topics, for example, prod*. This allows you to define virtual multi-tenancy.

Adding application permissions

Application permissions define how a user can interact with applications and linked resources associated with those datasets.

Application permissions cover:

1. Viewing or resetting Consumer group offsets linked to a group's namespaces

2. Deploying or viewing connectors linked to a group's namespaces

3. Deploying or viewing SQL Processors linked to a group's namespaces

Additionally, application permissions define whether a group can access a specified Connect cluster.

Adding administration permissions

Admin permissions refer to activities that are in the global scope of Lenses and affect all the related entities.

Authentication instances too old or in the future

The error that you see

Authentication issue instant is too old or in the future

What causes it

1. The user’s session in the SSO provider is too old.

2. The system clocks of the SSO provider and the Lenses instance are out of sync.

For security purposes, Lenses prevents authenticating SSO users that have remained logged in SSO for a very long time.

Example: You use Okta SSO and, you logged in to Okta a year ago. Okta might allow you to remain logged in along that year without having to re-authenticate. Lenses has a limit of 100 days. In that case, Lenses will receive an authenticated user that originally logged in before the 100 days mark.

How to solve it

1. Ensure that the SSO and Lenses system clocks are in sync.

2. If the SSO provider supports very long sessions either:

   1. Log out of the SSO and log back in. This explicitly renews the SSO session.

   2. Increase the Lenses limit to more than 100 days.

Example:

lenses.security.saml.idp.session.lifetime.max = 365days

Groups are case-sensitive and mapped to roles, by name, with OneLogin

Integrate your user roles with Lenses using the Keycloak role names. Create a group in Lenses using the same case-sensitive role name as in OneLogin.

For example, if the Engineers role is available in OneLogin, create a group with the same name.

Set up OneLogin IdP

Lenses is available in the OneLogin Application catalog.

Visit OneLogin’s Administration console. Select Applications > Applications > Add App

1. Search and select Lenses

2. Optionally add a description and click save

Add Lenses via the Application Catalog

1. In the Configuration section set the base path from the url of the Lenses installation e.g. lenses-dev.example.com ( without the https://)

2. Click Save

Download the IdP XML file

1. Use the More Actions button

2. Click and download the SAML Metadata

3. You will reference this file’s path in the security.conf configuration file.

lenses.security.saml.idp.metadata.file="/path/to/OneLoginIDPMetadata.xml"

Kerberos uses SPNEGO (Simple and Protected GSSAPI Negotiation Mechanism) for authentication.

Kerberos will automatically log in authorized users when using the /api/auth REST endpoint. If using Microsoft Windows, logging into your Windows domain is usually sufficient to issue your Kerberos credentials.

On Linux, if you use Kerberos with PAM, your Kerberos credentials should be already available to Kerberos-enabled browsers. Otherwise, you will need to authenticate to the KDC manually using kinit at the command line and start your browser from the same terminal.

Configuration

In order to use Kerberos authentication in Lenses, both a static configuration and Kerberos Connection is required.

• Static configuration To set up Kerberos you need a Kerberos principal and a password-less keytab. Add them in security.conf before starting Lenses:

lenses.security.kerberos.service.principal="HTTP/lenses.url[@REALM]"
lenses.security.kerberos.keytab=/path/to/lenses.keytab

• Kerberos Connection A Kerberos Connection should be defined in order to use a proper krb5.conf

Service accounts require an authentication token to be authenticated and must belong to at least one group for authorization.

Service accounts are commonly used for automation, for example, when using Lenses CLI or APIs, or any other application or service to interact with Lenses.

Creating a service account

To create a new Service Account, navigate to the Admin and select Users and New Service Account.

Authentication token

You can manually enter the authentication token or autogenerate it. If you select to auto-generate tokens, then you will receive a one-time token for this service account. Follow the instructions and copy and store this token. You can now use this token to authenticate via API and CLI.

Editing a service account

You can only change the groups and owner of services accounts. Go to the service account and select Edit Info, from the Actions menu.

Revoking a service account

To change the token, go to the service account and select Revoke Token from the Actions menu.

Using a service account

To use the service account you need to prefix the token with its name separated by a colon. You then include that in the corresponding header.

Example

For a service account named myservice and a token da6bad50-55c8-4ed4-8cad-5ebd54a18e26 then the combination looks like this:

myservice:28ab4195-18cf-426a-abda-c41a451e001a

To use the CLI with a service account for CI/CD you need to pass these options:

lenses-cli topics \
  --token=<service-account-name>:<service-account-token> \
  --host=<lenses-url-host>

# Real Example
lenses-cli topics \
  --token=ci:58d86476-bcc6-47e2-a57e-0c6bbd9c88b9 \
  --host=http://<your-lenses-url>:9991

Users must be assigned to a group. SSO and LDAP users are mapped to a group matching the group name provided by the Idp.

Multiple types of users can be supported at the same time.

Adding a user with Basic Authentication

Select Admin->Users->New User->Basic Auth.

Mapping SSO users to groups

By default, users are mapped to the group provided by the SSO provider. If you wish to override the group mapping from your SSO, users can be created directly in Lenses and you can manually map the user to a group.

Mapping LDAP users to groups

By default, users are mapped to the group provided by the LDAP server. If you wish to override the group mapping you manually map the user to a group.

See LDAP user group mapping.

Listing Users

Lenses allows you to view users and:

1. Authentication type

2. Groups they belong to

3. Last login

Go to Admin -> Users.

Admin permissions

This matrix shows both display name (first column) and code name (second column) for permissions. Knowing code name may be helpful while using API / CLI.

Data permissions

Application permissions

This matrix shows both display name (first column) and code name (second column) for permissions. Knowing code name may be helpful while using API / CLI.

The JMX endpoint is managed by the lenses.jmx.port option. To disable the JMX leave the option empty.

To enable monitoring of Lenses metrics:

To export via Prometheus exporter:

The Lenses Docker image (lensesio/lenses) automatically sets up the Prometheus endpoint. You only have to expose the 9102 port to access it.

Setting up the JMX Agent with Basic Auth.

This will be done in two parts. The first part is about setting up the required files that JMX Agent will require and the second is about the options we need to pass to the agent.

Setting up required files

First let’s create a new folder called jmxremote

To enable basic auth JMX, first create two files:

• jmxremote.access

• jmxremote.password

JMX.Password file

The password file has the credentials that the JMX agent will check during client authentication

The above code is registering 2 users.

• UserA:

  • username admin

  • password admin

• UserB:

  • username: guest

  • password: admin

JMX.Access file

The access file has authorization information, like who is allowed to do what.

In the above code, we can see that the admin user can do read and write operations in JMX, while guest user can only read the JMX content.

Enable JMX with Basic Auth Protection

Now, to enable JMX with basic auth protection, all we need to do is pass the following options in the JRE’s env that will run the Java process you need to protect the jmx.

Let’s assume this java process is Kafka.

Change the permissions on both files so only owner can edit and view them.

If you do not change the permissions to 0600 and to the user that will run the jre process, then JMX will Agent will cause an error complaining that the Process is not the owner of the files that will be used for authentication and authorization.

Finally export the following options in the user’s env which will run Kafka.

Secure JMX with TLS Encryption

First setup JMX with basic auth as shown in the Secure JMX: Basic Auth page.

To enable TLS Encryption/Authentication in JMX you need a jks keystore and truststore.

Please note that both JKS Truststore and Keystore should have the same password.

The reason for this is because the javax.net.ssl class will use the password you pass to the Keystore as the keypassword

Let’s assume this java process is Kafka and that you have installed the keystore.jks and truststore.jks under `/etc/certs``

Export the following options in the user’s env which will run Kafka.

Lenses state can be stored:

• on the local filesystem - (quick start and default option)

• in a PostgreSQL database - (recommended) and takes preference when configured

Local storage

By default, Lenses will store its internal state in the storage folder. We advise explicitly setting this location, ensuring the Lenses process has permission to read and write files in this directory and have an upgrade and backup policy.

PostgreSQL

Lenses can persist their internal state to a remote PostgreSQL database server.

Current minimum requirements:

• Postgres server running version 9.6 or higher

The recommended configuration is to create a dedicated login role and database for the agent, setting the agent role as the database owner. This will mean the agent will only be able to manage that database and require no superuser privileges.

Example psql command for initial setup:

You can then configure Lenses as so:

Migration of local storage to PostgreSQL

Enabling PostgreSQL storage for an existing Lenses installation means the data will be automatically migrated to the PostgreSQL schema on the first run.

After this process has succeeded, a lensesdb.postgresql.migration file will be created in the local storage directory to flag that the migration has already been run. You can then delete the local storage directory and remove the lenses.storage.directory configuration.

If, for whatever reason, you want to re-run the migration to PostgreSQL, deleting the lensesdb.postgresql.migration file will cause Lenses to re-attempt migration on the next restart. The migration process will fail if it encounters any data that can’t be migrated into PostgreSQL, so re-running the migration should only be done on an empty PostgreSQL schema to avoid duplicate record failures.

Connection pooling

Lenses use the HikariCP library for high-performance database connection pooling.

The default settings should perform well but can be overridden via the lenses.storage.hikaricp configuration prefix. The supported parameters can be found in the HikariCP documentation.

For example:

Lenses runs as a JVM app; you can tune runtime configurations via environment variables.

Global Truststore

To use a non-default global truststore, set the path in accordingly with the LENSES_OPTS variable.

Custom Truststore

To use a custom truststore set the following in security.conf. Supported types: jks, pkcs12.

Mutual TLS

To enable mutual TLS, set your keystore accordingly.

When your Kafka cluster is configured with an authorizer which enforces ACLs, Lenses will need a set of permissions to function correctly.

Common practice is to give Lenses superuser status or the complete list of available operations for all resources. The fine-grained permission model of Lenses can then be used to restrict the access level per user.

Minimal Permissions

The agent needs permission to manage and access their own internal Kafka topics:

• __topology

• __topology__metrics

It also needs to read and describe permissions for the consumer offsets and Kafka Connect topics —if enabled:

• __consumer_offsets

• connect-configs

• connect-offsets

• connect-status

This same set of permissions is required for any topic that the agent must have read access.

Additional permissions are needed to produce topics or manage them.

Consumer Groups

Permission to at least read and describe consumer groups is required to take advantage of the Consumer Groups' monitoring capabilities.

Additional permissions are needed to manage groups.

ACLs

To manage ACLs, permission to the cluster is required:

The following implementations can be specified:

1. Serializers/Deserializers Plug your serializer and deserializer to enable observability over any data format (i.e., protobuf / thrift)

2. Custom authentication Authenticate users on your proxy and inject permissions HTTP headers.

3. LDAP lookup Use multiple LDAP servers or your group mapping logic.

4. SQL UDFs User Defined Functions (UDF) that extend SQL and streaming SQL capabilities.

Once built, the jar files and any plugin dependencies should be added to Lenses and, in the case of Serializers and UDFs, to the SQL Processors if required.

Adding plugins

On startup, Lenses loads plugins from the $LENSES_HOME/plugins/ directory and any location set in the environment variable LENSES_PLUGINS_CLASSPATH_OPTS. These locations Lenses is watching, and dropping a new plugin will hot-reload it. For the Lenses docker (and Helm chart) you use /data/plugins.

Any first-level directories under the paths mentioned above, detected on startup will also be monitored for new files. During startup, the list of monitored locations will be shown in the logs to help confirm the setup.

...
Initializing (pre-run) Lenses
Installation directory autodetected: /opt/lenses
Current directory: /data
Logback configuration file autodetected: logback.xml
These directories will be monitored for new jar files:
 - /opt/lenses/plugins
 - /data/plugins
 - /opt/lenses/serde
Starting application
...

Whilst all jar files may be added to the same directory (e.g /data/plugins), it is suggested to use a directory hierarchy to make management and maintenance easier.

An example hierarchy for a set of plugins:

├── security
│   └── sso_header_decoder.jar
├── serde
│   ├── protobuf_actions.jar
│   └── protobuf_clients.jar
└── udf
    ├── eu_vat.jar
    ├── reverse_geocode.jar
    └── summer_sale_discount.jar

SQL Processors in Kubernetes

There are two ways to add custom plugins (UDFs and Serializers) to the SQL Processors; (1) via making available a tar.gz archive at an HTTP (s) address, or (2) via creating a custom docker image.

Archive served via HTTP

With this method, a tar archive, compressed with gzip, can be created that contains all plugin jars and their dependencies. Then this archive should be uploaded to a web server that the SQL Processors containers can access, and its address should be set with the option lenses.kubernetes.processor.extra.jars.url.

Step by step:

1. Create a tar.gz file that includes all required jars at its root:

   Copy

   tar -czf [FILENAME.tar.gz] -C /path/to/jars/ *

2. Upload to a web server, ie. https://example.net/myfiles/FILENAME.tar.gz

3. Set

   Copy

   lenses.kubernetes.processor.extra.jars.url=https://example.net/myfiles/FILENAME.tar.gz

   For the docker image, set the corresponding environment variable

   Copy

   LENSES_KUBERNETES_PROCESSOR_EXTRA_JARS_URL=https://example.net/myfiles/FILENAME.tar.gz`

Custom Docker image

The SQL Processors that run inside Kubernetes use the docker image lensesio-extra/sql-processor. It is possible to build a custom image and add all the required jar files under the /plugins directory, then set lenses.kubernetes.processor.image.name and lenses.kubernetes.processor.image.tag options to point to the custom image.

Step by step:

1. Create a Docker image using lensesio-extra/sql-processor:VERSION as a base and add all required jar files under /plugins:

   Copy

   FROM lensesio-extra/sql-processor:4.2
   ADD jars/* /plugins

   Copy

   docker build -t example/sql-processor:4.2 .

2. Upload the docker image to a registry:

   Copy

   docker push example/sql-processor:4.2

3. Set

   Copy

   lenses.kubernetes.processor.image.name=example/sql-processor
   lenses.kubernetes.processor.image.tag=4.2

   For the docker image, set the corresponding environment variables

   Copy

   LENSES_KUBERNETES_PROCESSOR_IMAGE_NAME=example/sql-processor
   LENSES_KUBERNETES_PROCESSOR_IMAGE_TAG=4.2

Lenses can be used to define & deploy stream processing applications that read from Kafka and write back to Kafka with SQL. They are based on the Kafka Stream framework. They are known as SQL Processors.

SQL processing of real-time data can run in 2 modes:

• SQL In-Process - the workload runs inside of Lenses.

• SQL in Kubernetes - the workload runs & scale on your Kubernetes cluster.

Which mode the SQL Processors will run as should be defined within the lenses.conf before Lenses is started.

In-Process Mode

In this mode, SQL processors run as part of the Lenses process, sharing resources, memory, and CPU time with the rest of the platform.

Set the execution configuration to IN_PROC

# Set up Lenses SQL processing engine
lenses.sql.execution.mode = "IN_PROC"

Set the directory to store the internal state of the SQL Processors:

lenses.sql.state.dir = "/tmp/lenses-sql-kstream-state"

TLS connections to Kafka and Schema Registries

SQL processors use the same connection details that Lenses uses to speak to Kafka and Schema Registry. The following properties are mounted, if present, on the file system for each processor:

• Kafka

  1. SSLTruststore

  2. SSLKeystore

• Schema Registry

  1. SSL Keystore

  2. SSL Truststore

The file structure created by applications is the following: /run/[lenses_installation_id]/applications/

Kubernetes Mode

Kubernetes can be used to deploy SQL Processors. To configure Kubernetes, set the mode to KUBERNETES and configure the location of the kubeconfig file.

When Lenses is deployed inside Kubernetes, the lenses.kubernetes.config.file configuration entry should be set to an empty string. The Kubernetes client will auto-configure from the pod it is deployed in.

The SQL Processor docker image is live in Dockerhub.

lenses.sql.execution.mode = KUBERNETES
# kubernetes configuration
lenses.kubernetes.config.file = "/home/lenses/.kube/config"
lenses.kubernetes.service.account = "default"
#lenses.kubernetes.processor.image.name = "" # Only needed if you use a custom image
#lenses.kubernetes.processor.image.tag = ""  # Only needed if you use a custom image

# Only needed if you want to tune the buffer size for incoming events from Kubernetes
#lenses.deployments.errors.buffer.size = 1000

# Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication
#lenses.deployments.events.buffer.size = 10000

Custom Serdes

Custom serdes should be embedded in a new Lenses SQL processor Docker image.

To build a custom Docker image, create the following directory structure:

mkdir -p processor-docker/serde

Copy your serde jar files under processor-docker/serde.

Create Dockerfile containing:

FROM lensesioextra/sql-processor:4.2

ADD serde /opt/serde
ENV LENSES_SQL_RUNNERS_SERDE_CLASSPATH_OPTS=/opt/serde

Build the Docker.

cd processor-docker
docker build -t example/lsql-processor .

Once the image is deployed in your registry, please set Lenses to use it (lenses.conf):

lenses.kubernetes.processor.image.name = "your/image-name"
lenses.kubernetes.processor.image.tag = "your-tag"

Use Role/RoleBinging to deploy Lenses processors

To deploy Lenses Processors in Kubernetes the suggested way is to activate RBAC in Cluster level through Helm values.yaml:

rbacEnable: true

To achieve this you need to create a Role and a RoleBinding resource in the namespace you want the processors deployed to:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: [ROLE_NAME]
  namespace: [PROCESSORS_NAMESPACE]
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - persistentvolumes
    - persistentvolumeclaims
    - pods/log
  verbs:
    - list
    - watch
    - get
    - create
- apiGroups: ["", "extensions", "apps"]
  resources:
    - pods
    - replicasets
    - deployments
    - ingresses
    - secrets
    - statefulsets
    - services
  verbs:
    - list
    - watch
    - get
    - update
    - create
    - delete
    - patch
- apiGroups: [""]
  resources:
    - events
  verbs:
    - list
    - watch
    - get

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: [ROLE_BINDING_NAME]
  namespace: [PROCESSOR_NAMESPACE]
subjects:
- kind: ServiceAccount
  namespace: [LENSES_NAMESPACE]
  name: [SERVICE_ACCOUNT_NAME]
roleRef:
  kind: Role
  name: [ROLE_NAME]
  apiGroup: rbac.authorization.k8s.io

example for:

• Lenses namespace = lenses-ns

• Processor namespace = lenses-proc-ns

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role
  namespace: lenses-proc-ns
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - persistentvolumes
    - persistentvolumeclaims
    - pods/log
  verbs:
    - list
    - watch
    - get
    - create
- apiGroups: ["", "extensions", "apps"]
  resources:
    - pods
    - replicasets
    - deployments
    - ingresses
    - secrets
    - statefulsets
    - services
  verbs:
    - list
    - watch
    - get
    - update
    - create
    - delete
    - patch
- apiGroups: [""]
  resources:
    - events
  verbs:
    - list
    - watch
    - get

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role-binding
  namespace: lenses-proc-ns
subjects:
- kind: ServiceAccount
  namespace: lenses-ns
  name: default
roleRef:
  kind: Role
  name: processor-role
  apiGroup: rbac.authorization.k8s.io

Finally you need to define in Lenses configuration which namespaces can Lenses access. To achieve this amend values.yaml to contain the following:

lenses:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "[PROCESSORS NAMESPACE]"
        ]
      }      

example:

lenses:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "lenses-processors"
        ]
      }      

Careful monitoring of the data managed by the configured Schema Registry is paramount in order for Lenses to provide its users with the most up-to-date data.

For most cases, this monitoring doesn't cause any issues, but it might happen that, in some cases, Lenses is forced to access the Schema Registry too often.

If this happens, or if you want to make sure Lenses does not go over a rate limit imposed by the Schema Registry, it is possible to throttle Lenses usage of the Schema Registry's API.

In order to do so, it is possible to set the following Lenses configuration:

# schema registry
lenses.schema.registry.client.http.rate.type="sliding" 
lenses.schema.registry.client.http.rate.maxRequests= 200
lenses.schema.registry.client.http.rate.window="2 seconds"

# connect clusters
lenses.connect.client.http.rate.type="sliding"                 
lenses.connect.client.http.rate.maxRequests=200        
lenses.connect.client.http.rate.window="2 seconds"  

Doing so will make sure Lenses does not issue more than maxRequests over any window period.

The exact values provided will depend on things like the resources of the machine hosting the Schema Registry, the number of schemas, how often are new schemas added, so some trial and error is required. These values should however define a rate smaller than the one allowed by the Schema Registry.

All logs are emitted unbuffered as a stream of events to both stdout and to rotating files inside the directory logs/.

The logback.xml file is used to configure logging.

If customization is required, it is recommended to adapt the default configuration rather than write your own from scratch.

The file can be placed in any of the following directories:

• the directory where Lenses is started from

• /etc/lenses/

• agent installation directory.

The first one found, in the above order, is used, but to override this and use a custom location, set the following environment variable:

export LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/path/to/logback.xml"

Default configuration

The default configuration file is set up to hot-reload any changes every 30 seconds.

Log Level

The default log level is set to INFO (apart from some very verbose classes).

Log Format

All the log entries are written to the output using the following pattern:

%d{ISO8601} %-5p [%c{2}:%L] [%thread] %m%n

You can adjust this inside logback.xml to match your organization’s defaults.

Log Location

logs/ you will find three files: lenses.log, lenses-warn.log and metrics.log. The first contains all logs and is the same as the stdout. The second contains only messages at level WARN and above. The third one contains timing metrics and can be useful for debugging.

Log Buffering

The default configuration contains two cyclic buffer appenders: "CYCLIC-INFO” and “CYCLIC-METRICS”. These appenders are required to expose the Lenses logs within the Admin UI.

Lenses uses Websockets. It can be that your load balancers block them by default. Depending on your load balancer you need to allow websockets.

For example on NGINX:

annotations:
  kubernetes.io/ingress.class: nginx
  nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

Lenses can be placed behind a proxy, but you must allow websocket connections.

These two paths are used for WebSocket connections:

• /api/ws

• /api/kafka/ws

Disable proxy buffering for SSE (Server Sent Events) connections on this path:

• /api/sse

TLS termination

Lenses supports TLS termination out of the box, see Enabling TLS

Sample Apache configuration

# Add these settings to your httpd.conf or under the VirtualHost section
# for Lenses.
# The rewrite directives need the rewrite module:
#   LoadModule rewrite_module modules/mod_rewrite.so
# The proxy directives need the proxy, proxy_http and proxy_wstunnel modules:
#   LoadModule proxy_module modules/mod_proxy.so
#   LoadModule proxy_http_module modules/mod_proxy_http.so
#   LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so

RewriteEngine On
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule ^/(.*)$           ws://lenses.url:9991/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket [NC]
RewriteRule ^/(.*)$           http://lenses.url:9991/$1 [P,L]

ProxyRequests On
ProxyPreserveHost On
ProxyPass / http://lenses.url:9991/
ProxyPassReverse / http://lenses.url:9991/

Sample Caddy configuration

proxy /api/kafka/ws http://lenses.url:9991 {
    websocket
}
proxy /api/ws http://lenses.url:9991 {
    websocket
}
proxy / http://lenses.url:9991

Sample NGINX configuration

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen 80;
    server_name example.lenses.url;

    # websocket paths
    location /api/ws {
        proxy_pass http://lenses.url:9991;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_redirect off;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  Host $http_host;
    }
    location /api/kafka/ws {
        proxy_pass http://lenses.url:9991;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_redirect off;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  Host $http_host;
    }

    # SSE paths
    location /api/sse {
        proxy_pass http://lenses.url:9991;
        proxy_http_version 1.1;

        proxy_buffering off;
        proxy_redirect off;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  Host $http_host;
    }

    # all other paths
    location / {
        proxy_pass http://lenses.url:9991;
        proxy_http_version 1.1;

        proxy_redirect off;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  Host $http_host;
    }
}

Basics

Reference documentation of all configuration and authentication options:

Default system topics

System or control topics are created by services for their internal use. Below is the list of built-in configurations to identify them.

• _schemas

• __consumer_offsets

• _kafka_lenses_

• lsql_*

• lsql-*

• __transaction_state

• __topology

• __topology__metrics

• _confluent*

• *-KSTREAM-*

• *-TableSource-*

• *-changelog

• __amazon_msk*

Wildcard (*) is used to match any name in the path to capture a list of topics not just one. When the wildcard is not specified, Lenses matches on the entry name provided.

Security

TLS

LDAP

LDAP or AD connectivity is optional. All settings are string.

An additional configuration setting lenses.security.ldap.use.service.user.search when set to true will use the lenses.security.ldap.user account to read the groups of the currently logged user. The default behaviour (false) uses the currently logged user to read group memberships.

SSO SAML

Kerberos

Persistent storage

Common

H2

Postgres

Schema registries

If the records schema is centralized, the connectivity to Schema Registry nodes is defined by a Lenses Connection.

There are two static config entries to enable/disable the deletion of schemas:

Deployments

Options for specific deployment targets:

• Global options

• Kubernetes

Global options

Common settings, independently of the underlying deployment target:

Kubernetes

Kubernetes connectivity is optional. Minimum supported K8 version 0.11.10. All settings are string.

SQL snapshot (Explore & Studio)

Optimization settings for SQL queries.

Lenses internal Kafka topics

Lenses requires these Kafka topics to be available, otherwise, it will try to create them. The topics can be created manually before Lenses is run, or allow Lenses the correct Kafka ACLs to create the topics:

To allow for fine-grained control over the replication factor of the three topics, the following settings are available:

Advanced

All time configuration options are in milliseconds.

Connectors topology

Control how Lenses identifies your connectors in the Topology view. Catalogue your connector types, set their icons, and control how Lenses extracts the topics used by your connectors.

Lenses comes preconfigured for some of the popular connectors as well as the Stream Reactor connectors. If you see that Lenses doesn’t automatically identify your connector type then use the lenses.connectors.info setting to register it with Lenses.

Add a new HOCON object {} for every new Connector in your lenses.connectors.info list :

  lenses.connectors.info = [
      {
        class.name = "The connector full classpath"
        name = "The name which will be presented in the UI"
        instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
        sink = true
        extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
        icon = "file.png"
        description = "A description for the connector"
        author = "The connector author"
      }
  ]

This configuration allows the connector to work with the topology graph, and also have the RBAC rules applied to it.

Source example

To extract the topic information from the connector configuration, source connectors require an extra configuration. The extractor class should be: io.lenses.config.kafka.connect.SimpleTopicsExtractor. Using this extractor requires an extra property configuration. It specifies the field in the connector configuration which determines the topics data is sent to.

Here is an example for the file source:

  lenses.connectors.info = [
    {
      class.name = "org.apache.kafka.connect.file.FileStreamSource"
      name = "File"
      instance = "file"
      sink = false
      property = "topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
    }
  ]

Sink example

An example of a Splunk sink connector and a Debezium SQL server connector

  lenses.connectors.info = [
    {
      class.name = "com.splunk.kafka.connect.SplunkSinkConnector"
      name = "Splunk Sink",
      instance = "splunk.hec.uri"
      sink = true,
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "splunk.png",
      description = "Stores Kafka data in Splunk"
      docs = "https://github.com/splunk/kafka-connect-splunk",
      author = "Splunk"
    },
    {
      class.name = "io.debezium.connector.sqlserver.SqlServerConnector"
      name = "CDC MySQL"
      instance = "database.hostname"
      sink = false,
      property = "database.history.kafka.topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "debezium.png"
      description = "CDC data from RDBMS into Kafka"
      docs = "//debezium.io/docs/connectors/mysql/",
      author = "Debezium"
    }
  ]

External Applications