Security Configuration

Once you configured Lenses, it’s time to setup security as well. The application cannot work without a security config in place.

Security has two important parts: authentication and groups. Authentication is how users login to Lenses. Currently the supported modes are Basic, LDAP, Kerberos (SPNEGO) and Custom HTTP where a user provided class can provide authentication details based on HTTP Headers. There is also support for service accounts when automated access is needed, like in configuration management and CI/CD tools.

Groups have to do with authorization. Lenses provides permissions (called roles) for various actions, like read topic data. A group is a collection of roles and optionally a white or blacklist of topics the group has access to. When a user logs in, one or more groups are assigned to provide the combined permissions this user has. The group mapping depends on the authentication method.

The security configuration is driven by a separate file, by default security.conf —but may be set to anything inside lenses.conf, via the lenses.secret.file option. The reason for a separate file, is that it may contain sensitive information and have to be to handled appropriately; e.g readable only by the Lenses process user. Every option under lenses.security. should go into the security.conf file.

For a quickstart example, have a look at the basic mode sample setup.

User Authentication

The authentication mode is driven by the lenses.security.mode option. You will find information and examples on the different modes below. Please also take the time to familiarize with Groups and Roles.

Security Mode
Key Description Optional Type Default
lenses.security.mode
Security mode to use. One of:
  • BASIC
  • LDAP
  • KERBEROS
  • CUSTOM_HTTP
no string (enum) n/a

Basic

In BASIC mode the user accounts and groups mapping are stored in the configuration file itself. Users are directly assigned one or more groups and this determines which roles are granted. Users and groups are set via the lenses.security.users and lenses.security.groups options respectively. For more information on groups see Groups and Roles.

# Security by default to is set to BASIC:

lenses.security.mode=BASIC

# In BASIC mode users are added here and are directly mapped to groups:

lenses.security.users = [
  {"username": "admin", "password": "admin", "displayname": "Administrator", "groups": ["adminGroup"]},
  {"username": "read", "password": "read", "displayname": "Reader", "groups": ["readGroup"]}
]

# Set user groups and their roles. At least one user group needs to be set:

lenses.security.groups = [
  {
    "name": "adminGroup",
    "roles": ["Admin", "AlertsWrite", "TableStorageWrite", "DataPolicyWrite"]
  },
  {
    "name": "readGroup",
    "roles": ["Read", "AlertsRead", "TableStorageRead"],
    "topic": {"blacklist": ["payments.*"]}
  }
]

LDAP

In this mode, Lenses queries an LDAP server for user authentication and groups. Active Directory (AD) and OpenLDAP (with the memberOf overlay) servers are tested and supported in general. Due to the LDAP standard ambiguity it is impossible to support all the configurations in the wild. The most usual pain point is the group mapping. If the default class that extracts and maps LDAP groups to Lenses groups does not work, it is possible to implement your own.

Before setting up an LDAP connection, we advise to familiriaze with LDAP and/or have access to your LDAP and/or Active Directory administrators.

An LDAP setup example is shown below:

lenses.security.mode=LDAP

# LDAP connection details

lenses.security.ldap.url="ldaps://example.com:636"
## For the LDAP user please use the distinguished name (DN).
## The LDAP user must be able to list users and their groups.
lenses.security.ldap.user="cn=lenses,ou=Services,dc=example,dc=com"
lenses.security.ldap.password="[PASSWORD]"

# LDAP user search settings

lenses.security.ldap.base="ou=Users,dc=example,dc=com"
lenses.security.ldap.filter="(&(objectClass=person)(sAMAccountName=<user>))"

# LDAP group search and mapping settings

lenses.security.ldap.plugin.class="com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin"
lenses.security.ldap.plugin.group.extract.regex="(?i)CN=(\\w+),ou=Groups.*"
lenses.security.ldap.plugin.memberof.key="memberOf"
lenses.security.ldap.plugin.person.name.key = "sn"

# Lenses Groups (not LDAP specific)

lenses.security.groups=[
  {"name": "LensesAdmin", "roles": ["Admin"]}
]

In the example above you can distinguish three key sections for LDAP:

  • the connection settings,
  • the user search settings,
  • and the group search and mapping settings.

Each key is explained in detail at the LDAP Configuration Options Table.

Lenses uses the connection settings to connect to your LDAP server. The account provided should be able to list users under the base path and their groups. The default group plugin only needs access to the memberOf attributes for each user, but your custom implementation may need different permissions.

When a user tries to login, a query is sent to the LDAP server for all accounts that are under the lenses.security.ldap.base and match the lenses.security.ldap.filter. The result needs to be unique; a distinguished name (DN) —the user that will login to Lenses.

In the example, the application would query the LDAP server for all entities under ou=Users,dc=example,dc=com that satisfy the LDAP filter (&(objectClass=person)(sAMAccountName=<user>)) where <user> would be replaced by the username that tries to login to Lenses. A more simple filter could be cn=<user>, which for user Mark would return the DN cn=Mark,ou=Users,dc=example,dc=com.

Once the user has been verified, Lenses queries for the users’ groups and tries to map them to Lenses groups. For every LDAP group that matches a Lenses group, the user is granted the permissions of it. This process is done by the Group Extract Plugin.

Group Extract Plugin

The group extract plugin is a class that implements an LDAP query that retrieves a user’s groups and makes any necessary transformation to match the LDAP group to a Lenses group name.

The default class implementation that comes with Lenses is com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin. If your LDAP server supports the memberOf functionality, where each user has his/her group memberships added as attributes to his/her entity, you can use it by setting the lenses.security.ldap.plugin.class option to this class:

lenses.security.ldap.plugin.class=com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin

The configuration settings for the default group extract plugin can be found at the LDAP Configuration Options Table. Below you will see a brief example of its setup.

# Set the full classpath that implementes the group extraction
lenses.security.ldap.plugin.class="com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin"

# The plugin uses the 'memberOf' attribute. If this attribute has a different
# name in your LDAP, set it here.
lenses.security.ldap.plugin.memberof.key="memberOf"

# This regular expression should return the group common name. If it matches
# a Lenses group name, the user is granted its permissions.
# As an example if there is a 'memberOf' attribute with value:
#   cn=LensesAdmins,ou=Groups,dn=example,dn=com
# The regular expression will return 'LensesAdmins'.
# Group names are case sensitive.
lenses.security.ldap.plugin.group.extract.regex="(?i)cn=(\\w+),ou=Groups.*"

# This is the LDAP attribute that holds the user's full name. It's optional.
lenses.security.ldap.plugin.person.name.key = "sn"

As an example, the memberOf search may return two attributes for user Mark:

attribute  value
---------  ------------------------------------------
memberOf   cn=LensesAdmin,ou=Groups,dc=example,dc=com
memberOf   cn=RandomGroup,ou=Groups,dc=example,dc=com

The regular expression (?i)cn=(\\w+),ou=Groups.* will return these two regex group matches:

LensesAdmin
RandomGroup

If any of these groups exists in Lenses, Mark will be granted the permissions of the matching groups.

To learn more about permissions in Lenses, see the Groups and Roles section.

Custom LDAP Plugin

If your LDAP doesn’t offer the memberOf functionality, or it isn’t enough alone —for example in Active Directory there are groups of groups— you can provide your own implementation.

The project template for a custom implementation can be found on Github (Lenses LDAP Plugin Template) . Once you build your implementation, drop the jar file into the plugins directory and set the lenses.security.ldap.plugin setting to point to your implementation’s full classpath.

Don’t forget to grant to the Lenses LDAP account any permissions it may need for your plugin to work.

Options Table

LDAP Configuration Options Table
Key Description Optional Type Default
lenses.security.ldap.url
The LDAP server url. TLS, StartTLS and
unencrypted connections are supported.
Example: ldaps://example.com:636
no string “n/a”
lenses.security.ldap.user
The LDAP account for Lenses. Must be able to
list users and their groups. The distinguished
name (DN) must be used. Example:
cn=lenses,ou=Services,dc=example,dc=com
no string n/a
lenses.security.ldap.password The LDAP account password. so string n/a
lenses.security.ldap.base
The LDAP base path for querying user accounts.
All user accounts that will be able to access
Lenses should be under this path.
Example: ou=Users,dc=example,dc=com
no string n/a
lenses.security.ldap.filter
The LDAP query filter for matching users. Lenses
will request all entries under the base path
that satisfy this filter. The result should be
unique: the user that logs in to Lenses. The
keyword <user> is replaced at runtime with
the Username that requests access.
yes string
(&
 (objectClass=person)
 (sAMAccountName=<user>)
)
lenses.security.ldap.plugin.class
The full classpath for the class that implements
the LDAP query for the user’s groups and
maps them to Lenses groups. You can use
if your LDAP setup is supported.
yes string n/a
lenses.security.ldap.plugin.memberof.key
This key is used by the included LDAP plugin class
LdapMemberOfUserGroupPlugin. It expects the
LDAP user attribute that provides memberOf
information. In most implementations the attribute
has the same name, so you don’t have to set
anything.
yes string memberOf
lenses.security.ldap.plugin.group.extract.regex
This key is used by the included LDAP plugin class
LdapMemberOfUserGroupPlugin. It expects a
regular expression that will be used to extract
a part of the user’s groups. If this part matches a
Lenses group, the user will be granted all the
permissions of this group.
Lenses checks against the list of memberOf
attribute values and uses the first regex group
that is returned.
yes string (?i)CN=(\\w+),ou=Groups.*
lenses.security.ldap.plugin.person.name.key
This key is used by the included LDAP plugin class
LdapMemberOfUserGroupPlugin. It expects the
LDAP user attribute that provides the full name
of the user.
yes string sn

Note

The configuration entries lenses.security.ldap.plugin.memberof.key, lenses.security.ldap.plugin.person.name.key, lenses.security.ldap.plugin.group.extract.regex, and lenses.security.ldap.plugin.person.name.key are specific to the default plugin class that comes with Lenses. A custom implementation may require different entries under lenses.security.ldap.plugin

Kerberos (SPNEGO)

In Kerberos mode, your browser will use the Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) to login to Lenses.

Before setting up Lenses with SPNEGO, it is important to make sure your browser does support this mechanism, you are adequately familiar with the protocol and you have access to your Kerberos and/or Active Directory administrators.

Let’s look at an example of a Kerberos security setup:

lenses.security.mode=KERBEROS

# Kerberos settings

lenses.security.kerberos.service.principal="HTTP/lenses.url[@REALM]"
lenses.security.kerberos.keytab=/path/to/lenses.keytab

# Kerberos user to groups mapping

lenses.security.mappings = [
  {"username": "mark@EXAMPLE.COM", "groups": ["LensesAdmin"]}
]

# Group settings (not Kerberos specific)

lenses.security.groups = [
  {"name": "LensesAdmin", "roles": ["Admin"]}
]

The SPNEGO-specific configuration is straightforward, you have to point Lenses to a password-less Kerberos keytab and specify the principal to use.

Your system should also provide a system-wide Kerberos configuration. Usually, for Linux distributions this resides in the file /etc/krb5.conf which is populated with information about your KDC, Kerberos realm and other authentication-related settings. If there isn’t a global krb5.conf that Lenses can use, please ask your Kerberos administrator for one, then point Lenses to it via the LENSES_OPT environment variable:

export LENSES_OPTS="-Djava.security.krb5.conf=/path/to/krb5.conf

Once setup, your users should be automatically logged in whenever they visit the Lenses web interface or make a call to the /api/auth REST endpoint. If you are on a Microsoft Windows system, then logging into your Windows domain is usually sufficient to issue your Kerberos credentials. On a Linux environment, if you use Kerberos with PAM, your Kerberos credentials should be already available to kerberos-enabled browsers. Otherwise you will typically need to authenticate to the KDC manually using kinit at the command line and start your browser from the same terminal.

Group Mapping

A shortcoming of Kerberos is that by design it is a purely authentication mechanism. It doesn’t provide any information about the user it authenticates except the user’s principal. As such it falls to the Lenses administrator to map users to groups.

This is done by the lenses.security.mappings option. For each user you wish to give access to Lenses, you need to map at least one group:

# Define the users and link each one to the group(-s) it belongs:

lenses.security.mappings = [
  { "username": "mark@EXAMPLE.COM", "groups": ["adminGroup"] },
  { "username": "john@EXAMPLE.COM", "groups": ["userGroup", "alertsGroup"]}
]

To learn more about the permission model of Lenses, see the Groups and Roles section.

SPNEGO Principal

The Kerberos principal is not random in the SPNEGO protocol. When your browser tries to connect to an SPNEGO service it needs a ticket for it which is granted by the Key Distribution Center (KDC). In SPNEGO, the browser will always request a key from the KDC in the format HTTP/service.url@REALM. So, if your service is at service.example.com and your realm is EXAMPLE.COM, you browser will always request a ticket for the principal HTTP/service.example.com@EXAMPLE.COM. The realm usually can be ommited as it is part of the system-wide Kerberos settings.

To repeat one more time, if you setup Lenses at lenses.example.com, you need to ask your Kerberos or Active Directory administrator to create the principal HTTP/lenses.example.com and provide you with a password-less keytab for it.

Options Table

Kerberos Configuration Options Table
Key Description Optional Type Default
lenses.security.kerberos.service.principal
The Kerberos principal Lenses should use.
It must be present in the keytab and in
form below (SPNEGO requirement):
HTTP/lenses.address@REALM.COM
no string n/a
lenses.security.kerberos.keytab
Path to a Kerberos keytab that contains
the service principal. It should not
be password protected.
no string (path) n/a
lenses.security.kerberos.debug
Enable Java’s JAAS debugging information.
yes boolean false
lenses.security.mappings
Array of principal to group(s) mapping.
See Kerberos Group Mapping.
no string (array) n/a

Custom HTTP

In this mode a custom, user-provided class takes over authentication and authorization to Lenses using the users’ HTTP request headers. The authentication layer is separated from Lenses into your own authentication solution. As an example, it would be possible to use JSON Web Tokens (JWT) injected via an authentication-proxy in-front of Lenses or any other single sign-on tokens your organization may use. The user-implemented class should process the request HTTP headers to extract the tokens and any other information it needs, ideally verify them, then provide Lenses with the username and groups of the user.

An example of a security configuration with Custom HTTP:

lenses.security.mode=CUSTOM_HTTP

# Just one setting for CUSTOM_HTTP, the full classpath
# of the security plugin implementation

lenses.security.plugin=my.custom.plugin.class.path

# Group settings (not Custom HTTP specific)

lenses.security.groups=[
  {"name": "LensesAdmin", "roles": ["Admin"]}
]

The only available option for this mode is lenses.security.plugin, which should point to the full Java classpath of the class implementing the required interface (HttpAuthenticationPlugin). Once you build your code, you can drop the required jar(s) inside the plugins directory or a subdirectory under it.

The interface of the Custom HTTP class:

public interface HttpAuthenticationPlugin {
    UserAndGroups authenticate(HttpRequest request);
}

The return object UserAndGroups should contain the username accepted to log in and the groups the person belongs to or raise an exception if the user is not allowed. To learn more about the permissions model of Lenses, check Groups and Roles. To implement the interface, you need to create a project where this maven dependency needs to be added (here is the example for a Maven project):

<dependency>
    <groupId>com.landoop</groupId>
    <artifactId>lenses-security-http-plugin</artifactId>
    <version>1.0.0</version>
</dependency>

Example Code

A sample implementation can be found at GitHub. This code expects the header API_KEY based on which it performs a lookup to a mocked user-store. Let’s explore this example a bit further.

If you check the HeaderTokenAuthPlugin.java file inside the repository you will notice that the package has a classpath of io.lenses.security.auth.http.custom, therefore the full java classpath would be io.lenses.security.auth.http.custom.HeaderTokenAuthPlugin.

/*
First line of HeaderTokenAuthPlugin.java
*/

package io.lenses.security.auth.http.custom;

The expected header key is API_KEY [1]:

/*
First entry under public class in HeaderTokenAuthPlugin.java
*/

public class HeaderTokenAuthPlugin implements HttpAuthenticationPlugin {

  private static final String API_HEADER_NAME = "API_KEY";

The expected key values are one of [KEY-ADMIN, KEY-WRITE, KEY-READ]. Each value links to a different pair of user & group as shown under UserAndGroups.java.

/*
Entries under public class in UserAndGroups.java
*/

public class UserStore {

    private final Map<String, UserAndGroups> users = new HashMap<>();

    {
        users.put("KEY-ADMIN", new UserAndGroups("AdminUser", Collections.singleton("adminGroup")));
        users.put("KEY-WRITE", new UserAndGroups("WriteUser", Collections.singleton("writeGroup")));
        users.put("KEY-READ", new UserAndGroups("ReadUser", Collections.singleton("readGroup")));
        users.put("KEY-NODATA", new UserAndGroups("NoData", Collections.singleton("nodataGroup")));
    }

The security.conf syntax for the example above could be:

lenses.security.mode=CUSTOM_HTTP

lenses.security.plugin=io.lenses.security.auth.http.custom.HeaderTokenAuthPlugin

# Define the user groups and their roles. At least one user group needs to be set

lenses.security.groups=[
  {"name": "adminGroup", "roles": ["Admin"]},
  {"name": "writeGroup", "roles": ["Write"], topic: { blacklist: ["payment.*"] },
  {"name": "readGroup",  "roles": ["Read"], topic: { whitelist: [ "users.*" ] },
  {"name": "nodataGroup",  "roles": ["NoData"]}
]

The jar produced should be dropped inside the plugins directory. If you are using Lenses Box for development and testing, you may drop it under /plugins or /opt/lenses/plugins.

To test that everything works as expected, execute the following curl command.

curl --header "API_KEY: KEY-ADMIN" --compressed http://lenses.url/api/auth

The result should be the authentication response of Lenses:

{
  "user": "AdminUser",
  "schemaRegistryDelete": true,
  "permissions": [
    "datapolicyread",
    "nodata",
    "tablestoragewrite",
    "admin",
    "alertswrite",
    "tablestorageread",
    "read",
    "write",
    "datapolicywrite",
    "alertsread"
  ],
  "token": "1234ff43-2130-dabb-b33e-a4b909148eed"
}
[1]HTTP headers are case-insensitive. RFC 2616, section 4.2.

Service Accounts

When a user authenticates to Lenses, an authentication token is returned, which the browser uses for all subsequent requests instead of the user’s credentials. Service accounts allows to directly set authentication tokens in security.conf; they facilitate easier integration with the Lenses API. A typical use case is CI/CD or configuration management tools access, an important part of GitOps.

An example of a service accounts section:

# Service Accounts, similar to the BASIC authentication mode for users

lenses.security.service.accounts = [
  {
    "username": "jenkins",
    "token": "jenkins-token",
    "groups": ["group1", "group2"]
  },
  {
    "username": "lenses-cli",
    "token": "lenses-cli-token",
    "groups": ["group2"]
  }
]

# Group settings (not Service Accounts specific)

lenses.security.groups=[
  {
    "name": "group1",
    "roles": ["Admin"]
  },
  {
    "name": "group2",
    "roles": ["Read", "AlertsWrite"],
    "topic": {"blacklist": ["payment.*"]}
  }
]

Service accounts are set via the lenses.security.service.accounts option. Your CI system (like Jenkins) and the Lenses Go CLI tool can call into the API without having to first login. All that is required is for every HTTP request to contain the HTTP header: X-Kafka-Lenses-Token:[TOKEN]. Each service account is of course linked to a user group in order to restrict the actions it can execute.

These accounts are meant to be used by machines, no human needs to memorize a token. As such it is advised to keep the token long and complex enough to avoid any brute-force attack attempt.

To verify a service account, you can use curl. The example below gets the application’s logs via the API:

curl --header "X-Kafka-Lenses-Token: jenkins-token" --compressed http://lenses.url/api/logs/INFO

The result should be an array of log entries:

[
   {
     "timestamp": 1545091832442,
     "logger": "com.landoop.kafka.lenses.actors.ConnectConfigActor",
     "stacktrace": "",
     "thread": "default-akka.actor.default-dispatcher-28",
     "message": "Config connector event received. Name logs-broker in cluster dev",
     "time": "2018-12-18 00:10:32.442",
     "level": "INFO"
   },
   ...
]

Groups and Roles

The Lenses permission model uses four key-concepts: Users, Groups, Roles, and explicit topic Black/White Lists.

ROLES
Roles are the most granular permission level, examples include Read (data) and AlertsWrite. Roles can only be assigned to groups.
BLACK/WHITE LISTS
Black and whitelists refer to topics and apply to groups. A whitelist sets explicitly which topics a group can access. A blacklist sets explicitly which topics a group is forbidden to access.
GROUPS
Groups are collections of roles, and black and whitelists. Groups are assigned to users and service accounts and dictate which actions a user may or may not perform, as well as access to data, access to specific topics and more. A user can belong to more than one groups.
USERS
Users and Service Accounts are people or machines accessing Lenses. They may perform various actions, such as list topics, view data of a topic, create alerts and so on. Users are assigned one or more groups. What the user can and cannot do is dictated by the group the user belongs too.

Roles Matrix

Below you will find the complete list of supported roles (permissions) that may be applied to a group. Roles names are case insensitive. The NoData permission is granted implicitly by any other permission.

Permission Description Notes
General Permissions    
Admin Required to delete topics, processors, connectors, ..., edit ACLs and more also grants Write and Read
Write Required in order to create and modify, topics, processors, connectors, ... also grants Write
Read Required in order to view data
NoData This permission allows to list topics, processors, connectors, ... granted by any other permission
Policy Permissions    
DataPolicyWrite Allows to add, delete & modify polices also grants DataPolicyRead
DataPolicyRead Allows to view polices
DataPolicyDisabled Avoids applying the data policy rules when running the SQL queries
Alert Permissions    
AlertsWrite Allows to add, delete and modify alerts also grants AlertsRead
AlertsRead Allows to view the alerts that have been set
Table Storage Permissions    
TableStorageWrite Allows setting the Kafka topics Key and Value storage format also grants TableStorageRead
TableStorageRead Allows reading the Kafka topics Key and Value storage format

Security Group Syntax

Security groups are a mandatory option of lenses and at least one should always be present. If a user cannot be matched to any group the authentication will fail and the user will not have access to the Lenses API.

Groups are added as an array to the lenses.security.groups option. An example can be found below.

lenses.security.groups = [
  {
    "name": "adminGroup",
    "roles": ["Admin", "DataPolicyWrite", "AlertsWrite", "TableStorageWrite"]
  },
  {
    "name": "readGroup",
    "roles": ["Read", "AlertsRead"],
    "topic": {"blacklist": ["payments.*"]}
  }
]

Options Table

Group Configuration Options Table
Key Description Optional Type
name The group’s name. no String
roles A list of roles. no Array of strings
topic.blacklist
A list of regular expressions. Topics that match
any of them, will not be shown to the group members.
yes Array of regular expressions
topic.whitelist
A list of regular expressions. Only topics that match
any of them, will be shown to the group members.
Whitelist has a higher priority than a blacklist.
That means that if there is both a whitelist and a
blacklist exist, the blacklist will apply to the topics
that the whitelist permits the user to view.
yes Array of regular expressions

Black and Whitelist Example

Let’s assume a list of topics:

  • RegionA-Clients
  • RegionA-Orders
  • RegionA-Payments
  • RegionB-Clients
  • RegionB-Orders
  • RegionB-Payments
  • RegionC-Clients
  • RegionC-Orders
  • RegionC-Payments

And a list of groups:

lenses.security.grous = [
  {
    "name": "AccountantsGroup",
    "roles": ["Read"],
    "topic": {"whitelist": [".*Payments"]}
  },
  {
    "name": "RegionA_Administrators",
    "roles": ["Admin", "TableStorageWrite],
    "topic": {
      "blacklist": [".*Payments"],
      "whitelist": ["RegionA.*"],
    }
  },
  {
    "name": "ShippingGroup",
    "roles": ["Read"],
    "topic": {"blacklist": [".*Payments"]}
  }
]

The AccountantsGroup may only see the topics that end with Payments, due to the whitelist. As such it has access to RegionA-Payments, RegionB-Payments, RegionC-Payments topics. Any other topic will not be visible.

For the RegionA_Administrators, the whitelist allows them to only see topics from RegionA. The blacklist further restricts them from seeing payment topics. So they can only see RegionA-Clients and RegionA-Orders.

Finally the ShippingGroup, may see any topic except the blacklisted payments topics. So they can see the client and orders topics in all regions.