Security Configuration

Lenses allows you to easily secure access to the Lenses API and UI. You can password protect the UI and API as well as allow service accounts. More advanced features include LDAP and Kerberos authentication, role and group based access control, topic whitelisting/blacklisting. This section describes how to configure the security setup.

Lenses security options start with lenses.security.*** and should be stored in a separate file, named security.conf. You can set the path to the security configuration file in the main configuration file lenses.conf via the key lenses.secret.file. This way security.conf can be managed only by the administration team and have more tight access control than the rest of the configuration.

Lenses has support for the following login modes: BASIC, LDAP, KERBEROS and CUSTOM HTTP. The security mode is configured through the lenses.security.mode option. The permission matrix contains additional details on roles and access levels.

BASIC

If BASIC mode is set, then user accounts and groups are stored in the configuration file itself. Users are assigned one or more groups and this determines which roles they are granted. Users and groups are set via the lenses.security.users and lenses.security.groups options respectively. For example:

# Security by default to is set to BASIC, alternatively LDAP.
lenses.security.mode=BASIC

# Define the user groups and their roles. At least one user group needs to be set
lenses.security.groups=[
     {"name": "adminGroup", "roles": ["admin", "write", "read"]},
     {"name": "writeGroup", "roles": ["read", "write"], topic: { blacklist: ["payment.*"] },
     {"name": "readGroup",  "roles": ["read"], topic: { whitelist: [ "users.*" ] },
     {"name": "nodataGroup",  "roles": ["nodata"]}
]

#Define the users and link each one to the group(-s) it belongs
lenses.security.users=[
  {"username": "admin", "password": "admin999", "displayname": "Lenses Admin", "groups": ["adminGroup"]},
  {"username": "write", "password": "write1", "displayname": "Write User", "groups": ["writeGroup"]},
  {"username": "read", "password": "read1", "displayname": "Read Only", "groups": ["readGroup"]},
  {"username": "nodata", "password": "nodata1", "displayname": "No Data", "groups": ["nodataGroup"]}
]

The admin role inherits automatically the read and write privileges. The write role inherits read privilege.

Security modes require lenses.security.groups to be specified. In addition to specifying the roles for each group, you can optionally specify which topics can be accessed through a whitelist/blacklist approach. In the example above, the writeGroup will get to access all topics apart from the ones starting with payment. The readGroup has been set using the whitelisting approach, and will only allow access to topics with names starting with users. The entries for the topic.whitelist and topic.blacklist are expected to be regular expressions.

The permission matrix contains additional details on roles and access levels.

Note

Lenses offers topic centric security; you can control which topics a user group can access. A simple white/black list configuration enables you to handle multi-tenancy over Kafka. Leaving out the setting means the group will be able to access all topics.

LDAP

It is common for enterprises to use an LDAP server for user management and Lenses can integrate with your LDAP environment for user authentication and group assignment.

If you want to use LDAP support then the lenses.security.mode option must be set to LDAP and the ldap configuration section must be completed to provide the settings.

Each individual LDAP setup will assign membership to groups using a different hierarchy, therefore there is a plugin to allow custom implementations for retrieving the user group list.

The project template for a custom implementation can be found on Github. With the implementation ready, all that is required is to drop the jar file into Lenses lib folder and set the configuration entry lenses.security.ldap.plugin to point to the implementation full classpath.

Lenses provides out-of-the-box a default implementation via com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin class. Here is the template for the LDAP configuration section:

lenses.security.mode=LDAP
lenses.security.ldap.url="ldaps://mycompany.com:636"
lenses.security.ldap.base="OU=Users,DC=mycompany,DC=com"
lenses.security.ldap.user="$LDAP_USER"
lenses.security.ldap.password="$LDAP_USER_PASSWORD"
lenses.security.ldap.filter="(&(objectClass=person)(sAMAccountName=<user>))"

//LDAP roles retriever settings
lenses.security.ldap.plugin.class="com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin"
lenses.security.ldap.plugin.group.extract.regex="(?i)CN=(\\w+),ou=ServiceGroups.*"
lenses.security.ldap.plugin.memberof.key="memberOf"
lenses.security.ldap.plugin.person.name.key = "sn"
Key Description Optional Type Default
url
The LDAP server url.
For example: ldap://mycompany.com:10389
No String N/A
base
Your LDAP base.
For example: dc=jboss,dc=org
No String N/A
user
Your LDAP user.
For example: uid=admin,ou=system
No String N/A
password Your LDAP user password. No String N/A
filter
The LDAP search filter - must result in
a unique result. See default value. <user>
is required since is replaced
at runtime with the current
user id.
Yes String (&(objectClass=person)(sAMAccountName=<user>))
plugin.class
Contains the full classpath for
the LDAP roles retriever implementation
Yes string N/A
plugin.memberof.key
Your LDAP member of key entry.
This is the key for which a
role is attached to the user entry.
For example,
memberOf: cn=AdminR,ou=Groups,dc=jboss,dc=org
- links the AdminR role to the
current user entry.
Yes String memberOf
plugin.person.name.key
Your LDAP person entry attribute
containing the user full name.
The default value if the configuration is
not provided is sn.
yes string sn
plugin.group.extract.regex
The regular expression syntax to extra
the role for each memberof entry.
The default value matches the
earlier example for memberof.
Yes String (?i)CN=(\\w+),ou=Groups.*

Note

The configuration entries lenses.security.ldap.plugin.memberof.key, lenses.security.ldap.plugin.person.name.key and lenses.security.ldap.plugin.group.extract.regex are specific to the implementation Lenses provides out of the box. Any custom implementation may require different entries under lenses.security.ldap.plugin

Here is a sample configuration LDAP enabled Lenses:

Example of a user entry in LDAP server:

  # Lenses User Example with name userR, password userpsk and member of groupR
  dn: uid=userR,ou=Users,dc=landoop,dc=com
  objectClass: top
  objectClass: person
  objectClass: organizationalPerson
  objectClass: inetOrgPerson
  uid: userR
  cn: userR
  sn: groupR
  givenName: User Read
  title: Lenses Read User
  userPassword: userpsk
  description: Lenses User with read only permission
  memberOf: cn=groupR,ou=Group,dc=landoop,dc=com

Example of security.conf setup to use LDAP:

lenses.security.groups=[
  {"name": "admin", "roles": ["admin"]},
  {"name": "groupR", "roles": ["read"]},
  {"name": "groupW", "roles": ["write"]},
  {"name": "groupND", "roles": ["nodata"]}
  ]

lenses.security.mode=LDAP
lenses.security.ldap.url="ldaps://landoop.ldap.url:636"
lenses.security.ldap.base="DC=landoop,DC=com"
lenses.security.ldap.password=*****
lenses.security.ldap.user="UID=smiths,OU=Services,DC=landoop,DC=com"
lenses.security.ldap.filter="(&((objectclass=person)(CN=<user>)))"

lenses.security.ldap.plugin.class="com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin"
lenses.security.ldap.plugin.memberof.key="memberOf"
lenses.security.ldap.plugin.group.extract.regex="(?i)CN=(\\w+),ou=Group,dc=landoop,dc=com$"
lenses.security.ldap.plugin.person.name.key ="sn"

From the above example we setup Lenses with the LDAP URL and account. These are mandatory options required by Lenses in order to connect to LDAP and query for users.

Note

The Lenses LDAP account is used to query users, list their attributes and depending on the custom plugin to query other objects as well, such as groups. Please make sure the Lenses LDAP account has the permissions needed to perform its task.

The lenses.security.ldap.filter is what we should add to the base name so we can construct the distinguished name (DN) of the user. The <user> part is replaced by Lenses with your login handle. As an example, if your user’s DN is cn=Mark,ou=users,dc=example,dc=com, then you would set lenses.security.ldap.base to ou=users,dc=example,dc=com and lenses.security.ldap.filter to cn=<user>.

After a successful authentication, lenses proceeds by using the options from lenses.security.ldap.plugin.class and on to get the groups that the filtered user belongs. More specifically in this example we use the memberof class which appends in the ldapsearch also the lenses.security.ldap.plugin.memberof.key and then proceeds by filtering the result using lenses.security.ldap.plugin.group.extract.regex of which the returned value should match lenses.security.ldap.plugin.person.name.key=.

The regex should return a single string which must match with one of the groups that have been set above. In our case, if we authenticate with adminR, the extended filter (+memberof) will return this:

dn: cn=adminR,ou=Users,dc=landoop,dc=com
memberOf: cn=groupR,ou=Group,dc=landoop,dc=com

Which in turn, should match to this

Full match  52-88   `cn=groupR,ou=Group,dc=landoop,dc=com`
Group 1.    55-61   `groupR`

In the LDAP example above we have set sn=groupR, which is a match with our returned Group 1. groupR string. The group also matches one of the 4 groups that we set in the example and therefore the authentication is completed and the user is now redirected to the dashboard.

Kerberos

A popular solution for single sign on is Kerberos, which is often used in corporate environments. Clients obtain tickets from the Kerberos Key Distribution Centre (KDC) for a particular service and then that ticket is sent to the service when the client needs to authenticate against it.

If you want to enable Kerberos support then the lenses.security.mode option must be set to KERBEROS and the kerberos configuration section must be completed to provide the specific kerberos settings. The following table enumerates the kerberos config settings.

Key Description Optional Type Default
servicePrincipal The Lenses service principal. For example, HTTP/hostname No String N/A
keytab The path to the location of the service keytab No String N/A
debug Set to true to output Java’s JAAS debugging information Yes Boolean false

Here is a sample configuration for Kerberos enabled Lenses:

lenses.security.mode=KERBEROS
lenses.security.kerberos.service.principal="HTTP/lenses.host"
lenses.security.kerberos.keytab=/tmp/lenses.keytab
lenses.security.kerberos.debug=true

One of the limitations of Kerberos is that it is purely an authentication system for user principals without support for group membership. Therefore, in order to combine authentication with authorization it is necessary to map users to groups in a similar manner to the BASIC mode.

This is simply a mapping from username to group(s) specified in the security config, as in the following example:

#Define the users and link each one to the group(-s) it belongs
lenses.security.mappings = [
  { "username": "sam@LANDOOP.COM", "groups": ["adminGroup", "userGroup"] },
  { "username": "tom@LANDOOP.COM", "groups": ["userGroup"] }
]

Once Kerberos is enabled you should be implicitly logged in whenever you visit the lenses UI or directly to one of the REST API’s such as /api/auth. If you are using Kerberos on a windows system, then logging into your windows domain is usually sufficient to issue your Kerberos credentials. On a Linux environment, you will typically need to login manually using kinit at the command line before visiting any Kerberos enabled service.

Custom HTTP

In this mode, it’s possible to separate the authentication layer from Lenses into your own authentication solution. As an example, it would be possible to use JSON Web Tokens (JWT) injected via a proxy.

When setting CUSTOM_HTTP, a user-implemented class should be provided, that will be used to extract (and ideally verify) authentication and authorization information from the client HTTP headers.

Here is an example of the security configuration in this case:

lenses.security.mode=CUSTOM_HTTP
lenses.security.plugin=$my.custom.plugin.class.path

# Define the user groups and their roles. At least one user group needs to be set
lenses.security.groups=[
    {"name": "adminGroup", "roles": ["admin", "write", "read"]},
    {"name": "writeGroup", "roles": ["read", "write"], topic: { blacklist: ["payment.*"] },
    {"name": "readGroup",  "roles": ["read"], topic: { whitelist: [ "users.*" ] },
    {"name": "nodataGroup",  "roles": ["nodata"]}
]

Alongside the lenses.security.mode this configuration key lenses.security.plugin is required to be set. The value should be the full Java classpath to the class implementing the interface and drop the resulted Jar into the Lenses lib folder:

public interface HttpAuthenticationPlugin {
    UserAndGroups authenticate(HttpRequest request);
}

The return object UserAndGroups should contain the username accepted to log in and the user-group names the person belongs to, or raise an exception if the user is not allowed. To implement the interface, you need to create a project where this maven dependency needs to be added (here is the example for a Maven project):

<dependency>
    <groupId>com.landoop</groupId>
    <artifactId>lenses-security-http-plugin</artifactId>
    <version>1.0.0</version>
</dependency>

A sample implementation can be found on GitHub. There, the code expects a header API_KEY based on which a lookup to a mocked user-store is performed.

Example based on the above Github repository.

If you check the HeaderTokenAuthPlugin.java file inside the repository you will notice that our package has a classpath of io.lenses.security.auth.http.custom, therefore the full java classpath would be io.lenses.security.auth.http.custom.HeaderTokenAuthPlugin.

/*
First line of HeaderTokenAuthPlugin.java
*/

package io.lenses.security.auth.http.custom;

The expected header key is "API_KEY"

/*
First entry under public class in HeaderTokenAuthPlugin.java
*/

public class HeaderTokenAuthPlugin implements HttpAuthenticationPlugin {

  private static final String API_HEADER_NAME = "API_KEY";

The expected key values are one of [KEY-ADMIN, KEY-WRITE, KEY-READ]. Each value links to a different pair of user & group as shown under UserAndGroups.java.

/*
Entries under public class in UserAndGroups.java
*/

public class UserStore {

    private final Map<String, UserAndGroups> users = new HashMap<>();

    {
        users.put("KEY-ADMIN", new UserAndGroups("AdminUser", Collections.singleton("adminGroup")));
        users.put("KEY-WRITE", new UserAndGroups("WriteUser", Collections.singleton("writeGroup")));
        users.put("KEY-READ", new UserAndGroups("ReadUser", Collections.singleton("readGroup")));
        users.put("KEY-NODATA", new UserAndGroups("NoData", Collections.singleton("nodataGroup")));
    }

The security.conf syntax for the example above should be:

lenses.security.mode=CUSTOM_HTTP
lenses.security.plugin=io.lenses.security.auth.http.custom.HeaderTokenAuthPlugin

# Define the user groups and their roles. At least one user group needs to be set
lenses.security.groups=[
    {"name": "adminGroup", "roles": ["admin", "write", "read"]},
    {"name": "writeGroup", "roles": ["read", "write"], topic: { blacklist: ["payment.*"] },
    {"name": "readGroup",  "roles": ["read"], topic: { whitelist: [ "users.*" ] },
    {"name": "nodataGroup",  "roles": ["nodata"]}
]

Also, for lenses 2.2, the jar file must be placed under /opt/lenses/plugins/security.

To test that everything works as expected, execute the following curl command.

###
# curl --header "API_KEY: KEY-X" --request GET lenses-hostname:[PORT]/api/auth -w "\n" --compressed
# Example with lenses-hostname = localhost, PORT = 9991 & KEY=KEY-ADMIN
###

curl --header "API_KEY: KEY-ADMIN" --request GET localhost:9991/api/auth -w "\n" --compressed

{"user":"AdminUser",...,"permissions":["nodata","admin","read","write"],"token":"ee61..."}

Service Accounts

Service accounts allow easier integration with Lenses API. A typical use case is enabling your CI/CD tools to interact with Lenses. Via the lenses.security.service.accounts specific users and their authorization token can be defined. Here is how two service accounts can be created:

lenses.security.groups=[
      {"name": "group1", "roles": ["admin", "write", "read"]},
      {"name": "group2", "roles": ["read", "write"], topic: { blacklist: ["payment.*"] },
      {"name": "group3",  "roles": ["read"], topic: { whitelist: [ "users.*" ] },
]

lenses.security.service.accounts=[
  {
    "username": "jenkins",
    "token": "1231543kn!!1",
    "groups": ["group1", "group2"]
  },
  {
    "username": "goCli",
    "token": "12345678",
    "groups": ["group3"]
  }
]

Your CI system (like Jenkins) and the Lenses Go CLI tool can call into the API without having to first login. All that is required is for every HTTP request to contain the HTTP header: X-Kafka-Lenses-Token:$TOKEN.

Each service account is of course linked to a user group in order to restrict the actions it can execute.

# Example of listing Alerts with service account: jenkins (see above)

curl --header "X-Kafka-Lenses-Token: 1231543kn!!1" --request GET  http://localhost:9991/api/alerts --compressed -w "\n"

[{"endsAt":null,"startsAt":"2018-11-24T15:14:03.105Z","alertId":1002,...,
"labels":{"category":"Infrastructure",...,"generatorURL":"0.0.0.0:9991"}]

Security Permissions Matrix

Below you will find the complete list of supported permissions that may be applied to a group. Permission names are case insensitive. The NoData general permission is also granted by any non-general permission.

Permission Description Notes
General Permissions    
Admin Extends the actions that can be performed in lenses also grants Write and Read
Write Required in order to create and modify, topics, processors, connectors, ... also grants Write
Read Required in order to view data
NoData This permission removes the ability to view data also granted by any non-general permission
Policy Permissions    
DataPolicyWrite Allows to add, delete & modify polices also grants DataPolicyRead
DataPolicyRead Allows to view polices
DataPolicyDisabled Avoids applying the data policy rules when running the SQL queries
Alert Permissions    
AlertsWrite Allows to add, delete and modify alerts also grants AlertsRead
AlertsRead Allows to view the alerts that have been set
Table Storage Permissions    
TableStorageWrite Allows setting the Kafka topics Key and Value storage format also grants TableStorageRead
TableStorageRead Allows reading the Kafka topics Key and Value storage format