Upgrade Notes¶
To see the full list of changes made for this release check the Release Notes. Upgrading from the previous versions requires changes to the existing configuration file. This section covers all the changes to be made to the existing configuration file to make it work with this version.
This is a brief list of the important configuration options that have been altered and you should adjust before starting Lenses 2.0. Please read the full document for the full list and detailed examples for each change.
lenses.zookeeper.hosts, changedlenses.jmx.zookeepers, deprecatedlenses.zookeeper.chroot, added, no need to set if you don’t use a chrootlenses.zookeeper.security.enabled, added, important if you use SASL authentication (kerberos)lenses.schema.registry.urls, changedlenses.jmx.schema.registry, deprecatedlenses.connect.clusters, changedlenses.jmx.connect, deprecatedlenses.alert.consumers.lag.threshold, deprecated (in favor of new alerting system)lenses.security.users, changedlenses.security.groups, added (split fromlenses.security.users/ replaceslenses.security.ldap.roles.*)lenses.security.ldap.roles.*, deprecated (in favor oflenses.security.users)
If after an upgrade Lenses does not seem to detect JMX or show information about the brokers, please try
to delete the lenses.topics.cluster topic (by default set to _kafka_lenses_cluster) or set it to use a new topic.
The data in this topic are ephemeral, so this action won’t affect Lenses configuration and/or history.
Zookeeper Connection¶
In this release, the Zookeeper nodes connection and their JMX endpoints have been merged into one configuration entry. In addition, the (optional) zookeeper chroot entry has moved to its own configuration entry. When utilizing SASL (Kerberos) authentication to the brokers, the connection to zookeeper can optionally be authenticated as well.
Previous:
lenses.zookeeper.hosts="host.1:2181,host.2:2181,host.3:2181"
lenses.jmx.zookeepers="host.1:9585,host.2:9585,host.3:9585"
New:
lenses.zookeeper.hosts=[
{
url:"host.1:2181",
jmx:"host.1:9585"
},
{
url:"host.2:2181",
jmx:"host.2:9585"
},
{
url:"host.3:2181",
jmx:"host.3:9585"
}
]
lenses.zookeeper.chroot= "" # Optional in case a chroot path is in use, without leading slash
Zookeeper may optionally provide authentication via SASL GSSAPI (Kerberos). Zookeeper authentication in Kafka context is used in order to create nodes with (zookeeper) ACLs, which can only be written or deleted by the appropriate Kerberos principal.
By default, Lenses will not authenticate to Zookeeper. The behaviour is driven by a new configuration entry
lenses.zookeeper.security.enabled which by default is false. If this setting is enabled and authentication configuration
(via jaas.conf) is provided, Lenses will use Zookeeper ACLs when storing Kafka Quotas.
Warning
If lenses.zookeeper.security.enabled is set to true, then Lenses should be configured with the same Kerberos
principal as the brokers otherwise the brokers will fail to restart as they will be unable to manipulate the
Quota entries with their own Kerberos principal.
Schema Registry Connection¶
Following the trend mentioned earlier, the same changes have been applied for Schema Registry configuration.
Previous:
lenses.schema.registry.urls="http://host.1:8081,http://host.2:8081"
lenses.jmx.schema.registry="host.1:9582,host.2:9582"
New:
lenses.schema.registry.urls=[
{
url:"http://host.1:8081",
jmx:"host.1:9582"
},
{
url:"http://host.2:8081",
jmx:"host.2:9582"
}
]
Connect Configuration¶
Following the trend the following changes have been applied for the Connect cluster configuration details:
Previous:
lenses.connect.clusters=[
{
name: "dev",
url: "http://host.1:8083,http://host.2:8083,http://host.3:8083",
statuses: "connect-status-dev",
configs: "connect-configs-dev",
offsets: "connect-offsets-dev"
}
]
lenses.jmx.connect=[
{
dev: "host.1:9584,host.2:9584,host.3:9584"
}
]
New:
lenses.connect.clusters=[
{
name: "dev",
urls: [
{
url:"http://host.1:8083",
jmx:"host.1:9584"
},
{
url:"http://host.2:8083",
jmx:"host.2:9584"
},
{
url:"http://host.3:8083",
jmx:"host.3:9584"
}
],
statuses: "connect-status-dev",
configs: "connect-configs-dev",
offsets: "connect-offsets-dev"
}
]
Security¶
The concept of user groups has been added with this release. Therefore amendments to lenses.security are required.
Here is how to update the configuration when using BASIC authentication:
Previous:
lenses.security.mode = BASIC
lenses.security.users = [
{"username": "admin", "password": "admin", "displayname": "Lenses Admin", "roles": ["admin", "write", "read"]},
{"username": "write", "password": "write", "displayname": "Write User", "roles": ["read", "write"]},
{"username": "read", "password": "read", "displayname": "Read Only", "roles": ["read"]}
{"username": "nodata", "password": "nodata", "displayname": "No Data", "roles": ["nodata"]}
]
New:
lenses.security.mode=BASIC
lenses.security.groups=[
{"name": "adminGroup", "roles": ["admin", "write", "read"]},
{"name": "writeGroup", "roles": ["read", "write"]},
{"name": "readGroup", "roles": ["read"]},
{"name": "nodataGroup", "roles": ["nodata"]}
]
lenses.security.users=[
{"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]},
{"username": "write", "password": "write", "displayname": "Write User", "groups": ["writeGroup"]},
{"username": "read", "password": "read", "displayname": "Read Only", "groups": ["readGroup"]},
{"username": "nodata", "password": "nodata", "displayname": "No Data", "groups": ["nodataGroup"]}
]
If LDAP security mode is used, here is how to amend the current configuration:
Previous:
lenses.security.mode=LDAP
lenses.security.ldap.url="ldaps://LDAP_HOST:LDAP_HOST"
lenses.security.ldap.base="xxxx"
lenses.security.ldap.password="xxxx"
lenses.security.ldap.user="xxxx"
lenses.security.ldap.filter="(CN=<user>)"
lenses.security.ldap.roles.admin=[ "groupA" ]
lenses.security.ldap.roles.nodata=[ "groupB" ]
lenses.security.ldap.plugin.class="com.landoop.kafka.lenses.security.LdapMemberOfUserGroupPlugin"
lenses.security.ldap.plugin.group.extract.regex="(?i)CN=(\\w+),ou=ServiceGroups.*"
New:
lenses.security.mode=LDAP
lenses.security.groups=[
{"name": "groupA", "roles": ["admin", "write", "read"]},
{"name": "groupB", "roles": ["nodata"]}
]
lenses.security.ldap.url="ldaps://LDAP_HOST:LDAP_HOST"
lenses.security.ldap.base="xxxx"
lenses.security.ldap.password="xxxx"
lenses.security.ldap.user="xxxx"
lenses.security.ldap.filter="(CN=<user>)"
lenses.security.ldap.plugin.class="com.landoop.kafka.lenses.security.LdapMemberOfUserGroupPlugin"
lenses.security.ldap.plugin.group.extract.regex="(?i)CN=(\\w+),ou=ServiceGroups.*"
Deprecated configuration¶
Part of this release the following configuration entries have been deprecated, so you can remove them from your configuration file.
lenses.alert.consumers.lag.threshold=...
lenses.security.ldap.roles.admin = [...]
lenses.security.ldap.roles.read = [...]
lenses.security.ldap.roles.nodata = [...]
lenses.jmx.schema.registry = "" // "schema-host:jmx-port"
lenses.jmx.zookeepers = "" // "zkhost1:jmx-port,zkhost2:jmx-port,zkhost3:jmx-port"
lenses.jmx.connect = [] // [ {dev: "kafka-connect-dev-host-or-ip:jmp-port"} , .. ]