To use this option, the External Secret Operator (ESO) has to be installed and available for usage in K8s cluster your are deploying HQ.
When specifying passwordSecret.type: "externalSecret", thechart will:
create an ExternalSecretin the namespace where HQ is deployed;
a secret is mounted for HQ to use.
values.yaml
lensesHq:
storage:
postgres:
enabled: true
host: postgres-postgresql.playground.svc.cluster.local
port: 5432
username: lenses
database: lenseshq
passwordSecret:
type: "externalSecret"
# Secret name where database password will be read from
name: hq-password
# Key name under secret where database password is stored
key: password
externalSecret:
additionalSpecs: {}
secretStoreRef:
type: SecretStore # or ClusterSecretStore
name: secretstore-secrets
Make sure that secret you are going to use is already created in namespace where HQ will be installed.
values.yaml
lensesHq:
storage:
postgres:
enabled: true
host: postgres-postgresql.playground.svc.cluster.local
port: 5432
username: lenses
database: lenseshq
passwordSecret:
type: "precreated"
# Secret name where database password will be read from
name: hq-password
# Key from secret's data where database password is being stored
key: postgres-password
This option is NOT for PRODUCTION usage but rather just for demo / testing.
The chart will create a secret with defined values below and the same secret will be read by HQ in order to connect to Postgres.
values.yaml
lensesHq:
storage:
postgres:
enabled: true
host: [POSTGRES_HOSTNAME]
port: 5432
username: lenses
database: lenseshq
passwordSecret:
type: "createNew"
# name of a secret that will be created
name: [K8s_SECRET_NAME]
# Database password
password: [DATABASE_USER_PASSWORD]
Advanced Postgres settings
Sometimes to form the correct connection URI special parameters are needed. You can set the extra settings using params.
SAML / SSO is available only with Enterprise license.
The second pre-requirement to successfully run HQ is setting initial authentication.
You can choose between:
password-based authentication, which requires users to provide a username and password;
and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.
First to cover is the users property. Users Property: The users property is defined as an array, where each entry includes a username and a password. Passwords must be hashed using bcrypt before being placed within the password property, for security purposes, ensuring that they are stored correctly and securely.
Second, to cover will be administrators. It serves as the definition of user emails have the highest level of permissionsupon authentication to HQ.
Third attribute is saml.metadata field, needed for setting SAML / SSO authentication. In this step, you will need metadata.xml file which can be set in two ways:
Referencing metadata.xml file through pre-created secret;
The third pre-requirement to successfully run HQ is the http definition. As previously mentioned, this parameter defines everything around the HTTP endpoint of the HQ itself and how users will interact with it.
Second part of HTTP definition would be enabling TLS and TLS definition itself. As previously defined for lensesHq.agents.tlssame way of configuring TLS can be used for lensesHq.http.tls definition as well.
4
Configure agent's connection endpoint
After correctly configuring the authentication strategy and connection endpoint, the agent handling is the last most important box to tick.
The Agent's object is defined as follows:
lensesHq:
agents:
# which port to listen on for agent requests
address: ":10000"
tls:
enabled: false
verboseLogs: false
cert:
privateKey:
Enabling TLS
By default TLS for the communication between Agent and HQ is disabled. In case the requirement is to enable it, fthe ollowing has to be set:
lensesHq.agents.tls - certificates to manage the connection between HQ and the Agents
lensesHq.http.tls- certificates to manage connection with HQ's API
Unlike private keys which can be referenced and obtained only through a secret, Certificates can be referenced directly in values.yamlfile as a string or as a secret.
Metrics are optionally available in a Prometheus format and by default served on port 9090.
The port can be changed in the following way:
values.yaml
lensesHq:
metrics:
prometheusAddress: ":9090"
(Optional) Configure Ingress & Services
Whilst the chart supports setting TLS on Lenses HQ itself we recommend placing it on the Ingress resource
Ingress and service resources are optionally supported.
The http ingress is intended only for HTTP/S traffic, while the agents ingress is designed specifically for TCP protocol. Ensure appropriate ingress configuration for your use case.
Enable an Ingress resource in the values.yaml:
values.yaml
ingress:
http:
enabled: true
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
host: example.com
ingressClassName: ""
tls:
enabled: false
# The TLS secret must contain keys named tls.crt and tls.key that contain the certificate and private key to use for TLS.
secretName: ""
agent:
enabled: true
agentIngressConfig:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: agents
spec:
entryPoints:
- agents
routes:
- match: HostSNI(`example.com`) # HostSNI to match TLS for TCP
services:
- name: lenses-hq # Replace with your service name
port: 10000 # Agent default TCP port
tls: {}
Be aware that example of values.yaml shows only how all parameters should look at the end. Please fill them with correct values otherwise Helm installation might not be successful.
Example of values.yaml
More about default values for Lenses HQ Helm Chart can be found in values.yaml. An example is below:
To see be able to view and drilling to your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.
Agent key within provisioning.yaml can be referenced as a:
environment variable shown in example above
inline string
2
Configure Kafka
Bare in mind that each Agent connects to one Kafka cluster.
There are many Kafka flavours today in the market. Good news is that Lenses support all flavours of Kafka and we are trying hard to keep documention up to date.
In the following link you can find provisioning examples for the most common Kafka flavours.
Other connections
There are also provisioning examples for other components:
Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.
Alternatively, the lenses.conf can be mounted directly as
/mnt/settings/lenses.conf
Docker volumes
The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:
/data/storage
/data/plugins
/data/logs
/data/kafka-streams-state
Storage volume
Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Agent upgrades, the volume must be managed externally (persistent volume).
Plugins volume
Resides under /data/plugins it’s where classes that extend Agent may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.
Logs volume
Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.
KStreams state volume
Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.
Agent TLS and Global JVM Trust Store
By default, the the serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.
This capability is optional, and users can mount such files under custom paths and configure lenses.conf manually via environment variables, or lenses.append.conf.
There are two ways to use the File/Variable names of the table below.
Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets
Set them as environment variables.
All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.
File / Variable Name
Description
FILECONTENT_JVM_SSL_TRUSTSTORE
The SSL/TLS trust store to use as the global JVM trust store. Add to LENSES_OPTS the property javax.net.ssl.trustStore
FILECONTENT_JVM_SSL_TRUSTSTORE_PASSWORD
Τhe trust store password. If set, the startup script will add automatically to LENSESOPTS the property javax.net.ssl.trustStorePassword (**_base64 not supported**)
FILECONTENT_LENSES_SSL_KEYSTORE
The SSL/TLS keystore to use for the TLS listener for the Agent
Process UID/GUI
The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting the Agent.
If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the license, settings, and data) have the correct permission set.