Deploy HQ
Learn how to deploy HQ with Helm.
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
This page describes installing Lenses HQ in Kubernetes via Helm.
Lenses HQ is prerequisite for installation of Lenses Agent
Kubernetes 1.23+
Helm 3.8.0+
Running Postgres instance:
database for HQ;
username (and password) that has access to HQ database;
Optional External secret operator (in case of ExternalSecret usage)
In order to configure properly HQ we have to understand parameter groups that the Chart offers.
Under the lensesHq parameter there some key parameter groups that are used to setup HQ:
definition of connection towards database (Postgres is the only storage option)
Password based authentication configuration
SAML / SSO configuration
definition of administrators or first users to access the HQ
defines port under which HQ will be available for end users
defines values of special headers and cookies
types of connection such as TLS and non-TLS definitions
defines connection between HQ and the Agent such as port where HQ will be listening for agent connections.
types of connection such as TLS and non-TLS definitions
license
controls the metrics settings where Prometheus alike metrics will be exposed
definition of logging level for HQ
Moving forward, in the same order you can start configuring your Helm chart.
Postgres is the only available storage option.
Prerequisite:
Running Postgres instance;
Created database for HQ;
Username (and password) which has access to created database;
In order to successfully run HQ, storage within values.yaml has to be defined first.
Definition of storage object is as follows:
Alongside Postgres password, which can be referenced / created through Helm chart, there are few more options which can help while setting up HQ.
There are two ways how username can be defined:
The most straight forward way if username is not being changed by just defining it within username parameter such as
In case Postgres username is being rotated or frequently changed it can be referenced from pre-created secret
Postgres password can be handled in three ways using:
External Secret via ExternalSecretOperator;
Pre-created secret;
Creating secret on the spot through values.yaml;
To use this option, the External Secret Operator (ESO) has to be installed and available for usage in K8s cluster your are deploying HQ.
When specifying passwordSecret.type: "externalSecret", the chart will:
create an ExternalSecret in the namespace where HQ is deployed;
a secret is mounted for HQ to use.
Make sure that secret you are going to use is already created in namespace where HQ will be installed.
This option is NOT for PRODUCTION usage but rather just for demo / testing.
The chart will create a secret with defined values below and the same secret will be read by HQ in order to connect to Postgres.
Sometimes to form correct connection URI special parameters are needed. In order to od the same you can set extra settings using params.
Example:
SAML / SSO is available only with Enterprise license.
Second pre-requirement to successfully run HQ is setting initial authentication.
You can choose between:
password-based authentication, which requires users to provide a username and password;
and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.
Definition of auth object is as follows:
First to cover is users property. Users Property: The users property is defined as an array, where each entry includes a username and a password. The passwords are hashed using bcrypt for security purposes, ensuring that they are stored securely.
Second to cover will be administrators. It serves as definition of user emails which will have highest level of permissions upon authentication to HQ.
Third attribute is saml.metadata field needed for setting SAML / SSO authentication. In this step, you will need metadata.xml file which can be set in two ways:
Referencing metadata.xml file through pre-created secret;
Placing metadata.xml contents inline as a string.
Third pre-requirement to successfully run HQ is the http definition. As previously mentioned, this parameter defines everything around HTTP endpoint of the HQ itself and how users will interact with.
Definition of HTTP object is as follows:
Second part of HTTP definition would be enabling TLS and TLS definition itself. As previously defined for lensesHq.agents.tls same way of configuring TLS can be used for lensesHq.http.tls definition as well.
After correctly configuring authentication strategy and connection endpoint , agent handling is the last most important box to tick.
The Agent's object is defined as follows:
By default TLS for the communication between Agent and HQ is disabled. In case requirement is to enabled it, following has to be set:
lensesHq.agents.tls - certificates to manage connection between HQ and the Agents
lensesHq.http.tls- certificates to manage connection with HQ's API
Unlike private keys which can be referenced and obtained only through a secret, Certificates can be referenced directly in values.yaml file as a string or as a secret.
Whilst the chart supports setting TLS on Lenses HQ itself we recommend placing it on the Ingress resource
Ingress and service resources are optionally supported.
The http ingress is intended only for HTTP/S traffic, while the agents ingress is designed specifically for TCP protocol. Ensure appropriate ingress configuration for your use case.
Enable an Ingress resource in the values.yaml:
Enable a service resource in the values.yaml:
Lenses HQ, by default, uses default Kubernetes service account but user can choose to use specific one.
If user defines following:
The chart will create new service account in the the defined namespace for HQ to use.
There are two options user can choose between:
rbacEnable: true - will enable creation of ClusterRole and ClusterRoleBinding for service account mentioned in snippet above
rbacEnable: true and namespaceScope: true - will enable creation of Role and RoleBinding which is more restrictive.
There are different logging modes and levels that can be adjusted.
First, add the Helm Chart repository using the Helm command line:
Be aware that for the time being and for alpha purposes usage of --version is mandatory when deploying Helm chart through Helm repository.
This page describes installing Lenses HQ and Agent in Kubernetes via Helm.
Only Helm 3 is supported.
This page describes deploying an Lenses Agent via Docker.
The Agent docker image can be configured via environment variables or via volume mounts for the configuration files.
Please check "File configuration" before running the command
In order for command above provisioning yaml has to be configured.
There are two mandatory connections:
HQ which requires AgentKey (LENSESHQ_AGENT_KEY) - this key is being created once user registers "New environment" in HQ;
Kafka connection
Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.
Alternatively, the lenses.conf can be mounted directly as
/mnt/settings/lenses.conf
The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:
/data/storage
/data/plugins
/data/logs
/data/kafka-streams-state
Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Agent upgrades, the volume must be managed externally (persistent volume).
Resides under /data/plugins it’s where classes that extend Agent may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.
Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.
Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.
By default, the the serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.
This capability is optional, and users can mount such files under custom paths and configure lenses.conf manually via environment variables, or lenses.append.conf.
There are two ways to use the File/Variable names of the table below.
Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets
Set them as environment variables.
All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.
The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting the Agent.
If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the license, settings, and data) have the correct permission set.
This page describes installing Lenses Agent in Kubernetes via Helm.
Kubernetes 1.23+
Helm 3.8.0+
Running Postgres instance
(in case of ExternalSecret usage)
First, add the Helm Chart repository using the Helm command line:
Installing using cloned repository:
Installing using Helm repository:
Be aware that for the time being and for alpha purposes usage of --version is mandatory when deploying Helm chart through Helm repository.
To automatically provision the connections to Kafka, HQ and other systems set the .Values.lenses.provision.connections to be the YAML definition of your connections.
The chart will render the full YAML specified under this setting as the provisioning.yaml file.
Alternatively you can use a second YAML file, which contains only the connections pass them at the command line when installing:
The chart uses:
Secrets to store Postgres credentials and authentication credentials
Secrets to store connection credentials such as Kafka SASL_SCRAM password or password for SSL JKS stores.
Secrets to hold the base64 encoded values of the JKS stores
Secrets to store AGENT KEY for connection to Lenses HQ
ConfigMap for Lenses configuration overrides
Cluster roles and role bindings (optional).
Secrets and config maps are mounted as files under the mount /mnt:
settings - holds the lenses.conf
provision-secrets - holds the secrets for connections in the provisioning.yaml file
provision-secrets/files - holds any file needed for a connection, e.g. JKS files.
Connection to Lenses HQ is straight forward process which requires two steps:
Storing that same key in Vault or as a K8s secret
The agent communicates with HQ via a secure custom binary protocol channel. To establish this channel and authenticate the Agent needs and AGENT KEY.
Once the AGENT KEY has been copied, store it inside of Vault or any other tool that has integration with Kubernetes secrets.
There are three available options how agent key can be used:
ExternalSecret via External Secret Operator (ESO)
Pre-created secret
Inline string
To use this option, the External Secret Operator (ESO) has to be installed and available for usage in K8s cluster your are deploying Agent.
When specifying secret.type: "externalSecret", the chart will:
create an ExternalSecret in the namespace where Agent is deployed;
a secret is mounted for Agent to use.
Make sure that secret you are going to use is already created in namespace where Agent will be installed.
This option is NOT for PRODUCTION usage but rather just for demo / testing.
The chart will create a secret with defined values below and the same secret will be read by Agent to connect to HQ.
This secret will be fed into the provisioning.yaml. The HQ connection is specified at line 30, below, where reference ${LENSESHQ_AGENT_KEY} is being set:
The Helm chart creates Cluster roles and bindings, that are used by SQL Processors, if the deployment mode is set to KUBERNETES. They are used so that Lenses can deploy and monitor SQL Processor deployments in namespaces.
To disable the creation of Kubernetes RBAC set: rbacEnabled: false
If you are not using SQL Processors and want to limit permissions given to Agent's ServiceAccount, there are two options you can choose from:
rbacEnable: true - will enable the creation of ClusterRole and ClusterRoleBinding for service account mentioned above;
rbacEnable: true and namespaceScope: true - will enable creation of Role and RoleBinding which is more restrictive.
To use Postgres as the backing store for the Agent set the details in the lenses.storage.postgres object.
The chart relies on secrets for sensitive information such as Passwords. Secrets can rotate and are commonly stored in an external store such as Azure KeyVault, Hashicorp Vault or AWS Secrets Manager.
For Postgres, add additional ENV variables via the lenses.additionalEnv object to point to your secret and set the username and password to external in the Postgres section.
Enable a service resource in the values.yaml:
To control the resources used by the Agent:
In case LENSES_HEAP_OPTS is not set explicitly it will be set implicitly.
Examples:
if no requests or limits are defined, LENSES_HEAP_OPTS will be set as -Xms1G -Xmx3G
If requests and limits are defined above defined values, LENSES_HEAP_OPTS will be set by formula -Xms[-Xmx / 2] -Xmx[limits.memory - 2]
If .Values.lenses.jvm.heapOpts it will override everything
To enable SQL processor in KUBERENTES mode and control the defaults:
To control the namespace Lenses can deploy processors, use the sql.namespaces value.
To achieve you need to create a Role and a RoleBinding resource in the namespace you want the processors deployed to.
For example:
Lenses namespace = lenses-ns
Processor namespace = lenses-proc-ns
Finally you need to define in the Agent configuration which namespaces the Agent has access to. Amend values.yaml to contain the following:
Prometheus metrics are automatically exposed on port 9102 under /metrics.
The main configurable options for lenses.conf are available in the values.yaml under the lenses object. These include:
Authentication
Database connections
SQL processor configurations
To apply other static configurations use lenses.append.conf, for example:
This page describes the supported installation methods for Lenses.
Lenses can be deployed in the following ways:
This page describes deploying Lenses HQ via docker.
The HQ docker image can be configured via volume mounts for the configuration file.
The HQ looks for the config.yaml in the current working directory. This is the root directory for Docker.
The main pre-requirements that has to be fulfilled before Lenses HQ container can be started and those are:
In demo purposes and testing the product you can use our community license
Main configuration file that has to be configured before running docker command is config.yaml.
Sample configuration file is following:
This page describes the install of the Lenses Agent via an archive on Linux.
To install the HQ from the archive you must:
Extract the archive
Configure the HQ
Start the HQ
Extract the archive using the following command
Inside the extract archive, you will find.
In order to properly configure HQ, two core components are necessary:
To set up authentication, there are multiple methods available.
You can choose between:
password-based authentication, which requires users to provide a username and password;
and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.
Both password based and SAML / SSO authentication methods can be used alongside each other.
First to cover is users property. Users Property: The users property is defined as an array, where each entry includes a username and a password. The passwords are hashed using bcrypt for security purposes, ensuring that they are stored securely.
Second to cover will be administrators. It serves as definition of user emails which will have highest level of permissions upon authentication to HQ.
Another part which has to be set in order to successfully run HQ is the http definition. As previously mentioned, this parameter defines everything around HTTP endpoint of the HQ itself and how users will interact with.
Definition of HTTP object is as follows:
More about setting up TLS can be read .
If you have meticulously followed all the outlined steps, your config.yaml file should mirror the example provided below, fully configured and ready for deployment. This ensures your system is set up correctly with all necessary settings for authentication, database connection, and other configurations optimally defined.
Start Lenses by running:
or pass the location of the config file:
If you do not pass the location of the config file, the HQ will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.
To stop HQ, press CTRL+C.
If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.
This page describes installing Lenses with Docker Image.
This page describes the install of the Lenses Agent via an archive on Linux.
To install the Agent from the archive you must:
Extract the archive
Configure the Agent
Start the Agent
Extract the archive using the following command
Inside the extract archive, you will find.
Once the agent files are configure you can continue to start the agent.
The configuration files are the same for docker and Linux, for docker we are simply mounting the files into the container.
To see be able to view and drilling to your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.
Agent key reference
Agent key within provisioning.yaml can be referenced as a:
environment variable shown in example above
inline string
Start Lenses by running:
or pass the location of the config file:
Provisioning file path
If you configured provisioning.yaml make sure to place following property:
If you do not pass the location of lenses.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.
To stop Lenses, press CTRL+C.
Set the permissions of the lenses.conf to be readable only by the lenses user.
The agent needs write access in 4-5 places in total:
[RUNTIME DIRECTORY]
When the Agent runs, it will create at least one directory under the directory it is run in:
[RUNTIME DIRECTORY]/logs
Where logs are stored
[RUNTIME DIRECTORY]/logs/sql-kstream-state
Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.
[RUNTIME DIRECTORY]/storage
Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.
/run (Global directory for temporary data at runtime)
Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.
/tmp (Global temporary directory)
Used for temporary files (if access /run fails), and JNI shared libraries.
Back-up this location for disaster recovery
The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.
You must either:
Mount /tmp without noexec
or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location
If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.
The Agent uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (JMX over TLS) we always rely on the system trust store.
It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:
Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.
Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:
Increase as a super-user the soft limit to 4096 with:
Use 8GB RAM /4 CPUs and 20GB disk space.
This page describes install the Lenses via a Linux archive
| File / Variable Name | Description |
|---|
For more information on provisioning see .
If you are curious about Provisioning API specs which can help you understand connection configs, you can find the same under page .
Creating Environment and obtaining AGENT KEY in HQ as described , if you already have not done so.
If you want to limit the permissions the Agent has against your Kubernetes cluster, you can use Role/RoleBinging resources instead. Follow in order to enable it.
You can also find examples in the .
More about configuration options you can find on the page.
Once HQ starts, it will be listening on the
To configure the agents connection to Postgres and its provisioning file. See here in the .
In the following you can find provisioning examples for the most common Kafka flavours.
FILECONTENT_JVM_SSL_TRUSTSTORE | The SSL/TLS trust store to use as the global JVM trust store. Add to LENSES_OPTS the property javax.net.ssl.trustStore |
FILECONTENT_JVM_SSL_TRUSTSTORE_PASSWORD | Τhe trust store password. If set, the startup script will add automatically to LENSESOPTS the property javax.net.ssl.trustStorePassword (**_base64 not supported**) |
FILECONTENT_LENSES_SSL_KEYSTORE | The SSL/TLS keystore to use for the TLS listener for the Agent |
