Configure storage (Postgres)

Prerequisite:

• Running Postgres instance;

• Created database for an Agent;

• Username (and password) which has access to the created database;

In order to successfully run HQ, storage is within values.yaml has to be defined first.

The definition of storage object is as follows:

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: ""
      port: 
      username: ""
      database: ""
      schema: ""
      params: {}

Alongside Postgres password, which can be referenced/created through Helm chart, there are a few more options which can help while setting up HQ.

There are two ways how username can be defined:

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: lenses

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: external  # use "external" to manage it using secrets
  additionalEnv:
    - name: LENSES_STORAGE_POSTGRES_USERNAME
      valueFrom:
        secretKeyRef:
          name: [SECRET_RESOURCE_NAME]
          key: [SECRET_RESOURCE_KEY]

Password reference types

Postgres password can be handled in three ways using:

1. Pre-created secret;

2. Creating secrets on the spot through values.yaml;

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.playground.svc.cluster.local
      port: 5432
      username: lenses
      database: lensesagent
      password: useOnlyForDemos         

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: lenses
      password: external   # use "external" to manage it using secrets
  additionalEnv:
    - name: LENSES_STORAGE_POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: [SECRET_RESOURCE_NAME]
          key: [SECRET_RESOURCE_KEY]

Configure HQ connection (agent key)

Connection to Lenses HQ is a straight forward process which requires two steps:

1. Creating Environment and obtaining AGENT KEY in HQ as described here, if you already have not done so.

2. Storing that same key in Vault or as a K8s secret

The agent communicates with HQ via a secure custom binary protocol channel. To establish this channel and authenticate the Agent needs and AGENT KEY.

Once the AGENT KEY has been copied, store it inside of Vault or any other tool that has integration with Kubernetes secrets.

There are three available options how the agent key can be used:

1. ExternalSecret via External Secret Operator (ESO)

2. Pre-created secret

3. Inline string

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "externalSecret"
        # Secret name where agentKey will be read from
        name: hq-password
        # Key name under secret where agentKey is stored
        key: key
        externalSecret:
          additionalSpecs: {}
          secretStoreRef:
            type: ClusterSecretStore # ClusterSecretStore | SecretStore
            name: [secretstore_name]

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "precreated"
        # Secret name where agentKey will be read from
        name: hq-password
        # Key name under secret where agentKey is stored
        key: key

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "createNew"
        # Secret name where agentKey will be read from
        name: "lenses-agent-secret-1"
        # Value of agentKey generated by HQ
        value: "agent_key_*"

This secret will be fed into the provisioning.yaml. The HQ connection is specified at line 30, below, where reference ${LENSESHQ_AGENT_KEY} is being set:

lensesAgent:
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [LENSES_HQ_FQDN_OR_IP]
            port:
              value: 10000
            agentKey:
              # This property shouldn't be changed as it is mounted automatically
              # based on secret choice for hq.agentKey above 
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false

Configure provisioning (Kafka / SchemaRegistry / Kafka Connect)

Provisioning offers various connections starting with:

• Kafka ecosystem components such as:

  • Kafka

  • Schema Registry

  • Kafka Connect

  • Zookeeper

• Alerts & Audits

• AWS

lensesAgent:
  provision:
    path: /mnt/provision-secrets
    connections:
      # Kafka Connection
      kafka:
        - name: Kafka
          version: 1
          tags: [my-tag]
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://your.kafka.broker.0:9092
                - PLAINTEXT://your.kafka.broker.1:9092
            protocol: 
              value: PLAINTEXT
            # all metrics properties are optional
            metricsPort: 
              value: 9581
            metricsType: 
              value: JMX
            metricsSsl: 
              value: false
      # Confluent Schema Registry Connection
      confluentSchemaRegistry:
        - name: schema-registry
          tags: ["tag1"]
          version: 1      
          configuration:
            schemaRegistryUrls:
              value:
                - http://my-sr.host1:8081
                - http://my-sr.host2:8081
            ## all metrics properties are optional
            metricsPort: 
              value: 9581
            metricsType: 
              value: JMX
            metricsSsl: 
              value: false
      # Kafka Connect connection
      connect:
        - name: my-connect-cluster-name
          version: 1    
          tags: ["tag1"]
          configuration:
            workers:
              value:
                - http://my-kc.worker1:8083
                - http://my-kc.worker2:8083
            metricsPort: 
              value: 9585
            metricsType: 
              value: JMX

More about provisioning and more advanced configuration options for each of these components can be found on the following link.

Cluster RBACs

The Helm chart creates Cluster roles and bindings, that are used by SQL Processors, if the deployment mode is set to KUBERNETES. They are used so that Lenses can deploy and monitor SQL Processor deployments in namespaces.

To disable the creation of Kubernetes RBAC set: rbacEnabled: false

If you want to limit the permissions the Agent has against your Kubernetes cluster, you can use Role/RoleBinging resources instead. Follow this link in order to enable it.

If you are not using SQL Processors and want to limit permissions given to Agent's ServiceAccount, there are two options you can choose from:

1. rbacEnable: true - will enable the creation of ClusterRole and ClusterRoleBinding for service account mentioned above;

2. rbacEnable: true and namespaceScope: true - will enable the creation of Role and RoleBinding which is more restrictive;

Configure storage (Postgres)

Prerequisite:

• Running Postgres instance;

• Created database for HQ;

• Username (and password) which has access to the created database;

In order to successfully run HQ, storage within values.yaml has to be defined first.

Definition of storage object is as follows:

Alongside Postgres password, which can be referenced / created through Helm chart, there are few more options which can help while setting up HQ.

Username reference types

There are two ways how username can be defined:

Password reference types

Postgres password can be handled in three ways using:

2. Pre-created secret;

3. Creating secret on the spot through values.yaml;

Advanced Postgres settings

Sometimes to form the correct connection URI special parameters are needed. You can set the extra settings using params.

Example:

Configure AUTH endpoint

The second pre-requirement to successfully run HQ is setting initial authentication.

You can choose between:

• password-based authentication, which requires users to provide a username and password;

• and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.

The definition of auth object is as follows:

First to cover is the users property. Users Property: The users property is defined as an array, where each entry includes a username and a password. Passwords must be hashed using bcrypt before being placed within the password property, for security purposes, ensuring that they are stored correctly and securely.

Second, to cover will be administrators. It serves as the definition of user emails have the highest level of permissions upon authentication to HQ.

Third attribute is saml.metadata field, needed for setting SAML / SSO authentication. In this step, you will need metadata.xml file which can be set in two ways:

1. Referencing metadata.xml file through pre-created secret;

2. Placing metadata.xml contents inline as a string.

In case SAML IdP requires certificate verification, same can be enabled and provided in the following way:

Configure HTTP endpoint

The third pre-requirement to successfully run HQ is the http definition. As previously mentioned, this parameter defines everything around the HTTP endpoint of the HQ itself and how users will interact with it.

Definition of HTTP object is as follows:

Configure agent's connection endpoint

After correctly configuring the authentication strategy and connection endpoint, the agent handling is the last most important box to tick.

The Agent's object is defined as follows:

Enabling TLS

By default TLS for the communication between Agent and HQ is disabled. In case the requirement is to enable it, fthe ollowing has to be set:

• lensesHq.agents.tls - certificates to manage the connection between HQ and the Agents

• lensesHq.http.tls- certificates to manage connection with HQ's API

Unlike private keys which can be referenced and obtained only through a secret, Certificates can be referenced directly in values.yaml file as a string or as a secret.

Configure license

License can be read in multiple ways:

• from a pre-created secret

• directly as a string defined in values.yaml file

Configure metrics endpoint

Metrics are optionally available in a Prometheus format and by default served on port 9090.

The port can be changed in the following way: