Helm

This page describes installing Standalone K2K in Kubernetes via Helm.

Prerequisites

• Kubernetes 1.23+

• Helm 3.8.0+

• Available local Kafka Clusters:

Configure K2K

To configure Lenses K2K properly we have to understand the parameter groups that the Chart offers. Under the k2k parameter there are some key parameter groups that are used to set up HQ:

1. licence​

   • Definition that configures EULA acceptance.

2. ​otelConfig

   • Defines metric, traces and log exporters

3. ​replicationConfig​

   • Defines core K2K configuration file which included:

     • connection to source and destination Kafka Cluster / Schema Registry

     • replication semantics, replication options and many more

Moving forward, in the same order you can start configuring your Helm chart.

1

Configure licence

Before using K2K as a standalone application, you must agree to the End User License Agreement (EULA) and request a free license token by contacting [email protected]. Ensure this section is included in the replicationConfig yaml values:

values.yaml

k2k:
  replicationConfig:
    license:
      acceptEULA: true
      token: <license token>

2

Configure OTEL options

If you would like to monitor your K2K applications and by monitor we mean, export:

• logs;

• metrics;

Then you would have to configure following block:

values.yaml

k2k:
  otelConfig:
    serviceName: "k2k"
    metricsExporter: "prometheus"
    tracesExporter: "none"
    logsExporter: "none"
    prometheusHost: "0.0.0.0"
    prometheusPort: 9090

Note: The export functionality for warning logs and traces is currently unavailable.

3

Replication Configuration

The configuration file is in YAML and has 8 basic sections:

• source: defines the source cluster details (required)

• target: defined the target cluster details (required)

• replication: defines the set of topics to replicate and how to replicate (required)

• coordination: defines the setting for the coordinator, for example, the offsets (required)

• features: defines the extra functionality, such as exactly once (optional)

• errorHandling: defines how to handle errors (optional)

• tracing: defines the open tracing components (optional)

More about configuration blocks and descriptions read it here: Configuration

Helm definition of replicationConfig parameter is as an object

k2k:
  replicationConfig: {}

Therefore, all the yaml parameters that one can find under the configuration document above can be freely copy/pasted.

Secrets can be created via k2k.additionalEnv property and be referenced in following way

foo: ${env:string:MY_ENV}
bar: ${env:number:MY_ENV}
bar2: ${env:base64:MY_ENV}
bar3: ${file:MY_ENV}

Example of Kafka2Kafka replicationConfig that can be used.

Plaintext connection

values.yaml

k2k:
  replicationConfig:
    name: "k2k-demo-env"
    features:
      exactlyOnce: disabled
      headerReplication: disabled
      schemaMapping: disabled
      optimizeOffsetCommitPartition: enabled
      tracingHeaders: disabled
      autoCreateControlTopics: enabled
      autoCreateTopics: enabled
    coordination:
      kafka:
        assignment:
          topic: "__k2k-app-eot-assignment"
        commit:
          topic: "__k2k-app-eot-consumer-offsets"
          group: "k2k.eot"
    source:
      kafka:
        common:
          "bootstrap.servers": "source-kafka:9092"
        consumer:
          "group.id": "k2k.eot"
    target:
      kafka:
        common:
          servers: "target-kafka:9092"
        producer:
    replication:
      - source:
          name: source
          topic:
            - "topic1"
            - "topic2"
      - sink:
          name: sink-source-topic
          topic:
            prefix: "k2k.eot."
          partition: source

AWS_MSK_IAM

SASL (with secrets)

SSL

(Optional) Configure Service Accounts

Lenses Kafka2Kafka, by default, uses the default Kubernetes service account but you can choose to use a specific one.

If the user defines the following:

values.yaml

# serviceAccount is the Service account to be used by Lenses to deploy apps
serviceAccount:
  create: true
  annotations: {}
  name: lenses-k2k

The chart will create a new service account in the defined namespace for Kafka2Kafka to use.

Add chart repository

First, add the Helm Chart repository using the Helm command line:

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

GitHub - lensesio/lenses-helm-charts at release/6.0GitHub

Installing Kafka2Kafka

Be aware that for the time being and for alpha purposes usage of --versionis mandatory when deploying Helm chart through Helm repository.

terminal

helm install lenses-k2k lensesio/lenses-k2k \
   --values values.yaml \
   --create-namespace --namespace lenses-k2k \
   --version 1.0.0