Helm

This page describes installing Standalone K2K in Kubernetes via Helm.

Prerequisites

• Kubernetes 1.23+

• Helm 3.8.0+

• Available local Kafka Clusters

Introduction

The K2K Helm chart deploys two modules onto your Kubernetes cluster:

K2K Replicator -- the core component that continuously replicates Kafka topics from a source cluster to a target cluster. It supports exactly-once delivery, schema replication via Confluent-compatible Schema Registries, flexible topic routing, and automatic topic creation on the target cluster.

Offset Mapper -- an optional companion module that maps consumer group offsets between the source and target clusters. This enables consumers to resume from the correct position on the target cluster after a migration or failover. The Offset Mapper shares the base replicationConfig from K2K and can optionally override specific values (e.g. its own consumer group ID).

Both modules are independently configurable under the k2k and offsetMapper sections in values.yaml. When the Offset Mapper is enabled, the chart enforces that both offsetMapper.enabled: true and k2k.replicationConfig.features.offsetMapping: "enabled" are set together -- omitting either one will cause the chart to fail with a clear error message.

Configure K2K

To configure Lenses K2K properly we have to understand the parameter groups that the Chart offers. Under the k2k parameter there are some key parameter groups that are used to set up HQ:

1. licence​

   • Definition that configures EULA acceptance.

2. ​otelConfig

   • Defines metric, traces and log exporters

3. ​replicationConfig​

   • Defines core K2K configuration file which included:

     • connection to source and destination Kafka Cluster / Schema Registry

     • replication semantics, replication options and many more

Moving forward, in the same order you can start configuring your Helm chart.

1

Configure licence

Before using K2K as a standalone application, you must agree to the End User License Agreement (EULA) and request a free license token by contacting [email protected]. Ensure this section is included in the replicationConfig yaml values:

values.yaml

k2k:
  replicationConfig:
    license:
      acceptEULA: true
      token: <license token>

2

Configure OTEL options

If you would like to monitor your K2K applications and by monitor we mean, export:

• logs;

• metrics;

Then you would have to configure following block:

values.yaml

k2k:
  otelConfig:
    serviceName: "k2k"
    metricsExporter: "prometheus"
    tracesExporter: "none"
    logsExporter: "none"
    prometheusHost: "0.0.0.0"
    prometheusPort: 9090

Note: The export functionality for warning logs and traces is currently unavailable.

3

Replication Configuration

The configuration file is in YAML and has 8 basic sections:

• source: defines the source cluster details (required)

• target: defined the target cluster details (required)

• replication: defines the set of topics to replicate and how to replicate (required)

• coordination: defines the setting for the coordinator, for example, the offsets (required)

• features: defines the extra functionality, such as exactly once (optional)

• errorHandling: defines how to handle errors (optional)

• tracing: defines the open tracing components (optional)

More about configuration blocks and descriptions read it here: K2K Replicator

Helm definition of replicationConfig parameter is as an object

k2k:
  replicationConfig: {}

Therefore, all the yaml parameters that one can find under the configuration document above can be freely copy/pasted.

Secrets can be created via k2k.additionalEnv property and be referenced in following way

foo: ${env:string:MY_ENV}
bar: ${env:number:MY_ENV}
bar2: ${env:base64:MY_ENV}
bar3: ${file:MY_ENV}

Example of Kafka2Kafka replicationConfig that can be used.

Plaintext connection

values.yaml

k2k:
  replicationConfig:
    name: "k2k-demo-env"
    features:
      exactlyOnce: disabled
      headerReplication: disabled
      schemaMapping: disabled
      optimizeOffsetCommitPartition: enabled
      tracingHeaders: disabled
      autoCreateControlTopics: enabled
      autoCreateTopics: enabled
    coordination:
      kafka:
        assignment:
          topic: "__k2k-app-eot-assignment"
        commit:
          topic: "__k2k-app-eot-consumer-offsets"
          group: "k2k.eot"
    source:
      kafka:
        common:
          "bootstrap.servers": "source-kafka:9092"
        consumer:
          "group.id": "k2k.eot"
    target:
      kafka:
        common:
          servers: "target-kafka:9092"
        producer:
    replication:
      - source:
          name: source
          topic:
            - "topic1"
            - "topic2"
      - sink:
          name: sink-source-topic
          topic:
            prefix: "k2k.eot."
          partition: source

AWS_MSK_IAM

SASL (with secrets)

SSL

4

(Optional) Configure Service Accounts

Lenses Kafka2Kafka, by default, uses the default Kubernetes service account but you can choose to use a specific one.

If the user defines the following:

values.yaml

# serviceAccount is the Service account to be used by Lenses to deploy apps
k2k:
  serviceAccount:
    create: true
    annotations: {}
    name: lenses-k2k

The chart will create a new service account in the defined namespace for Kafka2Kafka to use.

Configure Offset Mapper

To configure Lenses K2K Offset Mapper properly we have to understand the parameter groups that the Chart offers. Under the offsetMapper parameter there are some key parameter groups that are used to set up HQ:

1

Enablement

To enable K2K Offset Mapper following fields have to be set

values.yaml

k2k:
  replicationConfig:
    features:
      offsetMapping: enabled

offsetMapper:
  enabled: true

2

Core Configuration

Because the Offset Mapper is an optional companion module to the K2K it reuses the base k2k.replicationConfig (source/target Kafka connection details, replication settings, etc.) and optionally allows overriding specific values via offsetMapper.overrideConfig.

Therefore if you followed steps to configure K2K connections for the source and target, you do not have to do it again, but maybe just reference Configuration Referencepage for some additional overrides or additions to configuration such as overriding replication configuration in the example below. Only the values you specify in overrideConfig will be overridden. Everything else is inherited from the base K2K replication config.

values.yaml

offsetMapper:  
  overrideConfig:
    groups:
      consumerGroups: "k2k.test"
    target:
      kafka:
        common:
          group.id: "offset-mapping-k2k-test-pipeline"

3

Configure OTEL options

If you would like to monitor your Offset Mapper application, configure the following block:

values.yaml

offsetMapper:
  otelConfig:
    serviceName: "k2k-offset-mapping"
    metricsExporter: "prometheus"
    tracesExporter: "none"
    logsExporter: "none"
    prometheusHost: "0.0.0.0"
    prometheusPort: 9091

4

(Optional) Configure Service Account

The Offset Mapper, by default, uses the default Kubernetes service account. To use a specific one

values.yaml

offsetMapper:
  serviceAccount:
    create: true
    annotations: {}
    name: k2k-offset-mapper

5

Configure Deployment Options

Adjust the Offset Mapper deployment resources based on your workload:

values.yaml

offsetMapper:
  deployment:
    replicas: 1
    resources:
      requests:
        memory: 512Mi
      limits:
        memory: 1Gi

Example of K2K + Offset Mapper deployment via Helm Chart

values.yaml

nameOverride: ""
fullnameOverride: ""

commonLabels:
  team: data-platform
  environment: test

k2k:
  annotations:
    description: "K2K test deployment"
  labels:
    app.kubernetes.io/component: replicator
  service:
    enabled: true
    type: ClusterIP
  serviceAccount:
    create: true
    name: msk-sa
  deployment:
    replicas: 1
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        cpu: 1
        memory: 2Gi
  image:
    repository: lensesio/k2k
    tag: "2.0"
    pullPolicy: IfNotPresent
  otelConfig:
    serviceName: "k2k"
    metricsExporter: "prometheus"
    tracesExporter: "none"
    logsExporter: "none"
    prometheusHost: "0.0.0.0"
    prometheusPort: 9090
  livenessProbe:
    enabled: true
  replicationConfig:
    name: "k2k-test-pipeline"
    features:
      exactlyOnce: disabled
      headerReplication: disabled
      schemaMapping: disabled
      optimizeOffsetCommitPartition: enabled
      tracingHeaders: disabled
      autoCreateControlTopics: enabled
      autoCreateTopics: enabled
      offsetMapping: enabled
    license:
      acceptEula: true
      token: "<YOUR_LICENSE>"
    source:
      kafka:
        common:
          sasl.mechanism: "AWS_MSK_IAM"
          sasl.jaas.config: "software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName=msk-sa;"
          security.protocol: "SASL_SSL"
          sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
          bootstrap.servers: ""
        consumer:
          group.id: "k2k.test"
    target:
      kafka:
        common:
          sasl.mechanism: "AWS_MSK_IAM"
          sasl.jaas.config: "software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName=msk-sa;"
          security.protocol: "SASL_SSL"
          sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
          bootstrap.servers: ""
    replication:
      - source:
          name: source
          topic:
            - "inventory-events"
      - sink:
          name: sink
          topic:
            prefix: "k2k.test."
          partition: source

offsetMapper:
  enabled: true
  annotations:
    description: "Offset Mapper test deployment"
  labels:
    app.kubernetes.io/component: offset-mapper
  service:
    enabled: true
    type: ClusterIP
  serviceAccount:
    create: true
    name: msk-sa-om
  deployment:
    replicas: 1
    resources:
      requests:
        cpu: 250m
        memory: 512Mi
      limits:
        cpu: 500m
        memory: 1Gi

  image:
    repository: lensesio/k2k-offset-mapping
    tag: "2.0"
    pullPolicy: IfNotPresent
  otelConfig:
    serviceName: "k2k-offset-mapping"
    metricsExporter: "prometheus"
    tracesExporter: "none"
    logsExporter: "none"
    prometheusHost: "0.0.0.0"
    prometheusPort: 9091
  livenessProbe:
    enabled: false
  overrideConfig:
    groups:
      consumerGroups: "k2k.test"
    target:
      kafka:
        common:
          "group.id": "offset-mapping-k2k-test-pipeline"

Add chart repository

First, add the Helm Chart repository using the Helm command line:

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

GitHub - lensesio/k2k-helm-charts: Helm Charts source code for K2KGitHub

Installing K2K & Offset Mapper

Be aware that for the time being and for alpha purposes usage of --versionis mandatory when deploying Helm chart through Helm repository.

terminal

helm install lenses-k2k lensesio/lenses-k2k \
   --values values.yaml \
   --create-namespace --namespace lenses-k2k \
   --version 2.0.0