> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/k2k/configuration-reference/kafka-connectivity-authentication.md).

# Kafka connectivity and authentication

Source and target Kafka cluster connections.

Every Kafka client needs bootstrap servers, security settings, and credentials before it can produce or consume. Because K2K connects to two Kafka clusters, the pipeline must declare separate connection settings for the source Kafka cluster and the target Kafka cluster.

K2K passes standard Kafka client properties through `source.kafka` and `target.kafka`, inherited from `common` into `consumer`, `producer`, and `admin` blocks.

See [Clusters](https://docs.lenses.io/latest/k2k/glossary.md#clusters) and [Kafka clients](https://docs.lenses.io/latest/k2k/glossary.md#kafka-clients) in the Glossary. Once brokers are reachable, configure the [consumer group](https://docs.lenses.io/latest/k2k/configuration-reference/consumer-group.md) on the source Kafka cluster. To verify connectivity before replication starts, see [Startup validation](https://docs.lenses.io/latest/k2k/configuration-reference/startup-validation.md). If you use Schema Registry, see [Schema Registry connectivity and authentication](https://docs.lenses.io/latest/k2k/configuration-reference/schema-registry-connectivity-authentication.md).

## Prerequisites

* [License accepted](https://docs.lenses.io/latest/k2k/configuration-reference/licenses.md)
* Bootstrap server addresses for both Kafka clusters
* Network and security credentials for each cluster

## Example

Use this pipeline configuration:

```yaml
name: "my_pipeline"
source:
  kafka:
    common:
      "bootstrap.servers": "source:9092"
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "PLAIN"
      "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"user\" password=\"secret\";"
target:
  kafka:
    common:
      "bootstrap.servers": "target:9092"
      "security.protocol": "SASL_SSL"
replication:
  - source:
      topic: "orders.*"
  - sink:
      topic: source
      partition: source
```

## Configuration keys

### Source

| Key                     | Description                                               | Required    | Default      | Used by    |
| ----------------------- | --------------------------------------------------------- | ----------- | ------------ | ---------- |
| `source.kafka`          | Source Kafka cluster connection settings                  | Yes         | (none)       | Both       |
| `source.kafka.common`   | Properties inherited by source consumer and admin clients | Conditional | `{}`         | Both       |
| `source.kafka.consumer` | Source consumer properties                                | Conditional | See defaults | Replicator |
| `source.kafka.admin`    | Source admin client properties                            | No          | `{}`         | Both       |

### Target

| Key                     | Description                                                          | Required    | Default      | Used by    |
| ----------------------- | -------------------------------------------------------------------- | ----------- | ------------ | ---------- |
| `target.kafka`          | Target Kafka cluster connection settings                             | Yes         | (none)       | Both       |
| `target.kafka.common`   | Properties inherited by target consumer, producer, and admin clients | Conditional | `{}`         | Both       |
| `target.kafka.consumer` | Target consumer properties (control consumer and offset mapper)      | No          | See defaults | Both       |
| `target.kafka.producer` | Target producer properties                                           | No          | See defaults | Replicator |
| `target.kafka.admin`    | Target admin client properties                                       | No          | `{}`         | Both       |

## Kafka client inheritance

On each side, properties flow from `common` to the specific client blocks:

| Block            | Used by                                                     |
| ---------------- | ----------------------------------------------------------- |
| `kafka.common`   | Inherited by consumer, producer, and admin on that side     |
| `kafka.consumer` | Consumer only (source data reader, target control consumer) |
| `kafka.producer` | Producer only (target)                                      |
| `kafka.admin`    | Admin client only                                           |

Any standard [Apache Kafka client property](https://kafka.apache.org/documentation/#configuration) can be set under the appropriate block. Later keys in a more specific block override `common`.

### Required keys

| Key                                       | Description                                                             | Used by       |
| ----------------------------------------- | ----------------------------------------------------------------------- | ------------- |
| `source.kafka.common."bootstrap.servers"` | Source Kafka cluster brokers                                            | Both          |
| `target.kafka.common."bootstrap.servers"` | Target Kafka cluster brokers                                            | Both          |
| `source.kafka.consumer."group.id"`        | K2K consumer group on the source Kafka cluster                          | Replicator    |
| `target.kafka.consumer."group.id"`        | Consumer group the offset mapper joins to read the offset mapping topic | Offset mapper |

`target.kafka.consumer."group.id"` is required only when deploying the offset mapper, which subscribes to `offsetMapping.topic` as a consumer group member and has no default group. See [Offset mapping](https://docs.lenses.io/latest/k2k/configuration-reference/offset-mapping.md). For the replicator's group, see [Consumer group](https://docs.lenses.io/latest/k2k/configuration-reference/consumer-group.md).

## Client defaults

K2K applies these defaults when keys are omitted. Properties in `common` are inherited by the client-specific blocks, and any standard Kafka client property can override a default under the appropriate block.

| Client   | Side   | Defaults                                                                   |
| -------- | ------ | -------------------------------------------------------------------------- |
| Consumer | Source | `"client.id": "k2k-app"`, `"auto.offset.reset": "earliest"`                |
| Consumer | Target | `"auto.offset.reset": "earliest"`, `"isolation.level": "read_committed"`   |
| Producer | Target | `"retries": "2147483647"`, `"enable.idempotence": "true"`, `"acks": "all"` |

Admin clients on both sides default to `{}`.

## Authentication

Set security properties under `kafka.common` or in a client specific block. Common Kafka properties:

| Key                       | Purpose                                                         |
| ------------------------- | --------------------------------------------------------------- |
| `security.protocol`       | `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`             |
| `sasl.mechanism`          | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `GSSAPI`, and others |
| `sasl.jaas.config`        | JAAS login module configuration                                 |
| `ssl.truststore.location` | Truststore path for TLS                                         |
| `ssl.truststore.password` | Truststore password                                             |

For secrets in connection settings, see [Secrets and templates](https://docs.lenses.io/latest/k2k/configuration-reference/secrets-and-templates.md).

### Managed cloud offerings

K2K bundles the login handlers for AWS and Google managed Kafka, so no extra JARs or sidecars are needed. Because the source Kafka cluster and target Kafka cluster are configured independently, one side can use IAM while the other uses SASL/SCRAM.

**AWS MSK with IAM authentication.** Set these under the relevant `kafka.common` (or a specific client) block:

```yaml
source:
  kafka:
    common:
      "bootstrap.servers": "b-1.my-cluster.kafka.eu-west-1.amazonaws.com:9098"
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "AWS_MSK_IAM"
      "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
      "sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
```

Credentials are resolved by the standard AWS SDK provider chain—instance or pod IAM roles (IRSA on EKS), environment variables, or a profile. To assume a role explicitly, use the `awsRoleArn` option in `sasl.jaas.config`.

**Google Managed Service for Apache Kafka.** Uses SASL/OAUTHBEARER with Application Default Credentials:

```yaml
target:
  kafka:
    common:
      "bootstrap.servers": "bootstrap.my-cluster.europe-west1.managedkafka.my-project.cloud.goog:9092"
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "OAUTHBEARER"
      "sasl.jaas.config": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;"
      "sasl.login.callback.handler.class": "com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler"
```

Credentials come from Application Default Credentials—Workload Identity on GKE, an attached service account, or `GOOGLE_APPLICATION_CREDENTIALS`.

Generic SASL/OAUTHBEARER against any other token provider is also supported through the standard Kafka `sasl.oauthbearer.*` properties.

On managed offerings, some admin APIs and topic configuration are restricted by the vendor. If startup checks report problems that do not reflect real connectivity issues, see [Startup validation](https://docs.lenses.io/latest/k2k/configuration-reference/startup-validation.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.lenses.io/latest/k2k/configuration-reference/kafka-connectivity-authentication.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
