> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/connectors/release-notes/stream-reactor.md). # Stream Reactor This page contains the release notes for the Stream Reactor. ## 12.1.2 ### Cloud Sinks (AWS S3, GCP Storage, Azure Data Lake Gen2) — PARTITIONBY exactly-once #### Fix: records could go missing when partitioning by a timestamp added at processing time With exactly-once enabled, these connectors track delivery progress separately for each value produced by your KCQL `PARTITIONBY` clause. This only works correctly if a given record always produces the same value — for example, a date or ID that comes from the record itself. If instead you partition by a value that reflects *when the connector processed the record* — most commonly a "current time" header added by a transform from the [kafka-connect-smt](https://github.com/lensesio/kafka-connect-smt) project, such as `InsertWallclockHeaders` — that assumption breaks. If Kafka ever needs to redeliver a record (after a restart or a rebalance), it can arrive slightly later than the first time, land under a different timestamp, and be mistaken for a record already written. In that case, the record was silently dropped: it was never written anywhere, and never retried. This release closes that gap. #### New: a delivery-tracking mode that doesn't depend on the partitioning value A new setting, `commit.mode`, controls how the connector guarantees exactly-once delivery: * `granular` (the existing behaviour, still the default) — tracks progress separately for each `PARTITIONBY` value. Requires that value to be derived from the record itself. * `batch` (new) — tracks progress once per Kafka partition instead, so it no longer matters whether the `PARTITIONBY` value is stable across redeliveries. **Use this if you partition by a processing-time value, such as a header from `InsertWallclockHeaders`, `InsertRollingWallclockHeaders`, `InsertWallclock`, `InsertRollingWallclock` or `InsertWallclockDateTimePart`.** Trade-off: in `batch` mode, every `PARTITIONBY` value that currently has unwritten records is written out together, every time the connector flushes — not only the ones that individually reached their own size or record-count threshold. If you're writing to many `PARTITIONBY` values at once (for example, many regions or IDs), this can produce more, smaller files than before. Enable it with `connect..exactly.once.commit.mode=batch` (`connect.s3...`, `connect.gcpstorage...`, or `connect.datalake...` depending on the connector). Changing this setting requires stopping the connector first; for a clean switch, let it catch up and go idle before stopping it, otherwise a small number of already-written records may be written again once (never lost) when it restarts. `batch` mode fixes the missing-records problem above, but a record that is redelivered before it's ever written can still end up filed under the newer timestamp rather than the original one. #### Fix: a connector instance that lost a partition could still write to it During a rebalance, Kafka Connect can briefly run an old and a new task instance for the same partition. Previously, the old instance could keep writing until the new instance made its first commit. It's now blocked as soon as the new instance takes over. #### Enforced: the connector now refuses to start with an unsafe setup If exactly-once is on, `commit.mode` is left at the default `granular`, and `PARTITIONBY` uses a header from one of the five transforms listed above, the connector now fails to start with a clear error naming the transform and header involved, instead of running with a silent data-loss risk. The error tells you to either switch to `commit.mode=batch`, partition by a different (record-derived) value, or turn off exactly-once. This check only recognizes the five transforms named above; a custom transform producing a similar "current time" value is not detected. Four new JMX counters (`BatchCommitsTotal`, `BatchCommitFilesTotal`, `BatchCommitFailuresTotal`, `LegacyLocksPurgedTotal`) track batch-mode commits; see each connector's Metrics (JMX) reference for details. Applies to: AWS S3, GCP Storage, Azure Data Lake Gen2. *** The changelog for the Stream Reactor has moved [here](/latest/changelog/kafka-connectors-stream-reactor/stream-reactor.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.lenses.io/latest/connectors/release-notes/stream-reactor.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.