5.0

Release notes

4.1.0 

Note: S3 Connector Changes

From Version 4.10, S3 Connectors will use the AWS Client by default. You can revert to the jClouds version by setting the connect.s3.aws.client property.

  • All

    • Scala upgrade to 2.13.10
    • Dependency upgrades
    • Upgrade to Kafka 3.3.0
    • SimpleJsonConverter - Fixes mismatching schema error.
  • AWS S3 Sink Connector

    • Add connection pool config
    • Add Short type support
    • Support null values
    • Enabling Compression Codecs for Avro and Parquet
    • Switch to AWS client by default
    • Add option to add a padding when writing files, so that files can be restored in order by the source
    • Enable wildcard syntax to support multiple topics without additional configuration.
  • AWS S3 Source Connector

    • Add connection pool config
    • Retain partitions from filename or regex
    • Switch to AWS client by default
  • MQTT Source Connector

    • Allow toggling the skipping of MQTT Duplicates
  • MQTT Sink Connector

    • Functionality to ensure unique MQTT Client ID is used for MQTT sink
  • Elastic6 & Elastic7 Sink Connectors

    • Fixing issue with missing null values

4.0.0 

  • All

    • Scala 2.13 Upgrade
    • Gradle to SBT Migration
    • Producing multiple artifacts supporting both Kafka 2.8 and Kafka 3.1.
    • Upgrade to newer dependencies to reduce CVE count
    • Switch e2e tests from Java to Scala.
  • AWS S3 Sink Connector

    • Optimal seek algorithm - see documentation for more detail.
    • Parquet data size flushing fixes.
    • Adding date partitioning capability - see documentation for more detail.
    • Adding switch to use official AWS library - see documentation for more detail.
    • Add AWS STS dependency to ensure correct operation when assuming roles with a web identity token.
    • Provide better debugging in case of exceptions.
  • FTP Source Connector

    • Fixes to slice mode support.
  • Hazelcast Sink Connector

    • Upgrade to HazelCast 4.2.4. The configuration model has changed and now uses clusters instead of username and password configuration.
  • Hive Sink Connector

    • Update of parquet functionality to ensure operation with Parquet 1.12.2.
    • Support for Hive 3.1.3.
  • JMS Connector

    • Enable protobuf support.
  • Pulsar

    • Upgrade to Pulsar 2.10 and associated refactor to support new client API.

3.0.1 

  • All

    • Replace Log4j with Logback to overcome CVE-2021-44228
    • Bringing code from legacy dependencies inside of project
  • Cassandra Sink Connector

    • Ensuring the table name is logged on encountering an InvalidQueryException
  • HBase Sink Connector

    • Alleviate possible race condition

3.0.0 

  • All

    • Move to KCQL 2.8.9
    • Change sys.errors to ConnectExceptions
    • Additional testing with TestContainers
    • Licence scan report and status
  • AWS S3 Sink Connector

    • S3 Source Offset Fix
    • Fix JSON & Text newline detection when running in certain Docker images
    • Byte handling fixes
    • Partitioning of nested data
    • Error handling and retry logic
    • Handle preCommit with null currentOffsets
    • Remove bucket validation on startup
    • Enabled simpler management of default flush values.
    • Local write mode - build locally, then ship
    • Deprecating old properties, however rewriting them to the new properties to ensure backwards compatibility.
    • Adding the capability to specify properties in yaml configuration
    • Rework exception handling. Refactoring errors to use Either[X,Y] return types where possible instead of throwing exceptions.
    • Ensuring task can be stopped gracefully if it has not been started yet
    • ContextReader testing and refactor
    • Adding a simple state model to the S3Writer to ensure that states and transitions are kept consistent. This can be improved in time.
  • AWS S3 Source Connector

    • Change order of match to avoid scala.MatchError
    • S3 Source rewritten to be more efficient and use the natural ordering of S3 keys
    • Region is necessary when using the AWS client
  • Cassandra Sink & Source Connectors

    • Add connection and read client timeout
  • FTP Connector

    • Support for Secure File Transfer Protocol
  • Hive Sink Connector

    • Array Support
    • Kerberos debug flag added
  • Influx DB Sink

    • Bump influxdb-java from version 2.9 to 2.29
    • Added array handling support
  • MongoDB Sink Connector

    • Nested Fields Support
  • Redis Sink Connector

    • Fix Redis Pubsub Writer
    • Add support for json and json with schema

2.1.3 

Move to connect-common 2.0.5 that adds complex type support to KCQL

2.1.2 

  • AWS S3 Sink Connector
    • Prevent null pointer exception in converters when maps are presented will null values
    • Offset reader optimisation to reduce S3 load
    • Ensuring that commit only occurs after the preconfigured time interval when using WITH_FLUSH_INTERVAL
  • AWS S3 Source Connector (New Connector)
  • Cassandra Source Connector
    • Add Bucket Timeseries Mode
    • Reduction of logging noise
    • Proper handling of uninitialized connections on task stop()
  • Elasticsearch Sink Connector
    • Update default port
  • Hive Sink
    • Improve Orc format handling
    • Fixing issues with partitioning by non-string keys
  • Hive Source
    • Ensuring newly written files can be read by the hive connector by introduction of a refresh frequency configuration option.
  • Redis Sink
    • Correct Redis writer initialisation

2.1.0 

  • AWS S3 Sink Connector
  • Elasticsearch 7 Support

2.0.1 

  • Hive Source

    • Rename option connect.hive.hive.metastore to connect.hive.metastore
    • Rename option connect.hive.hive.metastore.uris to connect.hive.metastore.uris
  • Fix Elastic start up NPE

  • Fix to correct batch size extraction from KCQL on Pulsar

2.0.0 

  • Move to Scala 2.12
  • Move to Kafka 2.4.1 and Confluent 5.4
  • Deprecated:
    • Druid Sink (not scala 2.12 compatible)
    • Elastic Sink (not scala 2.12 compatible)
    • Elastic5 Sink(not scala 2.12 compatible)
  • Redis
    • Add support for Redis Streams
  • Cassandra
    • Add support for setting the LoadBalancer policy on the Cassandra Sink
  • ReThinkDB
    • Use SSL connection on Rethink initialize tables is ssl set
  • FTP Source
    • Respect connect.ftp.max.poll.records when reading slices
  • MQTT Source
    • Allow lookup of avro schema files with wildcard subscriptions