Releases: snowflakedb/snowflake-kafka-connector
v2.4.0
Notable changes
• Upgrade of Snowflake Ingest SDK to the latest 2.2.0 version - containing a critical fix for potential corruption when change_tracking is enabled. - see https://github.com/snowflakedb/snowflake-ingest-java/releases/tag/v2.2.0 for more details
We strongly recommend upgrading Kafka connector to this 2.4.0+ version if you are using Snowpipe Streaming functionality
• Upgrade of Snowflake JDBC driver from version 3.14.5 to version 3.18.0 - see https://docs.snowflake.com/en/release-notes/clients-drivers/jdbc-2024 for more details,
• Fixed issue with connector loosing data when reopening a channel fails before fully initializing channel: Fix to SNOW-1514185: Do assign the reopened channel unless Kafka offsets are fully reset (#875),
• Improved logging experience in various components for improved troubleshooting experience,
• Updated dependencies with known vulnerabilities.
What's Changed
• SNOW-1618257 - upgrade to 2.2.0 ingest-sdk by @sfc-gh-xhuang in (#910)
• NO_SNOW Update dependencies by @sfc-gh-xhuang in (#898)
• SNOW-1229108 Add warning about buffer config when single buffer enabled by @sfc-gh-mbobowski in (#896)
• NO-SNOW Ignore OAuth test by @sfc-gh-mbobowski in (#906)
• Bump org.sonatype.plugins:nexus-staging-maven-plugin from 1.6.7 to 1.7.0 in (#857)
• Bump io.dropwizard.metrics:metrics-core from 4.2.3 to 4.2.26 in (#863)
• Bump org.assertj:assertj-core from 3.25.3 to 3.26.3 (#881)
• Bump dev.failsafe:failsafe from 3.2.1 to 3.3.2 (#859)
• SNOW-947731 Remove deprecated avro-python3 package by @sfc-gh-mbobowski in (#889) (
• SNOW-1541942 Extend Snowpipe initialization logging at INFO level by @sfc-gh-mbobowski in (#885)
• SNOW-1229110 Add single buffer usage data to telemetry by @sfc-gh-mbobowski in (#893)
• Upgrade com.google.protobuf to 3.24.4 by @sfc-gh-wtrefon in (#892)
• NO-SNOW Upgrade nexus-staging-maven-plugin version by @sfc-gh-xhuang in (#887)
• Remove insertRows DEBUG log (called once per row) by @sfc-gh-mbobowski in (#886)
• Adding logs to track schematization by @sfc-gh-mbobowski in (#884)
• NOSNOW: improve troubleshooting experience for scenario with skipped offsets by @sfc-gh-gjachimko in (#883)
• SNOW-1061855 Fix the E2E test for SchemaEvolutionDropTable with a single buffer by @sfc-gh-gjachimko in (#882)
• SNOW-1061851: Run SnowflakeSinkServiceV2IT for a single buffer by @sfc-gh-akowalczyk in (#876)
• SNOW-1514185: Do assign the reopened channel unless kafka offsets are fully reset by @sfc-gh-akowalczyk in (#875)
• NOSNOW: added utility to upload connector to our nexus for simplified k8s deployment by @sfc-gh-gjachimko in (#874)
Contributors
Assets 2
v2.3.0
Notable changes
- (Snowpipe Streaming) Timestamp of pushing data into Snowflake is added to record_metadata. Note according to the configured kafka buffers, the timestamp value may be have a slight delay between when it is pushed to the Snowpipe Streaming channel buffer and when it is flushed and available on the table.
- SNOW-1056407 Add SnowflakeConnectorPushTime to record_metadata by @sfc-gh-lshcharbaty in #833
- (Snowpipe Streaming) Optional parameter,
snowflake.streaming.closeChannelsInParallel.enabled, for Kafka connector to close channels in parallel which significantly speeds up rebalance time- SNOW-1369280 Close channels asynchronously by @sfc-gh-lshcharbaty in #841
- Updated dependencies with known vulnerabilities
- SGRC-4405 Update Kafka to 3.7.0 and Confluent to 7.6.0 by @sfc-gh-mbobowski in #854
What's Changed
- SNOW-1269024 Installing multiple Java versions for test purposes by @sfc-gh-krosinski in #824
- SNOW-1269025 Testing Confluent kafka connector using multiple JRE versions by @sfc-gh-krosinski in #826
- SNOW-1269038 Stress testing Confluent kafka connector using multiple JRE versions by @sfc-gh-krosinski in #827
- SNOW-1348656 Migrate RecordContentTest to JUnit5 by @sfc-gh-mbobowski in #828
- SNOW-1056407 Cleanup metadata properties class. Migrate records tests to junit5 by @sfc-gh-lshcharbaty in #831
- NO-SNOW Fix build warnings by declaring maven-plugin versions by @sfc-gh-xhuang in #837
- NO-SNOW Fix idempotent multiple column schema evolution by @sfc-gh-xhuang with contribution from @cmonty-paypal in #836
- SNOW-1348656 Send records that could not be parsed to DLQ by @sfc-gh-mbobowski in #830
- SNOW-1056407 Add tests MetaColumnTest for snowpipe streaming by @sfc-gh-lshcharbaty in #832
- SNOW-1369271 Ignore SFException when closing a channel by @sfc-gh-lshcharbaty in #839
- NO-SNOW Remove unused code path from RecordService by @sfc-gh-mbobowski in #838
- NO-SNOW Ugrade to ingest-sdk 2.1.1 by @sfc-gh-xhuang in #843
- SNOW-1167007: Migrate schema when receiving null values for not null cols by @sfc-gh-wtrefon in #849
- SNOW-1061848: Introduce TopicPartitionChannel interface by @sfc-gh-akowalczyk in #851
- SNOW-1061822: Add new properties for streaming client by @sfc-gh-wtrefon in #822
- SNOW-1061848: DirectTopicPartitionChannel - copy of the old implementation with removed buffer by @sfc-gh-akowalczyk in #852
- SNOW-1061848: Fix unit tests by @sfc-gh-akowalczyk in #855
- NO-SNOW Add dependabot by @sfc-gh-mbobowski in #856
- SNOW-1061848: Run TopicPartitionChannelIT against a non-buffered path by @sfc-gh-akowalczyk in #862
- SNOW-1475664: Move additional error handling from BufferedTopicPartitionChannel to fix E2E test by @sfc-gh-akowalczyk in #865
- NO_SNOW Add column comments from schemaregistry by @sfc-gh-akowalczyk with contribution from @wornjs in #870
- SNOW-1061854 Run e2e tests with single buffer by @sfc-gh-lshcharbaty in #867
- SNOW-1019628 Enable V2 file cleaner for Snowpipe ingestion by default by @sfc-gh-gjachimko in #868
- Release v2.3.0 by @sfc-gh-mbobowski in #873
New Contributors
- @sfc-gh-krosinski made their first contribution in #824
- @sfc-gh-nsienkiewicz-1 made their first contribution in #842
Full Changelog: v2.2.2...v.2.3.0
Contributors
Assets 2
v2.2.2
Notable changes
- Fixed bug that caused stage files being not deleted in Snowpipe ingestion
- SNOW-1019628 update file cleaner for snowpipe ingest by @sfc-gh-gjachimko in #807
What's Changed
- SNOW-1055524 E2E test with nullable SMT by @sfc-gh-lshcharbaty in #808
- SNOW-1057165 - Test-doubles for snowflake-ingest-java by @sfc-gh-achyzy in #809
- SNOW-1055524 E2E SMT without schema evolution by @sfc-gh-lshcharbaty in #811
- SNOW-1055524 Check table schema in e2e smt by @sfc-gh-lshcharbaty in #813
- SNOW-1055524 E2E SMT for a Snowpipe based connector by @sfc-gh-lshcharbaty in #812
- NO-SNOW Remove performance tests by @sfc-gh-mbobowski in #817
- SNOW-1055561: Check whether SMT returning null values no longer stops a data ingestion. by @sfc-gh-akowalczyk in #816
- Update to release 2.2.2 by @sfc-gh-rcheng in #819
New Contributors
- @sfc-gh-lshcharbaty made their first contribution in #808
- @sfc-gh-achyzy made their first contribution in #809
- @sfc-gh-akowalczyk made their first contribution in #816
- @sfc-gh-krosinski made their first contribution in #824
Full Changelog: v2.2.1...v2.2.2
Contributors
Assets 2
v2.2.1
Notable Changes
- SNOW-979849: Add offset verification logic by @sfc-gh-tzhang in #795
- SNOW-1117446 Snowpipe Streaming client provider map by @sfc-gh-japatel in #794
- SNOW-1049869 Cleanup streaming ingest threads when SinkTask stop() is called by @sfc-gh-mbobowski in #800
What's Changed
- SNOW-922795 Added third-party-licenses to .zip distribution by @sfc-gh-mbobowski in #784
- PROD-39429 Add PR template by @sfc-gh-japatel in #786
- Run subset of e2e tests by @sfc-gh-mbobowski in #788
- NO-SNOW: upgrade JDBC to 3.14.5 by @sfc-gh-tzhang in #799
- [external contributor] Upgrade dependency versions by @bansallovish merged by @sfc-gh-xhuang in #802
- Upgrade connect-api to 3.7.0 by @sfc-gh-mbobowski in #803
- SNOW-1057151: Add poc test setup for EmbeddedConnectCluster by @sfc-gh-wtrefon in #801
- SNOW-1049869 Rewrite client optimization unit tests by @sfc-gh-mbobowski in #804
- Release v2.2.1 by @sfc-gh-rcheng in #805
New Contributors
- @sfc-gh-mbobowski made their first contribution in #784
- @sfc-gh-wtrefon made their first contribution in #801
Full Changelog: v2.2.0...v2.2.1
Contributors
Assets 2
v2.2.0
Notable Changes
- Security fix that enforces client authentication that results in more clients being created. This may increase your client usage costs
- [SNOW-954150] Use map of clients with different configurations instead of one client for multiple connectors configurations by @sfc-gh-rcheng in #744
- Corruption fix that ensures exactly once delivery for schema evolution edge cases
- SNOW-943288, SNOW-938379: Fix two schema evolution cases that could cause non-exactly once delivery by @sfc-gh-tzhang in #775
What's Changed
- NO-SNOW Make schema evolution add columns idempotent by @sfc-gh-xhuang in #734
- NO-SNOW: Preserve the old data type that goes into an ARRAY column for Schematization by @sfc-gh-tzhang in #730
- SNOW-938038: Support AVRO Logical Types by @sfc-gh-tzhang in #722
- [SNOW-943288] Do not skip records when we're expecting the offset to be reset by @sfc-gh-rcheng in #729
- PROD-39429 Implement migrate sys func from new channel(Format V2) to old channel (V1) - Push to Main by @sfc-gh-japatel in #751
- [external contributor] Support config providers in validation by @swehner by @sfc-gh-rcheng in #753
- NO-SNOW: Expose the Ingest SDK MAX_CLIENT_LAG configuration in KC by @sfc-gh-tzhang in #758
- Add authentication for max lag parameter and fix IT by @sfc-gh-rcheng in #763
- SNOW-989387: Fix for java.lang.NoClassDefFoundError by @sfc-gh-wfateem in #774
- [SNOW-1015644] Update ingest sdk to 2.0.5 by @sfc-gh-rcheng in https://github.com/snowflakedb/snowflake-kafka-connector/pull/778f
- resolves issue with generating and building java libary by @sfc-gh-gjachimko in #780
- Update version to 2.2.0 for release by @sfc-gh-rcheng in #779
- Downgrade JDBC to 3.13.30 by @sfc-gh-rcheng in #783
New Contributors
- @sfc-gh-xhuang made their first contribution in #734
- @sfc-gh-gjachimko made their first contribution in #780
Full Changelog: v2.1.2...v2.2.0
Contributors
Assets 2
v2.1.2
IMPORTANT This version supplants previous 2.1.x versions and is the recommended version. Do not use earlier 2.1.x versions.
What's Changed
- SNOW-189106 Kafka Connector to Support External OAuth by @sfc-gh-alhuang in #671
- [SNOW-870373] Enable JMX metrics for Snowpipe Streaming by @sfc-gh-rcheng in #674
- [SNOW-851840] Enable tombstone record ingestion in Snowpipe Streaming by @sfc-gh-rcheng in #688, #709 and #700
- SNOW-903979 Set default to true for one client optimization by @sfc-gh-japatel in #695
- SNOW-808908: Handle Reserved Keywords and Special Characters in Schematization by @sfc-gh-tzhang in #693
- [Streaming Telemetry 2][SNOW-899866] Enables reportKafkaPartitionStart/Usage telemetry for Streaming by @sfc-gh-rcheng in #694 and #696
- [NO-SNOW] Prefix with channel name with connector name by @sfc-gh-japatel in #703
- PROD-39429 Implement migrate sys func from new channel(Format V2) to old channel (V1) by @sfc-gh-japatel in #750
Package Updates
- Ingest SDK 2.0.2 to 2.0.3 by @sfc-gh-rcheng in #699
- Avro 1.11.1 to 1.11.3 by @sfc-gh-snowflakedb-snyk-sa in #713 and #718
New Contributors
- @sfc-gh-alhuang made their first contribution in #683
- @sfc-gh-snowflakedb-snyk-sa made their first contribution in #713
Full Changelog: v2.0.1...v2.1.2
Contributors
Assets 2
2.0.2-rc
This release is on top of release https://github.com/snowflakedb/snowflake-kafka-connector/releases/tag/v2.0.1
Important commits between two are:
v2.0.1
Notable Changes
- SNOW-811265 Not clearing entire map of partitionToChannel, only remove by @sfc-gh-japatel in #687
- SNOW-858328 Add cache for schema evolution permission query by @sfc-gh-alhuang in #683
Other Changes
- Small changes to Snowpipe Telemetry and JMX by @sfc-gh-rcheng in #672
- Change pipe already registered log from error to warn level by @sfc-gh-rcheng in #679
Full Changelog: v2.0.0...v2.0.1
Contributors
Assets 2
v2.0.0
Snowpipe Streaming with Kafka Connector is now Generally Available
Reverts c5bb312 to close channel on rebalance
Full Changelog: v1.9.4...v2.0.0
1.9.4
One client
- Release 1.9.4 introduces the enable.streaming.client.optimization option, which is enabled by default.
- With this client optimization, only one client is created for multiple topic partitions per Kafka connector. This feature can reduce client runtime and lower migration cost by creating larger files.
- Note that in a high throughput scenario (for example, 50 MB/s per connector), we recommend that you disable this property if you see an increase in latency or costs.
What's Changed
- Refactor buffer threshold variables and methods to be more consistent by @sfc-gh-rcheng in #639
- [SNOW-840882] Disable flakey streaming client test by @sfc-gh-rcheng in #654
- SNOW-826772 Unification of Snowflake role and user for Snowpipe Streaming by @sfc-gh-japatel in #650
- NO-SNOW Kafka connector supports any encoding, not only UTF-8 by @sfc-gh-japatel in #652
- Upgrade to SDK 2.0.1 and fix corresponding override bdec version test by @sfc-gh-rcheng in #653
- no-snow update Google guava by @sfc-gh-japatel in #655
- enzo-cappa fix: add warn log when a file failed to be ingested by @sfc-gh-rcheng in #656
- Revert removal of TestSchemaEvolutionJson e2e test by @sfc-gh-rcheng in #659
- SNOW-811265 Do not close channel on rebalance by @sfc-gh-japatel in #651
- SNOW-850959: Fix a wrong result issue that offsets are skipped when schematization is enabled by @sfc-gh-tzhang in #658
- [SNOW-857447] Enable one client optimization by default by @sfc-gh-rcheng in #665
- Release version 1.9.4 by @sfc-gh-rcheng in #666
Changes made but reverted
- Add, attempt fix and ultimately remove flakey CLA bot
- Fix cla bot by using correct github token by @sfc-gh-rcheng in #640
- Remove CLA bot and update README.md to include cla and formatter requirements by @sfc-gh-rcheng in #643
- Change Kafka put and precommit log level to info and then revert
- Revert kafka put and precommit log message to info by @sfc-gh-rcheng in #637
- NO-SNOW Too many info logs for successfully called put and precommit,… by @sfc-gh-japatel in #660
Full Changelog: v1.9.3...v1.9.4