Add support for ingesting older iceberg snapshots #15348
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cryptoe
reviewed
Nov 15, 2023
| @@ -78,14 +82,16 @@ public IcebergInputSource( | |||
| @JsonProperty("namespace") String namespace, | |||
| @JsonProperty("icebergFilter") @Nullable IcebergFilter icebergFilter, | |||
| @JsonProperty("icebergCatalog") IcebergCatalog icebergCatalog, | |||
| @JsonProperty("warehouseSource") InputSourceFactory warehouseSource | |||
| @JsonProperty("warehouseSource") InputSourceFactory warehouseSource, | |||
| @JsonProperty("snapshotTime") DateTime snapshotTime | |||
There was a problem hiding this comment.
This can be nullable rite ? Might as well mark it nullable.
cryptoe
reviewed
Nov 15, 2023
| @@ -164,6 +170,12 @@ public IcebergFilter getIcebergFilter() | |||
| return icebergFilter; | |||
| } | |||
|
|
|||
| @JsonProperty | |||
There was a problem hiding this comment.
I generally mark the getter nullable as well.
|
@a2l007 Changes LGTM. Minor comments only. |
|
@cryptoe Thanks, I've addressed your comments. |
cryptoe
approved these changes
Nov 17, 2023
|
Test failure due to : apache/druid-website#252 which removed the wikipedia data set from apache druid website. |
writer-jill
pushed a commit
to writer-jill/druid
that referenced
this pull request
Nov 20, 2023
This patch introduces a param snapshotTime in the iceberg inputsource spec that allows the user to ingest data files associated with the most recent snapshot as of the given time. This helps the user ingest data based on older snapshots by specifying the associated snapshot time. This patch also upgrades the iceberg core version to 1.4.1
yashdeep97
pushed a commit
to yashdeep97/druid
that referenced
this pull request
Dec 1, 2023
This patch introduces a param snapshotTime in the iceberg inputsource spec that allows the user to ingest data files associated with the most recent snapshot as of the given time. This helps the user ingest data based on older snapshots by specifying the associated snapshot time. This patch also upgrades the iceberg core version to 1.4.1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This patch introduces a param
snapshotTimein the iceberg inputsource spec that allows the user to ingest data files associated with the most recent snapshot as of the given time. This helps the user ingest data based on older snapshots by specifying the associated snapshot time.This patch also upgrades the iceberg core version to
1.4.1This PR has: