-
Notifications
You must be signed in to change notification settings - Fork 564
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(storage): support different snapshot for streaming jobs #15896
Conversation
…mmock-snapshot-group
…mmock-snapshot-group
…mmock-snapshot-group
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implementation looks good but I have a general question:
Is SnapshotGroup
a meta only concept? In other words, do compute/frontent/compactor nodes need to be aware of this concept? Given that we put it in HummockVersionDelta
and all nodes uses HummockVersionDelta
to update their local version, it implies that SnapshotGroup
is "leaked" to all components.
I have changed to maintain snapshot per table. @hzxa21 PTAL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Discussed offline: we can also include compaction group id in StateTableInfo
and deprecate member_table_ids
.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Previously, all state tables shared a same global committed epoch and safe epoch. To introduce partial checkpoint, each streaming job will have different snapshot (committed and safe epoch). Therefore, in this PR, we introduce
SnapshotGroup
. The state table ids of a streaming job (table fragments) will be in the same group and share a same snapshot, while different streaming jobs can have different snapshot. Though in this PR we will support different snapshots for different streaming jobs, we still maintain that all streaming jobs will have the same snapshot. In the future when we implement and enable partial checkpoint, we can have different snapshots for different streaming jobs.Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.