Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): support different snapshot for streaming jobs #15896

Merged
merged 47 commits into from
Jun 5, 2024

Conversation

wenym1
Copy link
Contributor

@wenym1 wenym1 commented Mar 25, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Previously, all state tables shared a same global committed epoch and safe epoch. To introduce partial checkpoint, each streaming job will have different snapshot (committed and safe epoch). Therefore, in this PR, we introduce SnapshotGroup. The state table ids of a streaming job (table fragments) will be in the same group and share a same snapshot, while different streaming jobs can have different snapshot. Though in this PR we will support different snapshots for different streaming jobs, we still maintain that all streaming jobs will have the same snapshot. In the future when we implement and enable partial checkpoint, we can have different snapshots for different streaming jobs.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Base automatically changed from yiming/register-hummock-table-in-commit to main March 26, 2024 08:15
@wenym1 wenym1 requested a review from a team as a code owner May 24, 2024 10:45
Copy link
Collaborator

@hzxa21 hzxa21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation looks good but I have a general question:
Is SnapshotGroup a meta only concept? In other words, do compute/frontent/compactor nodes need to be aware of this concept? Given that we put it in HummockVersionDelta and all nodes uses HummockVersionDelta to update their local version, it implies that SnapshotGroup is "leaked" to all components.

proto/hummock.proto Outdated Show resolved Hide resolved
@wenym1
Copy link
Contributor Author

wenym1 commented May 30, 2024

I have changed to maintain snapshot per table. @hzxa21 PTAL

@wenym1 wenym1 requested a review from hzxa21 May 30, 2024 05:45
Copy link
Collaborator

@hzxa21 hzxa21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Discussed offline: we can also include compaction group id in StateTableInfo and deprecate member_table_ids.

@wenym1 wenym1 added this pull request to the merge queue Jun 5, 2024
Merged via the queue into main with commit 690b84a Jun 5, 2024
32 of 33 checks passed
@wenym1 wenym1 deleted the yiming/hummock-snapshot-group branch June 5, 2024 11:50
@zwang28 zwang28 mentioned this pull request Jun 6, 2024
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants