Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run gc on too many partial backup segments #7700

Merged
merged 1 commit into from
May 31, 2024
Merged

Conversation

petuhovskiy
Copy link
Member

The general partial backup idea is that each safekeeper keeps only one partial segment in remote storage at a time. Sometimes this is not true, for example if we uploaded object to S3 but got an error when tried to remove the previous upload. In this case we still keep a list of all potentially uploaded objects in safekeeper state.

This commit prints a warning to logs if there is too many objects in safekeeper state. This is not expected and we should try to fix this state, we can do this by running gc.

I haven't seen this being an issue anywhere, but printing a warning is something that I wanted to do and forgot in initial PR.

@petuhovskiy petuhovskiy requested a review from a team as a code owner May 10, 2024 13:33
@petuhovskiy petuhovskiy requested a review from arssher May 10, 2024 13:33
Copy link

3024 tests run: 2891 passed, 0 failed, 133 skipped (full report)


Flaky tests (5)

Postgres 15

  • test_partial_evict_tenant[relative_equal]: release
  • test_synthetic_size_while_deleting: release

Postgres 14

Code coverage* (full report)

  • functions: 31.4% (6324 of 20137 functions)
  • lines: 47.3% (47667 of 100781 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
1a97da1 at 2024-05-10T14:19:06.690Z :recycle:

Copy link
Contributor

@arssher arssher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if smth specific caused this PR.

@petuhovskiy
Copy link
Member Author

I wonder if smth specific caused this PR.

Not really. Just was thinking about S3 objects count and realized that if we have upload-error-retry loop we can have too many objects and it's hard to spot this.

@petuhovskiy petuhovskiy merged commit e98bc4f into main May 31, 2024
54 checks passed
@petuhovskiy petuhovskiy deleted the sk-partial-backup-limit branch May 31, 2024 23:18
a-masterov pushed a commit that referenced this pull request Jun 3, 2024
The general partial backup idea is that each safekeeper keeps only one
partial segment in remote storage at a time. Sometimes this is not true,
for example if we uploaded object to S3 but got an error when tried to
remove the previous upload. In this case we still keep a list of all
potentially uploaded objects in safekeeper state.

This commit prints a warning to logs if there is too many objects in
safekeeper state. This is not expected and we should try to fix this
state, we can do this by running gc.

I haven't seen this being an issue anywhere, but printing a warning is
something that I wanted to do and forgot in initial PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants