Evict WAL files from disk #8022

petuhovskiy · 2024-06-11T22:51:16Z

Add safekeeper support to switch between Present and Offloaded(flush_lsn) states. The offloading is disabled by default, but can be controlled using new cmdline arguments:

      --enable-offload
          Enable automatic switching to offloaded state
      --delete-offloaded-wal
          Delete local WAL files after offloading. When disabled, they will be left on disk
      --control-file-save-interval <CONTROL_FILE_SAVE_INTERVAL>
          Pending updates to control file will be automatically saved after this interval [default: 300s]

Manager watches state updates and detects when there are no actvity on the timeline and actual partial backup upload in remote storage. When all conditions are met, the state can be switched to offloaded.

In timeline.rs there is StateSK enum to support switching between states. When offloaded, code can access only control file structure and cannot use SafeKeeper to accept new WAL.

FullAccessTimeline is now renamed to WalResidentTimeline. This struct contains guard to notify manager about active tasks requiring on-disk WAL access. All guards are issued by the manager, all requests are sent via channel using ManagerCtl. When manager receives request to issue a guard, it unevicts timeline if it's currently evicted.

Fixed a bug in partial WAL backup, it used term instead of last_log_term previously.

After this commit is merged, next step is to roll this change out, as in issue #6338.

github-actions · 2024-06-11T23:00:02Z

2934 tests run: 2817 passed, 0 failed, 117 skipped (full report)

Flaky tests (1)

Postgres 15

test_tenant_delete_smoke: release

Code coverage* (full report)

functions: 32.6% (6890 of 21124 functions)
lines: 50.0% (53923 of 107928 lines)

* collected from Rust tests only

_{The comment gets automatically updated with the latest test results
9767942 at 2024-06-26T18:10:07.159Z :recycle:}

jcsp · 2024-06-19T14:58:49Z

Thinking about safekeepers that might re-download while some other safekeeper is uploading partial segments -- after this change, it's important that anything that is in a partial upload is committed data, since some other safekeeper might use it to re-download after eviction -- i.e. they must never upload partial segments that contain data that might not have achieved quorum write. Is that already the case?

petuhovskiy · 2024-06-19T15:09:12Z

since some other safekeeper might use it to re-download after eviction

Not really, each safekeeper uploads its own copy of partial segment. Safekeepers don't download partial segments from other safekeeper.

i.e. they must never upload partial segments that contain data that might not have achieved quorum write. Is that already the case?

No, currently safekeepers upload all bytes that are present on local disk, including the cases when flush_lsn is greater than commit_lsn.

safekeeper/src/bin/safekeeper.rs

safekeeper/src/lib.rs

safekeeper/src/wal_backup_partial.rs

safekeeper/src/timeline_manager.rs

safekeeper/src/timeline_eviction.rs

petuhovskiy · 2024-06-21T11:46:44Z

Control file upgrade is splitted to another PR: #8125
That PR is going to get deployed on prod early next week

This is a preparation for #8022, to make the PR both backwards and foward compatible. This commit adds `eviction_state` field to control file. Adds support for reading it, but writes control file in old format where possible, to keep the disk format forward compatible. Note: in `patch_control_file`, new field gets serialized to json like this: - `"eviction_state": "Present"` - `"eviction_state": {"Offloaded": "0/8F"}`

arssher

Looked through more than a half, LGTM, continuing, posting some comments accumulated so far.

BTW, TimelinePersistentState.partial_backup offloaded segment LSN always equals EvictionState::Offloaded LSN on evicted timelines, right?

safekeeper/src/timeline_manager.rs

safekeeper/src/timeline_eviction.rs

petuhovskiy · 2024-06-25T17:43:46Z

BTW, TimelinePersistentState.partial_backup offloaded segment LSN always equals EvictionState::Offloaded LSN on evicted timelines, right?

Yes.

safekeeper/src/timeline_eviction.rs

safekeeper/src/wal_backup_partial.rs

safekeeper/src/timeline_manager.rs

test_runner/regress/test_wal_acceptor.py

arssher

LGTM except
#8022 (comment)
though this PR shouldn't make the situation worse.

This is a preparation for #8022, to make the PR both backwards and foward compatible. This commit adds `eviction_state` field to control file. Adds support for reading it, but writes control file in old format where possible, to keep the disk format forward compatible. Note: in `patch_control_file`, new field gets serialized to json like this: - `"eviction_state": "Present"` - `"eviction_state": {"Offloaded": "0/8F"}`

Fixes #6337 Add safekeeper support to switch between `Present` and `Offloaded(flush_lsn)` states. The offloading is disabled by default, but can be controlled using new cmdline arguments: ``` --enable-offload Enable automatic switching to offloaded state --delete-offloaded-wal Delete local WAL files after offloading. When disabled, they will be left on disk --control-file-save-interval <CONTROL_FILE_SAVE_INTERVAL> Pending updates to control file will be automatically saved after this interval [default: 300s] ``` Manager watches state updates and detects when there are no actvity on the timeline and actual partial backup upload in remote storage. When all conditions are met, the state can be switched to offloaded. In `timeline.rs` there is `StateSK` enum to support switching between states. When offloaded, code can access only control file structure and cannot use `SafeKeeper` to accept new WAL. `FullAccessTimeline` is now renamed to `WalResidentTimeline`. This struct contains guard to notify manager about active tasks requiring on-disk WAL access. All guards are issued by the manager, all requests are sent via channel using `ManagerCtl`. When manager receives request to issue a guard, it unevicts timeline if it's currently evicted. Fixed a bug in partial WAL backup, it used `term` instead of `last_log_term` previously. After this commit is merged, next step is to roll this change out, as in issue #6338.

After #8022 was deployed to staging, I noticed many cases of timeouts. After inspecting the logs, I realized that some operations are taking ~20 seconds and they're doing while holding shared state lock. Usually it happens right after redeploy, because compute reconnections put high load on disks. This commit tries to improve observability around slow operations. Non-observability changes: - `TimelineState::finish_change` now skips update if nothing has changed - `wal_residence_guard()` timeout is set to 30s

petuhovskiy force-pushed the sk-s3-eviction branch from 4e6173c to c828cde Compare June 18, 2024 20:18

petuhovskiy changed the title ~~WIP: SK eviction~~ Evict WAL files from disk Jun 19, 2024

petuhovskiy marked this pull request as ready for review June 19, 2024 11:38

petuhovskiy requested a review from a team as a code owner June 19, 2024 11:38

petuhovskiy requested review from jcsp and arssher June 19, 2024 11:38

jcsp reviewed Jun 19, 2024

View reviewed changes

petuhovskiy mentioned this pull request Jun 20, 2024

Add eviction_state to control file #8125

Merged

petuhovskiy force-pushed the sk-s3-eviction branch from 52646b0 to 58f4fb8 Compare June 21, 2024 16:14

petuhovskiy added 16 commits June 24, 2024 13:19

Spawn partial backup on demand

274fbc1

Implement ManagerCtl

2a98f64

Add StateSK

619c96a

Fix tests

0abaf14

Fix guards usage in manager

dc394ae

WIP: add test_s3_eviction

24227e3

Implement sk state switch

87c9a65

Enable offloading

97b5f78

Enable automatic offloading switch

1a2fcab

Make eviction mostly work

5e5c397

Rebase

d0e9049

Restructure code

17e61b2

Fix test

200b302

Fix full_access_guard in start_snapshot

e682738

Rebase

87bfd3c

Add debug status to manager

c9cfef4

petuhovskiy added 5 commits June 24, 2024 13:19

Fix deadlock in backup task

ab8c74d

Add atomic rename

2e82866

Rename FullAccessTimeline to WalResidentTimeline

c29454a

Fix clippy warnings

a15f54f

Update remote_consistent_lsn in offloaded state

026868e

petuhovskiy force-pushed the sk-s3-eviction branch from 0950055 to 026868e Compare June 24, 2024 13:34

petuhovskiy added 2 commits June 24, 2024 13:50

Rename timeline_access.rs

d226955

Fix test_s3_eviction

6bef78a

petuhovskiy force-pushed the sk-s3-eviction branch from 992b565 to 6bef78a Compare June 24, 2024 18:20

petuhovskiy mentioned this pull request Jun 24, 2024

Rename "WAL backup" to "WAL upload" #8146

Open

petuhovskiy added 6 commits June 24, 2024 19:02

Add control_file_save_interval

e88a8c7

Add comments

02077ad

Add comments after self-review

63af675

Fix misc

a1fb21d

Fix test flakiness

1f0ed2e

Revert testing conf

1864721

arssher reviewed Jun 25, 2024

View reviewed changes

Fix review comments

8be0efe

arssher reviewed Jun 26, 2024

View reviewed changes

safekeeper/src/timeline_eviction.rs Show resolved Hide resolved

safekeeper/src/wal_backup_partial.rs Outdated Show resolved Hide resolved

safekeeper/src/timeline_manager.rs Show resolved Hide resolved

test_runner/regress/test_wal_acceptor.py Outdated Show resolved Hide resolved

arssher approved these changes Jun 26, 2024

View reviewed changes

petuhovskiy added 2 commits June 26, 2024 15:27

Assert uneviction in test

4f6feb6

Fix review comments

9767942

petuhovskiy force-pushed the sk-s3-eviction branch from 8d9e72d to 9767942 Compare June 26, 2024 16:57

petuhovskiy merged commit 76fc3d4 into main Jun 26, 2024
61 of 62 checks passed

petuhovskiy deleted the sk-s3-eviction branch June 26, 2024 17:58

petuhovskiy mentioned this pull request Jun 27, 2024

Improve slow operations observability in safekeepers #8188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Evict WAL files from disk #8022

Evict WAL files from disk #8022

petuhovskiy commented Jun 11, 2024 •

edited

Loading

github-actions bot commented Jun 11, 2024 •

edited

Loading

Postgres 15

jcsp commented Jun 19, 2024

petuhovskiy commented Jun 19, 2024

petuhovskiy commented Jun 21, 2024

arssher left a comment

petuhovskiy commented Jun 25, 2024

arssher left a comment

Evict WAL files from disk #8022

Evict WAL files from disk #8022

Conversation

petuhovskiy commented Jun 11, 2024 • edited Loading

github-actions bot commented Jun 11, 2024 • edited Loading

2934 tests run: 2817 passed, 0 failed, 117 skipped (full report)

Postgres 15

Code coverage* (full report)

jcsp commented Jun 19, 2024

petuhovskiy commented Jun 19, 2024

petuhovskiy commented Jun 21, 2024

arssher left a comment

Choose a reason for hiding this comment

petuhovskiy commented Jun 25, 2024

arssher left a comment

Choose a reason for hiding this comment

petuhovskiy commented Jun 11, 2024 •

edited

Loading

github-actions bot commented Jun 11, 2024 •

edited

Loading