Epic: evict inactive timelines on safekeepers to s3

[arssher]

Tasks

Beta Give feedback

1. Upload partial WAL segments to S3 #6336
   c/storage/safekeeper +1
   
2. Simplify backup task spawning in safekeepers #7751
   c/storage/safekeeper +1
   
3. Refactor safekeepers to support special timeline modes #6337
   c/storage/safekeeper +1
   
4. Partial upload is not working well together will pull_timeline #8542
   c/storage/safekeeper t/bug +2
   
5. Roll out safekeeper eviction with --delete-offloaded-wal #6338
   c/storage/safekeeper +1
   

[problame]

Progress on #6337 planned this week, but, multi-week project.

[problame]

This week:

• monitor release of PR that was merged last week (TODO: link)
• impl & create PR for Refactor safekeepers to support special timeline modes #6337

[koivunej]

Last week:

• Refactoring PR no. 2 was released
• Arseny found a bug related to refactoring PR, it was fixed too many timelines in broker_active_set #7967

This week:

• need to create the last PR to implement eviction
• then this issue should be ready

[petuhovskiy]

Last week:

• eviction PR was merged to staging, but not enabled anywhere
• then I discovered a bunch of issues and merged some fixes
• verified that no critical issues are left on staging

This week:

• waiting for production deploy to ensure nothing break with DISABLED eviction
• after that, we can enable eviction in staging and look for new issues

Next week:

• if everything will be good in staging, we should enable eviction in prod

Later:

• enable file deletion gradually in all regions (first staging, then prod)

[jcsp]

Status: #6338 is blocked on #8542