safekeeper: add eviction_min_resident to stop evictions thrashing

[jcsp]

Problem

• The condition for eviction is not time-based: it is possible for a timeline to be restored in response to a client, that client times out, and then as soon as the timeline is restored it is immediately evicted again.
• There is no delay on eviction at startup of the safekeeper, so when it starts up and sees many idle timelines, it does many evictions which will likely be immediately restored when someone uses the timeline.

Summary of changes

• Add eviction_min_resident parameter, and use it in ready_for_eviction to avoid evictions if the timeline has been resident for less than this period.
• This also implicitly delays evictions at startup for eviction_min_resident
• Set this to a very low number for the existing eviction test, which expects immediate eviction.

The default period is 10 minutes. The general reasoning for that is that in the worst case where we thrash ~10k timelines on one safekeeper, downloading 16MB for each one, we should set a period that would not overwhelm the node's bandwidth.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3042 tests run: 2927 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (1)

Postgres 14

• test_isolation[None]: debug

Code coverage* (full report)

• functions: 32.5% (6931 of 21297 functions)
• lines: 49.9% (54511 of 109162 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
d261b08 at 2024-07-10T18:48:08.982Z :recycle:}

[arssher]

I'd suggest to

• Explain in comments to the option that it shouldn't be normally needed because partial_backup_timeout already prevents eviction its value seconds/minutes after the write. However, it guards against eviction-read only access-quick deeviction pattern. I haven't seen yet evidence we do it though.
• Have the same default value for partial_backup_timeout and eviction_min_resident.

[jcsp]

Explain in comments to the option that it shouldn't be normally needed because partial_backup_timeout already prevents eviction its value seconds/minutes after the write. However, it guards against eviction-read only access-quick deeviction pattern. I haven't seen yet evidence we do it though.

When we saw that happen today, I think it was because clients had triggered the un-eviction, but then timed out before writing anything, so they left the timeline in the state where it was eligible for eviction (nothing pending upload)

Have the same default value for partial_backup_timeout and eviction_min_resident.

Makes sense, changed.

[arssher]

When we saw that happen today, I think it was because clients had triggered the un-eviction, but then timed out before writing anything, so they left the timeline in the state where it was eligible for eviction (nothing pending upload)

Probably, but I doubt it, see
https://neondb.slack.com/archives/C033RQ5SPDH/p1720626053871819?thread_ts=1720601531.744029&cid=C033RQ5SPDH
I think postgres has very little chance to start up without writing anything.