fix: EphmeralFiles can outlive their Timeline, new gate version

[koivunej]

Ephemeral files cleanup on drop but did not delay shutdown, leading to problems with restarting the tenant. The solution is:

• make ephemeral files carry the gate guard to delay Timeline::shutdown
• flush in-memory layers and strong references to those on Timeline::shutdown
• add a different gate LayerManager::ephmeral_files which is closed after all Timeline operations are closed

Fixes: #7830

[koivunej]

The two-gate idea comes from @jcsp following the larger restructuring on #8229. Compared to the original solution where we track whether LayerManager is open or closed (shutdown), this version:

• has a lot less changed lines (though some are "extra" on 8229)
• now instead of type system we get the same safety from the runtime state of the two gates

[koivunej]

Will push the doc fix after regress tests.

[github-actions]

3042 tests run: 2926 passed, 1 failed, 115 skipped (full report)

---

Failures on Postgres 14

• test_pg_regress[4]: debug

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_pg_regress[debug-pg14-4]"

Flaky tests (1)

Postgres 16

• test_isolation[None]: debug

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
607c044 at 2024-07-10T10:50:17.257Z :recycle:}

[problame]

now instead of type system we get the same safety from the runtime state of the two gates

I don't get how we get any of the safety I asked for in #8229 (comment)

[koivunej]

now instead of type system we get the same safety from the runtime state of the two gates

I don't get how we get any of the safety I asked for in #8229 (comment)

Earlier comment:

By simply taking out the in-memory layers, Timeline::get will be giving out wrong answers because we're not resetting last_record_lsn.

The only thing that prevents us from actually giving out the wrong answer is that we check for the cancellation token. Too brittle for my taste.

The two gates is the answer:

1. Timeline::cancel is cancelled
2. Timeline::gate is closed
   • everything except ephemeral files (should) use that
3. After that we drop the in-memory layers and close the added LayerManager::ephmeral_files: Gate

I couldn't spot any dire implementation error on my part, so could you please be more specific in case your question was not of general nature...?

[problame]

I think we just disagree on whether we need robustness in depth or not.
This PR isn't implementing the robustness in depth that I was asking for in #8229 (comment) (enum LayerMap { ... }).

[problame]

Ok, Joonas and I had a call and he explained this PR to me.

It's a clever idea with few LoC but I would much prefer the robustness-in-depth that we gain from an explicitly tracked shutdown state.

See also my comment here: #8229 (comment)

[koivunej]

Per Christians notes above, closing this down. Risk of no longer reading a should-be-present in-memory layer could cause us a lot of corruption.