stacked atop #5824

We can probably get rid of the entire trait, but, that's for another time.

…uilder-refactor' into problame/integrate-tokio-epoll-uring/write-path/simplify-write-at-api

This reverts commit b5a00b0.

…ath/refactor-blob-writer

…ath/refactor-blob-writer

…r-blob-writer' into problame/integrate-tokio-epoll-uring/write-path/refactor-virtual-file

…ath/refactor-blob-writer

…r-blob-writer' into problame/integrate-tokio-epoll-uring/write-path/virtual-file-write-all

…kio-epoll-uring/write-path/refactor-blob-writer

…r-blob-writer' into problame/integrate-tokio-epoll-uring/write-path/virtual-file-write-all

…kio-epoll-uring/more-ops

…w that IoBuf needs to be Send; PITA but feasible

…ring/write-path/virtual-file-write-all-at' into problame/integrate-tokio-epoll-uring/more-ops

…ath/virtual-file-write-all

…-file-write-all' into problame/integrate-tokio-epoll-uring/write-path/virtual-file-write-all-at

…-file-write-all-at' into problame/integrate-tokio-epoll-uring/more-ops

…ath/virtual-file-write-all-at

…ath/virtual-file-write-all-at

…kio-epoll-uring/write-path/virtual-file-write-all-at

…-file-write-all-at' into problame/integrate-tokio-epoll-uring/more-ops

The `writer.finish()` methods already fsync the inode,
using `VirtualFile::sync_all()`.

All that the callers need to do is fsync their directory,
i.e., the timeline directory.

Note that there's a call in the new compaction code that
is apparently dead-at-runtime, so, I couldn't fix up
any fsyncs there [Link](https://github.com/neondatabase/neon/blob/502b69b33bbd4ad1b0647e921a9c665249a2cd62/pageserver/src/tenant/timeline/compaction.rs#L204-L211).

In the grand scheme of things, layer durability probably doesn't
matter anymore because the remote storage is authoritative at all times
as of #5198. But, let's not break the discipline in htis commit.

part of #6663

…sync-cleanups' into problame/integrate-tokio-epoll-uring/more-ops

…nc_all()

Except for the involvement of the VirtualFile fd cache, this is
equivalent to what happened before at runtime.

Future PR #6378 will implement `VirtualFile::sync_all()` using
tokio-epoll-uring if that's configured as the io engine.
This PR is preliminary work for that.

As pointed out in the comments added in this PR:
the in-memory state of the filesystem already has the layer file in its final place.
If the fsync fails, but pageserver continues to execute, it's quite easy
for subsequent pageserver code to observe the file being there and
assume it's durable, where it really isn't.

It can happen that we get ENOSPC during the fsync.
However,
1. the timeline dir is small (remember, the big layer _file_ has already been synced).
   Small data means ENOSPC due to delayed allocation races etc are less likely.
2. what elase are we going to do in that case?

If we decide to bubble up the error, the file remains on disk.
We could try to unlink it and fsync after the unlink.
If that fails, we would _definitely_ need to error out.
Is it worth the trouble though?

Side note: all this logic about not carrying on after fsync failure
implies that we `sync` the filesystem successfully before we restart
the pageserver. Our systemd unit currently does not do that, but should.

…kio-epoll-uring/layer-write-path-fsync-cleanups

…sync-cleanups' into problame/integrate-tokio-epoll-uring/create-layer-fatal-err-on-fsync

…kio-epoll-uring/layer-write-path-fsync-cleanups

…sync-cleanups' into problame/integrate-tokio-epoll-uring/create-layer-fatal-err-on-fsync

…-err-on-fsync' into problame/integrate-tokio-epoll-uring/ioengine-par-fsync

…' into problame/integrate-tokio-epoll-uring/more-ops

This reverts commit 0548374.