pageserver: fix vectored read path delta layer index traversal

[VladLazar]

Problem

Last weeks enablement of vectored get generated a number of panics.
From them, I diagnosed two issues in the delta layer index traversal logic

1. The key >= range.start && lsn >= lsn_range.start
   was too aggressive. Lsns are not monotonically increasing in the delta
   layer index (keys are though), so we cannot assert on them.
2. Lsns greater or equal to lsn_range.end were not skipped. This
   caused the query to consider records newer than the request Lsn.

Summary of changes

• Fix the issues mentioned above inline
• Refactor the layer traversal logic to make it unit testable
• Add unit test which reproduces the failure modes listed above.

Reviewing

I've applied the fixes before doing the refactoring and adding the new test,
so it might be easier to go by commit.

Closes #6978

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2484 tests run: 2361 passed, 0 failed, 123 skipped (full report)

---

Flaky tests (2)

Postgres 15

• test_secondary_downloads: release
• test_vm_bit_clear_on_heap_lock: debug

Code coverage* (full report)

• functions: 28.8% (6969 of 24209 functions)
• lines: 47.4% (42795 of 90373 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
9036ded at 2024-03-05T12:17:23.029Z :recycle:}

[koivunej]

Some comments before I forget to post them, I don't see anything critical.

[problame]

Nice, a stream!

And a nice test, love it!