tests: stabilize test_timeline_size_quota_on_startup

[jcsp]

Problem

test_timeline_size_quota_on_startup assumed that writing data beyond the size limit would always be blocked. This is not so: the limit is only enforced if feedback makes it back from the pageserver to the safekeeper + compute.

Closes: #6562

Summary of changes

• Modify the test to wait for the pageserver to catch up. The size limit was never actually being enforced robustly, the original version of this test was just writing much more than 30MB and about 98% of the time getting lucky such that the feedback happened to arrive before the tests for loop was done.
• If the test fails, log the logical size as seen by the pageserver.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3067 tests run: 2952 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (3)

Postgres 16

• test_isolation[None]: debug
• test_sharding_split_compaction[compact-shard-ancestors-localonly]: debug

Postgres 14

• test_tenant_creation_fails: debug

Code coverage* (full report)

• functions: 32.6% (6938 of 21280 functions)
• lines: 50.0% (54539 of 109057 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
366105d at 2024-07-08T20:14:01.160Z :recycle:}

[lubennikovaav]

lgtm
thank you for fixing it!