Test test_backpressure_received_lsn_lag does not pass when failpoint is properly enabled

[LizardWizzard]

In #1571 I discovered that failpoints were not enabled, and after fixing failpoint integration test_backpressure_received_lsn_lag started to fail. Currently, it fails because it reaches statement_timeout. I tried to increase it up to 10 minutes but it didnt help. Needs more investigation. @lubennikovaav could you please take a look?

[stepashka]

this test confims that backpressure works
backpressure ensures that when compute has a very intensive write workload, we don't time out on reads

[LizardWizzard]

Merging #596 into this one.

Cross posting the idea:

Lets also test backpressure via applying CPU limits on pageserver

[kelvich]

What that should prove? If the pageserver is slow enough then back pressure kicks in and slows down the compute. What is the goal or suspected misbehavior?

[LizardWizzard]

Not sure I follow the last question.

If the pageserver is slow enough then back pressure kicks in and slows down the compute.

Yes, we need to prove that it works as intended.

[kelvich]

Yes, we need to prove that it works as intended.

Sure, what would be the proof or the test to validate that? My point is that if due to the failpoints (aka slow pageserver) compute times out that means that back pressure works as expected.

[LizardWizzard]

Yes, thats correct.

[koivunej]

The skip still exists, so I reopened this issue. The fail point used in the test with the long unsupported psql fail points command (replaced with an HTTP API) uses pause on a plain fail::fail_point!, which freezes the runtime thread.

While looking around "do we have any backpressure tests":

• test_runner/regress/test_timeline_size.py::test_timeline_size_quota
  • uses a bespoke wait_for_last_flush_lsn via neon.backpressure_lsns
  • unsure why we would even wait (it's a boostrapped timeline, so we have accurate logical size)
• test_runner/performance/test_wal_backpressure.py uses backpressure_lsns() to determine how late is pageserver
  • this is reused in another generate series benchmark as well
• test_runner/regress/test_sharding.py::test_sharding_backpressure
  • uses a failpoint to sleep on a runtime thread
  • fails as expected with max_replication_write_lag=100MB due to lsn difference between the shards
  • need to check how Wait for completion of the upload queue in flush_frozen_layer #8550 affects this

Cc: #7317