error: could not read block xx in rel xx from pageserver at lsn xx

[surquer]

After deploying neon according to the steps described in the README.md, use pgbench to test the performance, and the following error occurs：

[SomeoneToIgnore]

Hard to say for sure what happened without seeing pageserver and safekeeper logs, ideally etcd logs either (you can find those in .neon/ directory ) and the workload details.

I would guess, that the pgbench test you've launched might have caused big writes which caused pageserver's walreceiver connection to flicker, hence failing to receive WAL timely.
There's a patch #2253 aiming to fix most of such problems, might be good to check that if you can and want to.

[petuhovskiy]

I would guess, that the pgbench test you've launched might have caused big writes which caused pageserver's walreceiver connection to flicker, hence failing to receive WAL timely.

That's a plausible theory, but it also could be our usual backpressure issue (see #2028). We've already noticed Timed out while waiting for WAL record when running benchmarks, and it seems that it's still a relevant issue.

[hlinnaka]

Also for performance testing, make sure you're building in release mode:

BUILD_TYPE=release make -j`nproc`

and use the binaries from target/release instead of targer/debug. The debug-mode binaries are significantly slower.

[surquer]

Also for performance testing, make sure you're building in release mode:

BUILD_TYPE=release make -j`nproc`

and use the binaries from target/release instead of targer/debug. The debug-mode binaries are significantly slower.
Yes, it is tested with the release version

[surquer]

I would guess, that the pgbench test you've launched might have caused big writes which caused pageserver's walreceiver connection to flicker, hence failing to receive WAL timely.

That's a plausible theory, but it also could be our usual backpressure issue (see #2028). We've already noticed Timed out while waiting for WAL record when running benchmarks, and it seems that it's still a relevant issue.

Yes, it happened while doing the read and write test with lots of reads and writes at the same time

[surquer]

``> Hard to say for sure what happened without seeing pageserver and safekeeper logs, ideally etcd logs either (you can find those in .neon/ directory ) and the workload details.

I would guess, that the pgbench test you've launched might have caused big writes which caused pageserver's walreceiver connection to flicker, hence failing to receive WAL timely. There's a patch #2253 aiming to fix most of such problems, might be good to check that if you can and want to.

I updated the code and tested it, the problem still exists, there are many "Timed out while waiting for WAL record at" in pageserver.log

pageserver.zip

[knizhnik]

I wonder which backpressure setting you are using?
If it is default max_replication_write_lag=500MB, then such wait timeout errors are expected (or at least known) behavior.