Feat/postgres 16

[MMeent]

Problem

Hopefully none

Summary of changes

This adds postgres-v16 as a vendored postgresql version, and adapts the code to support this version

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2466 tests run: 2352 passed, 0 failed, 114 skipped (full report)

---

Flaky tests (4)

Postgres 16

• test_partial_evict_tenant: release, debug
• test_get_tenant_size_with_multiple_branches: debug

Postgres 14

• test_download_remote_layers_api[local_fs]: release

Code coverage (full report)

• functions: 53.0% (7640 of 14428 functions)
• lines: 81.0% (44787 of 55282 lines)

---

_{The comment gets automatically updated with the latest test results
585e4eb at 2023-09-12T12:14:29.919Z :recycle:}

[hlinnaka]

One thing came up in a conversation with @knizhnik today: We have made a couple of small change to some WAL record format. Heap update is one example that I remember, to add the command id, not sure if there are others.

Now would be a good chance to revisit that. We still need the command id for WAL generated by neon, but does our pageserver code handle both vanilla Postgres WAL records and the ones created with neon-patched postgres? If the pageserver can handle both, that makes it possible to stream WAL from unmodified Postgres to Neon storage. If we need to make changes the format, add a flag or something, we should do it now for v16.

[petuhovskiy]

Safekeeper and walproposer changes LGTM

[knizhnik]

One thing came up in a conversation with @knizhnik today: We have made a couple of small change to some WAL record format. Heap update is one example that I remember, to add the command id, not sure if there are others.

Now would be a good chance to revisit that. We still need the command id for WAL generated by neon, but does our pageserver code handle both vanilla Postgres WAL records and the ones created with neon-patched postgres? If the pageserver can handle both, that makes it possible to stream WAL from unmodified Postgres to Neon storage. If we need to make changes the format, add a flag or something, we should do it now for v16.

Here is the list of changes in WAL format we have made:

@@ -108,6 +108,7 @@ typedef struct xl_heap_delete
 {
        TransactionId xmax;                     /* xmax of the deleted tuple */
        OffsetNumber offnum;            /* deleted tuple's offset */
+       uint32      t_cid;
        uint8           infobits_set;   /* infomask bits */
        uint8           flags;
 } xl_heap_delete;
@@ -145,6 +146,7 @@ typedef struct xl_heap_header
 {
        uint16          t_infomask2;
        uint16          t_infomask;
+       uint32      t_cid;
        uint8           t_hoff;
 } xl_heap_header;
 
@@ -186,6 +188,7 @@ typedef struct xl_multi_insert_tuple
        uint16          datalen;                /* size of tuple data that follows */
        uint16          t_infomask2;
        uint16          t_infomask;
+       uint32      t_cid;
        uint8           t_hoff;
        /* TUPLE DATA FOLLOWS AT END OF STRUCT */
 } xl_multi_insert_tuple;
@@ -215,9 +218,9 @@ typedef struct xl_heap_update
        OffsetNumber old_offnum;        /* old tuple's offset */
        uint8           old_infobits_set;       /* infomask bits to set on old tuple */
        uint8           flags;
+       uint32       t_cid;
        TransactionId new_xmax;         /* xmax of the new tuple */
        OffsetNumber new_offnum;        /* new tuple's offset */
-
        /*
         * If XLH_UPDATE_CONTAINS_OLD_TUPLE or XLH_UPDATE_CONTAINS_OLD_KEY flags
         * are set, xl_heap_header and tuple data for the old tuple follow.
@@ -279,6 +282,7 @@ typedef struct xl_heap_lock
 {
        TransactionId locking_xid;      /* might be a MultiXactId not xid */
        OffsetNumber offnum;            /* locked tuple's offset on page */
+       uint32       t_cid;
        int8            infobits_set;   /* infomask and infomask2 bits to set */
        uint8           flags;                  /* XLH_LOCK_* flag bits */
 } xl_heap_lock;

So there are several affected WAL records and several functions where this structures are serialized/deserialized.
Unfortunately we do not have any Neon specific flag/format/version number in XLOG which can be used to determine format. We can add such flag now but then backward compatibility will be lost.
Alternatively we can somehow pass this flag to xlogreader, specifying with which format we are working now.

Just want to remember why it may be needed: there are many complaints from customers concerning slow import of data in Neon. Right now it mostly can be done only using pg_dump/pg_restore, which actually reconstruct heap an indexes from scratch. Using pg_basebackup is certainly more efficient because it allows just copy files.
But there are several problems with basebackup:

1. User should use exactly the same version of Postgres which is supported by Neon.
2. We should provide new way of launching compute. Right now basebackup is sent to compute from PS. And we need to be able to use client's basebacklup file
3. pg_basebackup include WAL, so we need our compute to be able to process vanilla WAL (without added t_cid)

Right now we are speaking about 3), although 2) can be also very non-trivial task and 1) can be show stopper.

It seems to be that least invasive solution will be to add some global variable or may be even GUC, (i.e. neon_wal_compatibility) and use it in redo handlers. I can make such change if there are no better proposals.

[hlinnaka]

It seems to be that least invasive solution will be to add some global variable or may be even GUC, (i.e. neon_wal_compatibility) and use it in redo handlers. I can make such change if there are no better proposals.

Let's make it so that you can distinguish the record types from the WAL record itself. An extra flag on the records or something. That way, we don't need any extra bookkeeping of what WAL was generated with unmodified PostgreSQL and which with neon.

[hlinnaka]

Summary of the current status:

• All the extension compilation errors have been fixed. v16 compute image builds successfully now. I had to disable a bunch of extensions to make that work, so we'll need to revisit them later, but we can release v16 with some extensions missing.
• I added libicu67 to the 'neon' docker image, so initdb should work now. Need to ensure that our storage nodes in staging and production have that library installed too
• One regression failure on v14. I believe it's a flaky test and will probably go away if we rerun the tests
• A few regression failures on v16. These passed at one point, so I believe they are caused by the recent changes to WAL-logging UPDATE records. @MMeent, you are working on those, right?

[hlinnaka]

Fixed an incorrect reference to XlNeonHeapUpdate and cleaned up some leftover structs that are no longer used. That was causing regression tests to hang. There are still some failures:

2023-09-12T06:53:15.536960Z ERROR wal_connection_manager{tenant_id=d54b4bfbf23ccf2176def12c31205783 timeline_id=744d48c600f902ad9f0fb583453c2a2c}:connection{node_id=1}:panic{thread=walreceiv
er worker location=/home/nonroot/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bytes-1.4.0/src/buf/buf_impl.rs:254:9}: assertion failed: self.remaining() >= dst.len()

Stack backtrace:
   0: utils::logging::tracing_panic_hook
             at /home/heikki/git-sandbox/neon/libs/utils/src/logging.rs:166:21
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/boxed.rs:2007:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:709:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:595:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:151:18
   5: rust_begin_unwind
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:593:5
   6: core::panicking::panic_fmt
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:67:14
   7: core::panicking::panic
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:117:5
   8: bytes::buf::buf_impl::Buf::copy_to_slice
             at /home/nonroot/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bytes-1.4.0/src/buf/buf_impl.rs:254:9
   9: bytes::buf::buf_impl::Buf::get_u32_le
             at /home/nonroot/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bytes-1.4.0/src/buf/buf_impl.rs:481:9
  10: pageserver::walrecord::v16::rm_neon::XlNeonHeapUpdate::decode
             at /home/heikki/git-sandbox/neon/pageserver/src/walrecord.rs:364:31
  11: pageserver::walingest::WalIngest::ingest_neonrmgr_record::{{closure}}
             at /home/heikki/git-sandbox/neon/pageserver/src/walingest.rs:743:37
  12: pageserver::walingest::WalIngest::ingest_record::{{closure}}
             at /home/heikki/git-sandbox/neon/pageserver/src/walingest.rs:111:18
...

but I didn't investigate that yet

[MMeent]

There are still some failures [..] but I didn't investigate that yet

A case of a bad copy-pasted constant in pg_constants leading to confusion about the type of the record.

[hlinnaka]

This PR includes a bunch of test changes that are unrelated to v16. I think they're good refactorings, but let's do them as separate PRs

[hlinnaka]

I merged main to this branch, to get latest changes. And I reverted the unrelated test changes.