fix: cleanup of layers from the future can race with their re-creation

[problame]

fixes #5878
obsoletes #5879

Before this PR, it could happen that load_layer_map schedules removal of the future
image layer. Then a later compaction run could re-create the same image layer, scheduling a PUT.
Due to lack of an upload queue barrier, the PUT and DELETE could be re-ordered.
The result was IndexPart referencing a non-existent object.

Summary of changes

• Add support to pagectl / Python tests to decode IndexPart
  • Rust
    • new pagectl Subcommand
    • IndexPart::{from,to}_s3_bytes() methods to internalize knowledge about encoding of IndexPart
  • Python
    • new NeonCli subclass
• Add regression test
  • Rust
    • Ability to force repartitioning; required to ensure image layer creation at last_record_lsn
  • Python
    • The regression test.
• Fix the issue
  • Insert an UploadOp::Barrier after scheduling the deletions.

[github-actions]

2400 tests run: 2272 passed, 0 failed, 128 skipped (full report)

---

Flaky tests (8)

Postgres 16

• test_import_from_pageserver_small: debug
• test_ondemand_download_timetravel[mock_s3]: debug
• test_emergency_mode: release

Postgres 15

• test_gc_cutoff: debug
• test_timeline_deletion_with_files_stuck_in_upload_queue: release
• test_remote_storage_backup_and_restore[False-real_s3]: debug
• test_restarts_frequent_checkpoints: release

Postgres 14

• test_timeline_delete_works_for_remote_smoke[real_s3]: debug

Code coverage (full report)

• functions: 54.2% (8966 of 16549 functions)
• lines: 81.5% (52289 of 64160 lines)

---

_{The comment gets automatically updated with the latest test results
f8dd44b at 2023-11-23T12:34:07.482Z :recycle:}

[koivunej]

Oki now I found the st_mtime_ns changes as well, probably just did not see them. I'm fine with them or the original inodes.

Please do reconsider turning the if file_mismatch: log + break into assert as I suggested in the unresolved thread, I at least cannot see an upside from tolerating this.

[arpad-m]

thanks, this is a really great PR

[arpad-m]

CI failure is legit. Black wants this change:

-    time.sleep(1.1) # so that we can use change in pre_stat.st_mtime to detect overwrites
+    time.sleep(1.1)  # so that we can use change in pre_stat.st_mtime to detect overwrites