-
Notifications
You must be signed in to change notification settings - Fork 409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Release 2024-05-27 #7888
Release 2024-05-27 #7888
Commits on May 22, 2024
-
refactor(test): duplication with fullbackup, tar content hashing (#7828)
"taking a fullbackup" is an ugly multi-liner copypasted in multiple places, most recently with timeline ancestor detach tests. move it under `PgBin` which is not a great place, but better than yet another utility function. Additionally: - cleanup `psql_env` repetition (PgBin already configures that) - move the backup tar comparison as a yet another free utility function - use backup tar comparison in `test_import.py` where a size check was done previously - cleanup extra timeline creation from test Cc: #7715
1Configuration menu - View commit details
-
Copy full SHA for df9ab1b - Browse repository at this point
Copy the full SHA df9ab1bView commit details -
CI(report-benchmarks-failures): fix condition (#7820)
## Problem `report-benchmarks-failures` got skipped if a dependent job fails. ## Summary of changes - Fix the if-condition by adding `&& failures()` to it; it'll make the job run if the dependent job fails.
1Configuration menu - View commit details
-
Copy full SHA for d1d55bb - Browse repository at this point
Copy the full SHA d1d55bbView commit details -
openapi: document timeline ancestor detach (#7650)
The openapi description with the error descriptions: - 200 is used for "detached or has been detached previously" - 400 is used for "cannot be detached right now" -- it's an odd thing, but good enough - 404 is used for tenant or timeline not found - 409 is used for "can never be detached" (root timeline) - 500 is used for transient errors (basically ill-defined shutdown errors) - 503 is used for busy (other tenant ancestor detach underway, pageserver shutdown) Cc: #6994
1Configuration menu - View commit details
-
Copy full SHA for ce44dfe - Browse repository at this point
Copy the full SHA ce44dfeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8901ce9 - Browse repository at this point
Copy the full SHA 8901ce9View commit details -
Make postgres_version action input default to a string
This is "required" by GitHub Actions, though they must do some coersion on their side.
1Configuration menu - View commit details
-
Copy full SHA for 900f391 - Browse repository at this point
Copy the full SHA 900f391View commit details -
CI(promote-images): simplify & fix the job (#7826)
## Problem Currently, `latest` tag is added to the images in several cases: ``` github.ref_name == 'main' || github.ref_name == 'release' || github.ref_name == 'release-proxy' ``` This leads to a race; the `latest` tag jumps back and forth depending on the branch that has built images. ## Summary of changes - Do not push `latest` images to prod ECR (we don't use it) - Use `docker buildx imagetools` instead of `crane` for tagging images - Unify `vm-compute-node-image` job with others and use dockerhub as a first source for images (sync images with ECR) - Tag images with `latest` only for commits in `main`
1Configuration menu - View commit details
-
Copy full SHA for 325f378 - Browse repository at this point
Copy the full SHA 325f378View commit details -
CI: build multi-arch images (#7696)
## Problem We don't build our docker images for ARM arch, and that makes it harder to run images on ARM (on MacBooks with Apple Silicon, for example). ## Summary of changes - Build `neondatabase/neon` for ARM and create a multi-arch image - Build `neondatabase/compute-node-vXX` for ARM and create a multi-arch image - Run `test-images` job on ARM as well
1Configuration menu - View commit details
-
Copy full SHA for a7f31f1 - Browse repository at this point
Copy the full SHA a7f31f1View commit details -
safekeeper: use CancellationToken instead of watch channel (#7836)
## Problem Safekeeper Timeline uses a channel for cancellation, but we have a dedicated type for that. ## Summary of changes - Use CancellationToken in Timeline
1Configuration menu - View commit details
-
Copy full SHA for e015b2b - Browse repository at this point
Copy the full SHA e015b2bView commit details -
fix(Layer): carry gate until eviction is complete (#7838)
the gate was accidentially being dropped before the final blocking phase, possibly explaining the resident physical size global problems during deletions. it could had caused more harm as well, but the path is not actively being tested because cplane no longer puts locationconfigs with higher generation number during normal operation which prompted the last wave of fixes. Cc: #7341.
1Configuration menu - View commit details
-
Copy full SHA for 62aac6c - Browse repository at this point
Copy the full SHA 62aac6cView commit details -
Fix confusion between 1-based Buffer and 0-based index (#7825)
The code was working correctly, but was incorrectly using Buffer for a 0-based index into the BufferDesc array.
Configuration menu - View commit details
-
Copy full SHA for 3404e76 - Browse repository at this point
Copy the full SHA 3404e76View commit details -
Fix issues with determining request LSN in read replica (#7795)
Don't set last-written LSN of a page when the record is replayed, only when the page is evicted from cache. For comparison, we don't update the last-written LSN on every page modification on the primary either, only when the page is evicted. Do update the last-written LSN when the page update is skipped in WAL redo, however. In neon_get_request_lsns(), don't be surprised if the last-written LSN is equal to the record being replayed. Use the LSN of the record being replayed as the request LSN in that case. Add a long comment explaining how that can happen. In neon_wallog_page, update last-written LSN also when Shutdown has been requested. We might still fetch and evict pages for a while, after shutdown has been requested, so we better continue to do that correctly. Enable the check that we don't evict a page with zero LSN also in standby, but make it a LOG message instead of PANIC Fixes issue #7791
1Configuration menu - View commit details
-
Copy full SHA for 9217564 - Browse repository at this point
Copy the full SHA 9217564View commit details -
Make 'neon.protocol_version = 2' the default, take two (#7819)
Once all the computes in production have restarted, we can remove protocol version 1 altogether. See issue #6211. This was done earlier already in commit 0115fe6, but reverted before it was released to production in commit bbe730d because of issue #7692. That issue was fixed in commit 22afaea, so we are ready to change the default again.
1Configuration menu - View commit details
-
Copy full SHA for 37f8128 - Browse repository at this point
Copy the full SHA 37f8128View commit details -
feat(pageserver): auto-detect previous aux file policy (#7841)
## Problem If an existing user already has some aux v1 files, we don't want to switch them to the global tenant-level config. Part of #7462 --------- Signed-off-by: Alex Chi Z <chi@neon.tech>
1Configuration menu - View commit details
-
Copy full SHA for 64577cf - Browse repository at this point
Copy the full SHA 64577cfView commit details -
proxy password threadpool (#7806)
## Problem Despite making password hashing async, it can still take time away from the network code. ## Summary of changes Introduce a custom threadpool, inspired by rayon. Features: ### Fairness Each task is tagged with it's endpoint ID. The more times we have seen the endpoint, the more likely we are to skip the task if it comes up in the queue. This is using a min-count-sketch estimator for the number of times we have seen the endpoint, resetting it every 1000+ steps. Since tasks are immediately rescheduled if they do not complete, the worker could get stuck in a "always work available loop". To combat this, we check the global queue every 61 steps to ensure all tasks quickly get a worker assigned to them. ### Balanced Using crossbeam_deque, like rayon does, we have workstealing out of the box. I've tested it a fair amount and it seems to balance the workload accordingly
1Configuration menu - View commit details
-
Copy full SHA for 9cfe08e - Browse repository at this point
Copy the full SHA 9cfe08eView commit details -
chore(pageserver): use kebab case for aux file flag (#7840)
part of #7462 --------- Signed-off-by: Alex Chi Z <chi@neon.tech>
1Configuration menu - View commit details
-
Copy full SHA for ddd8ebd - Browse repository at this point
Copy the full SHA ddd8ebdView commit details -
tests: refine test_secondary_background_downloads (#7829)
## Problem This test relied on some sleeps, and was failing ~5% of the time. ## Summary of changes Use log-watching rather than straight waits, and make timeouts more generous for the CI environment.
1Configuration menu - View commit details
-
Copy full SHA for 014f822 - Browse repository at this point
Copy the full SHA 014f822View commit details -
tests: add a couple of allow lists for shutdown cases (#7844)
## Problem Failures on some of our uglier shutdown log messages: https://neon-github-public-dev.s3.amazonaws.com/reports/main/9192662995/index.html#suites/07874de07c4a1c9effe0d92da7755ebf/51b365408678c66f/ ## Summary of changes - Allow-list these errors.
1Configuration menu - View commit details
-
Copy full SHA for f98fdd2 - Browse repository at this point
Copy the full SHA f98fdd2View commit details -
chore(pageserver): add force aux file policy switch handler (#7842)
For existing users, we want to allow doing a force switch for their aux file policy. Part of #7462 --------- Signed-off-by: Alex Chi Z <chi@neon.tech>
1Configuration menu - View commit details
-
Copy full SHA for 4a278cc - Browse repository at this point
Copy the full SHA 4a278ccView commit details -
chore(pageserver): use kebab case for compaction algorithms (#7845)
Signed-off-by: Alex Chi Z <chi@neon.tech>
1Configuration menu - View commit details
-
Copy full SHA for ff560a1 - Browse repository at this point
Copy the full SHA ff560a1View commit details -
Bump vm-builder v0.28.1 -> v0.29.3 (#7849)
One change: runner: allow coredump collection (#931)
1Configuration menu - View commit details
-
Copy full SHA for eb0c026 - Browse repository at this point
Copy the full SHA eb0c026View commit details
Commits on May 23, 2024
-
pageserver: fix API-driven secondary downloads possibly colliding wit…
…h background downloads (#7848) ## Problem We've seen some strange behaviors when doing lots of migrations involving secondary locations. One of these was where a tenant was apparently stuck in the `Scheduler::running` list, but didn't appear to be making any progress. Another was a shutdown hang (neondatabase/cloud#13576). ## Summary of changes - Fix one issue (probably not the only one) where a tenant in the `pending` list could proceed to `spawn` even if the same tenant already had a running task via `handle_command` (this could have resulted in a weird value of SecondaryProgress) - Add various extra logging: - log before as well as after layer downloads so that it would be obvious if we were stuck in remote storage code (we shouldn't be, it has built in timeouts) - log the number of running + pending jobs from the scheduler every time it wakes up to do a scheduling iteration (~10s) -- this is quite chatty, but not compared with the volume of logs on a busy pageserver. It should give us confidence that the scheduler loop is still alive, and visibility of how many tasks the scheduler thinks are running.
1Configuration menu - View commit details
-
Copy full SHA for a43a1ad - Browse repository at this point
Copy the full SHA a43a1adView commit details -
test_attach_tenant_config: add allowed error (#7839)
[evidence] of quite rare flaky. the detach can cause this with the right timing. [evidence]: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-7650/9191613501/index.html#suites/7745dadbd815ab87f5798aa881796f47/2190222925001078
1Configuration menu - View commit details
-
Copy full SHA for 58e31fe - Browse repository at this point
Copy the full SHA 58e31feView commit details -
Skip unnecessary shared state updates in safekeepers (#7851)
I looked at the metrics from #7768 on staging and it seems that manager does too many iterations. This is probably caused by background job `remove_wal.rs` which iterates over all timelines and tries to remove WAL and persist control file. This causes shared state updates and wakes up the manager. The fix is to skip notifying about the updates if nothing was updated.
1Configuration menu - View commit details
-
Copy full SHA for 8f3c316 - Browse repository at this point
Copy the full SHA 8f3c316View commit details -
[proxy] Do not fail after parquet upload error (#7858)
## Problem If the parquet upload was unsuccessful, it will panic. ## Summary of changes Write error in logs instead.
1Configuration menu - View commit details
-
Copy full SHA for cd6d811 - Browse repository at this point
Copy the full SHA cd6d811View commit details -
1
Configuration menu - View commit details
-
Copy full SHA for 545f7e8 - Browse repository at this point
Copy the full SHA 545f7e8View commit details -
remove march=native from pgvector Makefile's OPTFLAGS (#7854)
## Problem By default, pgvector compiles with `-march=native` on some platforms for best performance. However, this can lead to `Illegal instruction` errors if trying to run the compiled extension on a different machine. I had this problem when trying to run the Neon compute docker image on MacOS with Apple Silicon with Rosetta. see https://github.com/pgvector/pgvector/blob/ff9b22977e3ef19866d23a54332c8717f258e8db/README.md?plain=1#L1021 ## Summary of changes Pass OPTFLAGS="" to make.
1Configuration menu - View commit details
-
Copy full SHA for 95a49f0 - Browse repository at this point
Copy the full SHA 95a49f0View commit details -
test_import_from_pageserver_small: try to make less flaky (#7843)
With #7828 and proper fullbackup testing the test became flaky ([evidence]). - produce better assertion messages in `assert_pageserver_backups_equal` - use read only endpoint to confirm the row count [evidence]: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-7839/9192447962/index.html#suites/89cfa994d71769e01e3fc4f475a1f3fa/49009214d0f8b8ce
1Configuration menu - View commit details
-
Copy full SHA for 49d7f9b - Browse repository at this point
Copy the full SHA 49d7f9bView commit details -
Warn if a blob in an image is larger than 256 MiB (#7852)
We'd like to get some bits reserved in the length field of image layers for future usage (compression). This PR bases on the assumption that we don't have any blobs that require more than 28 bits (3 bytes + 4 bits) to store the length, but as a preparation, before erroring, we want to first emit warnings as if the assumption is wrong, such warnings are less disruptive than errors. A metric would be even less disruptive (log messages are more slow, if we have a LOT of such large blobs then it would take a lot of time to print them). At the same time, likely such 256 MiB blobs will occupy an entire layer file, as they are larger than our target size. For layer files we already log something, so there shouldn't be a large increase in overhead. Part of #5431
1Configuration menu - View commit details
-
Copy full SHA for d5d15eb - Browse repository at this point
Copy the full SHA d5d15ebView commit details -
fix(pageserver): make wal connstr a connstr (#7846)
The list timeline API gives something like `"wal_source_connstr":"PgConnectionConfig { host: Domain(\"safekeeper-5.us-east-2.aws.neon.build\"), port: 6500, password: Some(REDACTED-STRING) }"`, which is weird. This pull request makes it somehow like a connection string. This field is not used at least in the neon database, so I assume no one is reading or parsing it. Signed-off-by: Alex Chi Z <chi@neon.tech>
1Configuration menu - View commit details
-
Copy full SHA for e28e46f - Browse repository at this point
Copy the full SHA e28e46fView commit details -
Use Timeline::create_image_layer_for_rel_blocks in tiered compaction (#…
1Configuration menu - View commit details
-
Copy full SHA for 75a52ac - Browse repository at this point
Copy the full SHA 75a52acView commit details -
chore(pageserver): reduce logging related to image layers (#7864)
* Reduce the logging level for create image layers of metadata keys. (question: is it possible to adjust logging levels at runtime?) * Do a info logging of image layers only after the layer is created. Now there are a lot of cases where we create the image layer writer but then discarding that image layer because it does not contain any key. Therefore, I changed the new image layer logging to trace, and create image layer logging to info. Signed-off-by: Alex Chi Z <chi@neon.tech>
1Configuration menu - View commit details
-
Copy full SHA for 6b31642 - Browse repository at this point
Copy the full SHA 6b31642View commit details -
refactor(rtc): remove the duplicate IndexLayerMetadata (#7860)
Once upon a time, we used to have duplicated types for runtime IndexPart and whatever we stored. Because of the serde fixes in #5335 we have no need for duplicated IndexPart type anymore, but the `IndexLayerMetadata` stayed. - remove the type - remove LayerFileMetadata::file_size() in favor of direct field access Split off from #7833. Cc: #3072.
1Configuration menu - View commit details
-
Copy full SHA for 7cf726e - Browse repository at this point
Copy the full SHA 7cf726eView commit details -
## Problem ## Summary of changes ## Checklist before requesting a review - [ ] I have performed a self-review of my code. - [ ] If it is a core feature, I have added thorough tests. - [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard? - [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section. ## Checklist before merging - [ ] Do not forget to reformat commit message to not include the above checklist
Sasha Krassovsky committedMay 23, 2024 1Configuration menu - View commit details
-
Copy full SHA for ea2e830 - Browse repository at this point
Copy the full SHA ea2e830View commit details -
Rework PageStream connection state handling: (#7611)
* Make PS connection startup use async APIs This allows for improved query cancellation when we start connections * Make PS connections have per-shard connection retry state. Previously they shared global backoff state, which is bad for quickly getting all connections started and/or back online. * Make sure we clean up most connection state on failed connections. Previously, we could technically leak some resources that we'd otherwise clean up. Now, the resources are correctly cleaned up. * pagestore_smgr.c now PANICs on unexpected response message types. Unexpected responses are likely a symptom of having a desynchronized view of the connection state. As a desynchronized connection state can cause corruption, we PANIC, as we don't know what data may have been written to buffers: the only solution is to fail fast & hope we didn't write wrong data. * Catch errors in sync pagestream request handling. Previously, if a query was cancelled after a message was sent to the pageserver, but before the data was received, the backend could forget that it sent the synchronous request, and let others deal with the repercussions. This could then lead to incorrect responses, or errors such as "unexpected response from page server with tag 0x68"
1Configuration menu - View commit details
-
Copy full SHA for 0e4f182 - Browse repository at this point
Copy the full SHA 0e4f182View commit details
Commits on May 24, 2024
-
Export db size metrics for 10 user databases (#7857)
## Problem One database is too limiting. We have agreed to raise this limit to 10. ## Checklist before requesting a review - [x] I have performed a self-review of my code. - [ ] If it is a core feature, I have added thorough tests. - [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard? - [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section. ## Checklist before merging - [ ] Do not forget to reformat commit message to not include the above checklist
1Configuration menu - View commit details
-
Copy full SHA for c1f4028 - Browse repository at this point
Copy the full SHA c1f4028View commit details -
pageserver: post-shard-split layer rewrites (2/2) (#7531)
## Problem - After a shard split of a large existing tenant, child tenants can end up with oversized historic layers indefinitely, if those layers are prevented from being GC'd by branchpoints. This PR follows #7531, and adds rewriting of layers that contain a mixture of needed & un-needed contents, in addition to dropping un-needed layers. Closes: #7504 ## Summary of changes - Add methods to ImageLayer for reading back existing layers - Extend `compact_shard_ancestors` to rewrite layer files that contain a mixture of keys that we want and keys we do not, if unwanted keys are the majority of those in the file. - Amend initialization code to handle multiple layers with the same LayerName properly - Get rid of of renaming bad layer files to `.old` since that's now expected on restarts during rewrites.
1Configuration menu - View commit details
-
Copy full SHA for 3860bc9 - Browse repository at this point
Copy the full SHA 3860bc9View commit details -
pageserver: revert concurrent secondary downloads, make DownloadStrea…
…m always yield Err after cancel (#7866) ## Problem Ongoing hunt for secondary location shutdown hang issues. ## Summary of changes - Revert the functional changes from #7675 - Tweak a log in secondary downloads to make it more apparent when we drop out on cancellation - Modify DownloadStream's behavior to always return an Err after it has been cancelled. This _should_ not impact anything, but it makes the behavior simpler to reason about (e.g. even if the poll function somehow got called again, it could never end up in an un-cancellable state) Related #neondatabase/cloud#13576
1Configuration menu - View commit details
-
Copy full SHA for 1455f5a - Browse repository at this point
Copy the full SHA 1455f5aView commit details -
chore: lower gate guard drop logging threshold to 100ms (#7862)
We have some 1001ms cases, which do not yield gate guard context.
1Configuration menu - View commit details
-
Copy full SHA for a3f5b83 - Browse repository at this point
Copy the full SHA a3f5b83View commit details -
CI(release): tune Storage & Compute release PR title (#7870)
## Problem A title for automatic proxy release PRs is `Proxy release`, and for storage & compute, it's just `Release` ## Summary of changes - Amend PR title for Storage & Compute releases to "Storage & Compute release"
1Configuration menu - View commit details
-
Copy full SHA for 71a7fd9 - Browse repository at this point
Copy the full SHA 71a7fd9View commit details -
proxy: fix websocket buffering (#7878)
## Problem Seems the websocket buffering was broken for large query responses only ## Summary of changes Move buffering until after the underlying stream is ready. Tested locally confirms this fixes the bug. Also fixes the pg-sni-router missing metrics bug
Configuration menu - View commit details
-
Copy full SHA for 43f9a16 - Browse repository at this point
Copy the full SHA 43f9a16View commit details
Commits on May 25, 2024
-
safekeeper: test pull_timeline with WAL gc.
Do pull_timeline while WAL is being removed. To this end - extract pausable_failpoint to utils, sprinkle pull_timeline with it - add 'checkpoint' sk http endpoint to force WAL removal. After fixing checking for pull file status code test fails so far which is expected.
Configuration menu - View commit details
-
Copy full SHA for 3797566 - Browse repository at this point
Copy the full SHA 3797566View commit details -
1
Configuration menu - View commit details
-
Copy full SHA for b2d34a8 - Browse repository at this point
Copy the full SHA b2d34a8View commit details
Commits on May 27, 2024
-
Fix connect to PS on MacOS/X (#7885)
## Problem After [0e4f182] which introduce async connect Neon is not able to connect to page server. ## Summary of changes Perform sync commit at MacOS/X ## Checklist before requesting a review - [ ] I have performed a self-review of my code. - [ ] If it is a core feature, I have added thorough tests. - [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard? - [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section. ## Checklist before merging - [ ] Do not forget to reformat commit message to not include the above checklist --------- Co-authored-by: Konstantin Knizhnik <knizhnik@neon.tech>
Configuration menu - View commit details
-
Copy full SHA for 068c158 - Browse repository at this point
Copy the full SHA 068c158View commit details