Skip to content

Commit

Permalink
Warn if a blob in an image is larger than 256 MiB (#7852)
Browse files Browse the repository at this point in the history
We'd like to get some bits reserved in the length field of image layers
for future usage (compression). This PR bases on the assumption that we
don't have any blobs that require more than 28 bits (3 bytes + 4 bits)
to store the length, but as a preparation, before erroring, we want to
first emit warnings as if the assumption is wrong, such warnings are less
disruptive than errors.

A metric would be even less disruptive (log messages are more slow, if
we have a LOT of such large blobs then it would take a lot of time to
print them). At the same time, likely such 256 MiB blobs will occupy an
entire layer file, as they are larger than our target size. For layer
files we already log something, so there shouldn't be a large increase
in overhead.

Part of #5431
  • Loading branch information
arpad-m authored May 23, 2024
1 parent 49d7f9b commit d5d15eb
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion pageserver/src/tenant/blob_io.rs
Original file line number Diff line number Diff line change
Expand Up @@ -238,10 +238,13 @@ impl<const BUFFERED: bool> BlobWriter<BUFFERED> {
io_buf,
Err(Error::new(
ErrorKind::Other,
format!("blob too large ({} bytes)", len),
format!("blob too large ({len} bytes)"),
)),
);
}
if len > 0x0fff_ffff {
tracing::warn!("writing blob above future limit ({len} bytes)");
}
let mut len_buf = (len as u32).to_be_bytes();
len_buf[0] |= 0x80;
io_buf.extend_from_slice(&len_buf[..]);
Expand Down

1 comment on commit d5d15eb

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3196 tests run: 3055 passed, 1 failed, 140 skipped (full report)


Failures on Postgres 14

  • test_basebackup_with_high_slru_count[github-actions-selfhosted-sequential-10-13-30]: release
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_basebackup_with_high_slru_count[release-pg14-github-actions-selfhosted-sequential-10-13-30]"
Flaky tests (1)

Postgres 16

  • test_vm_bit_clear_on_heap_lock: debug

Code coverage* (full report)

  • functions: 31.4% (6449 of 20544 functions)
  • lines: 48.3% (49861 of 103277 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
d5d15eb at 2024-05-23T13:55:27.541Z :recycle:

Please sign in to comment.