Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pageserver): integrate lsn lease into synthetic size #8220

Merged
merged 17 commits into from
Jul 4, 2024

Conversation

yliang412
Copy link
Contributor

@yliang412 yliang412 commented Jul 1, 2024

Part of #7497, closes #8071. (accidentally closed #8208, reopened here)

Problem

After the changes in #8084, we need synthetic size to also account for leased LSNs so that users do not get free retention by running a small ephemeral endpoint for a long time.

Summary of changes

This PR integrates LSN leases into the synthetic size calculation. We model leases as read-only branches started at the leased LSN (except it does not have a timeline id).

Other changes:

  • Add new unit tests testing whether a lease behaves like a read-only branch.
  • Change /size_debug response to include lease point in the SVG visualization.
  • Fix /lsn_lease HTTP API to do proper parsing for POST.

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

yliang412 and others added 6 commits June 28, 2024 18:13
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
@yliang412 yliang412 requested a review from a team as a code owner July 1, 2024 14:44
@yliang412 yliang412 requested review from skyzh, problame and koivunej and removed request for skyzh July 1, 2024 14:44
@yliang412
Copy link
Contributor Author

Leases vs. Equivalent RO Timeline

Leases
image

RO Timeline
image

@yliang412 yliang412 self-assigned this Jul 1, 2024
Copy link

github-actions bot commented Jul 1, 2024

3018 tests run: 2903 passed, 0 failed, 115 skipped (full report)


Code coverage* (full report)

  • functions: 32.6% (6929 of 21236 functions)
  • lines: 50.0% (54437 of 108800 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
9f4ffb3 at 2024-07-04T15:05:17.367Z :recycle:

test_runner/regress/test_tenant_size.py Outdated Show resolved Hide resolved
pageserver/src/tenant/size.rs Outdated Show resolved Hide resolved
pageserver/src/tenant/size.rs Show resolved Hide resolved
Copy link
Member

@koivunej koivunej left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the added documentation breadcrumbs this is good.

yliang412 and others added 6 commits July 2, 2024 15:45
Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Copy link
Contributor

@problame problame left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the svg dra changes: why not store the bool (or even better, an enum BranchKind { Branch, Lease }) in SvgDraw::branches instead of for each segment?
Feels more natural there to me.

(The enum instead of bool is a usual readability pet peeve of mine, kind of orthogonal to where we store the information)


The insert_and_create_ro_branch and insert_and_acquire_lease look almost identical.

For clarity, I think it makes sense to DRY by parametrizing them with "lease" | "branch".


This looks like what we discussed.
Have some minor nits, but, good job!

test_runner/regress/test_tenant_size.py Outdated Show resolved Hide resolved
pageserver/src/tenant/size.rs Outdated Show resolved Hide resolved
pageserver/src/tenant/size.rs Outdated Show resolved Hide resolved
yliang412 and others added 5 commits July 3, 2024 13:57
Co-authored-by: Christian Schwarz <christian@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Signed-off-by: Yuchen Liang <yuchen@neon.tech>
@yliang412 yliang412 enabled auto-merge (squash) July 4, 2024 14:10
@yliang412 yliang412 merged commit 19accfe into main Jul 4, 2024
58 checks passed
@yliang412 yliang412 deleted the yuchen/add-lease-to-synthetic-size-calc branch July 4, 2024 15:09
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
Part of #7497, closes #8071. (accidentally closed #8208, reopened here)

## Problem

After the changes in #8084, we need synthetic size to also account for
leased LSNs so that users do not get free retention by running a small
ephemeral endpoint for a long time.

## Summary of changes

This PR integrates LSN leases into the synthetic size calculation. We
model leases as read-only branches started at the leased LSN (except it
does not have a timeline id).

Other changes:
- Add new unit tests testing whether a lease behaves like a read-only
branch.
- Change `/size_debug` response to include lease point in the SVG
visualization.
- Fix `/lsn_lease` HTTP API to do proper parsing for POST.



Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Co-authored-by: Christian Schwarz <christian@neon.tech>
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
Part of #7497, closes #8071. (accidentally closed #8208, reopened here)

## Problem

After the changes in #8084, we need synthetic size to also account for
leased LSNs so that users do not get free retention by running a small
ephemeral endpoint for a long time.

## Summary of changes

This PR integrates LSN leases into the synthetic size calculation. We
model leases as read-only branches started at the leased LSN (except it
does not have a timeline id).

Other changes:
- Add new unit tests testing whether a lease behaves like a read-only
branch.
- Change `/size_debug` response to include lease point in the SVG
visualization.
- Fix `/lsn_lease` HTTP API to do proper parsing for POST.



Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Co-authored-by: Christian Schwarz <christian@neon.tech>
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
Part of #7497, closes #8071. (accidentally closed #8208, reopened here)

## Problem

After the changes in #8084, we need synthetic size to also account for
leased LSNs so that users do not get free retention by running a small
ephemeral endpoint for a long time.

## Summary of changes

This PR integrates LSN leases into the synthetic size calculation. We
model leases as read-only branches started at the leased LSN (except it
does not have a timeline id).

Other changes:
- Add new unit tests testing whether a lease behaves like a read-only
branch.
- Change `/size_debug` response to include lease point in the SVG
visualization.
- Fix `/lsn_lease` HTTP API to do proper parsing for POST.



Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Co-authored-by: Christian Schwarz <christian@neon.tech>
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
Part of #7497, closes #8071. (accidentally closed #8208, reopened here)

## Problem

After the changes in #8084, we need synthetic size to also account for
leased LSNs so that users do not get free retention by running a small
ephemeral endpoint for a long time.

## Summary of changes

This PR integrates LSN leases into the synthetic size calculation. We
model leases as read-only branches started at the leased LSN (except it
does not have a timeline id).

Other changes:
- Add new unit tests testing whether a lease behaves like a read-only
branch.
- Change `/size_debug` response to include lease point in the SVG
visualization.
- Fix `/lsn_lease` HTTP API to do proper parsing for POST.



Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Co-authored-by: Christian Schwarz <christian@neon.tech>
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
Part of #7497, closes #8071. (accidentally closed #8208, reopened here)

## Problem

After the changes in #8084, we need synthetic size to also account for
leased LSNs so that users do not get free retention by running a small
ephemeral endpoint for a long time.

## Summary of changes

This PR integrates LSN leases into the synthetic size calculation. We
model leases as read-only branches started at the leased LSN (except it
does not have a timeline id).

Other changes:
- Add new unit tests testing whether a lease behaves like a read-only
branch.
- Change `/size_debug` response to include lease point in the SVG
visualization.
- Fix `/lsn_lease` HTTP API to do proper parsing for POST.



Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Co-authored-by: Christian Schwarz <christian@neon.tech>
VladLazar pushed a commit that referenced this pull request Jul 8, 2024
Part of #7497, closes #8071. (accidentally closed #8208, reopened here)

## Problem

After the changes in #8084, we need synthetic size to also account for
leased LSNs so that users do not get free retention by running a small
ephemeral endpoint for a long time.

## Summary of changes

This PR integrates LSN leases into the synthetic size calculation. We
model leases as read-only branches started at the leased LSN (except it
does not have a timeline id).

Other changes:
- Add new unit tests testing whether a lease behaves like a read-only
branch.
- Change `/size_debug` response to include lease point in the SVG
visualization.
- Fix `/lsn_lease` HTTP API to do proper parsing for POST.



Signed-off-by: Yuchen Liang <yuchen@neon.tech>
Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Co-authored-by: Christian Schwarz <christian@neon.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update synthetic size calculation to account for leases.
3 participants