pageserver: location configuration API, attachment modes, secondary locations #5299

jcsp · 2023-09-13T09:54:31Z

Problem

These changes are part of building seamless tenant migration, as described in the RFC:

docs/rfcs: add RFC for fast tenant migration/failover #5029

Summary of changes

A new configuration type LocationConf supersedes TenantConfOpt for storing a tenant's configuration in the pageserver repo dir. It contains TenantConfOpt, as well as a new mode attribute that describes what kind of location this is (secondary, attached, attachment mode etc). It is written to a file called config-v1 instead of config -- this prepares us for neatly making any other profound changes to the format of the file in future. Forward compat for existing pageserver code is achieved by writing out both old and new style files. Backward compat is achieved by checking for the old-style file if the new one isn't found.
The TenantMap type changes, to hold TenantSlot instead of just Tenant. The Tenant type continues to be used for attached tenants only. Tenants in other states (such as secondaries) are represented by a different variant of TenantSlot.
Where Tenant & Timeline used to hold an Arc<Mutex>, they now hold a reference to a AttachedTenantConf, which includes the extra information from LocationConf. This enables them to know the current attachment mode.
The attachment mode is used as an advisory input to decide whether to do compaction and GC (AttachedStale is meant to avoid doing uploads, AttachedMulti is meant to avoid doing deletions).
A new HTTP API is added at PUT /tenants/<tenant_id>/location_config to drive new location configuration. This provides a superset of the functionality of attach/detach/load/ignore:
- Attaching a tenant is just configuring it in an attached state
- Detaching a tenant is configuring it to a detached state
- Loading a tenant is just the same as attaching it
- Ignoring a tenant is the same as configuring it into Secondary with warm=false (i.e. retain the files on disk but do nothing else).

Caveats:

AttachedMulti tenants don't do compaction in this PR, but they do in the follow on pageserver: finer-grained handling of migration states #5397
Concurrent updates to the location_config API are not handled elegantly in this PR, a better mechanism is added in the follow on pageserver: add InProgress tenant map state, use a sync lock for the map #5367
Secondary mode is just a placeholder in this PR: the code to upload heatmaps and do downloads on secondary locations will be added in a later PR (but that shouldn't change any external interfaces)

Closes: #5379

Checklist before requesting a review

I have performed a self-review of my code.
If it is a core feature, I have added thorough tests.
Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

Do not forget to reformat commit message to not include the above checklist

github-actions · 2023-09-13T16:50:16Z

2238 tests run: 2122 passed, 0 failed, 116 skipped (full report)

Flaky tests (2)

Postgres 16

test_wal_lagging: release

Postgres 15

test_gc_cutoff: debug

Code coverage (full report)

functions: 52.5% (8116 of 15462 functions)
lines: 81.2% (47454 of 58472 lines)

_{The comment gets automatically updated with the latest test results
3b1b1d8 at 2023-10-04T17:11:12.357Z :recycle:}

problame

Didn't review mgr.rs nor config.rs yet. Will do that tomorrow. In the meantime, left you comment #5299 (comment) to worry about / figure out what to do.

pageserver/src/tenant/config.rs

pageserver/src/tenant.rs

pageserver/src/tenant/mgr.rs

pageserver/src/tenant/config.rs

pageserver/src/tenant/mgr.rs

pageserver/src/tenant/config.rs

Co-authored-by: Christian Schwarz <christian@neon.tech>

pageserver/src/tenant/mgr.rs

problame

Apart from the two comments I added, I approve these changes here.

Not sure if we should merge them before the existing generation numbers support has gotten exposure to prod.

…tions-pt1

…e map (#5367) ## Problem Follows on from #5299 - We didn't have a generic way to protect a tenant undergoing changes: `Tenant` had states, but for our arbitrary transitions between secondary/attached, we need a general way to say "reserve this tenant ID, and don't allow any other ops on it, but don't try and report it as being in any particular state". - The TenantsMap structure was behind an async RwLock, but it was never correct to hold it across await points: that would block any other changes for all tenants. ## Summary of changes - Add the `TenantSlot::InProgress` value. This means: - Incoming administrative operations on the tenant should retry later - Anything trying to read the live state of the tenant (e.g. a page service reader) should retry later or block. - Store TenantsMap in `std::sync::RwLock` - Provide an extended `get_active_tenant_with_timeout` for page_service to use, which will wait on InProgress slots as well as non-active tenants. Closes: #5378 --------- Co-authored-by: Christian Schwarz <christian@neon.tech>

…ration none/broken usages (#7947) ## Problem In #5299, the new config-v1 tenant config file was added to hold the LocationConf type. We left the old config file in place for forward compat, and because running without generations (therefore without LocationConf) as still useful before the storage controller was ready for prime-time. Closes: #5388 ## Summary of changes - Remove code for reading and writing the legacy config file - Remove Generation::Broken: it was unused. - Treat missing config file on disk as an error loading a tenant, rather than defaulting it. We can now remove LocationConf::default, and thereby guarantee that we never construct a tenant with a None generation. - Update some comments + add some assertions to clarify that Generation::None is only used in layer metadata, not in the state of a running tenant. - Update docker compose test to create tenants with a generation

jcsp added t/feature Issue type: feature, for new features or requests c/storage/pageserver Component: storage: pageserver labels Sep 13, 2023

jcsp mentioned this pull request Sep 13, 2023

pageserver: consistent validation of tenant conf #5300

Closed

jcsp changed the title ~~WIP: secondary locations prototype~~ pageserver: secondary locations prototype Sep 13, 2023

jcsp force-pushed the jcsp/secondary-locations-pt1 branch from ca2270d to 9b6bda9 Compare September 13, 2023 16:43

jcsp changed the title ~~pageserver: secondary locations prototype~~ pageserver: location configuration API, attachment modes, secondary locations Sep 14, 2023

jcsp force-pushed the jcsp/secondary-locations-pt1 branch 3 times, most recently from a435997 to 665647c Compare September 16, 2023 12:23

jcsp mentioned this pull request Sep 25, 2023

pageserver: add InProgress tenant map state, use a sync lock for the map #5367

Merged

jcsp force-pushed the jcsp/secondary-locations-pt1 branch from 665647c to 50bdc2e Compare September 26, 2023 09:15

This was referenced Sep 26, 2023

pageserver: full reconciliation of tenant locations (attached & secondary) on startup #5377

Closed

pageserver: implement API for secondary locations, attachment modes #5379

Closed

jcsp force-pushed the jcsp/secondary-locations-pt1 branch from 50bdc2e to 9239e92 Compare September 26, 2023 09:37

jcsp mentioned this pull request Sep 26, 2023

pageserver: implement "emergency mode" for starting without a control plane #5381

Closed

jcsp marked this pull request as ready for review September 26, 2023 10:00

jcsp requested review from a team as code owners September 26, 2023 10:00

jcsp requested review from save-buffer, shanyp and problame and removed request for a team, save-buffer and shanyp September 26, 2023 10:00

jcsp force-pushed the jcsp/secondary-locations-pt1 branch from 9239e92 to 1c5e6cc Compare September 26, 2023 16:29

jcsp added 3 commits September 26, 2023 17:29

pageserver: add LocationConf type

8a4e18d

pageserver: persist LocationConf instead of TenantConf

d566b48

pageserver: forward & backward compat load/store of config

0f00d1d

problame reviewed Sep 27, 2023

View reviewed changes

problame reviewed Sep 28, 2023

View reviewed changes

jcsp and others added 12 commits September 29, 2023 09:18

Apply suggestions from code review

cf4350d

Co-authored-by: Christian Schwarz <christian@neon.tech>

Apply suggestions from code review

4d6fc9a

Co-authored-by: Christian Schwarz <christian@neon.tech>

Revise LocationConf::try_from

316f346

Use an exhaustive match

bc8874f

Remove faulty default-on-error for LocatinoConf

0f1cad1

Pass by ref

69e80d0

improve naming of update_generation

a327f56

Improve LocationConf load on start

e578ed5

clippy

708324a

remove stale comment.

030627c

rename LocationConf::new

50543eb

Don't expose whole LocationConf to Tenant

117213a

jcsp requested a review from problame September 29, 2023 10:54

problame reviewed Sep 29, 2023

View reviewed changes

pageserver/src/tenant/mgr.rs Outdated Show resolved Hide resolved

problame reviewed Sep 29, 2023

View reviewed changes

pageserver/src/tenant/mgr.rs Show resolved Hide resolved

problame approved these changes Sep 29, 2023

View reviewed changes

Mark tenants broken on error loading config

9fcea2d

jcsp mentioned this pull request Oct 2, 2023

Epic: Pageserver should support 200K tenants #4816

Closed

Merge remote-tracking branch 'upstream/main' into jcsp/secondary-loca…

519fcb8

…tions-pt1

jcsp mentioned this pull request Oct 4, 2023

pageserver: implement emergency mode for operating without control plane #5469

Merged

Merge remote-tracking branch 'upstream/main' into jcsp/secondary-loca…

3b1b1d8

…tions-pt1

jcsp enabled auto-merge (squash) October 4, 2023 16:35

jcsp merged commit baa5fa1 into main Oct 5, 2023
39 checks passed

jcsp deleted the jcsp/secondary-locations-pt1 branch October 5, 2023 08:55

jcsp mentioned this pull request Jun 3, 2024

pageserver: remove legacy tenant config code, clean up redundant generation none/broken usages #7947

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pageserver: location configuration API, attachment modes, secondary locations #5299

pageserver: location configuration API, attachment modes, secondary locations #5299

jcsp commented Sep 13, 2023 •

edited

Loading

github-actions bot commented Sep 13, 2023 •

edited

Loading

Postgres 16

Postgres 15

problame left a comment

problame left a comment

pageserver: location configuration API, attachment modes, secondary locations #5299

pageserver: location configuration API, attachment modes, secondary locations #5299

Conversation

jcsp commented Sep 13, 2023 • edited Loading

Problem

Summary of changes

Checklist before requesting a review

Checklist before merging

github-actions bot commented Sep 13, 2023 • edited Loading

2238 tests run: 2122 passed, 0 failed, 116 skipped (full report)

Postgres 16

Postgres 15

Code coverage (full report)

problame left a comment

Choose a reason for hiding this comment

problame left a comment

Choose a reason for hiding this comment

jcsp commented Sep 13, 2023 •

edited

Loading

github-actions bot commented Sep 13, 2023 •

edited

Loading