-
Notifications
You must be signed in to change notification settings - Fork 394
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Epic: Pageserver should support 200K tenants #4816
Comments
Lets mention our assumption on how many of those 200k are active. I'd say that it is ~1-3%. Taking the average of 2% gives us 4k active tenants. Is it accurate? Current problems are:
|
Our latest thinking on this topic is that it does not make sense to attach idle tenants and then layer a low-energy mechanism on top of that. Instead, low energy tenant should not be attached at all. The Secondary mode in #5299 is also suitable for idle tenants: whereas a normal active tenant would have one attached location and one secondary location, a long-idle tenant can just have one secondary layer with warm=false. Its layers will gradually fall off disk due to disk pressure. Eventually, it will make sense to not even have secondary locations (these are cheaper than attachments but still have some cost), especially for small tenants whose layers can rapidly be downloaded from S3 on-demand. The responsibility for orchestrating this gets pushed up to the control plane, or to some new intermediate service that manages pageservers. To make this complete, it is also necessary to handle:
Nice to have:
|
Motivation
To reduce costs and support higher density PS
DoD
Pageserver P90 get_page_request is the same with 200K tenants
Implementation ideas
Detaching in-active tenants
Reducing metrics
Tasks
Other related tasks and Epics
The text was updated successfully, but these errors were encountered: