feat: convert k8s submissions from pods to jobs (#9296) #9438
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update our Kubernetes resource manager to submit one job per Determined task instead of many pods. This is a complicated change but we think it is worth it because: - Jobs play nice with resource quotas and other Kubernetes features out of the box. - Eventually we can delegate restarts, TTL, pause/resume (using suspend), and more to jobs. - They allow us to integrate with kueue immediately. - If we want to support VolcanoJobs we are much closer (and it is easier to maintain Job+VolcanoJob than Pods+VolcanoJob). This change also contains some unrelated CI QoL changes I found useful while working.
✅ Deploy Preview for determined-ui canceled.
|
|
As of 66f9d1e CI is very happy so this should be a good starting point for any last minute refactors. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9438 +/- ##
==========================================
+ Coverage 48.60% 49.03% +0.43%
==========================================
Files 1233 1233
Lines 158981 159202 +221
Branches 2778 2778
==========================================
+ Hits 77271 78063 +792
+ Misses 81536 80965 -571
Partials 174 174
Flags with carried forward coverage won't be shown. Click here to find out more.
|
carolinaecalderon
approved these changes
May 30, 2024
stoksc
commented
May 30, 2024
stoksc
requested review from
azhou-determined
and removed request for
MikhailKardash
azhou-determined
approved these changes
May 30, 2024
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change updates our Kubernetes resource manager to submit one job per Determined task instead of many pods. This is a complicated change but we think it is worth it because:
This commit is the result of several PRs, enumerated here for easier discovery.
Ticket
Description
This is a feature branch. It already contains the core of the change, but we'll be merging release notes, docs, extra tests, and maybe some more adjustments (nothing major) before it lands later this week.
Test Plan
Checklist
docs/release-notes/.See Release Note for details.