Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add explicit GPU models for all makes #111

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

parikls
Copy link

@parikls parikls commented Oct 8, 2024

No description provided.

Copy link
Contributor

@zubenkoivan zubenkoivan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, lgtm once comments addressed

neuro_config_client/entities.py Outdated Show resolved Hide resolved
neuro_config_client/entities.py Outdated Show resolved Hide resolved
@@ -121,8 +121,12 @@ class NodePool:
disk_size: int | None = None
disk_type: str | None = None

gpu: int | None = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You cannot just drop these fields, it breaks SDK/CLI.
Some migration procedure is required.
I suggest keeping gpu / gpu_model here for a while (say, until the New Year).
gpu could be a sum of amd, intel and nvidia.
gpu_model could be a first available model or concatination of available modes. We don't parse these values, so nvidia-tesla-k80+amd-rizen-xxx would be fine.

Later, when all clients update CLI/FLOW on their machines, we can drop these fields.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asvetlov I've restored both fields for now to not break the compatibility with the SDK.

Just FYI:

gpu was already dropped from the platform config api at April: https://github.com/neuro-inc/platform-config/commit/fc83088224859d5e81b35beb426069888e05c336#diff-61fdfd6f40520f332212d51ba625421a707ce6ff111f58610dd10c84b27b1d9b

If gpu property is sent to the API, it's converted by the API to an nvidia_gpu under the hood - https://github.com/neuro-inc/platform-config/commit/fc83088224859d5e81b35beb426069888e05c336#diff-c2827afbcb2bed90d9593131e0055607f29b65656938e4b93d972c2d40c7e27eR118

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that gpu property is actually blank in a DB since April? @zubenkoivan can you elaborate please?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In April we renamed gpu to nvidia-gpu and added two other gpu kinds. Because before the April we didn't have non-nvidia systems.

I like your approach for keeping gpu as an alias for nvidia, e.g. nvidia_gpu=payload.get("nvidia_gpu") or payload.get("gpu"),

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should NodePoolOptions and Resource have triple nvidia/amd/intel values like NodePool?
If not please elaborate why.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asvetlov
Resource is used for idle jobs only. I've asked @zubenkoivan about that and he mentioned that it is not required to have a different manufacturers support there.

NodePoolOptions are used only be a cloud providers. As far as I understand - we don't need to add such support for clouds.

@zubenkoivan please correct me if I'm wrong.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct, we don't need to support this for NodePoolOptions and Resource classes

@parikls parikls force-pushed the explicit-gpu-make-and-model-f606fcc4960f branch from d7cf539 to 7ba9d6d Compare October 11, 2024 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants