[WIP] Add enhancement for Parameter Distribution #2059

tenzen-y · 2022-12-12T03:13:29Z

Signed-off-by: tenzen-y yuki.iwai.tz@gmail.com

What this PR does / why we need it:
I added an enhancement proposal for Parameter Distribution as discussed in this.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
related #1207

Checklist:

Docs included if any changes are user facing

google-oss-prow · 2022-12-12T03:13:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: tenzen-y <yuki.iwai.tz@gmail.com>

tenzen-y · 2022-12-12T03:24:40Z

/hold for the review

johnugeorge · 2022-12-12T04:58:40Z

docs/proposals/parameter-distribution.md

+type ParameterSpec struct {
+	Name          string        `json:"name,omitempty"`
+- 	ParameterType ParameterType `json:"parameterType,omitempty"`
+ 	Distribution  Distribution  `json:"distribution,omitempty"`


Great proposal @tenzen-y . One question, how do we provide backward compatibility?

@johnugeorge This is a good point.

For the time being (1~2 releases?), I think we can operate ParameterType and Distribution concurrently.
This means in the case of users determining ParameterType, suggestion-services operate as now; in the case of users determining Distribution, suggestion-services set distributions to sampler.

Also, we should add webhook validation to restrict ParameterType and Distribution so that only one of them is available. (ParameterType and Distribution are exclusive)

@andreyvelich @johnugeorge wdyt?
If you agree with this, I will add this to the Proposal.

SGTM. Also, add deprecation tag to ParameterType

Also, add deprecation tag to ParameterType

SGTM
I will add the tag to the following:

katib/pkg/apis/manager/v1beta1/api.proto

Line 83 in f941ec6

ParameterType parameter_type = 2; /// Type of the parameter.

If we add only new features to v1beta2 API, deprecation labels are unnecessary since we create a separate proto definition for v1beta2 API as discussed in #2059 (comment).

tenzen-y · 2022-12-12T05:04:56Z

docs/proposals/parameter-distribution.md

+search space using libraries provided in each framework.
+
+#### Chocolate
+TODO


blocked by #2058

andreyvelich

Thanks a lot for driving this @tenzen-y!
I left few comments.

andreyvelich · 2022-12-12T20:35:33Z

docs/proposals/parameter-distribution.md

+Currently, Katib does not support determining a distribution for search space that samplers pick up parameters by users.
+
+Katib should be able to determine it by users since
+almost hyperparameter tuning algorithms (framework) can determine it by users.


Please can you link the appropriate issue: #1207 to this proposal motivation?

Makes sense.

andreyvelich · 2022-12-12T20:40:12Z

docs/proposals/parameter-distribution.md

+| IntUniformDistribution      | space.Integer                |
+| IntLogUniformDistribution   | space.Integer                |


I guess, you can set uniform or loguniform in skopt using prior API: https://scikit-optimize.github.io/stable/modules/generated/skopt.space.space.Real.html#skopt.space.space.Real

Yes, that's right. We need to set the prior argument in skopt. Also, we need to set the log argument in optuna.
I will add them to this enhancement proposal.

ref: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.distributions.FloatDistribution.html#optuna-distributions-floatdistribution

andreyvelich · 2022-12-12T20:50:15Z

docs/proposals/parameter-distribution.md

+ 	IntUniformDistribution      Distribution = "intUniform"
+ 	IntLogUniformDistribution   Distribution = "intLogUniform"
+ 	FloatUniformDistribution    Distribution = "floatUniform"
+ 	FloatLogUniformDistribution Distribution = "floatLogUniform"


@johnugeorge @tenzen-y @gaocegege @anencore94 What do you think about following hyperopt model instead of int and float model (e.g. uniform, quniform, loguniform, qloguniform) ? From my point of view, it sounds more native to HP tuning and many HPs papers mention that distribution.
Also, we can change step to q and integrate base parameter for the log.
Many data scientists who do HP tuning are familiar with Hyperopt, so the API will look the same for them.

Also, Ray Tune follows the same model: https://docs.ray.io/en/latest/tune/api_docs/search_space.html, and NNI has the same APIs: https://nni.readthedocs.io/en/stable/hpo/search_space.html#quniform

What do you think about following hyperopt model instead of int and float model (e.g. uniform, quniform, loguniform, qloguniform) ? From my point of view, it sounds more native to HP tuning and many HPs papers mention that distribution.

@andreyvelich Sounds good. I would add the corresponding tables for the old ParameterType and new Distribution using the hyperopt model to this proposal.

Also, we can change step to q and integrate base parameter for the log.

@andreyvelich Sounds good. One question, Does integrate base parameter for the log mean adding the base field to struct FeasibleSpace?

SGTM. While I am thinking if this is a huge change to our YAML APIs.

SGTM. While I am thinking if this is a huge change to our YAML APIs.

Maybe, we need to change the API version to v1beta2.

SGTM

Is it possible to convert v1beta1 resource object to v1beta2? Will it drop some necessary info from the conversion?

I will create a correspondence table between v1beta1 and v1beta2. Maybe, we only need to create a table for the ParameterType and the FeasibleSpace.

When will the webhook be configured? Should we install it by default?

IIUC, we do not need to install manifests for conversion webhook to clusters.

ref:

https://book.kubebuilder.io/multiversion-tutorial/tutorial.html

https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/multiversion-tutorial/testdata/project

And when will we deprecate v1beta1?

IMO, we need to keep maintaining v1beta1 for at least one release version. This means if we introduce v1beta2 API in katib v0.16.0, we will remove v1beta1 API in katib v0.17.0.

@gaocegege Do you know how many release versions we kept maintaining v1alpha2 after we introduced v1beta1?

Yep, I think so. But we need a detailed design for this to see if it is possible.

@andreyvelich @gaocegege Maybe, custom (implemented by user) suggestion services using v1beta1 API will not work since gRPC calls are not through conversion webhook.

<------------------------------ [Updated] ------------------------------
So, we probably need to separate CRD version changes from Distribution introduces. And then I take up only Introducing Distribution in this proposal. We can follow up on Upgrading CRD version in other issues and PRs.

~~- Introducing Distribution: we keep using ParameterType and introducing Distribution and Base to FeasibleSpace like the following.~~

~~#1207 (comment)~~

So, I would like to work in the following:

~~- Upgrading CRD version:~~

------------------------------ [Updated] ------------------------------>

introduce a new field that represents the gRPC API version (v1beta1 or v1beta2) to the following of katib-config since the suggestion controller needs to use a different gRPC client for v1beta1 and v1beta2. This means we keep maintaining both v1beta1 and v1beta2 gRPC APIs (proto) for a while (only gRPC API, no maintaining v1beta1 controller). And then after we remove the v1beta1 API, remove the new field in katib-config.

katib/pkg/util/v1beta1/katibconfig/config.go

Lines 35 to 45 in db72ce1

// SuggestionConfig is the JSON suggestion structure in Katib config.

type SuggestionConfig struct {

Image string `json:"image"`

ImagePullPolicy corev1.PullPolicy `json:"imagePullPolicy,omitempty"`

Resource corev1.ResourceRequirements `json:"resources,omitempty"`

ServiceAccountName string `json:"serviceAccountName,omitempty"`

VolumeMountPath string `json:"volumeMountPath,omitempty"`

PersistentVolumeClaimSpec corev1.PersistentVolumeClaimSpec `json:"persistentVolumeClaimSpec,omitempty"`

PersistentVolumeSpec corev1.PersistentVolumeSpec `json:"persistentVolumeSpec,omitempty"`

PersistentVolumeLabels map[string]string `json:"persistentVolumeLabels,omitempty"`

}

<------------------------------ [Updated] ------------------------------

~~Consolidate ParameterType and FeasibleSpace.Distribution to Distribution~~ Remove ParameterType API and add Distribution API based on the hyperopt model like @andreyvelich mentioned at [WIP] Add enhancement for Parameter Distribution #2059 (comment).

------------------------------ [Updated] ------------------------------>

@johnugeorge @tenzen-y @gaocegege @anencore94 What do you think about following hyperopt model instead of int and float model (e.g. uniform, quniform, loguniform, qloguniform) ? From my point of view, it sounds more native to HP tuning and many HPs papers mention that distribution.
Also, we can change step to q and integrate base parameter for the log.
Many data scientists who do HP tuning are familiar with Hyperopt, so the API will look the same for them.

Also, Ray Tune follows the same model: https://docs.ray.io/en/latest/tune/api_docs/search_space.html, and NNI has the same APIs: https://nni.readthedocs.io/en/stable/hpo/search_space.html#quniform

@andreyvelich @gaocegege @johnugeorge @anencore94 wdyt?

Hmm, gRPC might be a problem, yes.
Do we know how Kubernetes maintain 2 version of their gRPC APIs ?
e.g. v1 version for apps and v1beta2 version for apps ?

@tenzen-y Also, are we going to rename intuniform to quniform and floatuniform to uniform as I proposed ?

Do we know how Kubernetes maintain 2 version of their gRPC APIs ?

@andreyvelich Kubernetes uses helper functions to convert multiple APIs.

https://github.com/kubernetes/kubernetes/blob/c1c0e4fe0bb4e7c0145d45a010577ed64619903a/pkg/apis/apps/v1beta2/conversion.go

Does that answer your question?

Also, are we going to rename intuniform to quniform and floatuniform to uniform as I proposed ?

Yes, I updated the above comment.

gaocegege · 2022-12-14T06:34:08Z

docs/proposals/parameter-distribution.md

+- 	ParameterTypeCategorical ParameterType = "categorical"
+ 	UnknownDistribution         Distribution = "unknown"
+ 	CategoricalDistribution     Distribution = "categorical"
+ 	IntUniformDistribution      Distribution = "intUniform"


Should we use camel case here? Personally prefer lower case intuniform

gaocegege · 2022-12-14T06:34:58Z

docs/proposals/parameter-distribution.md

+ 	IntUniformDistribution      Distribution = "intUniform"
+ 	IntLogUniformDistribution   Distribution = "intLogUniform"
+ 	FloatUniformDistribution    Distribution = "floatUniform"
+ 	FloatLogUniformDistribution Distribution = "floatLogUniform"


SGTM. While I am thinking if this is a huge change to our YAML APIs.

tenzen-y · 2023-01-07T17:12:11Z

I will work on this proposal after the kubeflow 1.7 feature freeze date.

github-actions · 2023-08-23T20:05:29Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tenzen-y · 2023-08-24T13:30:17Z

/remove-lifecycle stale

github-actions · 2023-11-22T15:05:20Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

andreyvelich · 2023-11-22T15:57:11Z

/lifecycle frozen

google-oss-prow · 2023-11-22T15:57:14Z

@andreyvelich: The lifecycle/frozen label cannot be applied to Pull Requests.

In response to this:

/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

github-actions · 2024-02-20T20:05:43Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tenzen-y · 2024-02-21T03:09:22Z

/remove-lifecycle stale

PeterWrighten · 2024-03-08T21:22:48Z

Hi, I'm interested in this project and Project5 related to GSoC 2024, and seeking for some docs or proposals for more details. If you can refer more details, it would help me a lot. Thanks! @tenzen-y @andreyvelich

github-actions · 2024-06-07T00:20:23Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tenzen-y · 2024-06-07T05:05:46Z

/remove-lifecycle stale

google-oss-prow bot requested review from andreyvelich, anencore94 and johnugeorge December 12, 2022 03:13

google-oss-prow bot added approved size/L labels Dec 12, 2022

tenzen-y force-pushed the add-proposal-log-uniform-scale branch from d240479 to 05ffd16 Compare December 12, 2022 03:18

Add enhancement for Parameter Distribution

93bdfc2

Signed-off-by: tenzen-y <yuki.iwai.tz@gmail.com>

tenzen-y force-pushed the add-proposal-log-uniform-scale branch from 05ffd16 to 93bdfc2 Compare December 12, 2022 03:21

google-oss-prow bot added the do-not-merge/hold label Dec 12, 2022

johnugeorge reviewed Dec 12, 2022

View reviewed changes

tenzen-y commented Dec 12, 2022

View reviewed changes

andreyvelich reviewed Dec 12, 2022

View reviewed changes

gaocegege reviewed Dec 14, 2022

View reviewed changes

tenzen-y changed the title ~~Add enhancement for Parameter Distribution~~ [WIP] Add enhancement for Parameter Distribution Jan 7, 2023

google-oss-prow bot added the do-not-merge/work-in-progress label Jan 7, 2023

tenzen-y mentioned this pull request May 15, 2023

[feature] Support log-uniform scale in search space definition #1207

Open

github-actions bot added the lifecycle/stale label Aug 23, 2023

google-oss-prow bot removed the lifecycle/stale label Aug 24, 2023

github-actions bot added the lifecycle/stale label Nov 22, 2023

github-actions bot removed the lifecycle/stale label Nov 22, 2023

github-actions bot added the lifecycle/stale label Feb 20, 2024

google-oss-prow bot removed the lifecycle/stale label Feb 21, 2024

shashank-iitbhu mentioned this pull request May 28, 2024

[GSOC] Support for various Parameter distributions in Katib #2334

Open

1 task

github-actions bot added the lifecycle/stale label Jun 7, 2024

google-oss-prow bot removed the lifecycle/stale label Jun 7, 2024

		\| IntUniformDistribution \| space.Integer \|
		\| IntLogUniformDistribution \| space.Integer \|

	// SuggestionConfig is the JSON suggestion structure in Katib config.
	type SuggestionConfig struct {
	Image string `json:"image"`
	ImagePullPolicy corev1.PullPolicy `json:"imagePullPolicy,omitempty"`
	Resource corev1.ResourceRequirements `json:"resources,omitempty"`
	ServiceAccountName string `json:"serviceAccountName,omitempty"`
	VolumeMountPath string `json:"volumeMountPath,omitempty"`
	PersistentVolumeClaimSpec corev1.PersistentVolumeClaimSpec `json:"persistentVolumeClaimSpec,omitempty"`
	PersistentVolumeSpec corev1.PersistentVolumeSpec `json:"persistentVolumeSpec,omitempty"`
	PersistentVolumeLabels map[string]string `json:"persistentVolumeLabels,omitempty"`
	}

[WIP] Add enhancement for Parameter Distribution #2059

Are you sure you want to change the base?

[WIP] Add enhancement for Parameter Distribution #2059

Conversation

tenzen-y commented Dec 12, 2022 • edited Loading

google-oss-prow bot commented Dec 12, 2022

tenzen-y commented Dec 12, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Dec 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Dec 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented Jan 7, 2023

github-actions bot commented Aug 23, 2023

tenzen-y commented Aug 24, 2023

github-actions bot commented Nov 22, 2023

andreyvelich commented Nov 22, 2023

google-oss-prow bot commented Nov 22, 2023

github-actions bot commented Feb 20, 2024

tenzen-y commented Feb 21, 2024

PeterWrighten commented Mar 8, 2024 • edited Loading

github-actions bot commented Jun 7, 2024

tenzen-y commented Jun 7, 2024

tenzen-y commented Dec 12, 2022 •

edited

Loading

tenzen-y Dec 22, 2022 •

edited

Loading

tenzen-y Dec 15, 2022 •

edited

Loading

PeterWrighten commented Mar 8, 2024 •

edited

Loading