Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added checks for dataset registration #200

Merged
merged 1 commit into from
Aug 4, 2022

Conversation

dhruvsgarg
Copy link
Contributor

@dhruvsgarg dhruvsgarg commented Jul 29, 2022

This pr maps the realm in the input dataset json to the registered computes. The realm in the dataset json specifies either region OR computeId. If the dataset is private, the specified region or computeId must be correct. If it is invalid, an error would be thrown and creation of the dataset would fail. In the case of public datasets, if the region or computeId does not match, the defaultRealm is selected which also specifies a default computeId. The dataset realm is used at the time of task creation to assign computeIds and the job tasks get assigned to that compute cluster.

@dhruvsgarg dhruvsgarg requested a review from myungjin July 29, 2022 04:44
@codecov-commenter
Copy link

codecov-commenter commented Jul 29, 2022

Codecov Report

Merging #200 (2410f20) into main (0d3b13e) will increase coverage by 0.15%.
The diff coverage is 62.50%.

@@            Coverage Diff             @@
##             main     #200      +/-   ##
==========================================
+ Coverage   21.26%   21.41%   +0.15%     
==========================================
  Files          35       35              
  Lines        1632     1639       +7     
==========================================
+ Hits          347      351       +4     
- Misses       1275     1277       +2     
- Partials       10       11       +1     
Impacted Files Coverage Δ
cmd/controller/app/job/builder.go 61.78% <62.50%> (-0.11%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d3b13e...2410f20. Read the comment docs.

Copy link
Contributor

@myungjin myungjin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left several comments.Two independent things handled in one PR. Since compute notify hardcoding fix is incorrect, it is better to focus on dataset creation check and to remove compute notify hardcoding fix.

cmd/controller/app/database/mongodb/compute.go Outdated Show resolved Hide resolved
cmd/controller/app/database/mongodb/compute.go Outdated Show resolved Hide resolved
cmd/controller/app/database/mongodb/compute.go Outdated Show resolved Hide resolved
cmd/controller/app/job/default_handler.go Outdated Show resolved Hide resolved
cmd/controller/app/job/default_handler.go Outdated Show resolved Hide resolved
pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
@dhruvsgarg dhruvsgarg changed the title Added check for dataset creation, removed compute notify hardcoding Added checks for dataset registration Jul 29, 2022
Copy link
Contributor

@myungjin myungjin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comments are left. Please consider to address them.

cmd/controller/app/job/builder.go Outdated Show resolved Hide resolved
pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
cmd/controller/app/database/mongodb/compute.go Outdated Show resolved Hide resolved
cmd/controller/app/database/mongodb/compute.go Outdated Show resolved Hide resolved
cmd/controller/app/database/mongodb/compute.go Outdated Show resolved Hide resolved
pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
Copy link
Contributor

@myungjin myungjin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the revision is almost there. please consider to address a few comments.

pkg/openapi/controller/api_datasets_service.go Outdated Show resolved Hide resolved
cmd/controller/app/job/builder.go Outdated Show resolved Hide resolved
This pr maps the realm in the input dataset json to the registered computes. The realm in the dataset json specifies either region OR computeId. If the dataset is private, the specified region or computeId must be correct. If it is invalid, an error would be thrown and creation of the dataset would fail. In the case of public datasets, if the region or computeId does not match, the defaultRealm is selected which also specifies a default computeId. The dataset realm is used at the time of task creation to assign computeIds and the job tasks get assigned to that compute cluster.
Copy link
Contributor

@myungjin myungjin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one minor issue to think about. But overall lgtm.

@@ -110,3 +141,41 @@ func (s *DatasetsApiService) UpdateDataset(ctx context.Context, user string, dat

return openapi.Response(http.StatusNotImplemented, nil), errors.New("UpdateDataset method not implemented")
}

func (s *DatasetsApiService) checkComputeForDatasetByRegion(datasetRealm string, datasetInfo *openapi.DatasetInfo) error {
eligibleComputes, err := s.dbService.GetComputeIdsByRegion(datasetRealm)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no check on err. Is this safe or can the error be ignored? If so, you may need to change it to something like the following:

eligibleComputes, _ := s.dbService.GetComputeIdsByRegion(datasetRealm)

@myungjin myungjin merged commit 65f073b into cisco-open:main Aug 4, 2022
@dhruvsgarg dhruvsgarg deleted the updated_dataset_reg branch August 9, 2022 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants