Merge consensus jobs #8042
Draft
Merge consensus jobs #8042
+7,224
−341
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rce_job_id under Job
…ouldn't be updated once Task is created
…ta model to Task model
…core_threshold which are set through extra configuration when creating task
…can be set through API call also when creating task initially
…multiple annotator mode
… passed from frontend to backend
…rted not gt or consensus jobs
… through the parent_job_id
I tried to build via `docker-compose -f docker-compose.yml -f docker-compose.dev.yml build` but the installation of h5py fails in line https://github.com/cvat-ai/cvat/blob/develop/Dockerfile#L81 with error `Failed building wheel for h5py` adding the depdency `libhdf5-dev` fixes the issue. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Updated Dockerfile to include the `libhdf5-dev` package for improved compatibility and support. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
<!-- Raise an issue to propose your change (https://github.com/cvat-ai/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context Related cvat-ai#6536 ### How has this been tested? <!-- Please describe in detail how you tested your changes. Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [x] I submit my changes into the `develop` branch - [x] I have created a changelog fragment <!-- see top comment in CHANGELOG.md --> - [ ] I have updated the documentation accordingly - [ ] I have added tests to cover my changes - [ ] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)) - [ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning)) ### License - [x] I submit _my code changes_ under the same [MIT License]( https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Resolved an issue where copying/pasting annotation guides with assets between different resources failed due to an error message. - **Refactor** - Renamed `GuidePage` component to `AnnotationGuidePage` and refactored logic for fetching and saving guides. - Updated component structure and behavior in the annotation guide editor. - **Style** - Adjusted the height property in the annotation guide editor for better layout consistency. - **New Features** - Added a method to handle updating related assets in the annotation guides. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Andrey Zhavoronkov <andrey@cvat.ai>
) <!-- Raise an issue to propose your change (https://github.com/cvat-ai/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context <!-- Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. Describe your changes in detail, add screenshots. --> There are 2 schedulers supported by django_rq and by python RQ: `rq_scheduler` and a newer, builtin queue scheduler in RQ. rq_scheduler seems to die slowly in favor of the builtin scheduler. The schedulers have compatible API, but not the implementation. The existing job retry implementation relies on `retry()` calls, which, in turn, rely on the builtin RQ scheduler. CVAT uses rq_scheduler a for some tasks, so it its executed. The builtin RQ scheduler needs the `--with-scheduler` startup parameter on the worker processes. Thus, the jobs were hanging in the scheduled state, as the builtin RQ scheduler was not running on the queues. As CVAT is currently using rq_scheduler, it's decided to continue using it to avoid disruption and use of 2 schedulers together. The implementation in this PR does best efforts to be correct, but it's has potential problems with multiple same jobs running in parallel. In future we need to migrate to the builtin RQ scheduler, as it is the only one maintained as of February 2023. ### How has this been tested? <!-- Please describe in detail how you tested your changes. Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [ ] I submit my changes into the `develop` branch - [ ] I have created a changelog fragment <!-- see top comment in CHANGELOG.md --> - [ ] I have updated the documentation accordingly - [ ] I have added tests to cover my changes - [ ] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)) - [ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning)) ### License - [ ] I submit _my code changes_ under the same [MIT License]( https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved export and export cache clean operations by adding a retry mechanism to handle job retries, preventing hangs. - **Chores** - Updated internal process for handling job retries using RQ scheduler for better reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes
Aggregating their annotationsandMerging them into the parent jobin #7973Checklist
developbranch- [ ] I have increased versions of npm packages if it is necessary(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)
License
Feel free to contact the maintainers if that's a concern.