Fix dataset downloading #7864
Merged
Fix dataset downloading #7864
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- prolong export cache lifetime on cache requests - change 500 to 404 on missing cache file during downloading - remove rq job removal after result retrieval - make export filenames more distinct - fix possible infinite deferring of dependent rq jobs
WalkthroughThe changes introduce new functionalities for handling concurrent export and cleanup processes in the CVAT application, alongside updates to package dependencies. Key additions include new classes and methods for multiprocessing, lock management, and export file handling. These improvements aim to enhance the reliability and efficiency of export operations. Additionally, multiple package versions have been upgraded to maintain compatibility and security. Changes
Sequence Diagram(s) (Beta)sequenceDiagram
participant User
participant CVATApp
participant ExportManager
participant LockManager
participant FileHandler
User->>CVATApp: Initiate export
CVATApp->>ExportManager: Start export process
ExportManager->>LockManager: Acquire lock
LockManager-->>ExportManager: Lock acquired
ExportManager->>FileHandler: Export data
FileHandler-->>ExportManager: Data exported
ExportManager->>LockManager: Release lock
LockManager-->>ExportManager: Lock released
ExportManager-->>CVATApp: Export completed
CVATApp-->>User: Export successful
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
![coderabbitai[bot]](https://github.com/avatars/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Out of diff range and nitpick comments (3)
cvat/apps/engine/views.py (3)
Line range hint
44-44: Remove unused import to clean up the code.- import cvat.apps.dataset_manager.views # pylint: disable=unused-import
Line range hint
1966-1966: Remove unnecessary f-string as it does not contain any placeholders.- server_address = request.scheme + '://' + server_address = str(request.scheme) + '://'
Line range hint
2713-2713: Remove unnecessary f-string as it does not contain any placeholders.- server_address += request.get_host() + server_address = server_address + str(request.get_host())
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (10)
- cvat/apps/dataset_manager/apps.py (1 hunks)
- cvat/apps/dataset_manager/default_settings.py (1 hunks)
- cvat/apps/dataset_manager/util.py (2 hunks)
- cvat/apps/dataset_manager/views.py (4 hunks)
- cvat/apps/engine/views.py (4 hunks)
- cvat/requirements/base.in (1 hunks)
- cvat/requirements/base.txt (10 hunks)
- cvat/requirements/development.txt (3 hunks)
- cvat/requirements/production.txt (1 hunks)
- utils/dataset_manifest/requirements.txt (2 hunks)
Files skipped from review due to trivial changes (5)
- cvat/apps/dataset_manager/apps.py
- cvat/apps/dataset_manager/default_settings.py
- cvat/requirements/base.in
- cvat/requirements/development.txt
- utils/dataset_manifest/requirements.txt
Additional Context Used
Ruff (4)
cvat/apps/dataset_manager/views.py (1)
27-27:
.util.LockModeimported but unusedcvat/apps/engine/views.py (3)
44-44:
cvat.apps.dataset_manager.viewsimported but unused
1966-1966: f-string without any placeholders
2713-2713: f-string without any placeholders
Additional comments not posted (6)
cvat/requirements/production.txt (1)
13-13: Update ofexceptiongroupto1.2.1appears safe and beneficial.cvat/apps/dataset_manager/util.py (1)
93-146: Introduced a robust locking mechanism usingRedlock. Ensure thorough testing, especially for edge cases like rapid lock acquisition and release.cvat/requirements/base.txt (1)
Line range hint
53-325: Updates to package versions inbase.txtare minor and likely include improvements or security patches. Verify compatibility with existing codebase.cvat/apps/dataset_manager/views.py (1)
Line range hint
27-149: Integration of new locking mechanism and error handling in dataset management views appears well-implemented. Recommend additional testing for robustness.cvat/apps/engine/views.py (2)
2962-2962: Ensure that the new caching logic in_export_annotationsis correctly implemented and tested.
3022-3022: Confirm that the error handling inexport_annotationscovers all potential failure scenarios effectively.Verification successful
The output from the revised script provides a comprehensive list of exception handling within the
export_annotationsmethod and related areas in thecvat/apps/engine/views.pyfile. The exceptions handled includeValidationError,PermissionDenied,NotFound,AttributeError,IntegrityError,APIException,ValueError,CloudStorageModel.DoesNotExist, and generalException. This indicates a robust error handling mechanism that covers a wide range of potential issues that could arise during the execution of theexport_annotationsmethod.Given this information, it appears that the error handling within the
export_annotationsmethod is indeed comprehensive, covering various specific and general exceptions that could occur. This aligns with the initial review comment's concern about ensuring that all potential failure scenarios are effectively covered.Conclusion:
The error handling in theexport_annotationsmethod appears to be comprehensive and well-implemented, addressing a variety of error conditions that could potentially disrupt the functionality.
- The method handles specific exceptions like
ValidationError,PermissionDenied, andNotFound, which are crucial for robust API behavior.- It also covers database-related errors through
AttributeErrorandIntegrityError.- General exceptions and unexpected errors are caught by broad
Exceptionhandling.- Special cases like
CloudStorageModel.DoesNotExistare also considered, which is important for operations involving external storage.This thorough approach ensures that the method can gracefully handle errors and provide appropriate responses, thereby enhancing the reliability and user experience of the dataset exporting functionalities.
Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check all branches of error handling in the `export_annotations` method to ensure comprehensive coverage. # Test: Search for error handling branches in the method. Expect: Each branch handles a specific error scenario appropriately. rg --type python $'export_annotations'Length of output: 68
Script:
#!/bin/bash # Description: Re-check all branches of error handling in the `export_annotations` method to ensure comprehensive coverage. # Test: Search for error handling branches in the method. Expect: Each branch handles a specific error scenario appropriately. rg 'export_annotations' --files-with-matches | xargs rg 'except'Length of output: 3895
|
/check |
|
❌ Some checks failed |
|
❌ Some checks failed |
There was a problem hiding this comment.
Actionable comments posted: 1
Out of diff range and nitpick comments (6)
cvat/apps/dataset_manager/util.py (1)
Line range hint
16-41: Consider adding more detailed docstrings for the utility functions to improve maintainability and clarity for other developers.+ """ + Creates a zip archive from a directory. + + Args: + src_path (str): The source directory path. + dst_path (str): The destination zip file path. + """tests/python/rest_api/test_resource_import_export.py (2)
Line range hint
32-57: Comprehensive test cases for exporting resources to cloud storage. Consider adding comments to explain the test scenarios in detail for better maintainability.+ """ + Test saving resources to cloud storage with specific configurations. + This test covers various object types and ensures that the export functionality + respects the specified cloud storage locations and settings. + """
Line range hint
182-207: Import functionality tests are well-covered. Ensure that edge cases, such as network failures or partial data availability, are also tested.Would you like me to help by adding some tests for edge cases involving network failures or partial data availability?
cvat/apps/engine/views.py (3)
Line range hint
44-44: Remove unused import to clean up the code.- import cvat.apps.dataset_manager.views # pylint: disable=unused-import
Line range hint
1967-1967: The f-string used here does not contain any placeholders, which is unnecessary. Consider using a regular string if no formatting is needed.- server_address = request.scheme + '://' + server_address = str(request.scheme) + '://'
Line range hint
2714-2714: Similar to the previous comment, an f-string is used here without placeholders. Use a regular string instead.- filename_pattern = build_annotations_file_name( + filename_pattern = str(build_annotations_file_name(
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (11)
- changelog.d/20240508_144202_mzhiltso_fix_dataset_downloading.md (1 hunks)
- cvat/apps/dataset_manager/default_settings.py (1 hunks)
- cvat/apps/dataset_manager/util.py (2 hunks)
- cvat/apps/dataset_manager/views.py (4 hunks)
- cvat/apps/engine/views.py (4 hunks)
- cvat/requirements/base.in (1 hunks)
- cvat/requirements/base.txt (10 hunks)
- cvat/requirements/development.txt (1 hunks)
- cvat/requirements/testing.txt (1 hunks)
- tests/python/rest_api/test_resource_import_export.py (5 hunks)
- utils/dataset_manifest/requirements.txt (2 hunks)
Files skipped from review due to trivial changes (1)
- cvat/requirements/testing.txt
Files skipped from review as they are similar to previous changes (4)
- cvat/apps/dataset_manager/default_settings.py
- cvat/requirements/base.in
- cvat/requirements/base.txt
- cvat/requirements/development.txt
Additional Context Used
Ruff (3)
cvat/apps/engine/views.py (3)
44-44:
cvat.apps.dataset_manager.viewsimported but unused
1967-1967: f-string without any placeholders
2714-2714: f-string without any placeholders
Additional comments not posted (5)
changelog.d/20240508_144202_mzhiltso_fix_dataset_downloading.md (1)
1-5: Changelog entry is clear and properly linked to the PR.utils/dataset_manifest/requirements.txt (2)
8-9: Ensure that non-binary flags forlxmlandxmlsecare required for your project's needs.Verification successful
The comprehensive search across the codebase confirms the consistent use of
--no-binaryflags forlxmlandxmlsecacross various requirement files and Dockerfiles. This indicates a deliberate and justified choice to compile these libraries from source, likely for compatibility or optimization reasons specific to the project's needs. The presence of related development libraries in Dockerfiles further supports this requirement.
- utils/dataset_manifest/requirements.txt: Both
lxmlandxmlsecare set to compile from source.- cvat/requirements/: Multiple environment-specific files (base, testing, development, production) also enforce non-binary installation.
- Dockerfile: Includes libraries like
libxmlsec1-devandlibxmlsec1-openssl, which are related toxmlsec, suggesting system-level dependencies that might require source compilation.Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify compatibility of updated package versions. # Test: Search for the package usage in the Python files. Expect: No deprecated functions used. rg --type py 'lxml|xmlsec'Length of output: 28
Script:
#!/bin/bash # Description: Comprehensive search for any references to 'lxml' or 'xmlsec' in the entire codebase. # Search all files for 'lxml' or 'xmlsec' to ensure no usage is missed. rg 'lxml|xmlsec'Length of output: 1026
21-21: Update totqdm==4.66.4looks good. Ensure it's tested for compatibility with your project's usage.cvat/apps/dataset_manager/util.py (1)
104-141: Locking mechanism usingRedlockis implemented correctly. Ensure thorough testing, especially for edge cases in concurrent environments.cvat/apps/dataset_manager/views.py (1)
179-227: Robust error handling inclear_export_cache. Ensure that all possible exceptions are considered and handled appropriately.
SpecLad
reviewed
May 13, 2024
… zm/fix-dataset-downloading
There was a problem hiding this comment.
Actionable comments posted: 6
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- cvat/apps/dataset_manager/tests/test_rest_api_formats.py (9 hunks)
Additional comments not posted (2)
cvat/apps/dataset_manager/tests/test_rest_api_formats.py (2)
17-17: Ensure proper usage of context managers to handle resources efficiently.Verification successful
Ensure proper usage of context managers to handle resources efficiently
The usage of
ExitStackandcontextmanagerincvat/apps/dataset_manager/tests/test_rest_api_formats.pyappears to be correct and follows proper resource management practices. Here are the key points:
@contextmanagerDecorators: These are used to define context managers for handling processes and locks, ensuring that resources are properly managed and released.ExitStack: This is used to manage multiple context managers, ensuring that all resources are properly cleaned up even if an error occurs.The code snippets provided show that these constructs are used appropriately to handle resources efficiently.
Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the usage of ExitStack and contextmanager in the file to ensure proper resource management. # Test: Search for ExitStack and contextmanager usage. Expect: Proper usage patterns. rg --type python $'ExitStack|contextmanager' cvat/apps/dataset_manager/tests/test_rest_api_formats.pyLength of output: 133
Script:
#!/bin/bash # Description: Verify the usage of ExitStack and contextmanager in the file to ensure proper resource management. # Test: Search for ExitStack and contextmanager usage. Expect: Proper usage patterns. rg 'ExitStack|contextmanager' cvat/apps/dataset_manager/tests/test_rest_api_formats.pyLength of output: 269
Script:
#!/bin/bash # Description: Extract surrounding lines of ExitStack and contextmanager usage for detailed verification. # Extract 5 lines before and after each match to understand the context. rg -C 5 'ExitStack|contextmanager' cvat/apps/dataset_manager/tests/test_rest_api_formats.pyLength of output: 2772
1792-1792: Ensure proper handling of file usage conflicts during cleanup operations.Verification successful
Ensure proper handling of file usage conflicts during cleanup operations
The search results indicate that
LockNotAvailableErrorandFileIsBeingUsedErrorare indeed referenced in the test filecvat/apps/dataset_manager/tests/test_rest_api_formats.py. This suggests that there are mechanisms in place to handle file usage conflicts during cleanup operations.
LockNotAvailableErroris imported and used multiple times.FileIsBeingUsedErroris also imported and used in assertions.These references confirm that the code includes error handling for file usage conflicts.
Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify that file usage conflicts are handled correctly in cleanup operations. # Test: Search for conflict handling in cleanup operations. Expect: Proper error handling mechanisms. rg --type python $'LockNotAvailableError|FileIsBeingUsedError' cvat/apps/dataset_manager/tests/test_rest_api_formats.pyLength of output: 151
Script:
#!/bin/bash # Description: Verify that file usage conflicts are handled correctly in cleanup operations. # Test: Search for conflict handling in cleanup operations. Expect: Proper error handling mechanisms. rg --type py $'LockNotAvailableError|FileIsBeingUsedError' cvat/apps/dataset_manager/tests/test_rest_api_formats.pyLength of output: 1084
zhiltsov-max
commented
May 17, 2024
zhiltsov-max
commented
May 17, 2024
zhiltsov-max
commented
May 17, 2024
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7864 +/- ##
===========================================
+ Coverage 83.35% 83.37% +0.02%
===========================================
Files 380 383 +3
Lines 40032 40171 +139
Branches 3766 3769 +3
===========================================
+ Hits 33367 33492 +125
- Misses 6665 6679 +14
|
SpecLad
reviewed
May 27, 2024
There was a problem hiding this comment.
Actionable comments posted: 10
Outside diff range and nitpick comments (1)
cvat/apps/dataset_manager/views.py (1)
99-129: Add detailed logging for export operations.Consider adding more detailed logging in the
exportfunction to provide better visibility into the export process. This could include logging the start and end of the export operation, any errors encountered, and the final status of the export. This would help in debugging and monitoring the export operations.
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (3)
- cvat/apps/dataset_manager/tests/test_rest_api_formats.py (9 hunks)
- cvat/apps/dataset_manager/util.py (2 hunks)
- cvat/apps/dataset_manager/views.py (3 hunks)
Additional Context Used
Ruff (1)
cvat/apps/dataset_manager/views.py (1)
29-29:
.util.EXPORT_CACHE_DIR_NAMEimported but unused
Learnings (1)
cvat/apps/dataset_manager/tests/test_rest_api_formats.py (1)
User: zhiltsov-max PR: cvat-ai/cvat#7864 File: cvat/apps/dataset_manager/tests/test_rest_api_formats.py:1266-1266 Timestamp: 2024-05-17T12:20:07.528Z Learning: The CVAT project prefers using the `attrs` module over `dataclasses` due to its additional features and is commonly used across the project.
Additional comments not posted (4)
cvat/apps/dataset_manager/tests/test_rest_api_formats.py (4)
1266-1266: Use Python'sdataclassesmodule for simplicity and efficiency.- @define + from dataclasses import dataclass - @dataclass class SharedBase:Likely invalid or redundant comment.
1265-1265: Enhance theExportBehaviorTestclass with additional comments for better understanding.+ # This class tests the behavior of concurrent export and cleanup operations. class ExportBehaviorTest(_DbTestBase):Likely invalid or redundant comment.
419-419: Optimize the cleanup of RQ jobs to enhance readability and understanding of its functionality.- self._clear_rq_jobs() # clean up from previous tests and iterations + self.clearResourceQueueJobs() # Clear resource queue jobs from previous tests and iterationsLikely invalid or redundant comment.
97-97: Consider renaming_DbTestBasetoDatabaseTestBasefor improved readability and adherence to Python naming conventions.- class _DbTestBase(ApiTestBase): + class DatabaseTestBase(ApiTestBase):Likely invalid or redundant comment.
There was a problem hiding this comment.
Actionable comments posted: 3
Outside diff range and nitpick comments (1)
cvat/apps/dataset_manager/tests/test_rest_api_formats.py (1)
421-421: Repeated calls to_clear_rq_jobscan be optimized.Consider refactoring the repeated calls to
_clear_rq_jobsacross different test methods into a setup or teardown method to reduce redundancy and improve maintainability.Also applies to: 528-528, 615-615, 865-865, 2071-2071, 2153-2153
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- cvat/apps/dataset_manager/tests/test_rest_api_formats.py (9 hunks)
- cvat/apps/dataset_manager/util.py (2 hunks)
Additional Context Used
Learnings (2)
cvat/apps/dataset_manager/util.py (1)
User: zhiltsov-max PR: cvat-ai/cvat#7864 File: cvat/apps/dataset_manager/util.py:149-157 Timestamp: 2024-05-27T14:31:27.282Z Learning: zhiltsov-max prefers error messages in the `get_export_cache_dir` function to remain less detailed to facilitate manual investigation.cvat/apps/dataset_manager/tests/test_rest_api_formats.py (2)
User: zhiltsov-max PR: cvat-ai/cvat#7864 File: cvat/apps/dataset_manager/tests/test_rest_api_formats.py:2061-2061 Timestamp: 2024-05-27T14:34:32.440Z Learning: The project uses a specific naming convention for functions and methods, which should be adhered to in suggestions and comments.User: zhiltsov-max PR: cvat-ai/cvat#7864 File: cvat/apps/dataset_manager/tests/test_rest_api_formats.py:1266-1266 Timestamp: 2024-05-17T12:20:07.528Z Learning: The CVAT project prefers using the `attrs` module over `dataclasses` due to its additional features and is commonly used across the project.
Additional comments not posted (14)
cvat/apps/dataset_manager/util.py (2)
149-157: Review error handling inget_export_cache_dir.The error handling in
get_export_cache_diris appropriate as it raises aFileNotFoundErrorwhen the base directory does not exist. This is crucial for debugging and ensuring that the file operations are performed on valid directories.
106-144: Review the use of locking mechanism.The implementation of
get_export_cache_lockusesRedlockfor distributed locking, which is generally a robust choice. However, consider the following:
- Verify that the
auto_release_timeset byttlis appropriate for the operations being locked.- Ensure that the lock acquisition logic (
blockandacquire_timeout) is tested under high concurrency to prevent deadlocks or long wait times.cvat/apps/dataset_manager/tests/test_rest_api_formats.py (12)
1267-1267: TheExportBehaviorTestclass effectively handles concurrency tests.This class is well-structured to test various concurrency issues related to dataset export and cleanup.
1268-1268: Usage ofattrsoverdataclassesis consistent with project standards.Good adherence to project conventions by using
attrsfor defining data classes.
1272-1272: Inheritance fromSharedBaseis correctly implemented inSharedBoolandSharedString.The inheritance hierarchy is clear and correctly utilizes shared functionality from
SharedBase.Also applies to: 1284-1284
1298-1298: Custom exception_LockTimeoutErroris well-defined for specific timeout errors.This custom exception helps in handling specific timeout scenarios clearly within the tests.
1301-1301: Method overloading forset_conditionis correctly used to handle different types.The use of method overloading here allows for flexible setting of conditions based on variable type, enhancing code robustness.
Also applies to: 1305-1305
1326-1326: Utility functionsside_effectandchain_side_effectsare well-implemented for test flexibility.These functions provide a robust way to handle side effects in a controlled manner during testing, which is crucial for accurate concurrency testing.
Also applies to: 1339-1339
1358-1358: Theprocess_closingcontext manager is a critical addition for safely handling process termination.This method ensures that processes are properly closed after tests, preventing potential resource leaks.
1370-1370: Thetest_concurrent_export_and_cleanupmethod is comprehensive in testing race conditions.This method effectively simulates and tests potential race conditions between export and cleanup operations, ensuring robustness in concurrent environments.
1593-1593: Thetest_concurrent_download_and_cleanupmethod thoroughly tests file handling under concurrency.This test method is well-designed to ensure that file download and cleanup operations handle concurrency correctly without data loss or corruption.
1794-1794: Thetest_export_can_create_file_and_cleanup_jobmethod correctly tests the export functionality under normal and edge conditions.This method ensures that the export functionality is robust, handling both file creation and subsequent cleanup effectively.
1881-1881: Thetest_cleanup_can_remove_filemethod effectively tests the cleanup functionality of the export system.This method ensures that files are properly removed after their intended lifecycle, preventing unnecessary storage usage.
99-99: Consider renaming_DbTestBasetoDatabaseTestBasefor improved readability.- class _DbTestBase(ApiTestBase): + class DatabaseTestBase(ApiTestBase):Likely invalid or redundant comment.
There was a problem hiding this comment.
Actionable comments posted: 3
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (5)
- cvat/apps/dataset_manager/tests/test_rest_api_formats.py (9 hunks)
- cvat/apps/dataset_manager/util.py (2 hunks)
- cvat/requirements/base.txt (13 hunks)
- cvat/requirements/development.txt (3 hunks)
- cvat/requirements/testing.txt (1 hunks)
Files skipped from review due to trivial changes (1)
- cvat/requirements/development.txt
Additional Context Used
Learnings (2)
cvat/apps/dataset_manager/util.py (1)
User: zhiltsov-max PR: cvat-ai/cvat#7864 File: cvat/apps/dataset_manager/util.py:149-157 Timestamp: 2024-05-27T14:31:27.282Z Learning: zhiltsov-max prefers error messages in the `get_export_cache_dir` function to remain less detailed to facilitate manual investigation.cvat/apps/dataset_manager/tests/test_rest_api_formats.py (2)
User: zhiltsov-max PR: cvat-ai/cvat#7864 File: cvat/apps/dataset_manager/tests/test_rest_api_formats.py:2061-2061 Timestamp: 2024-05-27T14:34:32.440Z Learning: The project uses a specific naming convention for functions and methods, which should be adhered to in suggestions and comments.User: zhiltsov-max PR: cvat-ai/cvat#7864 File: cvat/apps/dataset_manager/tests/test_rest_api_formats.py:1266-1266 Timestamp: 2024-05-17T12:20:07.528Z Learning: The CVAT project prefers using the `attrs` module over `dataclasses` due to its additional features and is commonly used across the project.
Additional comments not posted (13)
cvat/requirements/testing.txt (3)
9-9: Updated coverage tool version to 7.2.3.
12-14: Updated fakeredis and its dependency lupa for testing environments.
14-14: Updated sortedcontainers to maintain compatibility with the new fakeredis version.cvat/requirements/base.txt (3)
72-72: Updated Django to 4.2.13, ensuring the application is up-to-date with the latest security patches and bug fixes.
103-103: Updated django-health-check to 3.18.2 for improved health checking capabilities.
213-214: Added new dependency, pottery, for enhanced Redis functionalities, likely related to the new locking mechanisms.cvat/apps/dataset_manager/util.py (3)
105-144: The implementation of Redis locking using the Redlock algorithm inget_export_cache_lockis correctly handled. Good use of context management and error handling.
149-157: The functionget_export_cache_dircorrectly checks for the existence of the directory and raises an appropriate error if not found. This aligns with your preference for less detailed error messages to facilitate manual investigation.
190-231: The functionparse_export_file_patheffectively uses regular expressions to parse the export file path and extract necessary metadata, with robust error handling for parsing failures.cvat/apps/dataset_manager/tests/test_rest_api_formats.py (4)
1268-1268: Usingattrsfor defining data classes is consistent with project standards. Good use of the library's features for multiprocessing compatibility.
1794-1794: The methodtest_concurrent_download_and_cleanupeffectively tests TOCTOU issues with file handling. It's a critical test that ensures the robustness of file operations under concurrency.
99-99: Consider renaming_DbTestBasetoDatabaseTestBasefor better readability and adherence to Python naming conventions.- class _DbTestBase(ApiTestBase): + class DatabaseTestBase(ApiTestBase):Skipped due to learnings
User: zhiltsov-max PR: cvat-ai/cvat#7864 File: cvat/apps/dataset_manager/tests/test_rest_api_formats.py:2061-2061 Timestamp: 2024-05-27T14:34:32.440Z Learning: The project uses a specific naming convention for functions and methods, which should be adhered to in suggestions and comments.
2024-2024: TheProjectDumpUploadclass tests are comprehensive and cover a wide range of scenarios. Ensure that all new formats added to the system are included in these tests to maintain coverage.
SpecLad
approved these changes
May 28, 2024
There was a problem hiding this comment.
Actionable comments posted: 0
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- utils/dataset_manifest/requirements.txt (2 hunks)
Files skipped from review due to trivial changes (1)
- utils/dataset_manifest/requirements.txt
Merged
This was referenced Jun 17, 2024
2 tasks
SpecLad
pushed a commit
that referenced
this pull request
Jun 19, 2024
In #7864 the cache cleanup function was updated. The function was not supposed to be called for anything except datasets, but it was called for backups and events. This PR changes these clients to use their own functions. - Fixed `ValueError: Couldn't parse filename components in 'c71eba87-0914-4ccb-b883-a1bf1612fbf8.csv'` errors
Merged
zhiltsov-max
added a commit
that referenced
this pull request
Jun 25, 2024
<!-- Raise an issue to propose your change (https://github.com/cvat-ai/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context <!-- Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. Describe your changes in detail, add screenshots. --> In #7864, a new file naming scheme was introduced for dataset cache entries, while the old naming convention was deprecated. The old names were parsed incorrectly, leading to failing cache cleanup attempts in `clear_export_cache()`. A test was added in that PR, but it didn't reproduce the old behavior at the full extent. - Fixed old dataset filename parsing (`ValueError: Couldn't parse filename components in 'annotations_cvat-for-images-11.ZIP` errors) - Fixed the test for old cache cleanup ### How has this been tested? <!-- Please describe in detail how you tested your changes. Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> Unit tests. ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [ ] I submit my changes into the `develop` branch - [ ] I have created a changelog fragment <!-- see top comment in CHANGELOG.md --> - [ ] I have updated the documentation accordingly - [ ] I have added tests to cover my changes - [ ] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)) - [ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning)) ### License - [ ] I submit _my code changes_ under the same [MIT License]( https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved handling of export file paths to correctly extract `instance_timestamp` and `format_tag` in dataset exports. - Updated test cases to check for the correct export paths, ensuring robust verification of export functionalities. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Boris Sekachev <boris.sekachev@yandex.ru>
Merged
fahadb-kt
added a commit
to fahadb-kt/cvat
that referenced
this pull request
Aug 21, 2024
* Move rego files into their respective apps (#7806)
This is the promised sequel to #7734. After this change, the `iam` app
will no longer contain any code specific to other apps.
To make this work, the `/api/auth/rules` endpoint will now construct the
OPA bundle from a set of paths, which will be populated by
`load_app_permissions`.
Move OPA test files accordingly. Fortunately, `opa test` accepts
multiple directories, so it is trivial to adapt the testing
instructions.
Make the necessary adaptations to `generate_tests.py` to search for test
generators in every app. The original parameters of `generate_tests.py`
don't really make sense when there are multiple `rules` directory, so
remove them.
Instead, add a new `--apps-dir` parameter. This parameter isn't really
needed to test the open source version of CVAT, but I expect it to be
useful for testing the Enterprise version.
In addition, add some safety checks to `generate_tests.py`:
* Make sure that we find at least one test generator.
* Propagate exceptions from `call_generator` into the main thread.
### How has this been tested?
I tested the updated commands from the documentation manually, and
examined the rules bundle returned by `/api/auth/rules` to ensure that
it still contains all the `.rego` files.
* Fixed incorrect Cloud Storage request by ID (#7823)
* Opening update CS page sends infinite requests when CS id does not exist (#7828)
=
* Fixed duration of 'change:frame' event (#7817)
* Save video if test failed (#7807)
* Modernize Rego syntax (#7824)
Open Policy Agent v0.59 introduced a new directive (`import rego.v1`)
that ensures that the file is compatible with OPA v1 (to be released in
the future).
Add this directive to all Rego files and update the syntax accordingly.
Which involves the following:
* Rewrite all rules to use the `if` keyword, which is now mandatory.
* Where appropriate, use the `in` keyword, which is now available
without a future import. It's not mandatory, but it looks much nicer.
In addition, update Regal to the latest version, which now enforces the
use of `import rego.v1` by default.
* Optimized analytics requests to ClickHouse (#7804)
* Update the Nuclio version (#7787)
Old verison of Nuclio has some vulnerabilities and it needs to be
updated. Function dependencies have also been updated.
The `mask_rcnn` function has been removed because `mask_rcnn` using
python 3.6. In new version of Nuclio python3.6 is no longer supported.
Nuclio officially recommends using python3.9. Running `mask_rcnn` on
python3.9 causes errors within the function and package conflicts.
* Fixed: Cannot read properties of undefined (reading 'addClass') (#7834)
* fix[security]: Disable nginx server signature by default (#7814)
* Enhanced uploading files with tus protocol, enabled retries (#7830)
* Fixed exception when copy/paste a skeleton point (#7843)
* Added ability to call analytics report manually (#7805)
* Use CPU PyTorch for testing the SDK (#7825)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
Not only is the GPU version of PyTorch much bigger than the CPU version,
but it also pulls in CUDA, which is enormous. We don't (and can't) use
any GPU features in our tests, so we don't need the GPU version.
Using the CPU version saves ~4GB of disk space, which is a lot, because
the standard GitHub runners only offer 14 GB.
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- ~~[ ] I have created a changelog fragment~~ <!-- see top comment in
CHANGELOG.md -->
- ~~[ ] I have updated the documentation accordingly~~
- ~~[ ] I have added tests to cover my changes~~
- ~~[ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))~~
- ~~[ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))~~
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Enhanced the installation process by adding an extra index URL for
PyTorch CPU wheels to improve SDK setup reliability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Update server dependencies (#7845)
* Bump tqdm from 4.60.0 to 4.66.3 in /utils/dicom_converter (#7848)
* Do not allow to remove latest keyframe from UI (#7844)
* Optimized requests to analytics DB, using timestamps, to avoid going trough the whole table (#7833)
* Fix task creation with video file when there are no valid keyframes (#7838)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Fixed an issue where task creation from videos without valid keyframes
could cause errors.
- **New Features**
- Enhanced video stream handling to support videos without keyframes.
- Improved manifest management with new checks for empty states and
better index handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* [GSoC2024] Added feature to show tags corresponding to GT job and manual job in a separate row (#7774)
Fixes #7773 and #7749
Added feature to show tags corresponding to GT job and manual job in a
separate row. Along with the tags of the GT job have a mark of '(GT)' in
them.
### How has this been tested?
When we want to see both manual annotations and GT annotations:
<img width="1217" alt="image loading..."
src="https://github.com/cvat-ai/cvat/assets/72168180/362a1728-24f3-43cb-ac4d-1571ebc5faaf">
When we only want to see the annotations for the manual annotations job:
<img width="1217" alt="image loading..."
src="https://github.com/cvat-ai/cvat/assets/72168180/443fbf56-cd86-404b-bd6d-28351738dddf">
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
~- [ ] I have updated the documentation accordingly~
~- [ ] I have added tests to cover my changes~
- [x] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
~- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))~
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced display tags for Ground Truth (GT) and manual jobs in a
separate row, with GT tags marked for easy identification.
- Enhanced tag highlighting in the annotation interface to better
indicate conflicts.
- **Style**
- Implemented new styles for frame tags to improve visual distinction
when highlighted.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Kirill Lakhov <kirill.9992@gmail.com>
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
* Fixed vertical polylines difficult to select (#7860)
* Make `generate_tests.py` work with relative `--apps-dir` values (#7851)
In #7806 I goofed and made the `--apps-dir` option work only with
absolute paths. This patch fixes that.
* Fixed cannot read property 'annotations' of null (#7857)
* [GSoC2024] Added quality reporting for Tag annotations (#7582)
Fixes #7424
This PR adds quality computations for Tag annotations.
* Avoid fetching a list of shapes/tags from db, optimized fetching tracks (#7852)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [x] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Updated the method for counting objects in analytics reports to
improve accuracy.
- Made internal methods for initializing tags, shapes, and tracks
publicly accessible, enhancing external usability.
- **Bug Fixes**
- Fixed import paths for better module integration and reliability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Prevent losing tracked attributes when moving to a project (#7863)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Improved object tracking by adding a new model
`TrackedShapeAttributeVal` for enhanced performance and accuracy.
- Resolved issue of lost tracked attribute values when moving tasks to
projects.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Prepare release v2.13.0
* Update develop after v2.13.0
* Remove tasks by projectId from state after deleting project (#7854)
* helm-chart: prevent Traefik from ignoring the backend ingress rule (#7859)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
There is a condition that may occur during Kubernetes deployment, where
the frontend service already has an endpoint (i.e. the frontend pod),
but the backend service does not. For example, the backend pod may not
have started yet or the service controller may not have had time to
react to the backend pod.
In this case, when Traefik serves a request with an `/api/...` path, it
will see that it matches the `/api` rule, but since the corresponding
service has no endpoints, it will _skip_ that rule and try other rules.
And since the `/` rule matches everything, it will then route the
request to the frontend.
This is confusing and unhelpful, and more importantly, it makes health
checks return the wrong result. Since the frontend will serve
`index.html` to every request, a request to `/api/server/health/` or
`/api/server/about` will return a 200 code, even though the server isn't
actually up.
Because of this bug, I have observed weird failures in the Helm
workflow, where the "Wait for CVAT to be ready" step completes, but CVAT
is not actually ready. (FYI: The failures I've seen are actually in a
private repo, but the failure condition could occur in this repo too.
It's just more likely in a private repo, because GitHub uses smaller
runners in private repos.)
The fix is simple: use the `allowEmptyServices` Traefik setting, which
disables the rule skipping behavior. With this setting on, Traefik will
return a 503 response for backend URLs until the backend service gains
an endpoint.
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
I deployed the Helm chart, then ran a `kubectl delete deployments.apps
cvat-backend-server` to simulate the server being unavailable. Then I
curled the `/api/server/health/` endpoint.
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- ~~[ ] I have updated the documentation accordingly~~
- ~~[ ] I have added tests to cover my changes~~
- ~~[ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))~~
- ~~[ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))~~
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Fixed an issue to prevent incorrect 200 OK responses from API
endpoints before backend readiness.
- **New Features**
- Updated Helm chart to support configurations that allow empty services
in the Kubernetes Ingress provider.
- **Documentation**
- Updated version in Helm chart documentation from `0.12.0` to `0.12.1`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Fixed calculation of metrics for analytics reports (#7144)
* Check UI does not crash if to activate an object while frame fetching (#7873)
* Fix creating chunks with original quality from png images (#7899)
* Update helm (#7894)
Added ability to specify ServiceAccount for backend pods
Removed passing of DJANGO_MODWSGI_EXTRA_ARGS env variable to server pod
Do not set database host and port env variables if they are empty
* fixed server for duplicate attribute names (#7890)
* Fixed object count in analytics for skeletons and tracks (#7883)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Corrected an issue where analytics reported an incorrect count of
objects for skeleton tracks/shapes.
- Fixed a bug where the analytic report consistently showed one less
object for tracks than the actual count.
- **Improvements**
- Enhanced filtering logic for shapes and tracks in analytics, improving
the accuracy of annotation speed metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Fix CI-nightly tests and refactoring cypress config (#7908)
* Fixed analytics report: working time rounding to minimal 1 hour is not applied to annotation speed anymore (#7898)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
Depends on #https://github.com/cvat-ai/cvat/pull/7883
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Corrected an issue where analytic reports showed an incorrect count of
objects for skeleton tracks and shapes.
- **Improvements**
- Renamed metrics related to annotation speed from total to average for
jobs, tasks, and projects.
- Updated descriptions for annotation speed metrics to specify the
number of objects per hour.
- Removed unnecessary clamping function for working time statistics.
These changes enhance the accuracy and clarity of analytic reports,
providing more meaningful insights into annotation speeds and object
counts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Fixed exception: Cannot read properties of undefined (reading 'onBloc… (#7913)
* Fixed one way to create an empty mask (#7915)
* check creating task with video without valid keyframes
* fix before commands
* remove extra check
* Fixed updating job/task status after changing job state (#7901)
* Array.toReversed replaced by Array.reduceRight because of better comp… (#7916)
* [GSoC2024] Added additional security headers (#7752)
Added security headers for Referrer-Policy, X-Content-Type-Options.
Referring to Issue https://github.com/cvat-ai/cvat/issues/7398, Added
additional security headers. Added to address the deduction in security
score rating third party scanners.
- Referrer-Policy "strict-origin-when-cross-origin";: Limit the referrer
information sent when a user navigates away from the website
- X-Content-Type-Options "nosniff";: Prevent browsers from attempting to
MIME-sniff the content type of a response to reduce risk of XSS and
Content Injection
Co-authored-by: Roman Donchenko <rdonchen@outlook.com>
* Fixed skeleton selection algorithm (#7921)
* add rest api test
* remove extra video file
* remove unused task
* fix video file path
* Ignore ground truth jobs when compute analytics report for a task/project (#7919)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
* Prepare release v2.14.0
* Update develop after v2.14.0
* replace test video
* formatted code
* add copy videos folder
* Update cypress version (#7929)
* Upgrade React and Antd till the latest version (#7466)
* Fixed conflicts highlight crash in case of hidden by `zOrder` objects (#7917)
* Fixed couple of not stable Cypress tests (#7937)
* Fix missing serviceName field in kvrocks (issue #7741) (#7924)
Add the serviceName field to the kvrocks StatefulSet as per the
Kubernetes specification. This change ensures that the service name is
correctly associated with the StatefulSet pods, allowing for proper DNS
resolution and service discovery within the cluster.
Fixes #7741
### Motivation and context
The Helm installation is currently failing as reported in issue #7741
### How has this been tested?
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Resolved the issue of a missing `serviceName` field in `kvrocks`,
ensuring proper configuration and improved stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Fix login when email domain contains capital symbols and user was created after invitation to some org (#7906)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved email creation process to ensure the use of the normalized
email from the user object, enhancing data consistency and reducing
errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Fixed .ant-modal-wrapper kept after closing saving modal (#7948)
* use other method to get path
* Fix dataset downloading (#7864)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
This PR addresses several problems:
- when requesting a dataset download, it's possible to get the 500 error
with the message "The result file does not exist in export cache", which
isn't expected for this request
- when downloading the dataset the same error can be obtained if the
file is requested right before the cache expiration
- there are several
[TOCTOU](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use)
bugs related to dataset cache file existence checks
- under some conditions, it's possible that the export job is never
started
- the finished RQ jobs were removed automatically on result reporting
(after the client requested the result). This made it hard to debug
problems for admins, as the jobs were often removed already
This PR fixes the problems by the following:
- introduced dataset cache file locking (via redis) during reading,
writing, and removal
- the 500 error is changed to automatic reexporting attempt on export
status request
- the 500 error is changed to 404 when the file is not available for
downloading
- the exported files are now have different names for each instance
update time
- the lifetime of the exported files is now automatically prolonged on
each export request for the file (given the export is still valid)
- the deferred export jobs are now checked to have ghost dependencies.
If so, the requested job is restarted
- added several environment variables for configuration
- <s>finished RQ export jobs are not removed automatically on result
retrieval. Now, they just use the export cache lifetime instead (should
be continued in another PR)</s>
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Improved reliability of file handling during export and cleanup
processes.
- Introduced new functionality for managing export cache locks and
directories.
- **Bug Fixes**
- Addressed race conditions in concurrent export and cleanup operations.
- **Dependencies**
- Updated multiple packages to their latest versions for enhanced
security and performance:
- `cryptography` to `42.0.7`
- `django` to `4.2.13`
- `django-health-check` to `3.18.2`
- `freezegun` to `1.5.1`
- `jinja2` to `3.1.4`
- `limits` to `3.12.0`
- `lxml` to `5.2.2`
- `orjson` to `3.10.3`
- Added `pottery` version `3.0.0`
- Updated `tqdm` to `4.66.4`
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Fixed working time lost in click:element events (#7942)
* Aborted enabling black linter onsave in vscode (#7956)
* Delete extra comma (#7957)
* Fix a non-deterministic webhook test (#7952)
`test_two_project_webhooks_intersection` is supposed to trigger each
webhook once. However, the first one of these webhooks actually gets
triggered twice, because creating a task causes the project's
`updated_date` to be bumped, which triggers an `update:project` event.
The test still passes a lot of the time (I guess because the second
delivery doesn't appear immediately?), but sometimes it fails. It's very
easy to make it fail consistently, though - just add a `sleep(5)` before
the `get_deliveries` calls.
Fix this by changing the first webhook's second event to something that
will not be triggered.
* Improved `DatasetNotFound` error message (#7923)
The recent changes enhance the dataset import functionality across various dataset formats in the CVAT application by integrating specific importers from the Datumaro library. The updates streamline the detection of datasets, improve error handling, and introduce new tests to ensure robustness against incorrect file structures during import operations.
* Fix automatic `tag` annotation support (#7839)
* Update packages with vulnerability (#7951)
* Cannot set properties of undefined (setting 'serverID') (#7949)
* Fixed some deprecation warnings (#7970)
* Added license information regarding '/serverless' directory (#7967)
* Stabilized the cypress test for fix CI-nightly runs (#7966)
* Squashed `zoom:image` and `send:exception` client events (#7953)
* Fix memory consumption when exporting to azure blob storage (#7960)
Fix memory consumption when exporting to azure blob storage
* Fixed several issues related to creating tasks with cloud data (#7969)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
@coderabbitai summary
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [x] I have added tests to cover my changes (*partially*)
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Improved media download performance with parallel downloading.
- Enhanced file handling with the new `NamedBytesIO` class.
- Added support for specifying stop frames in task manifest generation.
- Enhanced `DatasetImagesReader` to handle generator sources.
- **Performance Improvements**
- Optimized image download methods to use threading for faster
processing.
- **Configuration**
- Introduced new settings for maximum threads and files per thread in
cloud data downloading.
These updates enhance the flexibility, performance, and configurability
of media handling and downloading in the application.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Using dedicated event to store working time (#7958)
- Parsing JSON payloads to get `working_time` in general leads to low
performance in Clickhouse requests. This patch will not fix it right
now, but with this patch, after a period of time we may switch to new
quick approach to calculate working time.
- There will not be a lot of `send:working_time` events, we may store
this scope of events for a longer time (e.g. 5 years instead of one by
default).
- Finally storing working time in such events like `click:element` or
`send:exception`, or `debug:info` seems not logical.
- Also, the history showed, that as result in different bugs, these
events may sometime lose information about `job_id`, `task_id`, etc.
Resolved #7884
* Update README.md (#7980)
* Check non-existent cloud storage update page (#7972)
* Annotation interface documentation updated (#7947)
* Bump requests from 2.31.0 to 2.32.2 in /tests/python (#7954)
* Updated icon (#7981)
* Fixed layout on create cloud storage page (#7985)
* Prepare release v2.14.1
* Update develop after v2.14.1
* Fixed: Queued jobs are not considered in deferring logic (#7907)
* Stabilized the cypress test for fix CI-nightly runs 2 (#7971)
* Update datumaro format description (#7992)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
Skeletons are not supported in this format
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Documentation**
- Updated the `Datumaro 1.0` format to support `Tags` instead of
`Tracks`.
- Expanded documentation to include support for additional annotation
types like Polylines, Masks, Points, Cuboids, and Tags in both export
and import operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Fixed ImageBitmap memory leak (#7995)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
Resolved #7909
Resolved #7850
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [x] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Addressed a significant memory leak issue by ensuring `ImageBitmap`
objects are properly closed after use.
- Updated various components to handle cleanup and termination of
workers and instances correctly, preventing potential resource leaks.
- **Version Updates**
- Updated `cvat-canvas` to version 2.20.3.
- Updated `cvat-core` to version 15.0.6.
- Updated `cvat-data` to version 2.1.0.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Updated documentation (one item was missing in the list of events saving triggers) (#8001)
* Prepare release v2.14.2
* Update develop after v2.14.2
* Rename kvrocks port (#8010)
Fix connection error issue in case of istio usage:
https://istio.io/v1.0/docs/setup/kubernetes/spec-requirements/#:~:text=Named%20ports%3A%20Service%20ports%20must,but%20name%3A%20http2foo%20is%20not.
* Fixed login with token without next parameter (#7999)
* Increased server health check timeout (#7993)
* Fixed: Cannot read properties of null (reading 'draw') (#7997)
* Remove unnecessary fields from the `/api/lambda/functions` response (#8004)
Remove several fields that haven't been used for one reason or another:
* `labels` and `attributes` have been replaced by `labels_v2`. Keeping
them around nearly triples the response length.
* `framework` hasn't been used by the UI since #5635, and IMO was never
useful to begin with. There are no decisions that the UI can take based
on this field, so it's essentially just a freeform text field, and we
already have a freeform text field - `description`. (Which... the UI
doesn't display either. But it could!)
Remove the `framework` field from the function descriptions as well,
since it has no other purpose.
* `state` has, as far I could determine, never been used by the UI. I
could see a field like this potentially being useful (e.g. the UI could
still display a function, but prevent it from being used if it's
unavailable), but since none of that is implemented right now, I see no
reason to have this field in the API.
* Fixed exception: State cannot be updated during editing, need to finish current editing first (#8019)
* Check creating cloud storage without manifest file (#7984)
* Number of Org Members (#8015)
Updated number of members
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Documentation**
- Updated details for the Solo and Team plans on CVAT.ai:
- Solo Plan: Adjusted the number of members allowed from "up to 3
members" to "up to 2 members".
- Team Plan: Adjusted the number of members required to pay for from "4
seats (3 annotators + 1 organization owner)" to "3 seats (2 annotators +
1 organization owner)".
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Merge pull request from GHSA-q684-4jjh-83g6
S3 storages support user-specified endpoint URLs, and Azure storages support
user-specified connection strings (which can contain endpoint URLs), so they
are susceptible to SSRF. Make S3 and Azure requests go through smokescreen
to fix this.
AFAIK, there is no way to configure a custom URL for Google Cloud storages,
so those aren't vulnerable.
Co-authored-by: Nikita Manovich <nikita@cvat.ai>
* Merge pull request from GHSA-jpf9-646h-4px7
* Mitigate a CSRF vulnerability in export and backup-related endpoints
While Django has built-in CSRF protection (which we use), it does not cover
GET requests, and AFAICS, there is no way to force it to do that.
Unfortunately, the many endpoints that initiate dataset exports and backups
do accept GET requests _and_ initiate side effects, making them susceptible.
The proper fix for this issue would be to redesign those endpoints to use
POST requests, but a) that's more complicated, and b) we should still keep
the old endpoints for backwards compatibility.
So apply a less proper fix, which is to disable session authentication for
the affected endpoints. It's a bit complex, because in some cases
(particularly when `action=download`) we _need_ session authentication to
work, because the UI redirects the user to such endpoints.
In addition, modify the handling logic for these endpoints in order to
ensure that when `action=download`, no side effects are triggered.
Previously, `action=download` would still queue an RQ job if none existed.
Even after this, `action=download` will still have two small side effects:
* An existing RQ job will be deleted if its results are out of date.
I don't think this is a problem, because such a job cannot be used anyway.
* A completed RQ job will be deleted too. This is a problematic design,
but I don't think an attacker can achieve anything by exploiting this. If
an attacker maliciously redirects the user to an `action=download` URL,
then they'll just download the export/backup as usual.
Some tests were making export requests incorrectly, so fix them.
* Add test for the CSRF workaround
* Prepare release v2.14.3
* Update develop after v2.14.3
* Remove `ModelKind.CLASSIFIER` (#8011)
I'd like the "kind" field in the API to identify the function's
"signature", or the types of values it receives as input and produces as
output. Classifiers have the same signature as detectors, so
`classifier` is a redundant value.
Besides improving semantic purity, removing this redundant value
simplifies the UI code.
The only meaningful difference between how the UI handles classifiers,
as compared to detectors, is that it shows the word "classifier" in the
model modal, which can be helpful. But we can achieve the same thing by
examining the function's `return_type` field. This lets us give a
special label to segmentation functions, as well.
"classifier" can't actually be returned by `/api/lambda/functions`, but
it _can_ be returned by the RoboFlow/Hugging Face function API in CVAT
Enterprise. So we'll need a small compatibility shim to transform this
value to "detector" until I fix that API to stop returning it too.
* Change minio host server definition (#8032)
* Stop editing when n key pressed (#7922)
* Allowed editing in single shape annotation mode (#8017)
* Fix server cache cleanup for backups and events (#8040)
In #7864 the cache cleanup function was updated. The function was not
supposed to be called for anything except datasets, but it was called
for backups and events. This PR changes these clients to use their own
functions.
- Fixed `ValueError: Couldn't parse filename components in
'c71eba87-0914-4ccb-b883-a1bf1612fbf8.csv'` errors
* CVAT Architecture documentation update (#8031)
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.githu…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and context
This PR addresses several problems:
This PR fixes the problems by the following:
finished RQ export jobs are not removed automatically on result retrieval. Now, they just use the export cache lifetime instead (should be continued in another PR)How has this been tested?
Checklist
developbranch(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)
License
Feel free to contact the maintainers if that's a concern.
Summary by CodeRabbit
New Features
Bug Fixes
Dependencies
cryptographyto42.0.7djangoto4.2.13django-health-checkto3.18.2freezegunto1.5.1jinja2to3.1.4limitsto3.12.0lxmlto5.2.2orjsonto3.10.3potteryversion3.0.0tqdmto4.66.4