After #8611 and #8866, it's no longer necessary to apply the formatters
to each module separately. So don't!
In addition, update the black exclusion list in `pyproject.toml` to only
contain files that were _not_ listed in `format_python_code.sh`. This
way, formatting will automatically be enforced for all new files.
For a few of the files reformatting only creates a small diff, so don't
even bother excluding them - reformat them instead.
Also, delete a few `pyproject.toml` files that are now redundant.
PR introduces the following changes:
- [_Export datasets|backups_] Cache cleaning moved into a separate cron
job. A separate RQ job is no longer enqueued after each export process
- [_Export datasets|backups_] Temporary directory is created inside the
common CVAT tmp directory (`data/tmp/`) instead of
`project|task|job/id/tmp/export_cache/`
- [_Export datasets_] One tmp directory (instead of 2) is created during
the export process
- Added the `exportcachecleanup` management command to remove outdated
`project|task|job/id/tmp/export_cache/` directories
- Added a cron job to clean up the `data/tmp/` directory. The new
setting `TMP_FILE_OR_DIR_RETENTION_DAYS` is used to determine whether a
file or directory should be removed
- Locks are now used when working with backups as they are used for
export
Notes:
- `data/project|task|job/id/tmp/` is still used during uploading
annotations/datasets, but this should be fixed in a separate PR.
Breaking changes:
- Enqueuing RQ jobs to backup a project/task: A filename argument was
removed from the background function signature. Previously enqueued jobs
will fail.
- Cleaning cache after exporting backups or datasets: Scheduled RQ jobs
will fail due to `cvat.apps.dataset_manager.views.clear_export_cache`
was moved and `cvat.apps.engine.backup._clear_export_cache` was deleted.
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
---------
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
- Changed honeypot selection algorithm in task creation to be more
uniform
- Changed random honeypot reroll algorithm to produce/maintain a uniform
distribution of validation frames in the task
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced uniformity in validation frame distribution for honeypot
tasks.
- Improved selection process for validation frames, ensuring fair
allocation across jobs.
- **Bug Fixes**
- Refined error handling for validation frame selection and task
configurations.
- **Tests**
- Added new tests for validation frames and honeypot management.
- Updated existing tests to improve robustness and error handling
related to task annotations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Maria Khrustaleva <maria@cvat.ai>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
1. Minimize payload for create:tags, create:shapes, create:tracks,
update:tags, update:shapes, update:tracks, delete:tags, delete:shapes,
delete:tracks
2. sending update/create/delete for Memberships, Invitations, Webhooks
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Refined event logging by removing unnecessary fields from processed
data, enhancing clarity and efficiency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
- Changed chunk generation from per-task chunks to per-segment chunks
- Fixed a memory leak in video reading on the server side (only in
media_extractors, so there are several more left)
- Fixed a potential hang in `import` worker or the server process on
process shutdown
- Disabled multithreading in video reading in endpoints (not in static
chunk generation)
- Refactored static chunk generation code (moved after job creation)
- Refactored various server internal APIs for frame retrieval
- Updated UI logic to access chunks, added support for non-sequential
frames in chunks
- Added a new server configuration option `CVAT_ALLOW_STATIC_CACHE`
(boolean) to enable and disable static cache support. The option is
disabled by default (it's changed from the previous behavior)
- Added tests for the changes made
- Added missing original chunk type field in job responses
- Fixed invalid kvrocks cleanup in tests for Helm deployment
- Added a new 0-based `index` parameter in `GET
/api/jobs/{id}/data/?type=chunk` to simplify indexing
- GT job chunks with non-sequential frames have no placeholders inside
When this update is applied to the server, there will be a data storage
setting migration for the tasks. Existing tasks using static chunks
(`task.data.storage_method == FILE_SYSTEM`) will be switched to the
dynamic cache (i.e. to `== CACHE)`). The remaining files should be
removed manually, there will be a list of such tasks in the migration
log file.
After this update, you'll have an option to enable or disable static
cache use during task creation. This allows, in particular, prohibit new
tasks using the static cache. With this option, any tasks using static
cache will use the dynamic cache instead on data access.
User-observable changes:
- Job chunk ids now start from 0 for each job instead of using parent
task ids
- The `use_cache = false` or `storage_method = filesystem` parameters in
task creation can be ignored by the server
- Task chunk access may be slower for some chunks (particularly, for
tasks with overlap configured, for chunks on segment boundaries, and for
tasks previously using static chunks)
- The last chunk in a job will contain only the frames from the current
job, even if there are more frames in the task
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced a new server setting to disable media chunks on the local
filesystem.
- Enhanced frame prefetching with a `startFrame` parameter for improved
chunk calculations.
- Added a new property, `data_original_chunk_type`, for enhanced job
differentiation in the metadata.
- **Bug Fixes**
- Resolved memory management issues to prevent leaks during video
processing.
- Corrected naming inconsistencies related to the `prefetchAnalyzer`.
- **Documentation**
- Included configuration for code formatting tools to ensure consistent
code quality across the project.
- **Refactor**
- Restructured classes and methods for improved clarity and
maintainability, particularly in media handling and task processing.
- **Chores**
- Updated formatting scripts to include additional directories for
automated code formatting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR fixes the following issues:
- [export API v1] do not reinitialize dataset export process when
downloading a result file if a resource (project|task|job) has been
updated since the first initialized export request
- [export API v1] return `rq_id` for all requests with 202 status code
(not only for initialization requests)
- [requests API] Fixed filtering by format && added resource to allowed
filters
REST API tests updates:
- Added tests to check requests filtration using simple filters
- Added tests to check specific requests retrieving
- Updated all tests that export project|task|job
datasets|annotations|backups:
- to test both API versions (including API mixing)
- to use only appropriate resources by checking the default export
location
- Added fixtures to filter projects/tasks assets
- Updated default target|source buckets to `import/export` bucket to
exclude the same bucket usage as a data source in several tests (when
bucket content is used as task data) and as a bucket for results
## Summary by CodeRabbit
- **New Features**
- Enhanced job handling for exports, improving error management and job
state tracking.
- Introduced a new `resource` field in the request handling system to
improve data categorization.
- Added new filtering capabilities for API queries, allowing users to
filter by the `resource` field.
- **Bug Fixes**
- Improved status checks and handling for job requests.
- Introduced exception handling for forbidden access during project
backup attempts.
- **Tests**
- Refactored test suites to improve coverage and ensure compatibility
across versions with new methods and exception handling.
- New tests added to validate request handling functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
Decided to split changes in this PR:
https://github.com/cvat-ai/cvat/pull/8223
1. Annotations import (https://github.com/cvat-ai/cvat/pull/8226)
2. Array fields optimization (this PR)
3. Logging function optimization
(https://github.com/cvat-ai/cvat/pull/8228)
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced the `LazyList` class for efficient, on-demand parsing of
list elements from strings.
- Added support for custom transformations through a converter function.
- Enhanced lazy evaluation with new decorators for improved performance
on list operations.
- **Tests**
- Implemented a comprehensive test suite for the `LazyList` class,
validating core functionalities and ensuring robustness.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The server part of #6039
- Added support for Ground Truth jobs in a task
- Added support for job creation and removal (only Ground Truth jobs can
be created or removed in a task)
- Added a component to autocompute quality metrics for a task
- Added tests
- Fixed https://github.com/opencv/cvat/issues/5971 (both parts - the
outside problem and the manifest problem, the manifest part fix is also
available in #6216)
Co-authored-by: klakhov <kirill.9992@gmail.com>
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Co-authored-by: Kirill Sizov <kirill.sizov@cvat.ai>
Roman Donchenko