Now that we've dropped Python 3.9 support, we can get rid of these. Some
of these were actually deprecated in Python 3.9 too, but they must've
slipped through after the last cleanup round.
Done with:
ruff check --select=UP006,UP035 --target-version=py310 --fix
--unsafe-fixes
plus a bit of manual cleanup.
Looks like one PR made it through with the placeholder number. Make sure
this doesn't happen again.
The URL regex works with any org and repo names, so we could potentially
reuse this in private repositories.
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment
- [x] I submit _my code changes_ under the same [MIT License]
---------
Co-authored-by: Petr Iosipov <petr.iosipov@cvat.ai>
This is the final patch in the series that started with #5720. With
this, all linters will be run on all files, thus ensuring that a PR
can't silently add linter warnings in files that it doesn't touch
directly.
In addition, this 1) pins the bandit version, and 2) fixes all remaining
bandit warnings.
Co-authored-by: Oleg Valiulin <oleg.valiulin@cvat.ai>
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Typos spell-checker integration
Run `typos` to show typos in project
Run `typos -w` to autofix
Additionally, change the workflow to scan the entire codebase. A change
in one file can cause a pylint warning in another, so this is the only
way to avoid committing code with warnings.
I disabled several new warnings, since there were too many instances of
those to fix in one sittings. They can be fixed separately.
After #8611 and #8866, it's no longer necessary to apply the formatters
to each module separately. So don't!
In addition, update the black exclusion list in `pyproject.toml` to only
contain files that were _not_ listed in `format_python_code.sh`. This
way, formatting will automatically be enforced for all new files.
For a few of the files reformatting only creates a small diff, so don't
even bother excluding them - reformat them instead.
Also, delete a few `pyproject.toml` files that are now redundant.
The only reason to display separate UI and server version (IMO) is that
sometimes they can diverge due to deployment issues, and the information
can help with diagnostics. So it seems much more useful to use the same
version numbering as for the server, since:
* This makes it much easier to see whether a divergence exists.
* We can automate updates to it, eliminating annoying busywork for
developers.
* We don't need to show multiple versions for different UI components,
simplifying the code and the UI.
PR introduces the following changes:
- [_Export datasets|backups_] Cache cleaning moved into a separate cron
job. A separate RQ job is no longer enqueued after each export process
- [_Export datasets|backups_] Temporary directory is created inside the
common CVAT tmp directory (`data/tmp/`) instead of
`project|task|job/id/tmp/export_cache/`
- [_Export datasets_] One tmp directory (instead of 2) is created during
the export process
- Added the `exportcachecleanup` management command to remove outdated
`project|task|job/id/tmp/export_cache/` directories
- Added a cron job to clean up the `data/tmp/` directory. The new
setting `TMP_FILE_OR_DIR_RETENTION_DAYS` is used to determine whether a
file or directory should be removed
- Locks are now used when working with backups as they are used for
export
Notes:
- `data/project|task|job/id/tmp/` is still used during uploading
annotations/datasets, but this should be fixed in a separate PR.
Breaking changes:
- Enqueuing RQ jobs to backup a project/task: A filename argument was
removed from the background function signature. Previously enqueued jobs
will fail.
- Cleaning cache after exporting backups or datasets: Scheduled RQ jobs
will fail due to `cvat.apps.dataset_manager.views.clear_export_cache`
was moved and `cvat.apps.engine.backup._clear_export_cache` was deleted.
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
---------
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
isort has a `--resolve-all-configs` options that allows you to use
different configs for different subdirectories. With this option, we can
reformat/check the whole codebase with one command. So let's just do
that.
It's a little janky, because isort's first party module detection
doesn't seem to work properly when you run it from the project root and
not the appropriate subdirectory. So I had to patch that up with
explicit `known_first_party` settings. I also didn't feel like doing
that for all of the serverless functions, so I just added them to the
ignore list for now.
These are currently duplicated all over the codebase, often with
mismatched constraints. Put them all in one requirements file instead.
Fix a few minor issues while I'm at it:
* Constrain black to the current major version to avoid New Year
surprises (black can change styles between major versions). Constrain
isort too, just in case.
* Remove usages of `egrep`, which is deprecated (and unnecessary here).
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
- Changed honeypot selection algorithm in task creation to be more
uniform
- Changed random honeypot reroll algorithm to produce/maintain a uniform
distribution of validation frames in the task
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced uniformity in validation frame distribution for honeypot
tasks.
- Improved selection process for validation frames, ensuring fair
allocation across jobs.
- **Bug Fixes**
- Refined error handling for validation frame selection and task
configurations.
- **Tests**
- Added new tests for validation frames and honeypot management.
- Updated existing tests to improve robustness and error handling
related to task annotations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Maria Khrustaleva <maria@cvat.ai>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
1. Minimize payload for create:tags, create:shapes, create:tracks,
update:tags, update:shapes, update:tracks, delete:tags, delete:shapes,
delete:tracks
2. sending update/create/delete for Memberships, Invitations, Webhooks
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Refined event logging by removing unnecessary fields from processed
data, enhancing clarity and efficiency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This protects against developers accidentally erasing the dash in line 1
of the list item, as well as not indenting lines 2+.
Fix all such mistakes in the current changelog.
Make the Black workflow just run black with default settings. Instead of
including specific directories, exclude the ones that haven't been
blackened yet via `pyproject.toml`.
Leave a few small files out of the exclusion list; blacken them instead.
This is one step towards project-wide code formatting. It also makes it
easier to format your entire patch when you touch multiple components,
since you don't have to remember which directories to apply `black` to.
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
- Changed chunk generation from per-task chunks to per-segment chunks
- Fixed a memory leak in video reading on the server side (only in
media_extractors, so there are several more left)
- Fixed a potential hang in `import` worker or the server process on
process shutdown
- Disabled multithreading in video reading in endpoints (not in static
chunk generation)
- Refactored static chunk generation code (moved after job creation)
- Refactored various server internal APIs for frame retrieval
- Updated UI logic to access chunks, added support for non-sequential
frames in chunks
- Added a new server configuration option `CVAT_ALLOW_STATIC_CACHE`
(boolean) to enable and disable static cache support. The option is
disabled by default (it's changed from the previous behavior)
- Added tests for the changes made
- Added missing original chunk type field in job responses
- Fixed invalid kvrocks cleanup in tests for Helm deployment
- Added a new 0-based `index` parameter in `GET
/api/jobs/{id}/data/?type=chunk` to simplify indexing
- GT job chunks with non-sequential frames have no placeholders inside
When this update is applied to the server, there will be a data storage
setting migration for the tasks. Existing tasks using static chunks
(`task.data.storage_method == FILE_SYSTEM`) will be switched to the
dynamic cache (i.e. to `== CACHE)`). The remaining files should be
removed manually, there will be a list of such tasks in the migration
log file.
After this update, you'll have an option to enable or disable static
cache use during task creation. This allows, in particular, prohibit new
tasks using the static cache. With this option, any tasks using static
cache will use the dynamic cache instead on data access.
User-observable changes:
- Job chunk ids now start from 0 for each job instead of using parent
task ids
- The `use_cache = false` or `storage_method = filesystem` parameters in
task creation can be ignored by the server
- Task chunk access may be slower for some chunks (particularly, for
tasks with overlap configured, for chunks on segment boundaries, and for
tasks previously using static chunks)
- The last chunk in a job will contain only the frames from the current
job, even if there are more frames in the task
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced a new server setting to disable media chunks on the local
filesystem.
- Enhanced frame prefetching with a `startFrame` parameter for improved
chunk calculations.
- Added a new property, `data_original_chunk_type`, for enhanced job
differentiation in the metadata.
- **Bug Fixes**
- Resolved memory management issues to prevent leaks during video
processing.
- Corrected naming inconsistencies related to the `prefetchAnalyzer`.
- **Documentation**
- Included configuration for code formatting tools to ensure consistent
code quality across the project.
- **Refactor**
- Restructured classes and methods for improved clarity and
maintainability, particularly in media handling and task processing.
- **Chores**
- Updated formatting scripts to include additional directories for
automated code formatting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR fixes the following issues:
- [export API v1] do not reinitialize dataset export process when
downloading a result file if a resource (project|task|job) has been
updated since the first initialized export request
- [export API v1] return `rq_id` for all requests with 202 status code
(not only for initialization requests)
- [requests API] Fixed filtering by format && added resource to allowed
filters
REST API tests updates:
- Added tests to check requests filtration using simple filters
- Added tests to check specific requests retrieving
- Updated all tests that export project|task|job
datasets|annotations|backups:
- to test both API versions (including API mixing)
- to use only appropriate resources by checking the default export
location
- Added fixtures to filter projects/tasks assets
- Updated default target|source buckets to `import/export` bucket to
exclude the same bucket usage as a data source in several tests (when
bucket content is used as task data) and as a bucket for results
## Summary by CodeRabbit
- **New Features**
- Enhanced job handling for exports, improving error management and job
state tracking.
- Introduced a new `resource` field in the request handling system to
improve data categorization.
- Added new filtering capabilities for API queries, allowing users to
filter by the `resource` field.
- **Bug Fixes**
- Improved status checks and handling for job requests.
- Introduced exception handling for forbidden access during project
backup attempts.
- **Tests**
- Refactored test suites to improve coverage and ensure compatibility
across versions with new methods and exception handling.
- New tests added to validate request handling functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
Decided to split changes in this PR:
https://github.com/cvat-ai/cvat/pull/8223
1. Annotations import (https://github.com/cvat-ai/cvat/pull/8226)
2. Array fields optimization (this PR)
3. Logging function optimization
(https://github.com/cvat-ai/cvat/pull/8228)
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced the `LazyList` class for efficient, on-demand parsing of
list elements from strings.
- Added support for custom transformations through a converter function.
- Enhanced lazy evaluation with new decorators for improved performance
on list operations.
- **Tests**
- Implemented a comprehensive test suite for the `LazyList` class,
validating core functionalities and ensuring robustness.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This automates the second half of the release process (the first being
automated by `prepare-release.yml`). After this workflow completes, the
only action that should remain for the releaser to do is to merge the
`dev-release-*` pull request. We can't do that as part of this workflow,
because CI has to finish first, and it seems pointless to create another
workflow just to merge 1 PR.
Apply some of the aspects of this pipeline to `prepare-release.yml` as
well:
* Make the release notes extraction process more sophisticated to work
around GitHub's frustrating handling of line breaks in PR and release
descriptions.
* Use GitHub app credentials in order to be able to trigger other
pipelines.
<!-- Raise an issue to propose your change
(https://github.com/opencv/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution
guide](https://opencv.github.io/cvat/docs/contributing/). -->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
This'll help prevent simple mistakes that would cause scriv to assemble
the changelog incorrectly.
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
Manual testing.
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- ~~[ ] I have created a changelog fragment~~ <!-- see top comment in
CHANGELOG.md -->
- ~~[ ] I have updated the documentation accordingly~~
- ~~[ ] I have added tests to cover my changes~~
- ~~[ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))~~
- ~~[ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))~~
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
This automates the first half of the current release process. The second
half will be implemented by another workflow.
The reason why it can't all be done in a single workflow is that it's
useful to let developers inspect what'll go into the release before
actually publishing it, and to apply any last-minute fixes, if
necessary. It also allows CI to complete for the release PR.
To support the new workflow, add a `--set` option to `update_version.py`
that sets the version to a custom value.
The main motivation was to add SDK and schema-related version updates,
which were missing from the original script. I also did a bunch of
refactoring to make changing the replacement rules easier, and added a
checking mode to make sure that version numbers are consistent across
the project (adding a corresponding check to CI).
The server part of #6039
- Added support for Ground Truth jobs in a task
- Added support for job creation and removal (only Ground Truth jobs can
be created or removed in a task)
- Added a component to autocompute quality metrics for a task
- Added tests
- Fixed https://github.com/opencv/cvat/issues/5971 (both parts - the
outside problem and the manifest problem, the manifest part fix is also
available in #6216)
Co-authored-by: klakhov <kirill.9992@gmail.com>
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Co-authored-by: Kirill Sizov <kirill.sizov@cvat.ai>
Maxim Zhiltsov