Currently, all source files are located in `/home/django`, and owned by
`django`. This means that if there's any vulnerability that lets an
attacker overwrite files in the server, they can replace source files
with their own code, and potentially get that code executed. That's
pretty bad, so I want to harden against that.
Make all source files owned by root, and move them to `/opt/cvat`. Add a
`manage.py` symlink in `/home/django` for backwards compatibility. It
happens that if a script is a symlink, Python does not add the symlink's
directory to `sys.path`, which is great for us, since that lets us avoid
a writable directory on there.
Still, even though `/home/django/manage.py` is owned by root, an
attacker could potentially be able to delete it and replace it with
their own malicious file. To be a bit more safe, replace `~/manage.py`
calls in backend scripts with `django-admin`.
To make sure CVAT can still find the data directory, add a new
environment variable, `CVAT_BASE_DIR` and set it in the Docker image.
This also fixes a minor bug: we no longer override the `HOME`
environment variable in the `Dockerfile`, so now it's automatically set
by `Docker` depending on the current user.
It's possible that some tasks with videos have no manifest. There are
several reasons for that, including manual playback speed manipulation
attepts. CVAT will generate no manifest for such videos, but the
chapters functionality will try to read the chapters from the manifest
regardless. It's technically possible to ignore bad keyframes, but it
can lead to invalid chapters.
- Fixed task access for tasks with videos with bad keyframes
This is a helper method that pretty much just wraps the double loop over
the packets and the frames in a packet. Using it makes the code simpler
and enables better type inference in IDEs, because the PyAV type stubs
are sufficient to infer that `container.decode(video_stream)` returns an
`Iterable[VideoFrame]` (whereas `packet.decode()` is declared to return
`list[SubtitleSet]` for some reason).
Also, add some imports and type annotations to ensure that the IDEs have
types to infer _from_.
This is a continuation of #10060.
I also added a couple of `TypeAlias` annotations where appropriate, and
changed the import style in `test_users.py` to be more consistent with
other files.
Now that we've dropped Python 3.9 support, we can get rid of these. Some
of these were actually deprecated in Python 3.9 too, but they must've
slipped through after the last cleanup round.
Done with:
ruff check --select=UP006,UP035 --target-version=py310 --fix
--unsafe-fixes
plus a bit of manual cleanup.
**API changes:**
- `POST /api/consensus/merges?rq_id=rq_id` returns 410 status code, this
endpoint no longer supports process status checking
- `GET /api/projects/id/dataset?action=import_status` returns 410 status
code, this endpoint no longer supports process status checking
- `POST /api/projects/backup?rq_id=rq_id` returns 410 status code, this
endpoint no longer supports process status checking
- `POST /api/tasks/backup?rq_id=rq_id` returns 410 status code, this
endpoint no longer supports process status checking
- `PUT /api/tasks/id/annotations?rq_id=rq_id&format=format` returns 410
status code, this endpoint no longer supports process status checking
- `PUT /api/jobs/id/annotations?rq_id=rq_id&format=format` returns 410
status code, this endpoint no longer supports process status checking
- `GET /api/events` is deprecated in favor of the following API:
- [new] `POST /api/events/export` (returns 202 with Request ID)
- GET /api/requests/rq_id
- [new] `GET /api/events/download?rq_id=rq_id` (private endpoint, should
be used only as result_url)
- `POST /api/quality/reports/rq_id=rq_id` is deprecated in favor of `GET
/api/requests/rq_id`
**Architecture visible changes:**
- Cache files containing events (created after using the API to export
events as a file) are stored in `/data/cache/export/` instead of
`/data/tmp/`. The `_clear_export_cache` function is deleted, since cache
files are deleted one day after creation (by default) by the
`cleanup_export_cache_directory` cron job.
**SDK backward-incompatible changes:**
- [requests_api.list] action/target/subresource filters now have string
type
- Several types were removed:
`DatasetWriteRequest`/`BackupWriteRequest`/`TaskAnnotationsWriteRequest`;
`JobAnnotationsUpdateRequest`/`TaskAnnotationsUpdateRequest`
---------
Co-authored-by: Roman Donchenko <roman@cvat.ai>
This PR introduces the following changes:
- Several server API endpoints have been deprecated or no longer handle
the process of exporting specific resources:
- [API] **GET /api/projects/id/dataset?action=import_status** is
deprecated
- [API] **GET /api/projects/id/dataset**(no action parameter or action
!= import_status) returns 410 status code - API endpoint no longer used
to handle export datasets flow
- [API] **GET /api/projects/id/annotations** returns 410 status code -
API endpoint no longer used to handle export annotations flow
- [API] **GET /api/projects/id/backup** returns 410 status code - API
endpoint no longer used to handle export backups flow
- [API] **GET /api/tasks/id/backup** returns 410 status code - API
endpoint no longer used to handle export backups flow
- [API] **GET /api/tasks/id/annotations?format=<format>** returns 410
status code - API endpoint no longer used to handle export annotations
flow
- [API] **GET /api/tasks/d/dataset** returns 410 status code - API
endpoint no longer used to handle export datasets flow
- [API] **GET /api/jobs/id/annotations?format=<format>** returns 410
status code - API endpoint no longer used to handle export annotations
flow
- [API] **GET /api/jobs/id/dataset** returns 410 status code - API
endpoint no longer used to handle export datasets flow
- Introduced new "private" endpoints to download prepared files (they
are hidden in the generated server schema):
- [API] GET /api/projects/id/dataset/download?rq_id=rq_id
- [API] GET /api/projects/id/backup/download?rq_id=rq_id
- [API] GET /api/tasks/id/dataset/download?rq_id=rq_id
- [API] GET /api/tasks/id/backup/download?rq_id=rq_id
- [API] GET /api/jobs/id/dataset/download?rq_id=rq_id
- Changed permissions used when downloading a prepared file (now admin
and RQ job owner have rights to download a prepared file related to a
specific background job (it's a questionable topic whether should be
resource access checked also))
- Added result filename saving to RQ metadata for export-related RQ
jobs:
```
{
...
"result_filename": str // is used as the final filename when downloading files/uploading to cloud storage
}
```
- Final file name is taken from meta when uploading a file to cloud
storage (`key` and `key_pattern` args have been removed)
- Added support for consensus task and consensus job merging (API and
UI)
- Added simple consensus settings
- Added server tests
- Added new `consensus` RQ queue and worker
- Updated skeleton comparisons: hidden points now also contribute to the
skeleton similarity. Only visibility is taken into account for invisible
points
Limitations:
- Merging is supported for all annotations except 2d and 3d cuboids. 3d
tasks are not supported
- Annotation groups are not supported (each annotation is considered
separate in a group)
- Polygons and masks are not interchangeable (each type is compared only
with the same type)
Co-authored-by: Kirill Lakhov <kirill.lakhov@cvat.ai>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
- Changed chunk generation from per-task chunks to per-segment chunks
- Fixed a memory leak in video reading on the server side (only in
media_extractors, so there are several more left)
- Fixed a potential hang in `import` worker or the server process on
process shutdown
- Disabled multithreading in video reading in endpoints (not in static
chunk generation)
- Refactored static chunk generation code (moved after job creation)
- Refactored various server internal APIs for frame retrieval
- Updated UI logic to access chunks, added support for non-sequential
frames in chunks
- Added a new server configuration option `CVAT_ALLOW_STATIC_CACHE`
(boolean) to enable and disable static cache support. The option is
disabled by default (it's changed from the previous behavior)
- Added tests for the changes made
- Added missing original chunk type field in job responses
- Fixed invalid kvrocks cleanup in tests for Helm deployment
- Added a new 0-based `index` parameter in `GET
/api/jobs/{id}/data/?type=chunk` to simplify indexing
- GT job chunks with non-sequential frames have no placeholders inside
When this update is applied to the server, there will be a data storage
setting migration for the tasks. Existing tasks using static chunks
(`task.data.storage_method == FILE_SYSTEM`) will be switched to the
dynamic cache (i.e. to `== CACHE)`). The remaining files should be
removed manually, there will be a list of such tasks in the migration
log file.
After this update, you'll have an option to enable or disable static
cache use during task creation. This allows, in particular, prohibit new
tasks using the static cache. With this option, any tasks using static
cache will use the dynamic cache instead on data access.
User-observable changes:
- Job chunk ids now start from 0 for each job instead of using parent
task ids
- The `use_cache = false` or `storage_method = filesystem` parameters in
task creation can be ignored by the server
- Task chunk access may be slower for some chunks (particularly, for
tasks with overlap configured, for chunks on segment boundaries, and for
tasks previously using static chunks)
- The last chunk in a job will contain only the frames from the current
job, even if there are more frames in the task
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced a new server setting to disable media chunks on the local
filesystem.
- Enhanced frame prefetching with a `startFrame` parameter for improved
chunk calculations.
- Added a new property, `data_original_chunk_type`, for enhanced job
differentiation in the metadata.
- **Bug Fixes**
- Resolved memory management issues to prevent leaks during video
processing.
- Corrected naming inconsistencies related to the `prefetchAnalyzer`.
- **Documentation**
- Included configuration for code formatting tools to ensure consistent
code quality across the project.
- **Refactor**
- Restructured classes and methods for improved clarity and
maintainability, particularly in media handling and task processing.
- **Chores**
- Updated formatting scripts to include additional directories for
automated code formatting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Fixed exporting the same dataset or backup twice in a row using
high-level SDK (switched to new export API version) (related
https://github.com/cvat-ai/cvat/issues/8256)
- Fixed exporting a dataset or backup using high-level SDK when the
default project or task location refers to cloud storage
- Added ability to explicitly specify location when exporting datasets
and backups using high-level SDK
## Summary by CodeRabbit
- **New Features**
- Introduced mixins for exporting datasets and downloading backups,
enhancing functionality across multiple classes.
- Added a new fixture for testing tasks with specified target storage,
improving test coverage.
- **Bug Fixes**
- Improved error handling in the file download process to ensure
validity before proceeding.
- **Refactor**
- Restructured the downloading mechanism for better modularity and
maintainability.
- Removed outdated methods in favor of mixin functionality, streamlining
class design.
- **Tests**
- Enhanced the test suite with additional scenarios and flexibility for
task management and dataset downloading.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
This PR introduces new page with information about data
processing(status and progress). For now it will support: task creation,
import/export.
For previous discussions refer to: #7537
Co-authored-by: Maria Khrustaleva <maria@cvat.ai>
Co-authored-by: Boris Sekachev <sekachev.bs@gmail.com>
* Mitigate a CSRF vulnerability in export and backup-related endpoints
While Django has built-in CSRF protection (which we use), it does not cover
GET requests, and AFAICS, there is no way to force it to do that.
Unfortunately, the many endpoints that initiate dataset exports and backups
do accept GET requests _and_ initiate side effects, making them susceptible.
The proper fix for this issue would be to redesign those endpoints to use
POST requests, but a) that's more complicated, and b) we should still keep
the old endpoints for backwards compatibility.
So apply a less proper fix, which is to disable session authentication for
the affected endpoints. It's a bit complex, because in some cases
(particularly when `action=download`) we _need_ session authentication to
work, because the UI redirects the user to such endpoints.
In addition, modify the handling logic for these endpoints in order to
ensure that when `action=download`, no side effects are triggered.
Previously, `action=download` would still queue an RQ job if none existed.
Even after this, `action=download` will still have two small side effects:
* An existing RQ job will be deleted if its results are out of date.
I don't think this is a problem, because such a job cannot be used anyway.
* A completed RQ job will be deleted too. This is a problematic design,
but I don't think an attacker can achieve anything by exploiting this. If
an attacker maliciously redirects the user to an `action=download` URL,
then they'll just download the export/backup as usual.
Some tests were making export requests incorrectly, so fix them.
* Add test for the CSRF workaround
1. When a task has non-zero overlap and exactly as many frames as needed
to create 1 or more complete segments, the current algorithm generates a
redundant segment at the end. For example, if size is 5, segment size is
3, and overlap is 1, it generates segments (0, 2), (2, 4), and (4, 4).
The algorithm attempts to compensate for this, but it only works in the
case where the segment size is unspecified (and defaults to the total
size).
Update the algorithm to handle this correctly in the general case.
2. The algorithm selects a default overlap size of 5 if the media file
is a video. However, this might not be a valid value if the task has a
very small segment size. In this case, a range of undesirable behaviors
may occur, depending on the segment size:
* segments getting generated such that more than 2 segments cover a
single frame;
* task creation crashing with an exception;
* a task being created with no segments at all.
Fix this by clamping the default overlap size the same way as a
user-specified one.
Fixes#7675.
It doesn't make much sense to encode the image as JPEG if the given file
name is "image.png". Instead, let PIL select the format based on the
requested file name.
The server part of #6039
- Added support for Ground Truth jobs in a task
- Added support for job creation and removal (only Ground Truth jobs can
be created or removed in a task)
- Added a component to autocompute quality metrics for a task
- Added tests
- Fixed https://github.com/opencv/cvat/issues/5971 (both parts - the
outside problem and the manifest problem, the manifest part fix is also
available in #6216)
Co-authored-by: klakhov <kirill.9992@gmail.com>
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Co-authored-by: Kirill Sizov <kirill.sizov@cvat.ai>
Fixed:
- wrong location of tmp file when importing job annotations
- ```Traceback (most recent call last):
File
"/home/maya/Documents/cvat/.env/lib/python3.8/site-packages/rq/worker.py",
line 795, in work
self.execute_job(job, queue)
File "/home/maya/Documents/cvat/cvat/rqworker.py", line 37, in
execute_job
return self.perform_job(*args, **kwargs)
File
"/home/maya/Documents/cvat/.env/lib/python3.8/site-packages/rq/worker.py",
line 1389, in perform_job
self.handle_exception(job, *exc_info)
File
"/home/maya/Documents/cvat/.env/lib/python3.8/site-packages/rq/worker.py",
line 1438, in handle_exception
fallthrough = handler(job, *exc_info)
File "/home/maya/Documents/cvat/cvat/apps/engine/views.py", line 2233,
in rq_exception_handler
rq_job.exc_info = "".join(
AttributeError: can't set attribute
```
Resolves https://github.com/opencv/cvat/issues/5773
Resolves https://github.com/opencv/cvat/issues/5563
- root causes of the issues:
- the annotation file was uploaded to the server by tus protocol and
rq job was created but no one next requests for checking status were not
made. (e.g. user closed the browser tab)
- the annotation file was uploaded to the server by tus protocol but
rq job has not yet been created (e.g cvat instance restarted)
- tasks/projects creation from backups with the same name at the
same time by different users
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
This PR is focused on server performance optimization for requests with
collections.
- Collections with user-controlled size are removed from server output
and replaced
with structures providing links to the matching paginated endpoints and
summary info:
- /api/projects/{id}.tasks -> /api/tasks?project_id={id}
- /api/tasks/{id}.segments -> /api/jobs?task_id={id}: .start_frame,
.stop_frame
- /api/jobs/{id}.issues -> /api/issues?job_id={id}
- /api/issues/{id}.comments -> /api/comments?issue_id={id}
- /api/projects | tasks | jobs/{id}.labels -> /api/labels?
project_id|task_id|job_id={id}
- Added Label-related endpoints:
- /api/labels (GET)
- /api/labels/{id} (GET, PATCH, DELETE)
- PATCH and DELETE both cannot be used for sublabels
- Added and updated tests
- Removed unused test server data
- Added several new cases with skeletons and ownerships
- Fixed possible problems with duplicate labels appearing on renaming
PR removes previews downloading from CS when task creating (skipped in
PR #5478). In addition, I had to change the test to check for the file
name existing in the message when the specified file is not found in the
bucket, because now the preview is no longer downloaded at the stage of
creating a task.
Extracted some enhancements from
https://github.com/opencv/cvat/pull/4819
- Extracted common s3 manipulations in tests
- Refactored import/export tests to be more clear
`pathlib` improves code readability and type safety. It is already used
in some of the tests; convert all remaining `os.path` usage to `pathlib`
equivalents.
Roman Donchenko
