Currently, all source files are located in `/home/django`, and owned by
`django`. This means that if there's any vulnerability that lets an
attacker overwrite files in the server, they can replace source files
with their own code, and potentially get that code executed. That's
pretty bad, so I want to harden against that.
Make all source files owned by root, and move them to `/opt/cvat`. Add a
`manage.py` symlink in `/home/django` for backwards compatibility. It
happens that if a script is a symlink, Python does not add the symlink's
directory to `sys.path`, which is great for us, since that lets us avoid
a writable directory on there.
Still, even though `/home/django/manage.py` is owned by root, an
attacker could potentially be able to delete it and replace it with
their own malicious file. To be a bit more safe, replace `~/manage.py`
calls in backend scripts with `django-admin`.
To make sure CVAT can still find the data directory, add a new
environment variable, `CVAT_BASE_DIR` and set it in the Docker image.
This also fixes a minor bug: we no longer override the `HOME`
environment variable in the `Dockerfile`, so now it's automatically set
by `Docker` depending on the current user.
### Summary
- Moved most enum-like fields from `search_fields` to `simple_filters`,
and `filter_fields` in viewsets
- Added field sorting for schema
### Motivation and context
Enum-like fields (e.g. `status`, `state`, `role`) were declared in
`search_fields` on several list endpoints. DRF's `SearchFilter` performs
case-insensitive substring matching across `search_fields`, so the
global `?search=` parameter would also match against these enum values,
which is rarely what users want. They are now exposed only via
`simple_filters` (exact match), e.g. `?status=completed`,
`?state=in_progress`, `?role=maintainer`.
In some cases, free-text fields are intentionally excluded from simple
filters, because exact matching is not very useful for them. By this
logic, `project_name` and `task_name` in task and job list endpoints
should be removed as well, but they are kept to avoid breaking changes
without a significant reason, as they're available for quite a long time
already.
The filters are now sorted for schema generation, which helps to avoid
spurious schema diffs.
#### Per-endpoint changes
| Endpoint | Removed from `search_fields` | Added to `simple_filters` |
Added to `filter_fields` |
|---|---|---|---|
| `GET /api/projects` | `status` | — | — |
| `GET /api/tasks` | `status`, `mode`, `dimension`, `validation_mode` |
— | — |
| `GET /api/jobs` | `state`, `stage` | — | — |
| `GET /api/memberships` | `role` | — | — |
| `GET /api/webhooks` | `type` | — | — |
| `GET /api/cloudstorages` | `provider_type`, `credentials_type` | — | —
|
| `GET /api/access_tokens` | — | `read_only` | — |
| `GET /api/invitations` | — | `user_id`, `accepted` | `id` |
| `GET /api/requests` | — | `org_id` | `org`, `org_id` |
For fields removed from `search_fields`: they remain exact-match
filterable as `simple_filters` (e.g. `?status=completed`).
For fields newly in `simple_filters`: they were previously available
only via the `filter` parameter and are now also exact-match filterable.
For fields newly in `filter_fields`: they were previously not available
for filtering.
The following endpoints were touched only for the construction-style
refactor (no field-level behavior change):
`GET /api/issues`, `GET /api/comments`, `GET /api/labels`, `GET
/api/users`, `GET /api/organizations`, `GET /api/consensus_settings`,
`GET /api/quality/conflicts`, `GET /api/quality/reports`, `GET
/api/quality/settings`.
### API response changes
The following list endpoints now return additional fields:
- `GET /api/requests` — `operation.org_id` (integer, nullable). Supports
`?org_id=` filtering.
- `GET /api/invitations` — `accepted` (boolean). Mirrors the existing
`accepted` filters.
### How has this been tested?
Added simple-filter test coverage for fields newly exposed in this PR:
- `tests/python/rest_api/test_access_tokens.py`: `read_only`.
- `tests/python/rest_api/test_invitations.py`: `user_id`, `accepted`.
- `tests/python/rest_api/test_requests.py`: `org`, `org_id`.
Filled in pre-existing simple-filter coverage gaps:
- `tests/python/rest_api/test_jobs.py`: `dimension`, `media_type`,
`mode`, `task_name`, `project_name`.
- `tests/python/rest_api/test_tasks.py`: `media_type`, `project_name`.
`TestRequestsListFilters` was refactored to class-scoped fixtures so the
heavy setup (3 projects + 3 tasks + ~13 RQ requests + the new
org-context project/task/exports) runs once for the whole class instead
of per parametrize case. Wall-clock for `pytest -k
TestRequestsListFilters` on the same hardware:
| | Before | After |
|---|---|---|
| Tests collected | 9 | 10 |
| Total time | 78.4 s | 13.1 s |
| Per-case `call` time | ~5.2 s | ~0.05 s |
| Per-case `setup` time | ~4.3 s every case | ~12.1 s once, then
negligible |
≈6× faster despite covering one more case.
The corrupted-meta test was also extracted out of the class to a
module-level function with `restore_db_per_function` /
`restore_redis_inmem_per_function` decorators since it has no class
dependencies.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The feature allows to export current list of Projects/Tasks/Jobs as .csv file via button in filters UI.
---------
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
This allows storing media data of "local" tasks in cloud storage,
transparently to the user. The CVAT administrator must explicitly move
tasks to backing cloud storage (or back) using the new management
commands.
Currently, only tasks with images using cached chunks are supported.
1. Don't capture stderr. Nobody needs it, and it seems unlikely that
anybody will.
2. Don't exit the process if the subprocess fails. Since this function
is now (indirectly) used within individual tests, this behavior causes
the testsuite to exit prematurely.
We don't really need to print a custom error message, because the
default exception message already contains the command line and status;
and since we no longer capture stderr, it'll be printed too. Stdout
will, unfortunately, not be printed, but it's not as important for
debugging.
Also, fix an issue where the subprocess status is not checked if
`capture_output` is false.
We are going to increase the possible number of replicas, and current
way of displaying replicas is not convenient if there are hundreds of
them.
So,
- flatten list of jobs
- on both jobs page and task page:
- every parent job has a tag "parent"
- every replica job has a tag "replica"
- every parent job has an action "go to replicas" which switches filters
on a page to show it's replicas
- every replica job has an action "go to parent" which switches filters
on a page to show it's parent
- task page be default does not show replicas
- added "replicas_count" field to api on job retrieving
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
Co-authored-by: Kirill Lakhov <kirill.lakhov@cvat.ai>
Frankly, I don't know why we're even doing this. It seems to only have
downsides:
* It slows down testsuite execution, which is especially annoying
because it happens _before_ detailed exception messages are printed.
* It erases evidence that could be used to investigate why a test
failed.
The DB gets reloaded every time the testsuite starts anyway, so a dirty
database should not affect any subsequent executions. To make sure
there's nothing left from a previous test execution, run `manage.py flush`
before `loaddata`.
Absolute paths don't fit very well here:
* Files can be stored in cloud storage, in which case the absolute path
stored in the database doesn't actually exist.
* They make it unnecessarily hard to move storage to a different base
directory.
* The code becomes simpler with relative paths.
This PR implements a simplified consensus scoring system for merged annotations. Instead of filtering annotations based on quorum thresholds, all annotations are now merged and assigned a consensus score (0.0 to 1.0) that represents the level of agreement among annotators.
Also PR contains a bunch of improvements to review mode allowing to unlock and edit objects, navigate in between objects using shortcuts.
It's possible that some tasks with videos have no manifest. There are
several reasons for that, including manual playback speed manipulation
attepts. CVAT will generate no manifest for such videos, but the
chapters functionality will try to read the chapters from the manifest
regardless. It's technically possible to ignore bad keyframes, but it
can lead to invalid chapters.
- Fixed task access for tasks with videos with bad keyframes
This is a helper method that pretty much just wraps the double loop over
the packets and the frames in a packet. Using it makes the code simpler
and enables better type inference in IDEs, because the PyAV type stubs
are sufficient to infer that `container.decode(video_stream)` returns an
`Iterable[VideoFrame]` (whereas `packet.decode()` is declared to return
`list[SubtitleSet]` for some reason).
Also, add some imports and type annotations to ensure that the IDEs have
types to infer _from_.
This is a continuation of #10060.
I also added a couple of `TypeAlias` annotations where appropriate, and
changed the import style in `test_users.py` to be more consistent with
other files.
Now that we've dropped Python 3.9 support, we can get rid of these. Some
of these were actually deprecated in Python 3.9 too, but they must've
slipped through after the last cleanup round.
Done with:
ruff check --select=UP006,UP035 --target-version=py310 --fix
--unsafe-fixes
plus a bit of manual cleanup.
Issue: https://github.com/cvat-ai/cvat/issues/9380
Incorrect exception handling for Cloud Storage non-existence and other
NOT_FOUND errors
I fixed the exception handling and tested all these cases on three types
of Cloud Storage.
Required to push our Helm chart to our public ECR repo
I've tested in separate workflow
(https://github.com/cvat-ai/cvat/actions/runs/17459911830/job/49582057271)
- [x] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues
- [x] I submit _my code changes_ under the same [MIT License]
---------
Co-authored-by: Petr Iosipov <petr.iosipov@cvat.ai>
Co-authored-by: Andrey Zhavoronkov <andrey@cvat.ai>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
This PR replaces `example.com` domain name with the new alias
`webhooks.internal` that exists inside a docker network. Previously
tests failed often because example.com does not return any response for
some requests, RQ had webhooks jobs that tried to finish within 10
seconds, but also failed due to timeout.
Also since there are webhooks created in a DB, they try to reach
example.com when a cluster starts, that sometimes fills up the worker's
queue.
Usage of subdomain is important because Django URL matcher does not
allow domains like `webhooks`
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [ ] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
### License
- [ ] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
Co-authored-by: Oleg Valiulin <oleg.valiulin@cvat.ai>
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Typos spell-checker integration
Run `typos` to show typos in project
Run `typos -w` to autofix
**API changes:**
- `POST /api/consensus/merges?rq_id=rq_id` returns 410 status code, this
endpoint no longer supports process status checking
- `GET /api/projects/id/dataset?action=import_status` returns 410 status
code, this endpoint no longer supports process status checking
- `POST /api/projects/backup?rq_id=rq_id` returns 410 status code, this
endpoint no longer supports process status checking
- `POST /api/tasks/backup?rq_id=rq_id` returns 410 status code, this
endpoint no longer supports process status checking
- `PUT /api/tasks/id/annotations?rq_id=rq_id&format=format` returns 410
status code, this endpoint no longer supports process status checking
- `PUT /api/jobs/id/annotations?rq_id=rq_id&format=format` returns 410
status code, this endpoint no longer supports process status checking
- `GET /api/events` is deprecated in favor of the following API:
- [new] `POST /api/events/export` (returns 202 with Request ID)
- GET /api/requests/rq_id
- [new] `GET /api/events/download?rq_id=rq_id` (private endpoint, should
be used only as result_url)
- `POST /api/quality/reports/rq_id=rq_id` is deprecated in favor of `GET
/api/requests/rq_id`
**Architecture visible changes:**
- Cache files containing events (created after using the API to export
events as a file) are stored in `/data/cache/export/` instead of
`/data/tmp/`. The `_clear_export_cache` function is deleted, since cache
files are deleted one day after creation (by default) by the
`cleanup_export_cache_directory` cron job.
**SDK backward-incompatible changes:**
- [requests_api.list] action/target/subresource filters now have string
type
- Several types were removed:
`DatasetWriteRequest`/`BackupWriteRequest`/`TaskAnnotationsWriteRequest`;
`JobAnnotationsUpdateRequest`/`TaskAnnotationsUpdateRequest`
---------
Co-authored-by: Roman Donchenko <roman@cvat.ai>
- Added quality estimation for projects
- Updated quality report contents and quality report summary server API
-- frame_count field - deprecated in favor of the new validation_frames field
-- frame_share field - deprecated in favor of the new validation_frame_share field
- Added support for job filters in quality settings. This allows to include or exclude jobs based on filter criteria
- Added support for quality estimation in consensus replicas
- Task quality checks now can use individual settings or inherit ones from the parent project
- Project quality updates now trigger task quality updates for all nested tasks. Relevant task quality reports are reused
- [Server API] Optimized GET api/quality/reports, GET api/quality/conflicts requests
- [Server API] Optimized permission checks in api/quality/* requests
- [Server API] Improved endpoint descriptions in the schema
Roman Donchenko