### Motivation and context
#### Fix UnboundLocalError on GET /api/tasks/<id>/data/meta for tasks
without media
#### Motivation
Production is throwing 500s on GET /api/tasks/<id>/data/meta:
```python
UnboundLocalError: cannot access local variable 'media' where it is not associated with a value
File "cvat/apps/engine/views.py", line 1935, in metadata
for item in media
```
#### Root cause
TaskViewSet.metadata() dispatches on db_task.media_type through an if /
elif / elif chain:
```python
if db_task.media_type == AUDIO and mode == INTERPOLATION:
media = [db_data.audio]; chapters = None; def serialize_media_item(...)
elif db_task.media_type in (IMAGE, POINT_CLOUD):
media = ...; chapters = ...
elif db_task.media_type: # ← truthiness guard
assert False, f"Unknown media type '{db_task.media_type}'"
# no else
```
Task.media_type is declared CharField(..., default="", blank=True), so a
freshly-created Task whose Data row exists but whose media has not been
populated yet has media_type == "". That value is falsy, so none of the
branches execute and media, chapters, and serialize_media_item are never
bound. The subsequent for item in media raises UnboundLocalError and DRF
returns a 500.
The empty-media state was already known at the top of the same method —
the prefetch() match block explicitly handles ("", "") as a no-op — but
that handling was never carried through to the body of the view. The
regression was introduced when the audio media type was added
https://github.com/cvat-ai/cvat/pull/10551, which restructured the
previous if/else into the current chain and put a truthy guard on the
final branch.
#### Fix
Raise ValidationError early, mirroring the neighbouring db_data is None
check just a few lines above:
```python
db_data = db_task.data
if db_data is None:
raise ValidationError("Data is not uploaded for the task yet")
if not db_task.media_type:
raise ValidationError("Task has no media data yet")
```
This turns the silent 500 into an actionable 400 with a clear message, is consistent with the existing "data not uploaded" pattern, and requires no changes to the dispatch chain below.
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes that apply.
If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the pull request.
If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in CHANGELOG.md -->
- [x] I have updated the documentation accordingly
- [x] I have added tests to cover my changes
- [x] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the project.
Feel free to contact the maintainers if that's a concern.
---------
Co-authored-by: Maxim Zhiltsov <maxim@cvat.ai>
Currently this either returns 200 and an empty list (if the parent
quality report is in an accessible organization or a personal
workspace), or 500 (if the parent is in an inaccessible organization).
This is inconsistent with all other resource-based filters, where the
endpoint returns 403 if the resource is inaccessible (and obviously we
should never return a 500 status regardless). It's also a (very minor)
security problem, since it lets an attacker learn some information about
a quality report they can't access.
The cause is that we don't explicitly check that the user has
permissions to view the parent, so do that.
Thanks to @geo-chen for the report.
There are actually two problems with it:
* The fixtures could execute in the wrong order, because all of the
fixtures in this test are class-level, and the order pytest executes
these in is indeterminate. So the DB/Redis reset could happen after the
creation of test data. Fix it by adding explicit fixture dependencies.
* It takes a while for all the setup to execute, which can time out.
Bump the timeout.
In addition, update the `wait_background_request` function to stop
waiting if a request fails. This lets the test fail early if something
goes wrong, instead of timing out.
### Summary
- Moved most enum-like fields from `search_fields` to `simple_filters`,
and `filter_fields` in viewsets
- Added field sorting for schema
### Motivation and context
Enum-like fields (e.g. `status`, `state`, `role`) were declared in
`search_fields` on several list endpoints. DRF's `SearchFilter` performs
case-insensitive substring matching across `search_fields`, so the
global `?search=` parameter would also match against these enum values,
which is rarely what users want. They are now exposed only via
`simple_filters` (exact match), e.g. `?status=completed`,
`?state=in_progress`, `?role=maintainer`.
In some cases, free-text fields are intentionally excluded from simple
filters, because exact matching is not very useful for them. By this
logic, `project_name` and `task_name` in task and job list endpoints
should be removed as well, but they are kept to avoid breaking changes
without a significant reason, as they're available for quite a long time
already.
The filters are now sorted for schema generation, which helps to avoid
spurious schema diffs.
#### Per-endpoint changes
| Endpoint | Removed from `search_fields` | Added to `simple_filters` |
Added to `filter_fields` |
|---|---|---|---|
| `GET /api/projects` | `status` | — | — |
| `GET /api/tasks` | `status`, `mode`, `dimension`, `validation_mode` |
— | — |
| `GET /api/jobs` | `state`, `stage` | — | — |
| `GET /api/memberships` | `role` | — | — |
| `GET /api/webhooks` | `type` | — | — |
| `GET /api/cloudstorages` | `provider_type`, `credentials_type` | — | —
|
| `GET /api/access_tokens` | — | `read_only` | — |
| `GET /api/invitations` | — | `user_id`, `accepted` | `id` |
| `GET /api/requests` | — | `org_id` | `org`, `org_id` |
For fields removed from `search_fields`: they remain exact-match
filterable as `simple_filters` (e.g. `?status=completed`).
For fields newly in `simple_filters`: they were previously available
only via the `filter` parameter and are now also exact-match filterable.
For fields newly in `filter_fields`: they were previously not available
for filtering.
The following endpoints were touched only for the construction-style
refactor (no field-level behavior change):
`GET /api/issues`, `GET /api/comments`, `GET /api/labels`, `GET
/api/users`, `GET /api/organizations`, `GET /api/consensus_settings`,
`GET /api/quality/conflicts`, `GET /api/quality/reports`, `GET
/api/quality/settings`.
### API response changes
The following list endpoints now return additional fields:
- `GET /api/requests` — `operation.org_id` (integer, nullable). Supports
`?org_id=` filtering.
- `GET /api/invitations` — `accepted` (boolean). Mirrors the existing
`accepted` filters.
### How has this been tested?
Added simple-filter test coverage for fields newly exposed in this PR:
- `tests/python/rest_api/test_access_tokens.py`: `read_only`.
- `tests/python/rest_api/test_invitations.py`: `user_id`, `accepted`.
- `tests/python/rest_api/test_requests.py`: `org`, `org_id`.
Filled in pre-existing simple-filter coverage gaps:
- `tests/python/rest_api/test_jobs.py`: `dimension`, `media_type`,
`mode`, `task_name`, `project_name`.
- `tests/python/rest_api/test_tasks.py`: `media_type`, `project_name`.
`TestRequestsListFilters` was refactored to class-scoped fixtures so the
heavy setup (3 projects + 3 tasks + ~13 RQ requests + the new
org-context project/task/exports) runs once for the whole class instead
of per parametrize case. Wall-clock for `pytest -k
TestRequestsListFilters` on the same hardware:
| | Before | After |
|---|---|---|
| Tests collected | 9 | 10 |
| Total time | 78.4 s | 13.1 s |
| Per-case `call` time | ~5.2 s | ~0.05 s |
| Per-case `setup` time | ~4.3 s every case | ~12.1 s once, then
negligible |
≈6× faster despite covering one more case.
The corrupted-meta test was also extracted out of the class to a
module-level function with `restore_db_per_function` /
`restore_redis_inmem_per_function` decorators since it has no class
dependencies.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
I inadvertently broke this in #10106 by changing the order of file
operations in `PcdReader.convert_bin_to_pcd`. The current version of
`read_raw_images` saves the `.bin` file with a `.pcd` extension and then
tries to convert it to the PCD format. `convert_bin_to_pcd` opens the
output file first; but in this case the input and output paths are the
same, so this truncates the input file to 0 bytes before
`convert_bin_to_pcd` can read it.
We could probably reorder the opens again to fix this, but IMO, this
scheme with saving the `.bin` file with a `.pcd` extension is rather
confusing and it's better to fix this instead. Save the file with its
original extension and let `convert_bin_to_pcd` save the output to a new
file instead.
In addition, fix a bug in `HeaderFirstDownloader` that prevents a task
from being created if a `.bin` file is small. This is needed for the
test to work. The problem is that
`_HeaderFirstPcdDownloader.try_parse_header` always returns False, so if
the entire file fits within `headers_to_try[0]` bytes, the loop will
then try to download a range that doesn't exist. Fix it by detecting
that there is no more data left to download and exiting early.
## Summary
This is the first PR in a planned series that splits the large
`tests-infra-kube-stable-v2` branch into smaller, reviewable changes.
This PR is intentionally narrow. It only fixes baseline test instability
that blocks reliable validation of the later runtime and CI refactors.
## What changed
- relaxed the REST video-frame comparison tolerance slightly to handle
small local decode variance
- replaced brittle clipboard shortcut simulation in the raw-labels E2E
flow with the editor paste path while keeping the UI responsible for
stripping server-assigned IDs
- removed two `intercept(...)` -> `wait(...)` races in task-opening
Cypress flows
- stabilized bulk-actions E2E checks by waiting on durable
request/results state instead of transient notifications/progress
wrappers
- hardened Cypress video cleanup so missing video files do not turn
passing specs into failed runs
## Files
- `tests/python/rest_api/_test_base.py`
-
`tests/cypress/e2e/actions_tasks/case_117_paste_labels_from_another_task.js`
- `tests/cypress/e2e/actions_tasks/task_rectangles_only.js`
- `tests/cypress/e2e/features/skeletons_pipeline.js`
- `tests/cypress/e2e/features/bulk_actions.js`
- `tests/cypress/plugins/index.js`
## Why this PR exists
The split PR series needs a stable baseline first. Without these fixes,
later PRs in the series keep failing for unrelated local/Cypress
flakiness, which makes review and validation noisy.
## PR series
This is **PR 1** in the split series.
Planned follow-ups will cover backend fixes, Helm/OPA fixes, dependency
compatibility, pytest runtime foundation, parallel runtime, kube
runtime, and CI/docs.
Task labels used to be mandatory, but now (see #9822) they are not. This
means that a lot of these parameters are no longer necessary and just
create clutter.
## Summary
- enforce an explicit `8..256` password length policy for newly set
passwords across registration, password change, and password reset
confirm flows
- keep the OpenAPI request component names stable while owning the
serializer limits in CVAT
- keep the current registration UX where the initial submit on an empty
form reveals all required fields, instead of disabling submit before any
validation feedback is shown
- keep unit-test coverage for the password policy and add the edge case
that an already-existing password longer than 256 can still be used as
`old_password`
This was a regression introduced in d471db6b. I hadn't realized that the
`logger` parameter of `_upload_file_data_with_tus` was a _function_
rather than a logger object, so I used it as if it was the latter.
I don't think the logger customization makes any sense here regardless,
so just drop this parameter and use the client logger.
The feature allows to export current list of Projects/Tasks/Jobs as .csv file via button in filters UI.
---------
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
This allows storing media data of "local" tasks in cloud storage,
transparently to the user. The CVAT administrator must explicitly move
tasks to backing cloud storage (or back) using the new management
commands.
Currently, only tasks with images using cached chunks are supported.
1. Don't capture stderr. Nobody needs it, and it seems unlikely that
anybody will.
2. Don't exit the process if the subprocess fails. Since this function
is now (indirectly) used within individual tests, this behavior causes
the testsuite to exit prematurely.
We don't really need to print a custom error message, because the
default exception message already contains the command line and status;
and since we no longer capture stderr, it'll be printed too. Stdout
will, unfortunately, not be printed, but it's not as important for
debugging.
Also, fix an issue where the subprocess status is not checked if
`capture_output` is false.
... and other similar endpoints.
Instead of manually building the URL, use reversing. This works
correctly whether the original URL has a trailing slash or not.
Fixes#10115.
When `frame_media` is empty (i.e. there are no related images for a
frame), the `truncate_common_filename_prefix` logic crashes, because
`os.path.commonpath` requires at least one path as input.
This only happens when the task is in cloud storage, because the
non-cloud branch constructs the `media` list in such a way that frames
with no related images are simply omitted. This works fine, so factor
out that logic and use it in the cloud case as well.
There is special logic to handle this case in `create_thread`, and I
broke it in one of the recent patches that introduced pathlib. Add a
test to prevent this from breaking again.
This PR implements a simplified consensus scoring system for merged annotations. Instead of filtering annotations based on quorum thresholds, all annotations are now merged and assigned a consensus score (0.0 to 1.0) that represents the level of agreement among annotators.
Also PR contains a bunch of improvements to review mode allowing to unlock and edit objects, navigate in between objects using shortcuts.
Fix: in get_interpolated_shapes the branch that decides which
interpolation function to use had
```
if dimension == DimensionType.DIM_3D:
```
followed by another independent if for
rectangle/cuboid/ellipse/skeleton. That could cause both branches to be
considered and lead to incorrect interpolation behavior. The code now
uses elif for the second condition so only one interpolation method runs
for a pair of keyframes.
Changed
```
if is_rectangle or is_cuboid or is_ellipse or is_skeleton:
```
To
```
elif is_rectangle or is_cuboid or is_ellipse or is_skeleton:
```
#### **Problem:**
The TUS implementation in `cvat/apps/engine/tus.py` was using
`request.body` to access uploaded data. Django's `request.body` property
attempts to read the entire request body at once, which:
- Consumes large amounts of memory for big files
Bug:
- Returns a 500 Internal Server Error status for HEAD requests after a
file upload is completed via TUS
#### **Solution:** Changed to streaming approach using `shutil.copyfileobj(chunk.request, file)`:
- Reads data in 64KB chunks instead of loading entire body
- Significantly reduces memory usage for large uploads
Alexey Zinoviev