<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
This PR adds file-based profile creation:
- `cvat-cli profile create --file <path>`
- Supports plain-text PAT files
- Supports JSON envelopes with `token`, optional `server`, and optional
`name`
- Explicit `--name` and `--server-host` override envelope values
- A complete JSON envelope can create a profile without extra CLI
arguments
It also includes:
- Tests for plain-file import, JSON import, and precedence rules
- CLI/SDK/profile documentation updates
- A changelog entry for `--file`
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [x] I have updated the documentation accordingly
- [x] I have added tests to cover my changes
- [x] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
---------
Co-authored-by: Maxim Zhiltsov <maxim@cvat.ai>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
This branch completes the CLI workflow for persistent authentication
profiles. Users can save a CVAT server URL and Personal Access Token
under a named profile, then reuse it with `--profile` instead of
repeatedly passing credentials on the command line.
It adds `cvat-cli profile` commands to:
- List saved profiles, including the active default.
- Create a profile from a PAT, optionally deriving its name from the
server.
- Set, print, or clear the default profile.
- Delete profiles safely.
The implementation builds on the auth/config resolution already present
in `develop`. It uses the SDK’s `AuthStore.put_profile` API, preserves
copy-on-write store updates, and applies the same server-port validation
used by normal CLI client creation.
URL normalization was also moved into shared SDK utilities so both
`Client` and profile creation use one implementation for default HTTPS
schemes, trailing-slash removal, and invalid-scheme validation.
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [x] I have added tests to cover my changes
- [x] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
---------
Co-authored-by: Maxim Zhiltsov <maxim@cvat.ai>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
This branch continues the persistent authentication work for the CVAT
SDK and CLI. It makes saved auth profiles usable from CLI workflows, so
users do not need to repeatedly pass `--server-host`, `--server-port`,
and `--auth` for every command.
The changes centralize CLI auth resolution around the SDK auth helpers,
add support for profile/default-server based connection selection, and
introduce `cvat-cli config default-server` for managing the fallback
server used outside profile-based flows. The branch also adds dispatch
support for CLI commands that do not need an authenticated client, plus
tests and changelog coverage for the new config command and auth
behavior.
### How has this been tested?
Automated and manual tests
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [x] I have added tests to cover my changes
- [x] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
---------
Co-authored-by: Maxim Zhiltsov <maxim@cvat.ai>
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [x] I have added tests to cover my changes
- [x] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
---------
Co-authored-by: Maxim Zhiltsov <maxim@cvat.ai>
Task labels used to be mandatory, but now (see #9822) they are not. This
means that a lot of these parameters are no longer necessary and just
create clutter.
This is a continuation of #10060.
I also added a couple of `TypeAlias` annotations where appropriate, and
changed the import style in `test_users.py` to be more consistent with
other files.
The underlying SDK functions already emit human-friendly log messages
with the ID of the created resource. Instead of printing largely the
same message twice, we can just print the ID. That way, the CLI can be
more easily integrated into other software.
Currently, the CLI has one level of subcommands, and all subcommands
work on tasks. This leaves no room for subcommands that work on other
CVAT resources.
This change redesigns the CLI interface by adding another level of
subcommand hierarchy. Instead of running `cvat-cli <action>`, the user
will now run `cvat-cli <resource> <action>`. Previously available
commands are left available as deprecated aliases.
As a proof of concept, this PR adds some basic project actions.
I have also used this opportunity to correct some of the task action
names, specifically `export`, `import`, `dump` and `upload`. These names
don't correspond to either SDK function names, API endpoints, or UI
labels corresponding to these actions. In the new subcommand hierarchy,
I renamed those commands to `backup`, `create-from-backup`,
`export-dataset` and `import-dataset`, which are more consistent with
how other CVAT components call these actions.
I rewrote the introduction and usage sections of the cli README and
reference in order to reduce clutter and remove the need to
resynchronize the help output after every interface change.
<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->
<!-- Provide a general summary of your changes in the Title above -->
Added:
- Prepare chunks in rq workers instead of application server process
- Increase TTL for cached preview images
### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](
https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary
([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),
[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),
[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and
[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))
### License
- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
Feel free to contact the maintainers if that's a concern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new debugging configurations for various RQ workers,
enhancing debugging capabilities.
- Added a new service for processing chunks in Docker and Kubernetes
configurations.
- Enhanced error handling in data retrieval methods to manage timeout
scenarios effectively.
- **Bug Fixes**
- Improved robustness of job-related functionalities with enhanced test
coverage and validation checks.
- **Documentation**
- Updated API documentation settings and configurations for better
clarity and usability.
- **Tests**
- Expanded test suite with new test cases and parameterization for
comprehensive validation of task and cloud storage functionalities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
While it is already possible to output mask shapes from AA functions,
which the driver _will_ accept, it's not convenient to do so. Improve
the practicalities of it in several ways:
* Add `mask` and `polygon` helpers to the interface module.
* Add a helper function to encode masks into the format CVAT expects.
* Add a built-in torchvision-based instance segmentation function.
* Add an equivalent of the `conv_mask_to_poly` parameter for Nuclio
functions.
Add another extra for the `masks` module, because NumPy is a fairly
beefy dependency that most SDK users probably will not need (and
conversely, I don't think we can implement `encode_mask` efficiently
without using NumPy).
My main motivation is to support a future feature, but I think this is a
good thing in its own right.
While it's already possible to create an AA function that lets you
customize the threshold (by adding a creation parameter), confidence
scoring is very common in detection models, so it makes sense to make
this easier to support, both for the implementer of the function, and
for its user.
Some of them are annotated with an `Iterator` return type. However...
It just occurred to me that `@contextmanager` cannot work with a
function that returns a plain iterator, since it relies on the generator
class's `throw` method. `contextmanager` is defined in typeshed as
accepting an iterator-returning function, but that appears to be a bug:
<https://github.com/python/typeshed/issues/2772>.
Change all such annotations to a `Generator` type instead.
Some annotations are also broken in other ways; fix them too.
Test it by starting an HTTPS server (that forwards all requests to CVAT)
and attempting to access that server via the CLI. This allows us to test
that the option actually works end-to-end, and avoids dependencies on
CLI internals.
The new test requires `main` to catch the exception that is raised in
the case of a certificate error, so implement that.
These serve as a replacement for YOLOv8n that was removed in #6632.
To support these functions, I also add an ability to define
parameterized functions for use with the CLI.
Introduce a `cvat-sdk auto-annotate` command that downloads data for a task,
then runs a function on the local computer on that data, and uploads
resulting annotations back to the task.
To support this functionality, add a new SDK module,
`cvat_sdk.auto_annotation`, that contains an interface that the
functions must follow, and a driver that applies a function to a task.
This will let users easily annotate their tasks with custom DL models.
It doesn't make much sense to encode the image as JPEG if the given file
name is "image.png". Instead, let PIL select the format based on the
requested file name.
Fixes#5061, #4179
- Added a way to declare custom file ordering for the local task data
uploads via TUS protocol
- Added an option to use a manifest to support the `predefined` sorting
method
- This file is required for the `predefined` sorting mode with image
archives
- Fixed file ordering when tasks are created from SDK or CLI in the
`predefined` sorting mode
- Added more tests for task data uploading API
The uploading protocol is implemented:
The user specifies `sorting_method=predefined` if the task creation
request. Then the data is uploaded.
1. Client files uploading
1.1. The files are uploaded as separate files (using the TUS protocol)
or grouped files (using the `Upload-Multiple` requests).
1.2. The `Upload-Finish` request comes (or its unlabeled legacy
equivalent). The new optional field can be supplied: `upload_file_order`
- a list of strings. It allows to override the input file order, if
necessary, and is only valid with the `predefined` sorting method
specified.
1.2.1. If the field is empty or missing, the client files in the data
requests are considered ordered.
1.2.2. If the field is not empty, a list containing the file list in the
required order is expected in the `upload_file_order` field.
1.2.2.1. If there are `client_files` in the request, the files are
sorted
1.2.2.2. If file lists mismatch, an explanatory error is raised.
2. Data processing
2.1. At this point, all `*_files` are considered ordered as requested.
2.2. Require a metafile for zip uploads with predefined sorting. The
file is expected to accompany the uploaded zip file, not to be inside of
the archive.
2.3. If there is a metafile in the input data, files are ordered after
the metafile.
2.3.1. If the data is extracted from cloud, only the specified subset of
the files is kept in the manifest.
2.3.2. If the upload data doesn't exist in the metafile, an error is
raised.
2.3.3. A `job_file_mapping` has higher priority than metafile, if
specified.
Co-authored-by: Roman Donchenko <roman@cvat.ai>
For user-facing functions, keep accepting `str` paths to maintain
compatibility and flexibility, but add support for arbitrary path-like
objects. For internal functions (in `downloading.py` and
`uploading.py`), don't bother and require `pathlib.Path`.
The only code that isn't converted is build-time code (e.g. `setup.py`)
and code that came from openapi-generator.
`pathlib` improves code readability and type safety. It is already used
in some of the tests; convert all remaining `os.path` usage to `pathlib`
equivalents.
- Added auth docs in API schema and SDK ApiClient/Config (fixes#1517)
- Added SDK docs with API, SDK and CLI docs
- Added `develop` branch in the docs
- Allowed unauthorized access to `api/docs`, `api/swagger`, `api/schema` endpoints
- Added `--insecure` env var to control host checks in CLI
- Refactored `build_docs.py` (backported https://github.com/openvinotoolkit/datumaro/pull/589)
- Extracted requirements file for site
Andrei Strongin