Drop accumulated dead code in the test-support package: helpers with
zero call sites, unused capability probes, dead FFI re-exports, and
orphaned pytest plumbing. Verified by repo-wide grep that nothing
references any of these.
tvm.testing's test-gating layer had a number of one-line helper
functions that add a name but no behavior. Inline the thin ones so call
sites name the underlying flag/feature/condition directly.
Pytest plugin (plugin.py): _target_to_requirement built its skip / gpu
marks through two one-line wrappers (_gpu_mark_and_skip / _skip_only)
plus a per-kind if ladder. Replace them with two frozensets (GPU- vs
CPU-family kinds) and resolve the skip probe by name:
marks.append(pytest.mark.skipif(not getattr(env, f"has_{kind}")(),
reason=f"need {kind}"))
The cuda+cudnn / cuda+cublas accelerator-library cases are remapped
inline (cudnn before cublas). Adds two direct unit tests for the
cudnn/cublas special-case and the unknown-kind ([]) fallback.
tvm.testing.env (env.py): inline the pure probe wrappers that just
forwarded to a primitive --
* build-flag (5): has_cutlass/rpc/nnapi/openclml/mrvl ->
env.build_flag_enabled ("USE_X"). The private _build_flag_enabled is
promoted to the public build_flag_enabled; the composed probes
(has_cudnn/cublas/nccl/hipblas) and the hexagon/adreno probes call it
too.
* cpu-feature (5 pure):
has_arm_dot/arm_fp16/aarch64_sve/aarch64_sme/x86_amx ->
env.has_cpu_feature("..."). The composed has_x86_vnni (avx512vnni OR
avxvnni) and has_x86_avx512 (a five-feature set) are kept -- not thin
wrappers.
Also drops the obsolete test_build_flag_probe_matches_libinfo self-test
and the matching _BOOL_PROBES entries.
The runtime device probes (has_cuda/has_rocm/...) are intentionally left
as-is: the pytest plugin resolves env.has_<kind>() from each target
kind, so those names are load-bearing rather than thin wrappers.
All in-tree uses of tvm.testing.parameters() were migrated to native
pytest.mark.parametrize in #19803, so remove the helper itself along
with the plugin machinery that only served it:
- python/tvm/testing/utils.py: delete the parameters() function and the
_parametrize_group counter.
- python/tvm/testing/plugin.py: delete
_parametrize_correlated_parameters and its call in
pytest_generate_tests.
- tests/python/testing/test_tvm_testing_features.py: drop the
joint-parameter tests that exercised parameters() (the parameter() and
fixture() tests stay).
This removes the public tvm.testing.parameters symbol;
tvm.testing.parameter (singular) and tvm.testing.fixture are unchanged.
Use pytest.mark.parametrize instead.
This pr updates the contributor guide and tvm.testing
docstrings/comments to describe the current gating API
---------
Co-authored-by: Tianqi Chen <tqchen@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This pr modernizes test gating. It replaces the heavy
`tvm.testing.Feature` machinery with a thin `tvm.testing.env` module of
`has_*()` capability probes, used via standard pytest.mark + skipif. And
markers move to `pyproject.toml`
This PR enables ruff pyupgrade (UP) rules with py310 target, auto-fixing
~5600 annotation modernizations (PEP 585 generics, PEP 604 unions,
deprecated typing imports).
Also removes from __future__ import annotations from ir/module.py and
rmsnorm.py, bumps requires-python to >=3.10, and removes absolute_import
aliases from topi/contrib files.
This PR phases out legacy target string format in favor of the json
style format that is more well formed. It also simplfies our overall
code in handling multiple formats.
This PR modernizes the FFI foundation of the project and introduce
a new minimal and lightweight module [tvm ffi](https://github.com/apache/tvm/tree/refactor-s3/ffi)
based on our lessons in the past few years. It implements a modern
version of the [Unified Packed and Object RFC](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0097-unify-packed-and-object.md)
that unifies the packed function call and object systems.
Summary of the change:
- A dedicated clean Any/AnyView that can store strong and weak
references of items
- Function(previously PackedFunc) system built on top of the Any/AnyView
- A minimal C API that backs the overall calls. We are stabilizing the
API with a goal to bring clean, stable FFI conventions for both compiled
and registered code
- A rewrite of core python binding and generated code based on the module
- Update existing code and test cases to the new module
- Latest dlpack support
The new module brings many benefits thanks to the cleaner design,
to name a few:
- Any can support both POD types(int) and object types.
- Containers (e.g. Array) can now also contain Any value, e.g. now
`Array<int>` is supported, no need for boxed types
- Error handling now upgrades to object-based, allowing cleaner
traceback across languages
- Map now preserves insertion orders
- Path toward isolated stabilize minimum core ABI/API foundation module
- Type traits based design that cleanly defines how values interact
with Any system
- Automatic conversion of different types based on traits if needed
Because FFI upgrade is at heart of the project, the change touches every
component of the system. Importantly, this is an upgrade of the ABI so the
change is not backward compatible. The code compiled under the old
FFI won't work under the new one. We did provide example ABI translation
(e.g. LegacyTVMArgValueToFFIAny) functions for compatibility.
The PR tries to leave files in their old places while creating redirections.
The goal is to have the first milestone landed and infrastructure in place,
so we can do further refactors to complete features and cleanup legacy code
as trackable PRs. As of now, python binding and compiled code are under the
new convention while RPC and some other bindings still relies on legacy ABI
translation. We will work on upgrades in the coming PRs, including areas such
as reflection, phasing out legacy redirections etc.
The current unittest folder is too large and contains too many files and
too many components. This PR refactors the unittest folder by moving the
files to the corresponding folders.
* [OpenCL][unit tests] Fix opencl cpp unit tests
After some changes in Hexagon, the run of cpp opencl tests leads to the
following error:
```
pluggy.manager.PluginValidationError: unknown hook 'pytest_configure_node' in plugin <module 'tvm.contrib.hexagon.pytest_plugin'
```
Added `pytest_plugin` for OpenCL CPP tests for avoiding this error and
processing gtest arguments.
* Fix fail than gtest_args option was already added
* Move `gtest_args` deginition to the main testing plugin
* Revert "[skip ci] Revert "[ci] Default to n=2 for test parallelism (#12376)" (#12413)"
This reverts commit 478b672f2b.
* [ci] Default to n=2 for test parallelism
This is attempt #2 of #12376 which was reverted in #12413. The changes
in `plugin.py` should keep all the tests on the same node so sporadic
failures don't happen due to scheduling.
Co-authored-by: driazati <driazati@users.noreply.github.com>
* [CI] Improved skip messages when using @tvm.testing.requires_*
Previously, the same message was given regardless of why a test
couldn't be run. This has been split up into separate checks for TVM
cmake options in `config.cmake`, enabled targets in `TVM_TEST_TARGETS`
environment variable, and checks for available hardware.
* Refactor to specify repeated feature marks, compile-only markers
* Fixed lint errors
* Import from contrib, not from a different import
* Removed use of requires_llvm() as a list of marks
* Corrected mark from requires_gpu to requires_cuda
* Adding missing "not"
* Added USE_CMSISNN as a requirement for corstone300.
A follow-up from https://github.com/apache/tvm/pull/9188. The
`item.location` tuple contains `(filename, line_number, test_name)`,
where the `test_name` includes a string representation of all
parameters. This change preserves pytest's sorting of parametrized
values within a parametrized test, rather than sorting by strings.
* [Pytest] Sort unit tests before running.
By default, pytest will sort tests to maximize the re-use of fixtures.
However, this assumes that all fixtures have an equal cost to
generate, and no caches outside of those managed by pytest. A fixture
for a `tvm.testing.parameter` is effectively free, while a fixture
maintaining a cache of reference data
`tvm.testing.utils._fixture_cache` be quite large.
Since most of the TVM fixtures are specific to a python function, sort
the test ordering by python function, so that
tvm.testing.utils._fixture_cache can be cleared sooner rather than
later.
* Updated TestTargetAutoParametrization
When sorting the tests, the order of parametrizations may change.
Therefore, the tests checking for automatic target parametrization
shouldn't depend on order.
* [UnitTest] Added ids argument to tvm.testing.parameters
This matches the usage in `tvm.testing.parameter`, and allows for
parameter sets to be referred to by a single name.
* [Pytest] Fixed ordering issue of tvm.testing.parametrize_targets and known_failing_targets
If an explicit list of targets is given, then the
`known_failing_targets` decorator would fail to apply. This commit
resolves the issue, and cleans up all target-specific marks to apply
in `tvm.testing.plugin._add_target_specific_marks`.
* [UnitTest][Vulkan] Runnable relay unit tests on Vulkan
This commit allows the relay test suite to be run targeting Vulkan with
`TVM_TEST_TARGETS="vulkan -from_device=0" pytest tests/python/relay`. All
tests that require a specific environment are skipped if that environment
isn't present. All tests that are known to fail when running on Vulkan
are marked as expected failure, and will be tracked in
https://github.com/apache/tvm/issues/8903.
- Failures during code generation
- Type mismatches, boolean vs int8
- tests/python/relay/test_any.py::test_any_reduce
- tests/python/relay/test_op_level3.py::test_sparse_reshape
- tests/python/relay/test_op_level4.py::test_reduce_functions
- tests/python/relay/test_vm.py::test_cond
- tests/python/relay/test_vm.py::test_simple_if
- Incorrect strategy selection, picks NCHWc implemenation for NHWC layout
- tests/python/relay/test_op_level2.py::test_conv2d_run
- Unresolved CallNode operation
- tests/python/relay/test_op_level1.py::test_unary_op[erf/tan/atan]
- tests/python/relay/test_op_level3.py::test_scatter_add
- tests/python/relay/test_op_level3.py::test_segment_sum
- Generates 64-bit calls to GLSL that have only 16-/32-bit support
- tests/python/relay/test_op_grad_level1.py::test_log_softmax_grad
- tests/python/relay/test_op_grad_level1.py::test_softmax_grad
- tests/python/relay/test_op_grad_level1.py::test_unary_op
- tests/python/relay/test_op_grad_level10.py::test_cross_entropy_grad
- Codegen raises error for variable size
- tests/python/relay/test_any.py::test_any_batch_matmul
- tests/python/relay/test_any.py::test_any_conv2d_NCHWc
- tests/python/relay/test_any.py::test_any_dense
- Failures when running
- Numeric differences (observed on GTX 1650 with NVIDIA driver)
- tests/python/relay/test_op_level3.py::test_take
- tests/python/relay/test_op_level5.py::TestCropAndResize
- tests/python/relay/test_op_level5.py::TestResize1D
- tests/python/relay/test_op_level5.py::TestResize2D
- Adds the decorator `tvm.testing.requires_ethosn`
- Marks all tests in `tests/python/contrib/test_ethosn` as requiring
ethosn instead of directly checking `ethosn_available()`. This way,
they show up as skipped rather than passing.
- Marks test_compile_tflite_module_with_external_codegen as requiring
ethosn.
* [UnitTests] Expose TVM pytest helpers as plugin
Previously, pytest helper utilities such as automatic parametrization
of `target`/`dev`, or `tvm.testing.parameter` were only available for
tests within the `${TVM_HOME}/tests` directory. This PR extracts the
helper utilities into an importable plugin, which can be used in
external tests (e.g. one-off debugging).
* [UnitTests] Refactor the plugin-specific logic out into plugin.py.
* [UnitTests] Moved marker definition out to global variable.
Shushi Hong