## Summary
Compiler Targets can carry device-type semantics that runtime
device-name parsing does not preserve.
- add `tvm.device_from_target` for canonical Target-to-Device
translation
- use explicit runtime constructors where the device kind is fixed
- update target-derived utilities, tests, and documentation to use the
explicit boundary
IR module cleanup benefits from using a single unique-name primitive
directly at module call sites. This PR renames NameSupply to
UniqueNameSupply and removes redundant wrappers around global variable
naming.
Main changes:
- Rename the public name supply API and header to UniqueNameSupply
- Replace GlobalVarSupply with direct iterator-seeded UniqueNameSupply
usage
- Remove obsolete access-path repr registration now covered by tvm-ffi
## Summary
`tvm::runtime::regex_match` was a thin C++ wrapper that bounced through
a
global `ffi::Function` back into Python's `re.match`. It was introduced
solely to avoid pulling `<regex>` into TVM (libstdc++ dual-ABI conflict
with
pre-cxx11 pytorch wheels). The only C++ caller is the DNNL JSON runtime,
where
every pattern reduces to substring containment — `re.match` anchors at
the
start only, so `.*X.*` is equivalent to `s.find(X) != npos`.
- Remove `src/runtime/regex.{h,cc}` and the Python
`tvm.runtime.regex_match`
global registration.
- Add file-local `contains` / `contains_any` helpers in
`dnnl_json_runtime.cc`
and inline `std::string::find` at the 15 call sites.
- Drop the dead `regex.h` include from
`src/relax/transform/update_param_struct_info.cc`.
No CMakeLists.txt change needed — `src/runtime/*.cc` is picked up by
glob.
`USE_DNNL` is OFF in the ci_gpu container, so DNNL-specific runtime
tests
are not exercised locally. The DNNL translation unit compiles cleanly
with
the inlined helpers, and the full TVM build (636 targets) passes.
## Summary
This PR adds the initial TIRx support needed for low-level programming
of Blackwell-class GPU architectures. As part of the ongoing TIRx
refactor, it introduces TVMScript support for directly scripting
advanced hardware features without relying on scheduling as the primary
programming interface.
The change keeps existing `s_tir` script support intact while making
direct scripting a first-class path for TIRx programs.
## Main Changes
- Add TIRx operator dispatch and layout infrastructure.
- Add TVMScript support for new low-level TIRx operations.
- Add analysis, transform, and lowering support for TIRx IR nodes.
- Add CUDA/Blackwell-oriented codegen and intrinsic coverage.
- Add Python and C++ integration points for TIRx scripting and runtime
support.
## Validation
- `pre-commit run --all-files`
- `ninja -C build -j32`
- `CUDA_VISIBLE_DEVICES=2 pytest tests/python/tirx/ -n 16`
- `1723 passed, 47 skipped, 32 warnings`
- `CUDA_VISIBLE_DEVICES=2 python -m pytest -v
tests/python/all-platform-minimal-test`
- `37 passed, 105 skipped`
- `TVM_TEST_TARGETS=llvm python -m pytest -v tests/python/tirx-analysis
tests/python/tirx-base tests/python/tirx-transform -n 16`
- `664 passed, 25 skipped, 9 xfailed, 1 xpassed`
## Local CI Notes
Some full CI-equivalent jobs were not locally reproducible because this
machine is missing parts of the Apache TVM CI environment, including
`llvm-config-15/17`, Vulkan, ROCm, Maven, Sphinx, Doxygen, Emscripten,
and ARM/QEMU cross-toolchain components. Metal-specific tests were
skipped locally because no Metal runtime is available.
## Summary
TVM-side cleanup that drops the `python/tvm/runtime/object.py` shim and
routes `tvm.runtime.Object` directly to `tvm_ffi.Object`. The
`tvm.runtime.Object` re-export is preserved (now a re-export of
`tvm_ffi.Object`) so external callers keep working.
The load-bearing `__object_repr__` install — which wires TVM IR objects
up to the rich C++ `ReprPrinter` registered through
`init_ffi_api("node", ...)` — moves into
`python/tvm/runtime/_ffi_node_api.py`.
That module is already imported as a side-effect-only module from
`python/tvm/runtime/__init__.py`, so the override fires at the right
time (after `init_ffi_api` registers the C++ printer).
`_ffi_node_api.AsRepr` itself is **kept**: `tvm_ffi`'s default repr is
primitive (`ClassName(ptr)`); TVM IR objects need the rich printer
registered via `init_ffi_api("node", ...)`. `AsRepr` is what bridges
that printer back into Python `repr(obj)` and is also the runtime-only
fallback when `libtvm.so` is unavailable.
The 7 in-tree importers of the deleted shim (plus one straggler in
`runtime/disco/session.py`) are switched to either
`from tvm.runtime import Object` or `from tvm_ffi import Object`,
depending on which pattern the file already uses.
## Test plan
- [x] `python -c "import tvm; print(repr(tvm.IRModule({})))"` produces
TVMScript-style output (rich repr preserved).
- [x] `pytest tests/python/all-platform-minimal-test/ -x` — 75 passed,
77 skipped (matches baseline).
- [x] `pytest tests/python/tirx-base/ -x` — 273 passed, 2 skipped.
- [x] `pre-commit run --files <changed files>` — all hooks pass.
- [ ] CI green.
This PR cleans up the python API to make things more consistent
with existing python array api and torch.
Device update
- device_id => index, to be consistent with torch
- device_type => dlpack_device_type() returns int
- added type property same as torch.device
API updates:
- Move the convenient method like cpu() out into tvm runtime to keep device minimal
- tvm_ffi._init_api => tvm_ffi.init_ffi_api
- tvm_ffi.register_func => tvm_ffi.register_global_func
This PR Updates the NDArray => Tensor.
Both tensor and ndarray are commonly used terms.
Because the term Tensor is getting more common in the context of ML,
we do the rename to stay more aligned with torch.Tensor and DLTensor.
* [FFI][REFACTOR] Establish tvm_ffi as a standalone python module
This PR establishes tvm_ffi as a standalone python module.
The ffi is structured as a minimal pip module that can be
directly install by path or url.
examples/get_started provided a minimal example.
This is a major change as we are decoupling tvm_ffi as a
separate package, users need to install tvm_ffi separately.
Thanks to its minimal dependency, tvm_ffi can be easily installed
even just from the source by pip install ./ffi
This change would enable future improvement for library plugins
to have lightweight dependencies by just working on top of
the tvm_ffi, while the main compiler toolchain and runtime
can be layered on top.
* [FFI] Improve traceback setups
This PR improves traceback related setups
This PR modernizes the FFI foundation of the project and introduce
a new minimal and lightweight module [tvm ffi](https://github.com/apache/tvm/tree/refactor-s3/ffi)
based on our lessons in the past few years. It implements a modern
version of the [Unified Packed and Object RFC](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0097-unify-packed-and-object.md)
that unifies the packed function call and object systems.
Summary of the change:
- A dedicated clean Any/AnyView that can store strong and weak
references of items
- Function(previously PackedFunc) system built on top of the Any/AnyView
- A minimal C API that backs the overall calls. We are stabilizing the
API with a goal to bring clean, stable FFI conventions for both compiled
and registered code
- A rewrite of core python binding and generated code based on the module
- Update existing code and test cases to the new module
- Latest dlpack support
The new module brings many benefits thanks to the cleaner design,
to name a few:
- Any can support both POD types(int) and object types.
- Containers (e.g. Array) can now also contain Any value, e.g. now
`Array<int>` is supported, no need for boxed types
- Error handling now upgrades to object-based, allowing cleaner
traceback across languages
- Map now preserves insertion orders
- Path toward isolated stabilize minimum core ABI/API foundation module
- Type traits based design that cleanly defines how values interact
with Any system
- Automatic conversion of different types based on traits if needed
Because FFI upgrade is at heart of the project, the change touches every
component of the system. Importantly, this is an upgrade of the ABI so the
change is not backward compatible. The code compiled under the old
FFI won't work under the new one. We did provide example ABI translation
(e.g. LegacyTVMArgValueToFFIAny) functions for compatibility.
The PR tries to leave files in their old places while creating redirections.
The goal is to have the first milestone landed and infrastructure in place,
so we can do further refactors to complete features and cleanup legacy code
as trackable PRs. As of now, python binding and compiled code are under the
new convention while RPC and some other bindings still relies on legacy ABI
translation. We will work on upgrades in the coming PRs, including areas such
as reflection, phasing out legacy redirections etc.
This refactor introduces a base Executable class and a `tvm.compile`
interface that can be used to compile both TIR and Relax programs.
`tvm.compile` will return an Executable object that can be used to call
either TIR or Relax functions.
This PR starts the step 0 to phase out relay from the current
development main branch. This PR focuses on the python
components of relay, autotvm, auto_scheduler. To make the change
manageable, we will also do followup steps on te.Schedule and
c++ components in followup PRs.
To continue support community members who depends on
legacy flows, the [v0.19.0](https://github.com/apache/tvm/tree/v0.19.0)
branch will continue contain these components.
As noted in [discussion on phasing out legacy components](https://discuss.tvm.apache.org/t/phasing-out-legacy-components/17703/30),
this would help us to do two purposes:
- By removing outdated or redundant elements, we can significantly
reduce complexity and improve maintainability.
- Unify our focus: Concentrating our efforts on the new unity flow
will allow for more efficient development and innovation.
It is also a good opportunity for us to revisit and reduce CI time.
The past relay legacy flow contains a lot of end to end tests that
requires hardware resources to run and causing long CI time.
Moving onwards, we can focus more on unit-tests that focuses
on structural equality and runs within seconds, while be mindful
about tests that requires hardware resources (by restricting them
to specific folders and CI nightly in some cases).
---
Co-authored-by: Siyuan Feng <hzfengsy@sjtu.edu.cn>
* Revert "Revert "[FFI][RUNTIME] Introduce runtime boxed types for int/float/bool" (#17252)"
This reverts commit 11be832620.
* [FFI] Re-introduce the boxed primitive values
Initially introduced in https://github.com/apache/tvm/pull/16183,
these changes were reverted in
https://github.com/apache/tvm/pull/17252 due to performance
degredation in some Relax models. This could occur when a model
contained a large number of calls to `"vm.builtin.tuple_getitem"`,
which may occur when model weights are provided as a tuple.
This PR re-applies the changes from
https://github.com/apache/tvm/pull/16183, but with the performance
degredation resolved. The root cause was unnecessary type-checking
when converting from an untyped `tvm::ArrayNode*` to the typed
`tvm::Array<T>`, in the case where `T` is `ObjectRef`.
* Correct typo from T to U
* [Container] Support non-nullable types in Array::Map
Prior to this commit, the `Array::Map` member function could only be
applied to nullable object types. This was due to the internal use of
`U()` as the default value for initializing the output `ArrayNode`, where
`U` is the return type of the mapping function. This default
constructor is only available for nullable types, and would result in
a compile-time failure for non-nullable types.
This commit replaces `U()` with `ObjectRef()` in `Array::Map`,
removing this limitation. Since all items in the output array are
overwritten before returning to the calling scope, initializing the
output array with `ObjectRef()` does not violate type safety.
* [FFI] Separate runtime types from IR types for int/float/bool
Prior to this commit, `int`, `float`, and `bool` arguments from Python
were converted to `IntImm`, `FloatImm`, and `Bool`. These are
subtypes of `PrimExpr`, and should only be used at compile-time. By
automatically applying this conversion as part of the FFI, these types
are required to be present whenever a primitive is converted to a
`tvm::ObjectRef`.
This can become especially fragile for an end-user when storing
objects into a TVM container. Because TVM containers require all
contents to be `ObjectRef` subclasses, an automatic conversion may be
applied on storing into a container, resulting in an unexpected type
being retrieved from the container. For example, this currently
occurs in Relax when extracting a `R.Prim` from a `R.Tuple`.
This commit introduces a `Box<T>` type for storage of boxed primitives
at runtime, distinct from the IR types.
* Primitive arguments provided to a PackedFunc that requires an
`ObjectRef` will be converted to the corresponding boxed type.
(e.g. Passing a Python `int` to a C++ function accepting `ObjectRef`
produces a `Box<int64_t>`.
* Boxed primitives provided to a PackedFunc that requires an unboxed
primitive will be converted to the corresponding primitive.
* PackedFunc return values of `ObjectRef` are converted to the
corresponding primitive, if present. (e.g. If a `tuple_getitem`
with static return type `ObjectRef` returns a `Box<int64_t>`, it
will be unwrapped to a python `int`.)
Together, these three rules provide backwards compatibility for
existing PackedFunc definitions, while avoiding exposing the user to
any container-induced type conversions betweeen primitive types and
`ObjectRef`.
* Fix unit test failure after merge
* Fix breakage in new unit test
Currently, the pytorch wheels available through `pip install` use the
pre-C++11 ABI by setting `-DUSE_CXX11_ABI=0` [0]. If TVM were to user
the pre-C++11 ABI, this would cause breakages with dynamically-linked
LLVM environments.
This commit adds a lint check to search for use of `#include <regex>`
in any C++ files. Use of this header should be avoided, as its
implementation is not supported by gcc's dual ABI. This ABI
incompatibility results in runtime errors either when `std::regex` is
called from TVM, or when `std::regex` is called from pytorch,
depending on which library was loaded first.
This restriction can be removed when a version of pytorch compiled
using `-DUSE_CXX11_ABI=1` is available from PyPI.
[0] https://github.com/pytorch/pytorch/issues/51039
This PR introduces `PrinterConfig`, a systematic way to configure
TVMScript printer without having to set global flags.
This PR enables more customization of printer behavior. More
specifically, now any TVM’s object in python, as long as it
inherits from `Scriptable`, it automatically gains two methods:
- `.script(tir_prefix=...)`
- `.show(...)`
Motivation: when two IR objects fail a structural equality check, currently there is no easy way to
find out which part of the IR caused the mismatch. In this PR, we modify the `StructuralEqual`
infrastructure to also optionally return a pair of `ObjectPath` objects that point to the mismatch.
(See https://github.com/apache/tvm/pull/11977). In the upcoming PRs, we will pass these paths to the
TIR printer, so that it could highlight the mismatch location nicely.
Tracking issue: https://github.com/apache/tvm/issues/11912
(See https://discuss.tvm.apache.org/t/byoc-supporting-cutlass-byoc-with-collage/12796/6 for
context, which in turn is part of Collage (https://github.com/apache/tvm-rfcs/blob/main/rfcs/0062-collage.md).
This adds a new 'DSO exportable' runtime module representing the contents of a .o file. It
allows external codegen toolchains to yield a result which:
- Like CSource modules, can be conveyed directly to the final export_library compilation
step for linking into the final .so and saved to a know location without risk the
underlying code artifact will be lost.
- Like DSOLibrary modules, are self contained so that no additional compile-time arguments
need be conveyed from the CSource module to the final export_library command line
Since this is the third flavor of 'DSO exportable' module, add a Module::IsDSOExportable.
Since adding the above, can't resist also adding a Module::ImplementsFunction virtual and
calling it from TEComplier to check if an external codegen function actually provided the
implementation it promised.
Note:
- I've left the existing implementation of runtime.load_module alone which
relinks .o files to .so files.
- Though also contained in the .o metadata, I require static libraries to always
carry their list of exported function names.
This is all pretty stop gap pending a good rework of TVM to supoprt the notion of artifacts
and, perhaps, build rules.
* [RUNTIME] Api to get number of runtime threads
Add `tvm::runtime::threading::NumThreads` and `tvm.runtime.num_threads`
as a way to get the number of threads in use by the TVM runtime.
* check if equal to hardware threads or hardware threads/2
* Add memory pools to Metadata classes.
* Move ShapeToJSON to utils.
* Track returned TensorType from AOTExecutorCodegen.
* Support calling Relay functions with Tuple.
* Expand supported TIR calling conventions to work with C++ runtime.
* Rename MetadataModule to ConstLoaderModule.
* Add runtime AOT executor module.
* Add AOT code-generation.
* Add a runtime Module to mux between .text Metadata and live Metadata.
* Move launch_param to namespace
* Add test of c++ AOT.
* Fix incongruity between kTvmRuntimeCrt constant
* Expand ExecutorCodegenMetadata to include AOT runtime metadata.
* commit cpp test
* Make Metadata compile under C.
* Ignore ephemeral metadata_module export_model_library_format.
* This module does not need to be exported, since it is merely a C++
wrapper around get_c_metadata, and get_metadata is not used in C.
* address manupa, kparszsyc, masahi comments.
* further address comments
* clang and python format
* Fix broken test
* Address lingering comments from masahi, kparszyzc
* [Codegen] Use "target.build.$TARGET_KIND" for all codegen functions.
- Removed special case for "micro_dev" target. Instead, register
BuildCHost as both "target.build.c" and "target.build.micro_dev".
- Renamed "target.build.build.aocl_sw_emu" to
"target.build.aocl_sw_emu". Appears to be a typo introduced in
#841725cc585
* [micro_dev] Removed references to non-existent micro_dev
device_api.micro_dev was removed in
745e542e4d, but several references still
remained.
Co-authored-by: Eric Lunderberg <elunderberg@octoml.ai>
* [PROFILER] Add CSV output to profiler
This patch changes the profiler output from a string to a Report object.
A Report can either output CSV or the usual human-readable table.
* no spaces after commas
* Update src/runtime/profiling.cc
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
* fix gcc
* fix test
* overall percent fix
* rename overall -> device_metrics
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
We use a single enum(TypeCode) to represent ArgTypeCode and DLDataTypeCode.
However, as we start to expand more data types, it is clear that argument
type code(in the FFI convention) and data type code needs to evolve separately.
So that we can add first class for data types without having changing the FFI ABI.
This PR makes the distinction clear and refactored the code to separate the two.
- [PY] Separate ArgTypeCode from DataTypeCode
- [WEB] Separate ArgTypeCode from DataTypeCode
- [JAVA] Separate ArgTypeCode from DataTypeCode
* [REFACTOR][TIR] Migrate all low-level passes to the Pass Manager.
This PR migrates the tvm.lower to return IRModule of PrimFuncs
instead of the LoweredFuncs.
* Remove LoweredFunc.
* [REFACTOR][PY-API] Polish tvm.runtime, tvm.runtime.module API update
This PR updates the tvm.runtime to use the new FFI style.
- Remove top-level tvm.module to avoid confusion between runtime.Module and IRModule
- API changes wrt to runtime.Module
- tvm.module.load -> tvm.runtime.load_module
- tvm.module.enabled -> tvm.runtime.enabled
- tvm.module.system_lib -> tvm.runtime.system_lib
- Remove dep on api_internal from runtime.
* Update module.load in the latest API
* [REFACTOR][PY] Establish tvm.runtime
This PR establishes the tvm.runtime namespace that contains the core runtime data structures.
The top-level API are kept inact for now via re-exporting.
We will followup later to cleanup some of the top-level APIs.
* Fix ndarray name
Tianqi Chen