This PR slims `tvm.libinfo` into a thin *info* layer that delegates path
discovery to the `tvm_ffi.libinfo` primitives and never loads libraries.
Loading responsibilities move to `tvm.base`, and the various ad-hoc
path-finding helpers are phased out in favor of the tvm-ffi resolvers.
## Changes
- **libinfo**: add `find_libtvm_runtime()` (resolves `libtvm_runtime`
via
`_find_library_by_basename` + `_resolve_and_validate`) and
`find_tvm_include_path()` (TVM's own `include/`). `find_include_path()`
now
returns `[find_tvm_include_path(), *tvm_ffi.libinfo.include_paths()]`,
folding
in the FFI + dlpack + python-helper include dirs. Remove
`find_lib_path`,
`get_dll_directories`, `use_runtime_lib`, `split_env_var`, and
`load_backend_libs`.
- **base**: receive `load_backend_libs` and the backend DSO list; the
runtime-only switch becomes a strict `TVM_USE_RUNTIME_LIB == "1"` check.
- **rpc**: `with_minrpc` uses `find_libtvm_runtime()` (the `runtime`
kwarg is
retained as an inert back-compat parameter); the rpc server
`load_library`
resolves the literal library name against the current working directory.
- **wasm**: move the `web/dist` asset search into `emcc.find_wasm_lib`,
used by
`emcc.create_tvmjs_wasm` and the tvmjs asset lookup.
- **hexagon**: fix a latent bug where `_get_hexagon_rpc_lib_dir` called
a
non-existent `tvm_ffi.libinfo.find_lib_path`; it now relies solely on
the
`HEXAGON_RPC_LIB_DIR` environment variable.
## Summary
Lifts 10 host-toolchain / CLI / process / utility modules from
`python/tvm/contrib/` to a new `python/tvm/support/` package, and
deletes two dead contrib shims.
`tvm.support` is the home for Python helpers that integrate TVM with
external CLIs and host-side tools — compilers, archivers, subprocess
pools, and build-info queries. These are load-bearing internal pieces
that TVM's compile/link/run paths depend on. `tvm.contrib` is reserved
for optional vendor SDK integrations and experimental features. The
distinction is documented in the `tvm.support` package docstring.
Moved (one commit each):
- `tvm.contrib.cc` → `tvm.support.cc`
- `tvm.contrib.nvcc` → `tvm.support.nvcc`
- `tvm.contrib.rocm` → `tvm.support.rocm`
- `tvm.contrib.ndk` → `tvm.support.ndk`
- `tvm.contrib.xcode` → `tvm.support.xcode`
- `tvm.contrib.clang` → `tvm.support.clang`
- `tvm.contrib.emcc` → `tvm.support.emcc`
- `tvm.contrib.popen_pool` → `tvm.support.popen_pool`
- `tvm.contrib.utils` → `tvm.support.utils`
- `tvm.contrib.tar` → `tvm.support.tar`
Deleted:
- `tvm.contrib.spirv` — single `optimize()` wrapping `spirv-opt`; zero
importers.
- `tvm.contrib.rpc` — self-deprecation shim with "removed in 0.5"
banner; honoring it.
Package conversion:
- `python/tvm/support.py` → `python/tvm/support/__init__.py` with
inclusion-rule docstring.
- `libinfo()` extracted into `python/tvm/support/libinfo.py`.
- `FrontendTestModule` dropped (audit confirmed zero callers outside its
own definition).
## Compatibility
Hard break — no `tvm.contrib.<mod>` re-export shims. All callers updated
in this PR.
C++-side FFI registry keys (`tvm.contrib.nvcc.*`, etc.) are unchanged —
only the Python module path moves. Renaming the FFI keys is a separate
follow-up.
Replace `str(target.kind)` with `target.kind.name` for `Target` objects
since `target.kind` is a `TargetKind` object while `target.kind.name`
yields a string describing the target
This PR enables ruff pyupgrade (UP) rules with py310 target, auto-fixing
~5600 annotation modernizations (PEP 585 generics, PEP 604 unions,
deprecated typing imports).
Also removes from __future__ import annotations from ir/module.py and
rmsnorm.py, bumps requires-python to >=3.10, and removes absolute_import
aliases from topi/contrib files.
This PR phases out legacy target string format in favor of the json
style format that is more well formed. It also simplfies our overall
code in handling multiple formats.
This PR cleans up the python API to make things more consistent
with existing python array api and torch.
Device update
- device_id => index, to be consistent with torch
- device_type => dlpack_device_type() returns int
- added type property same as torch.device
API updates:
- Move the convenient method like cpu() out into tvm runtime to keep device minimal
- tvm_ffi._init_api => tvm_ffi.init_ffi_api
- tvm_ffi.register_func => tvm_ffi.register_global_func
* [FFI][REFACTOR] Establish tvm_ffi as a standalone python module
This PR establishes tvm_ffi as a standalone python module.
The ffi is structured as a minimal pip module that can be
directly install by path or url.
examples/get_started provided a minimal example.
This is a major change as we are decoupling tvm_ffi as a
separate package, users need to install tvm_ffi separately.
Thanks to its minimal dependency, tvm_ffi can be easily installed
even just from the source by pip install ./ffi
This change would enable future improvement for library plugins
to have lightweight dependencies by just working on top of
the tvm_ffi, while the main compiler toolchain and runtime
can be layered on top.
* [FFI] Improve traceback setups
This PR improves traceback related setups
This PR refactors and establishes ffi.Module under the python tvm ffi api.
Also moves export_library method to executable so it aligns more with
compiled artifact.
This PR formalizes original runtime::Module into ffi
as ffi.Module and cleans the APIs around it.
The goal is to stablize the Module API as extra API that can benefit the overall
ffi interactions. We also refactors the c++ code that depends on the Module.
This PR updates the entry function mechanism to create a stub that redirects to the real function.
This new behavior helps to simplify the runtime logic supporting entry function.
Also updates the name to `__tvm_ffi_main__`
This PR phases out tvm._ffi redirections in favor of new FFI
new functions are now called via tvm.ffi.
We also enabled limited API support for python 3.12+
so the compiled binary can be forward compatible to future
python versions.
This PR modernizes the FFI foundation of the project and introduce
a new minimal and lightweight module [tvm ffi](https://github.com/apache/tvm/tree/refactor-s3/ffi)
based on our lessons in the past few years. It implements a modern
version of the [Unified Packed and Object RFC](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0097-unify-packed-and-object.md)
that unifies the packed function call and object systems.
Summary of the change:
- A dedicated clean Any/AnyView that can store strong and weak
references of items
- Function(previously PackedFunc) system built on top of the Any/AnyView
- A minimal C API that backs the overall calls. We are stabilizing the
API with a goal to bring clean, stable FFI conventions for both compiled
and registered code
- A rewrite of core python binding and generated code based on the module
- Update existing code and test cases to the new module
- Latest dlpack support
The new module brings many benefits thanks to the cleaner design,
to name a few:
- Any can support both POD types(int) and object types.
- Containers (e.g. Array) can now also contain Any value, e.g. now
`Array<int>` is supported, no need for boxed types
- Error handling now upgrades to object-based, allowing cleaner
traceback across languages
- Map now preserves insertion orders
- Path toward isolated stabilize minimum core ABI/API foundation module
- Type traits based design that cleanly defines how values interact
with Any system
- Automatic conversion of different types based on traits if needed
Because FFI upgrade is at heart of the project, the change touches every
component of the system. Importantly, this is an upgrade of the ABI so the
change is not backward compatible. The code compiled under the old
FFI won't work under the new one. We did provide example ABI translation
(e.g. LegacyTVMArgValueToFFIAny) functions for compatibility.
The PR tries to leave files in their old places while creating redirections.
The goal is to have the first milestone landed and infrastructure in place,
so we can do further refactors to complete features and cleanup legacy code
as trackable PRs. As of now, python binding and compiled code are under the
new convention while RPC and some other bindings still relies on legacy ABI
translation. We will work on upgrades in the coming PRs, including areas such
as reflection, phasing out legacy redirections etc.
This PR phases out the stackvm implementation.
StackVM historically was used for some host codegen.
As we move towards more comprehensive host codegen support,
we can leverage LLVM and C for that purpose, so it can be helpful
to phase it out to simplify maintainace.
* [Module] Implement custom imported modules serialization
When a module with imported modules is exported into a shared library,
the imported modules are serialized and embedded inside of that library.
This is done by generating a raw binary from the imported modules, which
is then assigned to a symbol `__tvm_dev_mblob` in the final shared
library.
The way it happens for targets that are not "llvm", is by creating a C
source file, and defining a statically-initialized array `__tvm_dev_mblob`
in it. The static initializer is the byte-by-byte hexadecimal represen-
tation of the serialized modules. While working with Hexagon, this has
presented us with two issues:
1. For most models, the embedded data is very large, taking significant
amout of time to compile the auto-generated C source.
2. There are some models for which the C source size has exceeded clang's
limits on the size of the input file, making it impossible to export the
corresponding module.
This PR allows users to provide a custom serialization routine to
`Module.export_library`. We then apply it in Hexagon to build the ELF
object file with `__tvm_dev_mblob` by using the objcopy tool from the
Hexagon toolchain. This bypasses the C compilation altogether, avoiding
both of the issues.
Using the same custom mechanism, similar method can be implemented for
many other targets which have utilities to manipulate object files
directly.
* Allow toolchain version 8.5.x as well: objcopy has necessary options
It seems like older toolchains may work as well, but the oldest supported
SDK is 4.5.0.3, which contains toolchain version 8.5.08.
This PR introduces an optional cache flush functionality to
`time_evaluator`. It is implemented by allocating two large empty
NDArrays on the device so that the L2 cache are flushed. This gives us
more accurate evaluation on the performance of a runtime function.
This PR enables combination of multiple system lib
into the same static library with a system_lib_prefix attribute.
This can open doors for multiple models to be compiled separately
then packaged into the same app via static library.
It resolves a previous issue that prevents multiple system
lib to be linked together when they come with extra binary component
such as CUDA due to symbol conflict.
Readonly detection can cause the kernel arg order
to be different from other shaders, add options to
optionally skip it.
Also makes export auto use emcc for wasm target.
Currently, we only classify whether the runtime module is DSO-exportable.
This PR further classifies each runtime module into the properties as follows:
- `kBinarySerializable`: we can serialize the module to the stream of bytes. CUDA/OpenCL/JSON runtime are representative examples.
- `kRunnable`: we can run the module directly. LLVM/CUDA/JSON runtime, executors (e.g, virtual machine) runtimes are runnable. Non-runnable modules, such as CSourceModule, requires a few extra steps (e.g,. compilation, link) to make it runnable.
- `kBinaryExportable`: when the module is kBinarySerializable and kRunnable, we consider this module as binary exportable. A binary exportable module can be integrated into final runtime artifact by being serialized as data into the artifact, then deserialzied at runtime. This class of modules must implement `SaveToBinary`, and have a matching deserializer registered as `runtime.module.loadbinary_<type_key>`.
- `kDSOExportable`: we can export the module as DSO. A DSO exportable module (e.g., a CSourceModuleNode of type_key 'c') can be incorporated into the final runtime artifact (ie shared library) by compilation and/or linking using the external compiler (llvm, nvcc, etc). DSO exportable modules must implement `SaveToFile`.
Please note that `kDSOExportable` is a mutual exclusive property with `kBinaryExportable`.
For some operations such as `__nop` or `__copy` the measured inference
time is equal to 0. In this case we are in infinite loop and we won't
exit from it. Added new parameter `limit_zero_time_iterations ` which specify the
maximum number of repeats then the inference time is equal to 0. When
we exceed this value then we will exit from a loop.
The blob-embedding code creates a new LLVM module for which is needs more
information than just the target triple. The `_get_target_triple` function
in LLVMModule returned the triple with additional options appended to the
string. Instead of piggy-backing those extra options on top of the triple,
replace `_get_target_triple` with `_get_target_string`, which will return
the entire target string.
(See https://discuss.tvm.apache.org/t/byoc-supporting-cutlass-byoc-with-collage/12796/6 for
context, which in turn is part of Collage (https://github.com/apache/tvm-rfcs/blob/main/rfcs/0062-collage.md).
This adds a new 'DSO exportable' runtime module representing the contents of a .o file. It
allows external codegen toolchains to yield a result which:
- Like CSource modules, can be conveyed directly to the final export_library compilation
step for linking into the final .so and saved to a know location without risk the
underlying code artifact will be lost.
- Like DSOLibrary modules, are self contained so that no additional compile-time arguments
need be conveyed from the CSource module to the final export_library command line
Since this is the third flavor of 'DSO exportable' module, add a Module::IsDSOExportable.
Since adding the above, can't resist also adding a Module::ImplementsFunction virtual and
calling it from TEComplier to check if an external codegen function actually provided the
implementation it promised.
Note:
- I've left the existing implementation of runtime.load_module alone which
relinks .o files to .so files.
- Though also contained in the .o metadata, I require static libraries to always
carry their list of exported function names.
This is all pretty stop gap pending a good rework of TVM to supoprt the notion of artifacts
and, perhaps, build rules.
* [RUNTIME] Api to get number of runtime threads
Add `tvm::runtime::threading::NumThreads` and `tvm.runtime.num_threads`
as a way to get the number of threads in use by the TVM runtime.
* check if equal to hardware threads or hardware threads/2
* introduce profile_all_alignments option
* add profile_all_alignment option to API
* wip
* fixed dynamic case
* black
* update gen_gemm too
* minor improvement
* fix
* all tests work
* add doc
* fixed for sm = 75 case
* fix typo
* remove unused import
* profile_all -> find_first_valid
* fix
* Allow export of C++ kernels using correct file extension
* [WIP] Set module_key=c for CSourceCrtMetadataModuleNode to temporarily fix failing tests
I realized that the module format `cc` is currently already used by the `CSourceCrtMetadataModuleNode` declared in `src/target/source/source_module.cc`.
This needs to be discussed first to decide if either the module_key should be changed or the test cases expecting the systemlib kernel (e.g. `default_lib0.c`) to have a `.c` extension.
* Update Makefiles used by tests/python/relay/aot/ to support C++ file extensions
AOT: Add c++ support to aot_test.mk
AOT: Add c++ support to corstone300.mk
* Add missing definition of GetFormat to cmsisnn and ethosn codegens (WIP)
* Resolve PR comments
* lint python/tvm/runtime/module.py
* fix EthosUModuleNode for CI
* Fix: detect empty module.format
* Add error message to assertion
* Lint python/tvm/runtime/module.py
* [GRAPH EXECUTOR,VM] Add benchmarking function to graph executor and vm
This new benchmarking function is just a convenience function for
calling time_evaluator on the underlying module. Hopefully this should
make it easier for users to get good benchmarks of their code.
* formatting
* import order
* more test, more comments, more precision
* fix tests
* add seconds descriptions to doc
* [Codegen] Use "target.build.$TARGET_KIND" for all codegen functions.
- Removed special case for "micro_dev" target. Instead, register
BuildCHost as both "target.build.c" and "target.build.micro_dev".
- Renamed "target.build.build.aocl_sw_emu" to
"target.build.aocl_sw_emu". Appears to be a typo introduced in
#841725cc585
* [micro_dev] Removed references to non-existent micro_dev
device_api.micro_dev was removed in
745e542e4d, but several references still
remained.
Co-authored-by: Eric Lunderberg <elunderberg@octoml.ai>
* Get basic verison of VM RPC working
* Test case passes
* Clean up PR
* Lint
* Format
* Address Andrew R and TK feedback
* Add comment for Andrew
* Address Zhi's comment
* Format
* Fix broken test
* Some docstring fixes.
* Couple of small fixes:
- Use `west attach` instead of `west debug` in commandline to prevent
debugger from resetting device.
- Fix warning on use of led_pin in zephyr-runtime/src/main.c.
* Adding Zephyr demo runtime.
* Cleanup of uTVM tests and demo runtime.
* Working on QEMU support.
Need to add board-specific prj.conf files.
* Adding board-specific prj.conf files.
* Some cleanup.
* Lots of hacking to get ONNX model to run on QEMU and nRF5340.
Added test_onnx unit test.
Still need to clean up tutorial.
* Adding data for unit tests.
* Cleanup demo_runtime code.
* Fix up tutorial.
* Couple of small fixes:
- Use `west attach` instead of `west debug` in commandline to prevent
debugger from resetting device.
- Fix warning on use of led_pin in zephyr-runtime/src/main.c.
* Adding Zephyr demo runtime.
* Cleanup of uTVM tests and demo runtime.
* Working on QEMU support.
Need to add board-specific prj.conf files.
* Adding board-specific prj.conf files.
* Some cleanup.
* Lots of hacking to get ONNX model to run on QEMU and nRF5340.
Added test_onnx unit test.
Still need to clean up tutorial.
* Lots of hacking to get ONNX model to run on QEMU and nRF5340.
Added test_onnx unit test.
Still need to clean up tutorial.
* Adding data for unit tests.
* Cleanup demo_runtime code.
* Fix up tutorial.
* Fix tutorial.
* Fix tutorial and runtime.
* Fix merge conflicts.
* Fix merge conflict.
* Remove redundant files.
* Revert dep.
* Fixup
* Add new files to check_file_type.py.
* Adding missing ONNX file.
* Fixup docs.
* Fix linting rule.
* small fixes
* Add missing file to check_file_type.py.
* clang-format this file.
* Fix formatting.
* Black formatting.
* Lint comments.
* Fix path for test.
* Bump CI.
* Update from_onnx.
* fix path
* Fixing
* Revert dmlc-core to 21cc7de0dc9fd6acb796e1be6181fa8e6b6c8f41
* Fix path again.
* Fix tutorial to not use actual Zephyr.
* Revert submodule version change
* Fix bad merge.
* Trying to fix this mess.
* Fix formatting.
* context -> device
* Removing tutorial since I can't get it to pass CI.
Co-authored-by: Mehrdad Hessar <mehrdad.hessar@gmail.com>
Co-authored-by: Andrew Reusch <areusch@octoml.ai>
* Introduce Model Library Format export format.
* This function produces a stable on-disk representation of TVM's
compiler output.
* It's intended just for use with the C runtime for microTVM right
now. It could be expanded for other use cases.
* This PR implements the Model Library Format RFC, which ultimately
is intended to support the Project Generator API (RFC
forthcoming).
* There may be some changes to the format without revving the version
number until downstream consumers are known. The Project Generator
API is the first such known downstream consumer.
* There are no plans currently to support generating old Model
Library Format from TVM. The version number is intended as a
compatibility check between the generator and downstream consumers.
* Created CSourceMetaData module for model metadata
* Currently, there is a MetaData module to capture constants
conditionaly if the runtime modules implement const init
PackedFuncs. However, this one relies on a load process
in which the metadata is created on volatile memory that
may be not usable in uTVM environments.
* There is a need for model level metadata that is valid
across all runtime modules such as the func registry
when creating a system-lib.
* This commit implements a CSoureMetaData module to hold
func registry that collects function names from the
runtime module and generates a c source file to be
linked with final artifact.
* Modified and added export_library for utvm
Change-Id: Ie2e8e2aea1a66520f03fe8af7cc5bdf27339ea10
* Created CSourceMetaData module for model metadata
* fixed llvm_module to return null pfs for
get_symbol and get_const_vars
Change-Id: I84810e0695d4d6fb314af2469117f965eed71b51
* Created CSourceMetaData module for model metadata
*fixed bundle_deploy tests
Change-Id: I0d1332a4abbb6830531784c59264021bbbd7148a
* Created CSourceMetaData module for model metadata
*fixed export_library not to insert "options" when targeting tar
*fixed unit tests
Change-Id: Ia1686889498b71af66f1a0311a059154ad3c2c3e
* Created CSourceMetaData module for model metadata
* enable wasm to support csource metadata module
* disabled non DSOExportables from using csource metadata module
Change-Id: Ie09beaad35cbc2ef738d1d24d91e249b5e099569
* Created CSourceMetaData module for model metadata
* changed const pfs to be called only on external modules
or DSOExportable modules
Change-Id: I6ad28f166c0fc27a2548c851bf9287ec805550d1
* Created CSourceMetaData module for model metadata
* CSourceMetadata module wrapper is only created for c/llvm targets
Change-Id: I13cb4140c17e2e1f91d495b15a1ff7eeab9fb14d
* Created CSourceMetaData module for model metadata
*target should be defined to use csourcemetdata module
Change-Id: Id8e55b23d0007a79c550334de2c0fec63d40171f
* Created CSourceMetaData module for model metadata
* reinstate llvm func registry
Change-Id: I53e0754b6fb533637f08b25e98064d8c04092de4
* Created CSourceMetaData module for model metadata
* addressed comments and fixed bugs
Change-Id: I26401685dc803aeaf7642c865df88d683419e859
* Created CSourceMetaData module for model metadata
* addressed a missed comment
Change-Id: I65e65c30bc780a946f3f1b8372c40a49a5c20582
* Created CSourceMetaData module for model metadata
* te build interface should only include c-source metadata if
targetting "c"
Change-Id: Ie23cb8c6231c1f2de6d2827084774e3510288098
* Created CSourceMetaData module for model metadata
* c_source modules should be created only if they are
non-DSO exportable
Change-Id: I53f2f8e9caa41f133446f8881b9dc541ebeee8cc
* Created CSourceMetaData module for model metadata
* documetation misalignment in source_module.cc
Change-Id: I83e2c29b1f2980ca65a694304720dc58a5cb7879
* Created CSourceMetaData module for model metadata
* typo : same object file written as a dependency in the Makefile
Change-Id: I8becc4196d286cfb6372768687b3c836799dcb78
* Created CSourceMetaData module for model metadata
* removed unused param from a brief
Change-Id: Ie4db2aca3b7ea147bd8c65ef5d1cc2146f530e76
* Created CSourceMetaData module for model metadata
* made export library use c as the format for c source modules
Change-Id: Ie2fd6204414f0fa43988a8082d18af7a3225e237
* Created CSourceMetaData module for model metadata
*addressed a nit
Change-Id: I6084b8c06ddfaaece295439dbab589e6e202b664
Tianqi Chen