This PR cleans up the python API to make things more consistent
with existing python array api and torch.
Device update
- device_id => index, to be consistent with torch
- device_type => dlpack_device_type() returns int
- added type property same as torch.device
API updates:
- Move the convenient method like cpu() out into tvm runtime to keep device minimal
- tvm_ffi._init_api => tvm_ffi.init_ffi_api
- tvm_ffi.register_func => tvm_ffi.register_global_func
* [FFI][REFACTOR] Establish tvm_ffi as a standalone python module
This PR establishes tvm_ffi as a standalone python module.
The ffi is structured as a minimal pip module that can be
directly install by path or url.
examples/get_started provided a minimal example.
This is a major change as we are decoupling tvm_ffi as a
separate package, users need to install tvm_ffi separately.
Thanks to its minimal dependency, tvm_ffi can be easily installed
even just from the source by pip install ./ffi
This change would enable future improvement for library plugins
to have lightweight dependencies by just working on top of
the tvm_ffi, while the main compiler toolchain and runtime
can be layered on top.
* [FFI] Improve traceback setups
This PR improves traceback related setups
This PR modernizes the FFI foundation of the project and introduce
a new minimal and lightweight module [tvm ffi](https://github.com/apache/tvm/tree/refactor-s3/ffi)
based on our lessons in the past few years. It implements a modern
version of the [Unified Packed and Object RFC](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0097-unify-packed-and-object.md)
that unifies the packed function call and object systems.
Summary of the change:
- A dedicated clean Any/AnyView that can store strong and weak
references of items
- Function(previously PackedFunc) system built on top of the Any/AnyView
- A minimal C API that backs the overall calls. We are stabilizing the
API with a goal to bring clean, stable FFI conventions for both compiled
and registered code
- A rewrite of core python binding and generated code based on the module
- Update existing code and test cases to the new module
- Latest dlpack support
The new module brings many benefits thanks to the cleaner design,
to name a few:
- Any can support both POD types(int) and object types.
- Containers (e.g. Array) can now also contain Any value, e.g. now
`Array<int>` is supported, no need for boxed types
- Error handling now upgrades to object-based, allowing cleaner
traceback across languages
- Map now preserves insertion orders
- Path toward isolated stabilize minimum core ABI/API foundation module
- Type traits based design that cleanly defines how values interact
with Any system
- Automatic conversion of different types based on traits if needed
Because FFI upgrade is at heart of the project, the change touches every
component of the system. Importantly, this is an upgrade of the ABI so the
change is not backward compatible. The code compiled under the old
FFI won't work under the new one. We did provide example ABI translation
(e.g. LegacyTVMArgValueToFFIAny) functions for compatibility.
The PR tries to leave files in their old places while creating redirections.
The goal is to have the first milestone landed and infrastructure in place,
so we can do further refactors to complete features and cleanup legacy code
as trackable PRs. As of now, python binding and compiled code are under the
new convention while RPC and some other bindings still relies on legacy ABI
translation. We will work on upgrades in the coming PRs, including areas such
as reflection, phasing out legacy redirections etc.
* Revert "Revert "[FFI][RUNTIME] Introduce runtime boxed types for int/float/bool" (#17252)"
This reverts commit 11be832620.
* [FFI] Re-introduce the boxed primitive values
Initially introduced in https://github.com/apache/tvm/pull/16183,
these changes were reverted in
https://github.com/apache/tvm/pull/17252 due to performance
degredation in some Relax models. This could occur when a model
contained a large number of calls to `"vm.builtin.tuple_getitem"`,
which may occur when model weights are provided as a tuple.
This PR re-applies the changes from
https://github.com/apache/tvm/pull/16183, but with the performance
degredation resolved. The root cause was unnecessary type-checking
when converting from an untyped `tvm::ArrayNode*` to the typed
`tvm::Array<T>`, in the case where `T` is `ObjectRef`.
* Correct typo from T to U
* [Container] Support non-nullable types in Array::Map
Prior to this commit, the `Array::Map` member function could only be
applied to nullable object types. This was due to the internal use of
`U()` as the default value for initializing the output `ArrayNode`, where
`U` is the return type of the mapping function. This default
constructor is only available for nullable types, and would result in
a compile-time failure for non-nullable types.
This commit replaces `U()` with `ObjectRef()` in `Array::Map`,
removing this limitation. Since all items in the output array are
overwritten before returning to the calling scope, initializing the
output array with `ObjectRef()` does not violate type safety.
* [FFI] Separate runtime types from IR types for int/float/bool
Prior to this commit, `int`, `float`, and `bool` arguments from Python
were converted to `IntImm`, `FloatImm`, and `Bool`. These are
subtypes of `PrimExpr`, and should only be used at compile-time. By
automatically applying this conversion as part of the FFI, these types
are required to be present whenever a primitive is converted to a
`tvm::ObjectRef`.
This can become especially fragile for an end-user when storing
objects into a TVM container. Because TVM containers require all
contents to be `ObjectRef` subclasses, an automatic conversion may be
applied on storing into a container, resulting in an unexpected type
being retrieved from the container. For example, this currently
occurs in Relax when extracting a `R.Prim` from a `R.Tuple`.
This commit introduces a `Box<T>` type for storage of boxed primitives
at runtime, distinct from the IR types.
* Primitive arguments provided to a PackedFunc that requires an
`ObjectRef` will be converted to the corresponding boxed type.
(e.g. Passing a Python `int` to a C++ function accepting `ObjectRef`
produces a `Box<int64_t>`.
* Boxed primitives provided to a PackedFunc that requires an unboxed
primitive will be converted to the corresponding primitive.
* PackedFunc return values of `ObjectRef` are converted to the
corresponding primitive, if present. (e.g. If a `tuple_getitem`
with static return type `ObjectRef` returns a `Box<int64_t>`, it
will be unwrapped to a python `int`.)
Together, these three rules provide backwards compatibility for
existing PackedFunc definitions, while avoiding exposing the user to
any container-induced type conversions betweeen primitive types and
`ObjectRef`.
* Fix unit test failure after merge
* Fix breakage in new unit test
Prior to this PR, though the `convert` function is capable of
converting a single Python function/lambda to TVM func, it is not able
to convert a container whose values inside are functions to TVM
object.
This PR adds function conversion to `convert_to_object` and redirects
`convert` to `convert_to_object`, so that now the conversion is always
recursive, and therefore will work well on function container value
type.
Co-authored-by: Chaofan Lin <1713833595@qq.com>
* Add more strict check in tir imm construction and folding.
* fix bool-compare compile error
* fix some illegal imm construction in testcases
* do not test i64 overflow behaviour because it is not consistent on cython and ctypes
* fix float32 testcase
* auto-inferred dtype should be int64 when value exceeds int32 range
* add floatimm range check for fp16 and fp32
* add more folding testcases and fix store fp32 folding result to double
* fix i386 fp16 cases
* [microNPU] Move the compilation to use Target Hooks.
This commits moves the current compilation flow
to use target hooks, so that the generated TIR
is provided to unified module to for unified
optimizations.
Change-Id: Ib3239a04ab201748e7f1b1ffa503cfe2aa7ccb7b
* [microNPU] Move the compilation to use Target Hooks.
*Fixing unpacked API tests
*Adding use_device_api target attr to example target hooks
Change-Id: I72c51caa57e9a0c2a538f40eb73939e28d4f112f
* [microNPU] Move the compilation to use Target Hooks.
* Modifed CLZ test case to support target hooks
* Modifed reference TIR for test to include allocate annotation
* TIR to CS translation tests are modified to run MakeUnpackedAPI
Change-Id: I3a3d28777a6995e7f2b8789e14c5cb0f280dc763
* [microNPU] Move the compilation to use Target Hooks.
* Added a missed documentation to changes in source module
* Skipping device api test for packed API as microNPU does not
support it.
Change-Id: I6da1adcf8fdd3f972ec9b37ff530ff673e93058c
* [microNPU] Move the compilation to use Target Hooks.
* fixed tvmc test use unpacked-api for microNPU compilation
Change-Id: Ib722d91ca3b3e4c6d13075ee0873acb86f487247
* [microNPU] Move the compilation to use Target Hooks.
* adjust target name.
Change-Id: I862957324440705fb6093939b97b1a00fa1d4b46
* [microNPU] follow up on using target hooks
* Fixed few typos and cleaned up as per suggestions
Change-Id: I2a744a4bc4015e1884dbef4165252aa13aa30b31
* [microNPU] follow up on using target hooks
Fixing some typos and change params to
const_dict as it seems more clearer
Change-Id: Ia36a4635a68f6490bcc3eeaa72eeeeaadb6aa7f6
* [microNPU] Move the compilation to use Target Hooks.
Fixing up lookup table tests to use new runtime module
import structure resulted from using target hooks.
Change-Id: I250aedef7cc73edad3812bb7e9aab013ed8bed5b
* [RUNTIME] Move Map into runtime
This allows us to use Map to store parameters needed at runtime.
* node.{Array|Map} -> runtime.{Array|Map}
* missed some renames
To make runtime.String to work as naturally as possible in the python side,
we make it sub-class the python's str object. Note that however, we cannot
sub-class Object at the same time due to python's type layout constraint.
We introduce a PyNativeObject class to handle this kind of object sub-classing
and updated the FFI to handle PyNativeObject classes.
* [RUNTIME] Introduce RValue reference(move) support to TypedPackedFunc
This PR introduces RValue reference support the PackedFunc calling convention to address the above issue.
Specifically, when an argument is a r-value reference, we will use a assign a different type code(`kObjectRValueRefArg`),
and pass `Object**` (the address to the Object pointer) instead through the values array.
The callee can choose to move out this Object pointer and set the original Object pointer from the caller side to be nullptr.
We also add an experimental move support to the python side(marked as _move so to indicate the dev nature).
This enhancement will enable copy on write optimizations through out the TVM stack.
* Address review comments
* fix compilation
* [REFACTOR][PY-API] Polish tvm.runtime, tvm.runtime.module API update
This PR updates the tvm.runtime to use the new FFI style.
- Remove top-level tvm.module to avoid confusion between runtime.Module and IRModule
- API changes wrt to runtime.Module
- tvm.module.load -> tvm.runtime.load_module
- tvm.module.enabled -> tvm.runtime.enabled
- tvm.module.system_lib -> tvm.runtime.system_lib
- Remove dep on api_internal from runtime.
* Update module.load in the latest API
* [REFACTOR][PY] Establish tvm.runtime
This PR establishes the tvm.runtime namespace that contains the core runtime data structures.
The top-level API are kept inact for now via re-exporting.
We will followup later to cleanup some of the top-level APIs.
* Fix ndarray name
Tianqi Chen