This PR enables ruff pyupgrade (UP) rules with py310 target, auto-fixing
~5600 annotation modernizations (PEP 585 generics, PEP 604 unions,
deprecated typing imports).
Also removes from __future__ import annotations from ir/module.py and
rmsnorm.py, bumps requires-python to >=3.10, and removes absolute_import
aliases from topi/contrib files.
This PR cleans up the python API to make things more consistent
with existing python array api and torch.
Device update
- device_id => index, to be consistent with torch
- device_type => dlpack_device_type() returns int
- added type property same as torch.device
API updates:
- Move the convenient method like cpu() out into tvm runtime to keep device minimal
- tvm_ffi._init_api => tvm_ffi.init_ffi_api
- tvm_ffi.register_func => tvm_ffi.register_global_func
This PR Updates the NDArray => Tensor.
Both tensor and ndarray are commonly used terms.
Because the term Tensor is getting more common in the context of ML,
we do the rename to stay more aligned with torch.Tensor and DLTensor.
* [FFI][REFACTOR] Establish tvm_ffi as a standalone python module
This PR establishes tvm_ffi as a standalone python module.
The ffi is structured as a minimal pip module that can be
directly install by path or url.
examples/get_started provided a minimal example.
This is a major change as we are decoupling tvm_ffi as a
separate package, users need to install tvm_ffi separately.
Thanks to its minimal dependency, tvm_ffi can be easily installed
even just from the source by pip install ./ffi
This change would enable future improvement for library plugins
to have lightweight dependencies by just working on top of
the tvm_ffi, while the main compiler toolchain and runtime
can be layered on top.
* [FFI] Improve traceback setups
This PR improves traceback related setups
This PR renames the filenames/namespaces of `relax_vm`
to `vm`.
Previously, both VMs of relay and relax exist, and to avoid the
name conflicts, we added the prefix `relax_` to relax VM.
With the Relay runtime being phased out, we can now rename
`relax_vm` to `vm` for conciseness.
This PR starts the step 0 to phase out relay from the current
development main branch. This PR focuses on the python
components of relay, autotvm, auto_scheduler. To make the change
manageable, we will also do followup steps on te.Schedule and
c++ components in followup PRs.
To continue support community members who depends on
legacy flows, the [v0.19.0](https://github.com/apache/tvm/tree/v0.19.0)
branch will continue contain these components.
As noted in [discussion on phasing out legacy components](https://discuss.tvm.apache.org/t/phasing-out-legacy-components/17703/30),
this would help us to do two purposes:
- By removing outdated or redundant elements, we can significantly
reduce complexity and improve maintainability.
- Unify our focus: Concentrating our efforts on the new unity flow
will allow for more efficient development and innovation.
It is also a good opportunity for us to revisit and reduce CI time.
The past relay legacy flow contains a lot of end to end tests that
requires hardware resources to run and causing long CI time.
Moving onwards, we can focus more on unit-tests that focuses
on structural equality and runs within seconds, while be mindful
about tests that requires hardware resources (by restricting them
to specific folders and CI nightly in some cases).
---
Co-authored-by: Siyuan Feng <hzfengsy@sjtu.edu.cn>
[Relay][ONNX] Add Optional, OptionalHasElement and OptionalGetElement based on Sequence op
Co-authored-by: Ehsan M. Kermani <ehsanmok@users.noreply.github.com>
There is python API function 'set_output' which save external outputs in VM outputs_ field (map) for specified func name. It looks like 'set_input' method.
During 'invoke' outputs_ are saved in register_file. For this the register indices of output tensors are found from code_ field. I observed in tests for different models that AllocTensor and AllocADT ops are used for result tensors. Let's consider these two cases: result index is destination for AllocTensor op or AllocADT op. At the first case instead of construction new NDArray the outside output tensor is used. At the second one the fields of AllocADT are analyzed and register indices are extracted. During tests I observed that ReshapeTensor operation is rarely used as final one (SqueezeNet-v1.0 and DUC). Mechanism for replacement by external output tensors was also implemented for this op.
* Add methods to read and restore late-bound constants on Executable.
* Add bindings for new functions
* Cleanup
* Fix function name
* Add tests for python API to access new load/save functions
* Add another tests for python API to access new load/save functions where there are no constants
For some operations such as `__nop` or `__copy` the measured inference
time is equal to 0. In this case we are in infinite loop and we won't
exit from it. Added new parameter `limit_zero_time_iterations ` which specify the
maximum number of repeats then the inference time is equal to 0. When
we exceed this value then we will exit from a loop.
* method of creating of NDArray from external DLTensor was implemented
* set input without copying for DLTensor source
* code clean up
* update description and comments after review
Co-authored-by: Valery Chernov <valery.chernov@deelvin.com>
* set_input_with_index was implemented for VM
* clean code
* add getInputIndexFromName. add function descriptions. lint fix
* fix lint
* transfer comparison of parameter names number and assigned devices number to VMFunction constructor
* add GetVMFunctionWithName to Executable API
* clean code
* add SetInputWithName (set_input_with_name) to VM API
* join SetInputWithIndex and SetInputWithName to SetOneInputTensor (set_one_input) to VM API, the joined methods were removed
* fix lint
* some fixes after review
* add set_one_input method to python API of VirtualMachine
* pytests for set_input and set_one_input methods of VirtualMachine were implemented and checked
* CI restart
* construct simple model for pytests by relay instead of onnx tools (need for correct CI)
Co-authored-by: Valery Chernov <valery.chernov@deelvin.com>
* [Relay] Support large constants.
This allows constant tensors at or above a given byte limit to be marked as
'late bound' and saved/reloaded to a file independently of the overall
executable. Since the executable is often embedded in the data segment of
generated runtime Modules this avoids problems with external tools which can't
handle multi-gigabyte data segments.
[ACE-466 in OctoML JIRA]
* [checkpoint] fix latent bytecode/code bug
We replace use of the TECompiler::{Lower,LowerShapeFunc} methods from the VM's
compiler.cc with LowerTEPass. This clears the way for performing post-lowering
IRModule->IRModule transformations which combine Relay and TIR analysis. In particular,
it will allow us to use the PlanDevices pass to propagate memory scope constraints
across PrimFuncs.
We run LowerTEPass fairly early in the pipeline, which required quite a few passes
to become 'post-lowering friendly'. In particular, ManifestAlloc is now run after
rather than before lowering, and so must now work in a mixed Function/PrimFunc world.
The "vm.shape_func" operator has been removed since a) lowering has already generated
the necessary dynamic shape function, and b) the call to that function can be
represented by an 'ordinary' vm.invoke_tvm_op call.
We worked our way through the following glitches:
- Dynamic shape functions are now given their true type (rather than the type of
the primitive function they are paired with).
- Lowering was choosing definitional GlobalVars which were not pointer-equal to the
referential GlobalVars left behind in the rewritten Calls. We fixed that in
te_compiler.cc, though better would be to push GlobalVars deeper into the
lowering machinery.
- device_copy was rewritten to a call to @__copy without any definition. Though we
tried adding it as a global this (obviously in retrospect...) won't typecheck if
there are multiple device_copies in the program. Instead leave device_copy unchanged
during lowering and update each executor codegen to look for them specially.
- Calls to already-compiled BYOC functions were indistinguishable from calls
to (non-primitive) Relay functions. We move them into the call_lowered calling
convention, and leave behind a Function tagged with "ExternalSymbol". Better would
be a first-class representatn for externals in the IRModule but one step at a time.
- Functions with dynamic shapes tagged for BYOC compilation were not tracking their
connection to their dynamic shape function. We now use exactly the same attributes
as for non-BYOC primitives.
- VerilatorRuntime can legitimately be deleted before initialized.
- IRModule attributes must be preserved. In particular, since LowerTEPass can
be invoked more than once we need to be careful to preserve any existing external
modules and other attributes gatherd from an earlier LowerTEPass.
- GetUniqueName accounts for existing definitions in the module, but is not used
for external functions since their intended names are communicated to the codegen
toolchain via the already fixed "global_symbol" attribute.
This is a grab bag of fallout changes from switching the VM to use LoweTEPass
which can be easily split out of the main #9483 PR.
- AnnotateSpans can be used from C++ (though, unfortunately, it didn't help
me with debugging since spans are universally dropped in most passes).
- Can get a human readable dump of the VM's PackedFunc names and indexes for
debugging.
- If TVM_LOG_DEBUG defined then include types and ids of GlobalVars. I had
a lot of difficulty tracking down where duplicate GlobalVars for the same
name_hint were getting created and propagated.
- GetCallLoweredProps follows same API as GetDeviceCopy and GetOnDevice
where will return 'null' properties if call/expr is not of call_lowered
form. Mildly more convenient, though switching all the above to ICHECK
and push 'if (op == the relevant op)' into all use sites would also be just
fine.
- Misc VLOG improvements made while tracking down issues in #9483.
* Switch PlanDevices pass to be w.r.t. SEScopes instead of DLDeviceTypes.
CAUTION: Breaking VM executable serialization change. I needed a new 'virtual devices' array in the executable so that instructions can continue to refer to devices by a simple index yet the VM can respect both the device type and id for runtime devices.
Continuing from #9313, and as part of apache/tvm-rfcs#38, we switch PlanDevices to plan with respect to SEScopes instead of just DLDeviceTypes. Our ultimate goal is to be able to flow memory scopes between PrimFuncs by re-running PlanDevices after the LowerTE pass. This PR at least gets us to being able to flow the memory scopes, but the actual changes to PlanDevices to look inside PrimFuncs is still two PR's in the future.
However, we get two nice side effects right away:
- Since SEScopes contain Targets we can isolate all the device-to-target resolution machinery within PlanDevices (with the help of CompilationConfig). After PlanDevices has run we can retrieve the Target for any sub-expression directly from that sub-expression's SEScope. For now we retain the one-Target-per-DLDeviceType constraint since it baked into the public 'TargetMap' API, but the path to breaking that constraint is clearer.
- Device ids are now respected all the way from annotation to executor. Previously though we had a bit of plumbing using Devices the device_id therein was ignored or defaulted to zero.
The Python "on_device" annotation helpers still work w.r.t. devices. Thus though they now respect device ids, they do not allow the user to specify a Target or memory scope as supported by the underlying SEScope.
* [checkpoint] Revert emitter.py, must have run 'black .' by mistake.
* [checkpoint] Address PR comments
Also add back SplitArgs pass in build_module.cc which somehow got lost in the shuffle.
(try again -- flaky test_crt.py test_autotune?)
* [checkpoint] Fix after rebase on CallLowered.
* BUG: alloc_tensor offset and reshape shape should be on the CPU
The VM ManifestAlloc pass was allocating constants in a few places I
forgot to tag with on_device for the host/cpu. As a result the runtime
would (silently) do the x-device copy, which destroys perf.
To make this easier to spot in the future added a 'constants' property
to the VM Executable to dump the shape & device for all VM constants.
This is CORE-102 in OctoML JIRA.
* [checkpoint] Older compilers can't handle << overload
* [checkpoint] Woops, forgot requires_cuda
* [PROFILING] Profiling over RPC
Allow for profiling over RPC by serializing the returned report before
sending it. Also remove collectors argument when profiling over rpc
because it cannot be serialized.
* lint
* fixes
* add comments
Add benchmarking that includes ovearhead of transfering inputs and
outputs to and from the device. This should give an accurate measurement
of the runtime a user would see when using the model. This is
accomplished by adding functions that run from inputs to return values
into the graph executor and the VM.
* [GRAPH EXECUTOR,VM] Add benchmarking function to graph executor and vm
This new benchmarking function is just a convenience function for
calling time_evaluator on the underlying module. Hopefully this should
make it easier for users to get good benchmarks of their code.
* formatting
* import order
* more test, more comments, more precision
* fix tests
* add seconds descriptions to doc
* Get basic verison of VM RPC working
* Test case passes
* Clean up PR
* Lint
* Format
* Address Andrew R and TK feedback
* Add comment for Andrew
* Address Zhi's comment
* Format
* Fix broken test
* [Relay][VM] Allow to config allocator type and refactor vm code structure
* fix doc
* fix
* update
* trigger ci
* trigger ci
* trigger ci
* trigger ci
* fix doc warning
Tianqi Chen