Motivation:
Same IR node object can be referenced in several different contexts inside a larger IR object. For example, a variable could be referenced in several statements within a block.
This makes it impossible to use an object pointer to uniquely identify a "location" within the larger IR object for error reporting purposes. The `ObjectPath` class addresses this problem by serving as a unique "locator".
Tracking issue: https://github.com/apache/tvm/issues/11912
* [Relay] Move TOpPattern registration for nn.* to C++
Some of the Collage machinery is best tested from C++, but
requires Relay ops to have their "TOpPattern" registered.
However since the nn.* ops register on the Python side tests
can't rely on those ops.
The easy fix is to just move the registration to the
RELAY_REGISTER_OP block. However since kOpaque is the
default I did not preserve those registrations.
There's still a few dozen more exotic ops still registered
on the Python side. I've left them be.
* - D'oh! Even kOpaque ops must be registered.
* [relay] Changed Executor and Runtime 'name' field to 'registry_name'
Changed 'name' field to 'registry_name' for Executor and Runtime python
wrappers as it clashed with tvm object attribute 'name' which made the latter
inaccessible from Python
Change-Id: I917755753549edfe1d3090ca9ca4512de552c4bd
changed name to registry_name
Change-Id: I9feb5b33b7b6f6f8421902e5721167f585cc4193
* more fixed unit tests
Change-Id: Ie2e96297fda119e1b726b196a59deae95b263a07
* typo fixed
Change-Id: Id579c50ab58dfb25fa18436265e0701ebbd9d554
* renamed registry_name to flag_registry_name
Change-Id: Iabbd81069959f05c073f9dbc8d10fb31dd05f7a3
* bugfix
* Compute common type for shape elements in BroadcastHelper
The corresponding dimensions in the input/output tensors in a broadcast
operations may have the same value, but different types (e.g. int32 vs
int64).
When the broadcast helper tries to unify the dimensions it also needs
to compute the common type to hold the dimension.
* Cast and simplify both members of `Range`
Only the `min` member was type-casted, which could lead to ranges with
different types for `min` and `extent`.
Move the casts to the argument of Simplify, so that they can be eliminated
if they aren't needed.
* Type-check iv domain ranges, use cast only if needed in MakeLoopNest
In some cases the domain ranges had the `min` and the `extent` values
be of different types (e.g. [(int64)0, 32)). This is an error, and it
can lead to compilation failures later on. Add a check for equal types
here to catch this early.
Also, only add the cast operation when the desired type differs from
the current one to keep the expressions simpler.
* Check that variable and substituted expression have same types
Add a check to IRSubstitute to detect when the type of a variable and
the type of the expression to replace it with have different types.
* Add testcase
* [TVMScript] Use void for lambda parameters, allow mismatch in Substitute
When the script parser deals with lambdas, it creates Var objects for each
parameter. Their actual types are not known at the time, and the properly
typed variables are subtituted in the body later. Since the default dtype
of a Var is "int32", this could lead to a type mismatch in Substitute.
To deal with this scenario, use "void" for newly created Vars in the
parser, and add an exception to Substitute to allow replacing void Vars
with expressions of any type.
* Fix type error in test_reduce_combiner_simplify
* Restart CI
Co-authored-by: Jiawei Liu <jaway.liu@gmail.com>
* support post-op swish
* support post-op clip
* enhance get_shape and get_dtype in dnnl.py to support efficientnet
* add checks for with_eltwise whether in supported list
* fix lint
* fix test
This PR introduced a new argument for `ApplyHistoryBest`'s `Query` interface to allow direct dispatch without querying the database, would be useful for debugging and benchmarking without interference.
* [microNPU] Calculate memory pressure for microNPU external functions
During the microNPU compilation stage, the "used_memory" annotations on
external microNPU functions are read to determine a memory pressure
value. This value is passed to the cascader to better approximate the
memory available for the optimization.
Change-Id: I11a311b0005e785637014cb451f4aed96edcda26
* fix get size from memory region
Change-Id: I41acfc83f05b2204075edb99f86a0eecaba00f71
* add test case for full offload
Change-Id: If3e672d402ab237fa82e34761bb972d2e9483ba9
* Common autotuning test
* Autotuned model evaluation utilities
* Bugfixes and more enablement
* Working autotune profiling test
* Refactoring based on PR comments
Bugfixes to get tests passing
Refactor to remove tflite model for consistency
Black formatting
Linting and bugfixes
Add Apache license header
Use larger chunk size to read files
Explicitly specify LRU cache size for compatibility with Python 3.7
Pass platform to microTVM common tests
Better comment for runtime bound
Stop directory from being removed after session creation
* Use the actual Zephyr timing library
Use unsigned integer
Additional logging
Try negation
Try 64 bit timer
Use Zephyr's timing library
Fix linting
Enable timing utilities
This PR introduces `Schedule.work_on`, which instructs
`Schedule.get_block` to find the correct PrimFunc to retrieve from
without having to specify `func_name` in every time if the PrimFunc's
name is not `main`.
* [BYOC] Switch TensorRT BYOC integration to IRModule-at-a-time using RelayToTIR hook
This does for the TensorRT integration what #11631 did for the CUTLASS integration.
- All compilation options are captured within the attributes of a Target of
kind "tensorrt" (instead of the "relay.ext.tensorrt.options" attribute in
PassContext). This means all BYOC configurations options needed by Collage can
be captured uniformly by a list-of-Targets. It also means RPC boundaries (as used
internally at OctoML) only need to worry about maintaining the fidelity of the
Target instance(s) rather than reaching into the PassContext.
- Compilation is switched from function-at-a-time (relying on the TECompiler) to
IRModule-at-a-time (using the RelayToTIR target-specific hook mechanism). Though
not strictly necessary for Collage I want to check the path is now clear to
deprecate the support for BYOC in TEComplier.
- Get all the TensorRT tests going again, except for a few I've disabled with
x-link to a new issue #11765. CAUTION: The TensorRT runtime is not supported in
CI so many of these tests are cosmetic.
- While trying to track down a 'free(): invalid pointer' error in test_tensorrt_int8_exp.py
made the TensorRT allocs/frees more robust, but turns out its also broken in main.
No harm leaving these changes in though.
* - Lints
* - Woops, fix test
* - lints
* - Use default tensorrt target if none given in targets list
* - fix free error
* - accidentally introduced 'transforms' namespace
- can't use default Target("tensorrt") arg
* - D'oh! Include ended up #if protected
* - restore mark for test_dynamic_offload
- handle missing runtime in versioning
- turn test_maskrcnn_resnet50 back on now that we have the
import-torch-first workaround.
* - wibble
* Working 8 bit vlut for relay take operator
* Formatting
* More formatting
* clang-format on codegen_hexagon.cc
* Update for llvm api
* Add return to VisitExpr(BufferLoadNode) function
* different llvm api
This PR enables extracting the embeddings of the workload in a tuning context, which further strengthens the feature extracting process. Workload embeddings are extracted based on names of each block in the IR module. If `extract_workload` is enabled, the extracted feature vectors will have length 164 + 8 = 172.
I tried to do to the TensorRT integration what #11631 did to the CUTLASS integration, viz:
- Make sure all compilation options are passed in Target instances. This helps Collage.
- Use a custom pass invoked via RelayToTIRTargetHooks instead of the relay.ext.$toolchain mechanism.
This helps use decouple external codegen from lowering.
This PR collects the prep for that change:
- TensorRT uses the JSONSerializer visitor to encode each partition function. Previously, when the
visitor encountered a Constant it simply generated and recorded a name for the constant. Then,
completely separately, and via a callback in TECompiler, the function is visited again in the
same order and with the same name generation convention by a ConstantUpdater to actually collect the
bindings, which are then encoded into a ConstLoaderModule to be made available at runtime.
However if all TensorRT compilation is to be done by a stand-alone pass there's no TECompiler callback
hackery available. So I've added a "const_name_to_ndarray" attribute to the IRModule of type
Map<String, runtime::NDArray> so that named constants can be accumulated throughout compilation by
any pass which needs to do so. Then the Graph, AOT and VM executors are all updated to merge those
constants into the final runtime artifact
(Compare with "Constants", the equivalent attribute for extracting TIR AllocateConsts.)
- The TensorRT tests use the create_executor interface but it wasn't quite ready for the
new more general form of passing list-of-targets.
- I want TensorRT compilation to work out of the box without the need for any special targets if
all the default options should apply. Go back and make the CUTLASS integration I did follow the
same convention.
- To test this I also switched the 'demo' "ccompiler" external codegen target to IRModule-at-a-time
style. This means we can test most of external codegen machinery in one place without depending on
any target which may not be enabled in CI (eg TensorRT):
- Target instances are plumbed correctly so compile-time options are available.
- External modules are conveyed to the final export library.
- Constant bindings are conveyed to the metadata module.
Fixes offloading a few corner cases of average pooling. Specifically
not offloading nn.avg_pool2d when:
* The attribute count_include_pad=True
* Padding exceeds the dimensions [3, 3, 4, 4]
* The pool size is greater than [8, 8] when the pool uses padding
Change-Id: I7be546e28ebe1f17482f3ed3cee56996a71bfcd1
Wuwei Lin