* [Texture] Add 2d memory support into static memory planner
Co-authored-by: Chris Sullivan <csullivan@octoml.ai>
* Add test verifying GraphPlanMemory work for 2d memory
Co-authored-by: Chris Sullivan <csullivan@octoml.ai>
* [Relay] Allow partial virtual device annotations.
Previously CompilationConfig::CanonicalVirtualDevice required
the argument virtual device to contain a device type. However
now that virtual devices may contain memory scopes that's
unnecessarily strict.
With this change it is possible to write virtual device
annotations with just memory scopes, and let PlanDevices
flow those constraints along with the usual device constraints.
* - Make sure CanonicalVirtualDevice reuses FullyUnconstrained
* [Collage] PruneCandidates and demo_collage_partition.py
See https://github.com/apache/tvm-rfcs/blob/main/rfcs/0062-collage.md.
This completes our checkin of our Collage 'sketch' branch into main. Special thanks
to Matthew Barrett for his help getting this over the line.
The only C++ functionality added here is for 'pruning' candidates. This is a somewhat
speculative algorithm (and I've called that out in the comments) which tries to
elide candidate partitions which will 'obviously' not contribute to the final optimal
partitioning. For largish models such as GPT2 this can significantly reduce the number of
candidates we need to actually measure latency on. I beefed up the MockCostEstimator to
make it possible to assert pruning occured from within the test_pass_collage_partition.py
unit test.
The rest of this PR adds the demo_collage_partition.py driver file we've been using
to test and measure perfomance differences against various baseline (though only
for the CUDA ecosystem). To eliminate loading time the models of interest are directly
expressed in Relay text form in menangerie.py.
* - lint
* [Collage] CollagePartition pass
See https://github.com/apache/tvm-rfcs/blob/main/rfcs/0062-collage.md.
This adds the main CollagePartition pass, which:
1. Inspects all the targets in the CompilationConfig and builds
PartitionSpecs describing how to generate speculative CandidatePartitions
for them.
2. Runs the above rules on the model to collect all the candidates.
3. Eliminates candidates whose target contradicts any constraints already
imposed by, eg, device planning.
4. Eagerly estimates the cost of each candidate.
5. Performs a shortest path search to chose an 'optimal' set of candidate
partitions so as to minimize estimated model latency, such that every sub-expression
node is contained in exactly one candidate partition.
6. Coalesces adjacent optimal candidates which ended up on the same target.
7. Rewrites the model according to the chosen optimal partitioning.
As for the existing partition_for_<external codegen name> methods, the result of
CollagePartition can then be built using regular TVM.
Very special thanks to @mbaret for authoring test_pass_collage_partition.py.
Logic to prune the candidates after step 3 will be in a follow up PR since it
deserves its own testing. A demonstration driver will also come as a follow up.
* - lints
* - more lints
* - use the _ffi_api properly
Motivation:
Same IR node object can be referenced in several different contexts inside a larger IR object. For example, a variable could be referenced in several statements within a block.
This makes it impossible to use an object pointer to uniquely identify a "location" within the larger IR object for error reporting purposes. The `ObjectPath` class addresses this problem by serving as a unique "locator".
Tracking issue: https://github.com/apache/tvm/issues/11912
* [Relay] Move TOpPattern registration for nn.* to C++
Some of the Collage machinery is best tested from C++, but
requires Relay ops to have their "TOpPattern" registered.
However since the nn.* ops register on the Python side tests
can't rely on those ops.
The easy fix is to just move the registration to the
RELAY_REGISTER_OP block. However since kOpaque is the
default I did not preserve those registrations.
There's still a few dozen more exotic ops still registered
on the Python side. I've left them be.
* - D'oh! Even kOpaque ops must be registered.
* [relay] Changed Executor and Runtime 'name' field to 'registry_name'
Changed 'name' field to 'registry_name' for Executor and Runtime python
wrappers as it clashed with tvm object attribute 'name' which made the latter
inaccessible from Python
Change-Id: I917755753549edfe1d3090ca9ca4512de552c4bd
changed name to registry_name
Change-Id: I9feb5b33b7b6f6f8421902e5721167f585cc4193
* more fixed unit tests
Change-Id: Ie2e96297fda119e1b726b196a59deae95b263a07
* typo fixed
Change-Id: Id579c50ab58dfb25fa18436265e0701ebbd9d554
* renamed registry_name to flag_registry_name
Change-Id: Iabbd81069959f05c073f9dbc8d10fb31dd05f7a3
* bugfix
* Compute common type for shape elements in BroadcastHelper
The corresponding dimensions in the input/output tensors in a broadcast
operations may have the same value, but different types (e.g. int32 vs
int64).
When the broadcast helper tries to unify the dimensions it also needs
to compute the common type to hold the dimension.
* Cast and simplify both members of `Range`
Only the `min` member was type-casted, which could lead to ranges with
different types for `min` and `extent`.
Move the casts to the argument of Simplify, so that they can be eliminated
if they aren't needed.
* Type-check iv domain ranges, use cast only if needed in MakeLoopNest
In some cases the domain ranges had the `min` and the `extent` values
be of different types (e.g. [(int64)0, 32)). This is an error, and it
can lead to compilation failures later on. Add a check for equal types
here to catch this early.
Also, only add the cast operation when the desired type differs from
the current one to keep the expressions simpler.
* Check that variable and substituted expression have same types
Add a check to IRSubstitute to detect when the type of a variable and
the type of the expression to replace it with have different types.
* Add testcase
* [TVMScript] Use void for lambda parameters, allow mismatch in Substitute
When the script parser deals with lambdas, it creates Var objects for each
parameter. Their actual types are not known at the time, and the properly
typed variables are subtituted in the body later. Since the default dtype
of a Var is "int32", this could lead to a type mismatch in Substitute.
To deal with this scenario, use "void" for newly created Vars in the
parser, and add an exception to Substitute to allow replacing void Vars
with expressions of any type.
* Fix type error in test_reduce_combiner_simplify
* Restart CI
Co-authored-by: Jiawei Liu <jaway.liu@gmail.com>
* support post-op swish
* support post-op clip
* enhance get_shape and get_dtype in dnnl.py to support efficientnet
* add checks for with_eltwise whether in supported list
* fix lint
* fix test
This PR introduced a new argument for `ApplyHistoryBest`'s `Query` interface to allow direct dispatch without querying the database, would be useful for debugging and benchmarking without interference.
* [microNPU] Calculate memory pressure for microNPU external functions
During the microNPU compilation stage, the "used_memory" annotations on
external microNPU functions are read to determine a memory pressure
value. This value is passed to the cascader to better approximate the
memory available for the optimization.
Change-Id: I11a311b0005e785637014cb451f4aed96edcda26
* fix get size from memory region
Change-Id: I41acfc83f05b2204075edb99f86a0eecaba00f71
* add test case for full offload
Change-Id: If3e672d402ab237fa82e34761bb972d2e9483ba9
* Common autotuning test
* Autotuned model evaluation utilities
* Bugfixes and more enablement
* Working autotune profiling test
* Refactoring based on PR comments
Bugfixes to get tests passing
Refactor to remove tflite model for consistency
Black formatting
Linting and bugfixes
Add Apache license header
Use larger chunk size to read files
Explicitly specify LRU cache size for compatibility with Python 3.7
Pass platform to microTVM common tests
Better comment for runtime bound
Stop directory from being removed after session creation
* Use the actual Zephyr timing library
Use unsigned integer
Additional logging
Try negation
Try 64 bit timer
Use Zephyr's timing library
Fix linting
Enable timing utilities
This PR introduces `Schedule.work_on`, which instructs
`Schedule.get_block` to find the correct PrimFunc to retrieve from
without having to specify `func_name` in every time if the PrimFunc's
name is not `main`.
* [BYOC] Switch TensorRT BYOC integration to IRModule-at-a-time using RelayToTIR hook
This does for the TensorRT integration what #11631 did for the CUTLASS integration.
- All compilation options are captured within the attributes of a Target of
kind "tensorrt" (instead of the "relay.ext.tensorrt.options" attribute in
PassContext). This means all BYOC configurations options needed by Collage can
be captured uniformly by a list-of-Targets. It also means RPC boundaries (as used
internally at OctoML) only need to worry about maintaining the fidelity of the
Target instance(s) rather than reaching into the PassContext.
- Compilation is switched from function-at-a-time (relying on the TECompiler) to
IRModule-at-a-time (using the RelayToTIR target-specific hook mechanism). Though
not strictly necessary for Collage I want to check the path is now clear to
deprecate the support for BYOC in TEComplier.
- Get all the TensorRT tests going again, except for a few I've disabled with
x-link to a new issue #11765. CAUTION: The TensorRT runtime is not supported in
CI so many of these tests are cosmetic.
- While trying to track down a 'free(): invalid pointer' error in test_tensorrt_int8_exp.py
made the TensorRT allocs/frees more robust, but turns out its also broken in main.
No harm leaving these changes in though.
* - Lints
* - Woops, fix test
* - lints
* - Use default tensorrt target if none given in targets list
* - fix free error
* - accidentally introduced 'transforms' namespace
- can't use default Target("tensorrt") arg
* - D'oh! Include ended up #if protected
* - restore mark for test_dynamic_offload
- handle missing runtime in versioning
- turn test_maskrcnn_resnet50 back on now that we have the
import-torch-first workaround.
* - wibble
* Working 8 bit vlut for relay take operator
* Formatting
* More formatting
* clang-format on codegen_hexagon.cc
* Update for llvm api
* Add return to VisitExpr(BufferLoadNode) function
* different llvm api
Luke Hutton