* [CUDNN] Add cuDNN as a Relay partitioning target (BYOC)
This adds infrastructure to support offloading of Relay
patterns to cuDNN. In this initial commit, only softmax
is supported.
* Refactor common TE BYOC code into separate file
* Add test guard
* [build] Update libinfo and add lint rule
This updates `tvm.support.libinfo()` to be in-line with the current tvm options. It also adds a lint rule to ensure these stay matched up in the future as well as a script to print out the options in more detail. This should add in communication when debugging (i.e. tell someone to run `python -c 'import tvm; tvm.support.describe()` to learn everything you need about their envrionment)
* Fix pylint
Co-authored-by: driazati <driazati@users.noreply.github.com>
As discussed in https://github.com/apache/tvm/pull/10856#discussion_r840324560, add a utility under `meta_schedule/testing/utils.py` to clean up the database boilerplate. Also using `DummyDatabase` instead of `JsonDatabase` for further clean up, as suggested by @junrushao1994 .
* Add per-channel quantization to QNN add/subtract/multiply
* Add feedback
* Add feedback - round 2
* Fix for arm test
* Add params to the test
* Try again
* Try int
* Move lhs_axis and rhs_axis
* Add as an attribute
* Add quotes
* Respect dtype in Scalarize.
* Add unittest.
* Fix lint.
* Promote dtype of IntImm to match loop_var in For.
* Fix dtype mismatches.
* Lint
* Lint.
* jostle ci
* Match dtype in hybrid parser.
* [RUNTIME] Api to get number of runtime threads
Add `tvm::runtime::threading::NumThreads` and `tvm.runtime.num_threads`
as a way to get the number of threads in use by the TVM runtime.
* check if equal to hardware threads or hardware threads/2
This PR adds DFPattern support for the TRT backend without removing the existing predicate registry.
Adds and extends the following:
In tensorrt.py: Add a pattern_table for all the supported ops and consumes the pre-existing op_registry checks
Adds an additional pass as unmerge_composites.cc. This is required for the TRT backend as it expects a single primitive
function to work with, while the MergeComposite and PartitionGraph will produce a single function for each Composite
pattern.
Adds test_inline_composites.py which tests the newly introduced pass.
Both the pattern-based and predicate-based pass sequences produce syntactically equivalent IRModules.
This is to ensure backwards compatibility."
Fix typo in comment about kill method in PopenWorker class used to kill
child processes created by the worker.
Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
Check that the function to be profiled is actually defined.
Check that the MetricCollector used actually can time the region
requested.
Default to using the module's entry_name instead of "main".
* introduce vm compile path
* support vm in tvmc
* cleanup + lint
* add profiler + simplify vm case in tvmcpackage
* address comments + parametrize tests
Co-authored-by: Margaret Qian <mqian@octoml.ai>
* [CUBLAS] Add support for nn.dense and nn.batch_matmul
This commit includes a fix for cublas.batch_matmul
when mixed precision is being used.
* Specify args in dense
There is no way to pass kwargs dictionary from C++ code, so the previous
way never worked. Use TVM's Map instead, and pass the target architecture
version to the linker to use libraries specific to the architecture.
In this PR we introduced more fine-grained loop spliting and reordering for Rewrite-Unbound-Block post processor based on given cuda target's attribute (`max_threads_per_block`). After this PR the performance of non-reductional kernels could improve by ~20%. Regression tests are also added.
This demonstrates how to selectively extract and tune tasks from a whole relay mod, and apply the tuned schedule during the final `relay.build(...)`.
This flow is entirely different from existing tests in `test_meta_schedule_tune_relay.py` where ALL ops are extracted and auto-scheduled by MS. My test extracts only int8 `dense` op, applies a manual TIR schedule on it, and leaves int8 `batch_matmul` to be scheduled by TE.
This also serves as an example of autotvm style manual template + tensorization. The manual TIR schedule is equivalent to TE VNNI `dense` schedule in https://github.com/apache/tvm/blob/ce335c3a74185df6cc1152e53c60695d8a418d8e/python/tvm/topi/x86/dense.py#L366-L375
* bump PyTorch version to 1.11
* disable some caffe2 ci
* Fix sub conversion in PyTorch frontend
* use fuse_modules_qat if available, fallback to fuse_modules for older PyTorch
* Re-Run CI
* [CUBLAS] Add cuBLAS as a Relay partitioning target (BYOC)
This PR adds a partitioning pass for cuBLAS so that
supported Relay patterns can be offloaded to cuBLAS.
This initial commit only adds offloading support
for nn.matmul.
Although cuBLAS is already enabled in TVM by using
strategy selection in TE, by exposing it explicitly
as a Relay partitioning target we can more precisely
describe how to execute a model in Relay. This is
desirable particularly in the Collage effort to
improve multi-backend graph partitioning.
* Refactor to remove boilerplate
* [microNPU] Fix layout transform matrix
One of the layout transforms currently causes the cascader to stripe
across B16 axis (which is not allowed), so change that and deal with
the implications to the get_valid_block_configs.
Change-Id: I04199f9f35fcc31618581567483cfb80d3b5aad2
* Reduce the duplication of layout transfrom matrices
* Change the nhcwb16_to_nhwc matrix for binary and unary elementwise
such that it matches the other NPU ops
* Reduce the number of places where the same layout transform matrices are
defined
* Add documentation to the layout transform matrices
* Register relay.ext.vitis_ai.available function
* Fix vitis-ai tests when running with USE_VITIS_AI OFF
* Replace skip_test with pytest skipif
* Add a function to see if vitis_ai is available
* Use requires_vitis_ai function for running tests
## Context
When dealing with end-to-end models, we note that some tensors may have large shapes. Thus, when designing graph-level IR, we sometimes use `int64` instead of `int32` for the shape. Below is an dense GeMM example which has `int64` input tensor shape:
```python
@tvm.script.ir_module
class Module:
@T.prim_func
def main(rxplaceholder: T.Buffer[(1, 512), "float32"], rxplaceholder_1: T.Buffer[(T.int64(1000), T.int64(512)), "float32"], T_matmul_NT: T.Buffer[(1, T.int64(1000)), "float32"]) -> None:
# function attr dict
T.func_attr({"global_symbol": "dense", "tir.noalias": True, "op_pattern": 3})
# body
# with T.block("root")
for i0_0, i1_0, i0_1, i1_1, i2_0, i0_2, i1_2, i2_1, i0_3, i1_3 in T.grid(1, 4, 1, 25, 8, 1, 10, 64, 1, 1):
with T.block("T_matmul_NT"):
i = T.axis.spatial(1, 0)
j = T.axis.spatial(T.int64(1000), i1_0 * T.int64(250) + i1_1 * T.int64(10) + i1_2)
k = T.axis.reduce(512, i2_0 * 64 + i2_1)
T.reads(T_matmul_NT[i, j], rxplaceholder[i, k], rxplaceholder_1[j, k])
T.writes(T_matmul_NT[i, j])
T.block_attr({"layout_free_placeholders":[rxplaceholder_1], "meta_schedule.tiling_structure":"SSRSRS"})
with T.init():
T_matmul_NT[i, j] = T.float32(0)
T_matmul_NT[i, j] = T_matmul_NT[i, j] + rxplaceholder[i, k] * rxplaceholder_1[j, k]
```
## Problem
Though our TVMScript printer can easily print `int64` constants, the parser had poor support for `int64`. So this PR introduces some parser support for `int64`, basically about the data type of loop variables, block iterators and block read/write regions.
Besides the parser, most of the TIR schedule primitives didn't take `int64` into account in their implementations. These schedule primitives will be fixed and updated in recent future, in followup PRs.
* [microNPU] Add a pass to move allocate nodes to the outer scope
Adds a pass called `HoistAllocates` to move allocate nodes to the top
of the body of the main function. In doing so, it opens the door to
other optimizations that need to swap the ordering of external calls.
Pass illustration:
(before)
```
allocate {
extern_call {
allocate {
extern_call {
}
}
}
}
```
(after)
```
allocate {
allocate {
extern_call
extern_call
}
}
```
Change-Id: Ibcfc3c75b15deebb5c6645a4923a6ddf683b37c4
* address comments
* uses prim func pass, rather than module pass.
* adds error message informing user to run this pass with LowerToTIR()
pass for now.
Change-Id: I57757b9dc5bff0208034a974a341c09cce0294bc
* Support allocates when not followed by a sequence statement
With a test to back this case up.
Change-Id: I670809f5ee53b583a15d9b783852dda3089756e9
* Add new directory tir/contrib/ethosu to cmake build
Change-Id: I3e9f24adfe992ace4e03238a18a8378b03257e1a
* [Hexagon] Improved ergonomics of HexagonLauncher in unit tests.
The goal of this commit is to reduce/eliminate common code required
through unit tests that interact with Hexagon hardware.
- New testing fixtures in `tests/python/contrib/test_hexagon`. A test
running on hexagon hardware should only need to use the
`hexagon_session` fixture.
- `rpc_server_port`: Iterates through port numbers, selecting an
unused port for each unit test. Avoids needing to explicitly
specify unique ports for each unit test.
- `tvm_tracker`: Starts a tracker on use, exits after test. Avoids
needing to manually start a tracker prior to running the unit
test.
- `hexagon_launcher`: Starts a `HexagonLauncher` server on use,
stops server after test. Avoids needing to call `start_server()`
and `stop_server()` in each test.
- `hexagon_session`: Starts a hexagon session using
`hexagon_laucnehr.start_session()`, exits after test.
- Added `Session.upload` function, which delegates to
`HexagonLauncher.upload`. Avoids needing to interact with both the
launcher and the session.
- Allowed `tvm.IRModule` as argument passed to `Session.load_module`,
which will automatically save/upload the module, then load it.
Avoids needing to handle save/upload of temporary files in each unit
test.
* Added default port for tracker if not already set.
* Pass through None from hexagon_launcher to hexagon_session.
* Updated launcher to use external tracker if specified.
* Avoid setting up the local tracker unless required.
* Declare previous_port as global, instead of list.
* Corrected type hints.
* Docstring updates
Builds upon the work in #10254 to remove identity operations sandwiched
between two non-compute operations (reshape/strided slice - concatenate
is handled differently), under certain conditions. Specifically, an
identity operation is not removed when the dimensionality between the
two non-compute operations is reduced, due to non-congruent values
being accessed incorrectly. For example,
```
strided_slice(dims=4) -> identity -> reshape(dims=4)
```
becomes...
```
strided_slice -> reshape
```
but,
```
strided_slice(dims=4) -> identity -> reshape(dims=2)
```
remains as...
```
strided_slice -> identity -> reshape
```
Change-Id: Ie28ba384fcb3230d6f4651c0c19e2b9526ebcc42
Currently when a non-existing FILE is passed to 'tvmc compile' it throws
a traceback because a FileNotFoundError exception is not handled. Since
there is no need for such abrupt exit, and the trace can also confuse
users, this commit fixes it by checking if FILE indeed exists, informing
the user about the non-existing FILE before exiting.
Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
* [ARM] Support NCHWc alter layout in the fallback mode
* remove fallback path
* add test
* fixed int32_lanes and add channel check
* fixed schedule dispatch bug
* add workaround fallback path for NHWC im2col based GEMM schedule
* int32_lanes=4 by default
* typo
* update test
An Wang