This has been working fine for a while, this code opens it up so it's
not limited to the authors in #9983.
Co-authored-by: driazati <driazati@users.noreply.github.com>
Following change introduced installing python dependencies inside
virtual environments: https://github.com/apache/tvm/pull/12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
[TIR][MetaSchedule] Support Tuple Reduction
This PR improves our TIR scheduling primitives/transformations (rfactor & cross-thread reduction)
designed for reduction operators, so that they can be applied to blocks of tuple-reduction.
Added operators pooling (avg, max), binary operators (add, subtract, multiply, min, max) and concat.
Clip operator with min=0 and max=6 is remapped to relu6 to take advantage of CLML acceleration
without sub graphing this to fallback path.
Added new test cases for above listed operators and also end-to-end network test cases for Resnet50
& InceptionV3.
CLML support FP16 arithmetic mode which gives significant performance boost over FP32. This PR
enhances FP16 usage based on Operator datatype in relay graph.
Co-authored-by: Krishna Raju quic_kvegiraj@quicinc.com
Co-authored-by: Shwetank Singh quic_shwesing@quicinc.com
This is a pass refactored out of the AOTExecutorCodegen. Instead of
combining all of the functionality of the AOTExecutorCodegen into a
single monolithic pass, this pass only handles the lowering of the
Relay main function into TIR. Tests for the pass are included.
* support fp32 constants in quantized bias add
* add a test
* clean up comment
* assert the bias is floating point as well as constant before requantizing
* [Hexagon] Increase max buffer size for tvm_rpc_android to 1GB.
* [Hexagon] Make errors more clear when unable to allocate VTCM buffers and throw an error to fail early.
* [Hexagon] Add mem_copy_DLTensor to enable directly calling DMA for mem copies.
* [Hexagon] Add new tests as examples of the performance to expect when copying data to VTCM.
* [Hexagon] Reduce rpc max size.
* [Hexagon] Fix test_parallel_hvx_load_vtcm.py test output to be human readable.
* Comment out tests that only work on 8Gen1 HDKs to get CI to pass
Previously, the test cases only tested TE-based schedules. This
commit runs the same tests for equivalent TIR-based schedules as
well. This is intended to catch Hexagon-specific regressions, such as
the one resolved in https://github.com/apache/tvm/pull/12652.
* [Hexagon] Add Hand written HVX conv2d
Co-authored-by: Krzysztof Parzyszek <kparzysz@quicinc.com>
* Address review comments
Co-authored-by: Krzysztof Parzyszek <kparzysz@quicinc.com>
* Add some more comments and a file rename
* Add gtest unit tests for blockize/deblockize
* Add gtest unit tests fp16 utils
Co-authored-by: Krzysztof Parzyszek <kparzysz@quicinc.com>
Updates the driver stack used by the NPU to the latest released version
(semantic version 3.1.0), while maintaining backwards compatibility for
the previous version 22.05 (semantic 3.0.1) during the migration period.
In addition, support for split is re-introduced as this is now supported
in 22.08.
Change-Id: I86bce3469f0b8ad52e66461ae055dec6717b3527
* Add more strict check in tir imm construction and folding.
* fix bool-compare compile error
* fix some illegal imm construction in testcases
* do not test i64 overflow behaviour because it is not consistent on cython and ctypes
* fix float32 testcase
* auto-inferred dtype should be int64 when value exceeds int32 range
* add floatimm range check for fp16 and fp32
* add more folding testcases and fix store fp32 folding result to double
* fix i386 fp16 cases
Previously, it was ambiguous whether `BlockNode::iter_vars` were
in-scope for `BlockRealizeNode::predicate`. `ConvertBlocksToOpaque`
treated them as in-scope, and applied a mapping from `iter_vars` to
`iter_values`. Similarly, TVMScript printing places `T.where`
statements below the `T.axis` statements, where `T.axis` definitions
are in scope. However, `BlockRealizeNode::SEqualReduce` and
`BlockRealizeNode::SHashReduce` do not visit the block and `iter_vars`
until after visiting the predicate, placing the `iter_vars` out of
scope.
This commit updates the printing of `T.where` to be above `T.axis`,
and updates `ConvertBlocksToOpaque` to report an error if the
predicate contains references to `BlockNode::iter_vars`. After this
commit, these three usages all consistently treat
`BlockNode::iter_vars` as out of scope for
`BlockRealizeNode::predicate`.
Base IRBuilder methods for `PrimFunc`
This PR introduces base IRBuilder methods for `PrimFunc`.
Co-authored-by: yongwww <yongcale@gmail.com>
Co-authored-by: yongwww <yongcale@gmail.com>
Prior to this commit, `ReplaceBufferMutator` only checks
`BufferRegionNode::buffer` to determine if a `BufferRegion` needs to
be replaced, and doesn't check the `BufferRegionNode::region`. As a
result, updating `T.reads(A[B[i]])` would fail to replace `B`.
This commit checks `BufferRegionNode::region` for buffer usage to
resolve this issue.
* [TIR] Moved tir.FlattenBuffer to occur before tir.LowerOpaqueBlock
For buffers with more than one physical axis, the `axis_separators`
are required in order to know which groups of logical axes to fuse
into each physical axis. The implementation in `tir.FlattenBuffer`
assumed that all buffers were being flattened to a single physical
axis. Because `tir.LowerOpaqueBlock` replaces the
`BlockNode::alloc_buffers` with `Allocate` nodes, `tir.FlattenBuffer`
no longer has access to the axis separators and performs inconsistent
flattening for `Allocate` as opposed to `BufferLoad`/`BufferStore`.
This was introduced in https://github.com/apache/tvm/pull/12172, which
decoupled the lowering/flattening steps.
The commit reorders the `tir.FlattenBuffer` to occur before
`tir.LowerOpaqueBlock`, to make use of the axis separators. Any
`Allocate` nodes that exist at that point (e.g. from hand-written
schedules) are still flattened to 1-d physical buffers, but the
`BlockNode::alloc_buffers` are flattened according to the axis
separators.
* Add unit test to validate non-flat memory after tvm.lower
* Explicitly write T.reads for test on BufferRegion updates
* Update incorrect docstring for test
* Use DeclBuffer information in FlattenBuffer
The DeclBuffer node can be inserted during LowerOpaqueBlock, then
provide the missing Buffer information required to flatten the
allocation.
* Use T.allocate in unit tests
With the insertion of `DeclBuffer` nodes, `LowerOpaqueBlock` no longer
needs to be before `FlattenBuffer`, and has been moved back to its
original position. Revering the tests to use `T.allocate` instead of
`T.alloc_buffer` more closely represents the functions as they are
being lowered.
* Fix usage of T.decl_buffer in updated tests
* Update LowerOpaqueBuffer to expect the DeclBuffer nodes
* Strip DeclBuffer annotation in FlattenBuffer
The DeclBuffer annotations aren't yet supported in all passes. This
restricts them to being introduced in LowerOpaqueBuffer, then
immediately removed in FlattenBuffer.
* Strip out all DeclBuffer nodes in FlattenBuffer
* Update unit tests to remove expectation of DeclBuffer nodes
Create a specific test dependency to map to USE_LIBTORCH, which
is disabled by deafult, and is independent from torch being
installed on the underlying machine, so it causes problems in
machines that have torch installed but TVM is build with
USE_LIBTORCH OFF.
Mark tests.python.contrib.test_libtorch_ops.test_backend with
this new decorator.
This patch marks two tests as xfail for further investigation:
* test_meta_schedule_integration_extract_from_resnet_with_filter_func
* test_meta_schedule_integration_extract_from_resnet
Some integration tests are not being run on CI due to the
configuration of the machine with onnx and torch not calling
the integration tests script.
This patch skips two more tests failing with the error message
below:
```
"OSError: /.../torch/lib/libgomp-d22c30c5.so.1:
cannot allocate memory in static TLS block"
```
Adds support for offloading transpose convolution with an optional bias
to the NPU.
Co-authored-by: Samuel Panijel <samuel.panijel@arm.com>
Co-authored-by: Leo Blonk <leo.blonk@arm.com>
Current LoopPartition doesn't check the value of attribute key "pragma_loop_partition_hint". Whatever I set pragma_loop_partition_hint to True or False, the result is same, which is confused for debug.
This PR fix pragma_loop_partition_hint attribute key should check it's value.
* [TVMC] Run module once by default
Currently executing `tvmc run module.tar` will run the input model
twice. For benchmaking this is to be expected as the first run is used
to prime caches etc before taking a measurement. However, this seems a
bit unintuitive to have as default, especially when benchmarking is not
always intended. In this sense, this commit aims to amend the
number of runs for the default: `tvmc run module.tar` to a single run.
After inspection, this seems to be down to the use of the `.benchmark()`
method which runs (1 + repeat * number) executions in total. This means
that at least two runs are required (i.e. when repeat=1, number=1). It
also seems that it is only necessary to benchmark the model when
`--print-time` has been set from the CLI POV. From the python interface
POV, benchmarking is always run, but this may not always be necessary.
This commit makes use of the `.run()` method to singularly execute the
model by default. From the CLI this will be used when `--print-time` is
set to False whereas from the python interface this will be used when
`benchmark=False`. Otherwise, the `.benchmark()` method will be used
as before. Complementary to this change `repeat`, `number` and
`end_to_end` parameters are only used when either `--print-time` or
`benchmark` are set to True - and the documentation has been updated to
indicate this.
Change-Id: I18a38a9d430d660264f7fce5caf0779aa059fed3
* improve documentation with number of exectuions when benchmarking
Change-Id: Iecf557594420fcc9f3abcec5ce7d952db2c94271
Motivation:
In case of quantized models nn.pad operation typically is not fused with QNN ops
and lives as a standalone operation. In this case it uses default injective
schedule for Hexagon target and it is not optimized very well (based on
analysis of real models like ResNet50 INT8).
What was done:
New schedule for Pad operation was implemented instead of default injective schedule.
For Hexagon target injective schedule does fusion of all axis and vectorization
on 128/64/32 (depends on dtype). It works fine for Add, Sub, etc... but not for Pad.
New optimized schedule does these steps (fusion+vectorization) only if last tensor
dimension is divisible by 128/64/32 (depends on dtype). It was done only for Hexagon,
for other targets (x86, cuda, etc.) there is no changes and it uses default injective
schedule.
Benchmark results on Snapdragon 888:
4d NHWC layout with ((0, 0), (1, 1), (1, 1), (0, 0)) padding, "uint8" dtype:
shape | default schedule, ms | optimized schedule, ms | speedup |
-------------------|----------------------|------------------------|-------------------|
(1, 112, 112, 32) | 10,03 | 0.2 | 50.1x times |
(1, 56, 56, 128) | 0,099 | 0,085 | ~1x (no speedup) |
---------------------------------------------------------------------------------------|
4d NCHW layout with ((0, 0), (0, 0), (1, 1), (1, 1)) padding, "uint8" dtype:
shape | default schedule, ms | optimized schedule, ms | speedup |
-------------------|----------------------|------------------------|-------------------|
(1, 128, 56, 56) | 10.96 | 1.38 | 7.9x times |
(1, 32, 126, 126) | 1.66 | 1.58 | ~1x (no speedup) |
(1, 32, 128, 128) | 13.98 | 2.66 | 5.25x times |
---------------------------------------------------------------------------------------|
5d NCHWc layout with ((0, 0), (0, 0), (1, 1), (1, 1), (0, 0)) padding, "uint8" dtype:
shape | default schedule, ms | optimized schedule, ms | speedup |
-------------------|----------------------|------------------------|-------------------|
(1, 4, 56, 56, 32) | 6.39 | 0.29 | 22x times |
(1, 56, 56, 128) | 0.15 | 0.15 | ~1x (no speedup) |
---------------------------------------------------------------------------------------|
Summary:
For some input tensors we get up to 50x times speedup, for other performance is the same.
No performance degradations were detected.
* [TIR][StorageRewrite] Allow in-place buffer reuse of non-flat memory
Previously, shared buffer use was entirely disabled for non-flat
memory, since the existing checks for shared memory assume flat 1-d
spaces. This was enforced in `FindAlloc` and validated in
`PrepareNewAlloc`. The validation in `PrepareNewAlloc` could trigger,
if the buffer sharing was due to an in-place operation, and not
through the `FindAlloc` function.
In-place operations do not require N-d packing, nor do they introduce
ambiguity in how different code generators may interpret non-flat
physical indices. Therefore, this commit relaxes the validation in
`PrepareNewAlloc`, allowing buffer reuse of non-flat buffers for
in-place operations.
* Update new StorageRewrite with correct allocate/buffer_decl usage
- Introduce 'global.ddr' memory scope:
- Like 'global', this allocates memory from the Hexagon SoC's
DDR memory.
- Like 'global.vtcm', the specified tensor shape must be 1d
or 2d, where 2d indicates Hexagon's "indirect tensor"
(i.e., discontiguous) allocation scheme.
- Change memory-alignment strategy to always be 2048-byte aligned
on Hexagon. (This can be refined in the future, but for now it
ensures all allocations meet the strictest alignment requirements
for any Hexagon operations.)
Using pytest parameterization helps identify the particular parameter combinations that are failing for a given test. Additionally, it can be useful when parallelizing the tests. This commit makes sure that "trials" have been replaced by parameterization as well as completing a general cleanup.
This fixes the issue where merging from GitHub Actions (i.e. with the default `GITHUB_TOKEN`) doesn't trigger post merge GitHub Actions on the commit it creates in `main`. Instead these jobs are triggered manually by a call to the Actions API after the merge has taken place.
This also updates the tvmbot testing code (and by extension some of the other CI testing code) to remove the fixtures for each test in favor of constructing them from a single sample at runtime, this makes it a lot easier to add new tests and see what is different between each data sample and clean up the testing anti-patterns that were there before (e.g. `run()` instead of `pytest.mark.parameterize`, but none of the tests in `test_ci.py` have changed)
Tested in https://github.com/driazati/tvm/pull/36 which ran https://github.com/driazati/tvm/actions/runs/2881047903
Following up #12520 and #12626, this PR introduces two database classes:
`UnionDatabase` and `OrderedUnionDatabase`, both of which allow users to
organically compose multiple databases together, so that the high-level
IR (Relay, Relax) could select the best tuning records according to
running time or a preferred order given by users.
To each query, `UnionDatabase` returns the best record among all the
databases given; Instead, `OrderedUnionDatabase` returns he record from
the first database that responds to the query.
Used together, users may specify complicated dispatching patterns like
below:
Examples below demonstrate the usecases of and difference between
UnionDatabase and OrderDatabase.
Assumption:
* db1, db2 do not have tuning records for the target workload.
* Each of db3, db4, db5 has tuning records r3, r4, r5 for target
workload respectively.
```python
#### Case 1. `UnionDatabase`:
merged_db = ms.database.UnionDatabase(
db1, # no record
db2, # no record
db3, # has r3
db4 # has r4
)
# returns the better one between r3 and r4
merged_db.query_tuning_record(..., target_workload)
### Case 2. `OrderedUnionDatabase`
merged_db = ms.database.OrderedUnionDatabase(
db1, # no record
db2, # no record
db3, # has r3
db4 # has r4
)
# returns r3
merged_db.query_tuning_record(..., target_workload)
### Case 3. Mix-use scenario
merged_db = ms.database.UnionDatabase(
db1, # no record
db2, # no record
db3, # has r3
ms.database.OrderedUnionDatabase( # returns r4
db4, # has r4
db5, # has r5
)
)
# returns the better one between r3 and r4
merged_db.query_tuning_record(..., target_workload)
### Case 4. Another mix-use scenario
merged_db = ms.database.UnionDatabase(
db1, # no record
db2, # no record
db3, # has r3
ms.database.UnionDatabase( # returns the better one between r4 and r5
db4, # has r4
db5, # has r5
)
)
# returns the best one among r3, r4 and r5
merged_db.query_tuning_record(..., target_workload)
### Case 5. Yet another mix-use scenario
merged_db = ms.database.OrderedUnionDatabase(
db1, # no record
db2, # no record
ms.database.UnionDatabase( # returns the better one between r3 and r4
db3, # has r3
db4, # has r4
)
db5, # has r5
)
# returns the better one between r3 and r4
merged_db.query_tuning_record(..., target_workload)
```
Co-authored-by: sunggg <49998730+sunggg@users.noreply.github.com>
driazati