This commit adds a MemoryPools argument for
the compilation flow according to RFC0029.
Moreover, it is used to provide support for
external pools from the application layer
that could be pinned for different memories
and/or be reused between multiple inferences
of a model.
PR #10218 was not enough for this fix so this should probably run through full CI to make sure it got everything.
cc @mousius @masahi
Co-authored-by: driazati <driazati@users.noreply.github.com>
* PackedFunction to return params from the .so module, show warning when no params are set
* Linter checkup
* Autoset of params, tests for get_graph_params
* Linter checkup
* Check that inputs were set before run
* Return the original implementation if Run
* Fix RPC behavior
* Initial implementation of Common Subexpression Elimination for TIR (#703)
The goal of this PR is to implement a Common Subexpression Elimination (CSE) pass for TIR, which aims at identifying redundant computations (both within statements and within expressions), and to replace them by a new fresh variable, introduced before the first occurrence of the redundant computation.
Note that it does not only try to do commoning on full expressions, but it is also able to do it on subexpressions. For instance, if the program computes the expression (w+x) + (y+z) and the expression (w+x)+u, it will introduce the subexpression (w+x) into a new variable.
If we want so, it will be easily possible in the future to make the notion of equivalence between terms more flexible, allowing for instance to identify expressions modulo commutativity (identifying for instance (x+y) with (y+x)), modulo associativity (identifying for instance (x+y)+z with x+(y+z)), etc. Replacing only the function bool EquivalentTerms(const PrimExpr& a, const PrimExpr& b) will be the only thing needed in order to do that. The typical way to rewrite it for such extensions would be to compute a canonical representant of a and a canonical representant of b and to then compare them with the strict syntactical equality.
The main CSE pass is declared and implemented respectively in the files common_subexpr_elim.h and common_subexpr_elim.cc.
The function Stmt CommonSubexpressionEliminator::VisitStmt(const Stmt& stmt) is a good entry point as it contains many comments about what the pass is doing.
The general idea of this pass is that it tries to introduce at the current level (the current root) the computations that are redundant and which are possible to introduce there (they should only contain variables that are in scope). This notion of variables in scope is implemented with a context, which is a vector of pairs (var, MaybeValue). The context is not only used for checking that variables that appear in candidate computations are known at this point, but also for checking if a computation has already been introduced into a variable.
For a greater flexibility in the future, there is a strong distinction already in place between :
- Syntactic computations, which are maintained in a hashtable which associates expressions (the computations already seen) to size_int (the number of times the computation has been seen).
- Semantic entities, which are obtained from the syntactic computations by merging equivalent computations (where this notion of "equivalent" is customizable). Semantic entities are stored into a vector of pairs (expr, size_int) where, again, the number is the number of times that expr or equivalent computations have been seen.
The VisitStmt() method starts by computing the syntactic computations (implemented in an auxiliary analysis), then it merges equivalent computations to obtain the semantic computations. Then it sorts these semantic computations from biggest to smallest in order to always consider first the biggest computations. The rest will essentially be a loop over all these candidates, which will stay sorted.
When dealing with a candidate computation, there are three cases that can happen:
1 - Rare case A variable in the context already contains this computation. This variable can't have been introduced by the CSE, as we would have performed the replacements at the same time (see case 2). So this is the case where the user himself (or the previous TIR passes) has written something like "let x = A in ...A...A...)"
-> In this case, we simply perform the replacements of A with x in the current result. These replacements are done by an auxiliary transform/Mutator, declared and implemented in replace_expr_selected.h and in replace_expr_selected.cc.
2 - Case where we need to introduce the current computation inside a new variable This is the case where all the variables used by the current computation are within scope (i.e. are present in the context) and where our internal heuristic/predicate tells us to introduce this computation into a new variable.
-> In this case, a new variable new_var_i is generated, all the locations that use this computation in result are replaced by this fresh variable (using the same auxiliary Mutator mentioned in 1.), and the current result is replaced by let new_var_i = currentComputation in result.
3 - Case where we can't or don't want to introduce this computation inside a new variable This is the case where we either can't introduce the current computation inside a new variable (because it contains variables that are not yet in scope there) or because our internal heuristic/predicate did not want to introduce it.
-> In this case, we will compute the direct sub-expressions of the current computation (implemented by an auxiliary analysis), and we will add them to the vector of semantic computations so that they have a chance to be considered later. Note that they are added while still preserving the order.
Note that we do not add all the sub-expressions of the current expression but only its direct subexpressions given the fact that we always consider them from biggest to smallest, and given that some candidates are mutually exclusive. Otherwise it would be computationally more intensive and it would pose the problem of cleaning the vector of candidate computations when one of them gets introduced into a variable. Evaluating them lazily by only looking at the direct sub-expressions is at the same time more efficient and simpler.
Once the entire vector of semantic computations has been tried, the main function VisitStmt() calls the general dispatcher , which will in turn call the appropriate handlers. The only specific task of overridden handlers will be to update the context appropriately as new variables are introduced into scope (via Let-In, via For loop, etc) or leave the current scope. Thus, they will update the context appropriately before and after the calls to VisitStmt() and VisitExpr() on the child nodes.
* Added empty newline at the end of every new file
* Rolled-back the pointer to the submodule vta-hw
* Improved the CSE by not commoning at the toplevel redundant computations that only appear in one of the possible execution path (for instance, only in the then/else branch of an IF statement). Redundant computations that appear only in a specific execution path are now being commoned at the entrance of their specific execution path instead of earlier at the toplevel. Introducing them at the toplevel was an anti-optimization as the redundant computation might not have been comptued at all. Added two additional tests for this too.
* Spelling and comment
* Improved the CSE by not commoning at the toplevel redundant computations that only appear in one of the possible execution path (for instance, only in the then/else branch of an IF statement). Redundant computations that appear only in a specific execution path are now being commoned at the entrance of their specific execution path instead of earlier at the toplevel. Introducing them at the toplevel was an anti-optimization as the redundant computation might not have been comptued at all. Added two additional tests for this too.
* Revert "Improved the CSE by not commoning at the toplevel redundant computations that only appear in one of the possible execution path (for instance, only in the then/else branch of an IF statement). Redundant computations that appear only in a specific execution path are now being commoned at the entrance of their specific execution path instead of earlier at the toplevel. Introducing them at the toplevel was an anti-optimization as the redundant computation might not have been comptued at all. Added two additional tests for this too."
This reverts commit c4138d9afc28e79f107a4eccf988a6d93221eb5a.
* Fixed reference used for no reason instead of normal variable.
* Added comment explaning why we do not need the union/intersection over N tables at the moment (because we would only use it for N=3)
* Did most of the changes suggested by upstream
* Continued to work on the remarks given on the public repo.
* Final remarks addressed, small formatting things, and fixing things reported by the linter
* Last linter fix.
* Fixing newline
* Adding newline missing.
* Minor commit for style fo conform with clang-format
* Removed trailing space at end of line
* And more minor style changes
* Fixing style of the python test files
* And one more for style in python tests!
* This linter is very annoying to force the style of indentation in a comment, in a test file. It makes it harder to read in this case! And that incitates people to not write comments
* Deactivate the CSE pass for the lowering tests as it would otherwise do some commoning, and improve the way the CSE recurse + test added for cascade commonings
* Fixing new lint offenses
* Removing debug statement
* Restore other test file to its previous state
* One more for the linter...
* Linter again, this time for the new test...
* again
* again...
* Deactivating the CSE pass for another lowering test as it does some commoning
* Disabling the CSE for the a test for GPU too
* Trying to fix a VTA test by disabling the CSE pass for it, as it probably does some commoning
* Complying with the linter
* Restarting the CI 1/2
* Restarting the CI 2/2
* Restarting CI 1/2
* Restarting CI 2/2
* Slightly reduce size of large pretty printer test, copied from https://github.com/apache/tvm/pull/10026/commits/ae98f9e7809cbf8d910fa16bfeac8364196e57d7
* Trying to resolve the problems on the weird tests
* Linter.
* Restarting CI which has skipped the MacOS build for no reason 1/2
* Restarting CI which has skipped the MacOS build for no reason 2/2
* Commented buggy tests
* Linter...
* Restore the VTA tests, and use trick kindly given by Masa to disable the CSE pass for the VTA tests, as vta.build() overwrittes the config
* New fix, which this time does not break the doc (VTA uses a set with {} for the disabled passes instead of a list with [] for some reason
* More VTA fixes
* vta tutorial fix
Co-authored-by: Masahiro Masuda <masahi129@gmail.com>
* initial tanh impl
* smalls error
* support uint and int lookup into tables
* reinterpret cast, working tanh tests
* refactor relay func creation
* basic casting tests
* explicitly say do not handle multi-channel lookups
* add example funcs
* fix silent fail
* fix some bugs with floating point funcs not working
* add TODO
* add tood
* canonicalizations
* refactor integer lookup ops into own folder
* fq2i stuff
* clean up existing tests
* flesh out todo
* more tests
* test on keeping shape good
* lookup table fix
* replace canonicalization for rsqrt
* remove canonicalization of rsqrt
* add asf headers
* topi tests
* gather supports unsigned integer tests
* fix things
* move to legalization
* jostle ci
* linting
* use take instead of gather
* remove gather changes
* undo changes
* undo changes
* undo changes
* move thing in range
* initial tanh impl
* smalls error
* support uint and int lookup into tables
* reinterpret cast, working tanh tests
* refactor relay func creation
* basic casting tests
* explicitly say do not handle multi-channel lookups
* add example funcs
* fix silent fail
* fix some bugs with floating point funcs not working
* add TODO
* add tood
* canonicalizations
* refactor integer lookup ops into own folder
* fq2i stuff
* clean up existing tests
* flesh out todo
* more tests
* test on keeping shape good
* lookup table fix
* replace canonicalization for rsqrt
* remove canonicalization of rsqrt
* add asf headers
* gather supports unsigned integer tests
* fix things
* move to legalization
* jostle ci
* linting
* use take instead of gather
* remove gather changes
* undo changes
* undo changes
* undo changes
* move thing in range
* lint
* remove unneeded line
* jostle
Co-authored-by: andrewzhaoluo (generated by with_the_same_user script) <andrewzhaoluo@system76-pc.localdomain>
* [TVMC] Add codegen args to tvmc
This enables external codegen arguments similar to those for `Target`s:
```
tvmc compile --target=cmsis-nn,c --target-cmsis-nn-mcpu=cortex-m55
```
* Add CMSIS-NN decorator to dependent tests
* [microNPU] Add support for pack and unpack
Pack is represented by a series of `expand_dims` operations
followed by a `concatenate` in Relay. Unpack is represented
by a `split` followed by a series of `squeeze` operations in
Relay. This commit legalizes `expand_dims` and `squeeze` to
reshape operations while making use of existing legalization
techniques for `split` and `concatenate` so that pack and
unpack can be offloaded to the NPU.
Change-Id: I3fbebb4ece5ca04598f8e587b9e6c0ddf280266d
* rebase and add tests for expand dims and squeeze
Change-Id: Ic6a9fd77b61368720328bfe82032490bcc66152c
Adding the support of mean on Ethos-N78, which is based on
an underlying pattern matching scheme.
The operator is tested with 2 shapes: 4 and 3 dimensions.
Co-authored-by: Samuel Panijel <samuel.panijel@arm.com>
* [relay] Fix stack overflow in device_planner observed on windows due to recursive function calls.
* Revert "[relay] Fix stack overflow in device_planner observed on windows due to recursive function calls."
This reverts commit 70581364771e2415b37b202a9fa6a937f275cfc6.
* [PyTorch] Add grid_sample with zeros and border padding mode for PyTorch.
* [CUTLASS] Add wgrad support (without split-k)
* run black
* wgrad tests now work under pytest
* dw conv2d properly supported for wgrad
* all tests work
* fixed for sm75
* cpplint
* fix conv2d grad test
* WIP VM memory planning
* tuple projection
* support if
* lint
* remove old comment
* WIP check in attempt at CFG analysis
* rewrite CFG analysis in stages, support ADTs
* lint
* fix small bug in alias elimination, try fix VM profiler error
* update DCE tests since allocations can be DCE'd
* optimize worklist to reduce runtime
* add docs, rename pass to ManifestLifetimes
* add tests, more comments, proper VM profiler fix
* lint
* ci please
* address nits
* retry ci again
* retry ci once again :)
* fix sneaky memory leak due to cyclic refs
* fix didn't work but retry ci anyway
* slightly reduce size of large pretty printer test
* works on resnet18 and deeplabv3
* yolo5 conversion worked
* fixed sigmoid
* [Torch] Support clamp_min, clamp_max
* fixed clamp_min
* fixed quantize for 1 dim input
* cleanup
* improve inline_qparam impl
* add clamp_min/max test
* add fx quant test
* cleanup
* skip build in testing
* black
* improve clamp conversion
* leave TODO on inf handling
Fixes the layout optimizer incorrectly assigning layouts for graphs with
more complex topologies than previously considered. Specifically, this
commit now ensures that intermediate layouts match (e.g. parent output =
child input) and that all consumers are taken into account when altering
the output layout - something not done previously due to an incorrect
traversal order.
Previously, the input layout was always altered if the producer was an
NPU operation without regard to the output layout of that operation.
Additionally, is was possible for the output layout to be incorrectly
set due to a depth-first post-order of traversal of the graph, meaning
it was possible for not all consumers to be taken into account when
altering the layout.
Now the `AnalyzeConsumers` pass is run before `LayoutOptimization` which
determines a mapping from NPU operation to list of boolean values that
represent whether or not each consumer is an NPU operation. Since this
is completed before `LayoutOptimization`, all consumers are guaranteed
to be taken into account when altering the output layout. In turn, the
input layouts can correctly be determined by checking whether the output
of the producer will be altered.
Change-Id: I04e9605da65fa9f12801109dd50c5e3f08cbc73c
* [TVMScript] Added unit tests demonstrating desired functionality
* [TVMScript] Implemented parsing of T.Ptr[...]
These can be generated when exporting to TVMscript, but were not
parsable after being generated.
* [TVMScript] Updated buffer_var printing
LetStmt and AllocateNode can both be used to generate handles that are
used in Buffer objects. In these cases, the Buffer declarations must
go after the handle declaration, not in the function header.
* Moved printing of var and buffer_decl into separate statements.
* Updated following @shingjan's review comments.
* [microNPU][3] Plan generation for the cascader
The cascader creates 'Plans' which describe how
to schedule subgraphs. As part of the cascading
algorithm, it's necessary to explore a large
variety of Plans which are Pareto optimal (in
terms of memory usage and performance). This is
done by the Plan generation algorithm.
This commit adds the TensorConfig and Plan data
structures which hold information on how to schedule
the tensors/operators. Additionally, it includes
functions to calculate Pareto frontiers which are
used to cull sub-optimal Plans.
Change-Id: Ia358b2a1b29bd810df4441027752ced75812ad4e
* Fixes to lint/test
Change-Id: If4e083a3c96af75a8ffa72510704818d21a477d9
* Improve python docs
Change-Id: I831137f8235665bc20ab4c060cc7049ffd48088a
* Fix enum hashing issue with old gcc
Change-Id: Ifbe97eb33b1ef313710f24c687a8155421a3c195
Adds support for legalizing transpose convolution to
a microNPU conv2d operation for the case when strides==(2, 2),
dilation==(1, 1) and no padding of the output is required.
Change-Id: I485e2571913b3dcd7c75c46304f2f9a82f630ee0
* [FIX,AUTOTVM] Add backtraces to tuning errors
Collects tracebacks in LocalBuilder and LocalRunner and adds them to the
error messages.
* formatting
* correctly unpack traceback and exception
* add assert
* fix?
* one remaining measureresult
* formatting
* fixed
* add conv2d transpose nhwc cudnn test
* support conv2d transpose nhwc direct offload to cudnn
* add cutlass dgrad support
* remove unused arg
* allow target none
* fix beta initiaization condition
* disable dynamic dense fp16 test since it fails on cuda 11.6
* [USMP] Add performance characteristics to PoolInfo
Scheduling algorithms that wish to optimize around
memory pools require further information about the
perfomance characteristics of those pools. This
commit adds clock frequency, bandwidth, latency and
burst length as optional fields to PoolInfo.
Change-Id: I4cf3f35324d093fb38e874f0f2e587cb84d4ba1e
* Remove unused import
Change-Id: I1e2ef885425f4361b80c2bab9261ec129e61a756
* [AUTOTVM] Use opt level 3 when extracting tasks
Autotvm was implicitly ignoring opt_level when extracting tasks because
pass opt_level is a thread local variable and extraction happens in a
new thread. Not having opt_level 3 causes alter op layout to not
fire, which in turn prevents tuning from finding all possible kernels.
* disable alter op layout
* [LLVM,TIR] Print LLVM intrinsic names instead of ids
This makes it much easy to understand what is happening with llvm
intrinsics.
* add test, version llvm
* [microNPU] Add support for nearest neighbor and bilinear upsampling
Adds support for 2x2 nearest neighbor and bilinear upsampling. In the
case of bilinear upsampling with align_corners set to true, the
upsampling size must be `2*input_size - 1` (as opposed to `2*input_size`).
Change-Id: I95d215eabfaac983629dcdedcda2b90efb8e0ddf
* rebase and add support for no-upsampling case.
Change-Id: I840d8ee3671a40c5c99f22119442c349dbed39cf
* Adds runtime to AOTExecutorFactoryModule
* Standalone CRT files are added to MLF tarball if runtime is crt
* external_dependencies info added to metadata.json for crt runtime
* microNPU demo Makefile references standalone crt files from MLF tarball
* [Fix Bug]fix the bugs of keras frontend when parsing LSTM, GRU, RNN layers.
* Reformat files with black formatter.
Co-authored-by: AndrewZhaoLuo <andrew.zhao.luo@gmail.com>
* [Runtime][PipelineExecutor] Pipeline Executor Sequential execution
In the first, adding the "get output" logic. Secondly, adding the the sequential executing
logic of pipeline executor. In the last, testing the pipeline executor interface and
checking the output data.
* Address review comments.
Co-authored-by: Cody Yu <comaniac0422@gmail.com>
* trigger build.
Co-authored-by: Cody Yu <comaniac0422@gmail.com>
Before this commit, microNPU creates PrimFunc as if
it accepts constants from the callee. This commit
changes the PrimFunc to remove the constants as an
argument to PrimFunc as they are not provided from
the main function.
Manupa Karunaratne