(See https://discuss.tvm.apache.org/t/byoc-supporting-cutlass-byoc-with-collage/12796/6 for
context, which in turn is part of Collage (https://github.com/apache/tvm-rfcs/blob/main/rfcs/0062-collage.md).
This adds a new 'DSO exportable' runtime module representing the contents of a .o file. It
allows external codegen toolchains to yield a result which:
- Like CSource modules, can be conveyed directly to the final export_library compilation
step for linking into the final .so and saved to a know location without risk the
underlying code artifact will be lost.
- Like DSOLibrary modules, are self contained so that no additional compile-time arguments
need be conveyed from the CSource module to the final export_library command line
Since this is the third flavor of 'DSO exportable' module, add a Module::IsDSOExportable.
Since adding the above, can't resist also adding a Module::ImplementsFunction virtual and
calling it from TEComplier to check if an external codegen function actually provided the
implementation it promised.
Note:
- I've left the existing implementation of runtime.load_module alone which
relinks .o files to .so files.
- Though also contained in the .o metadata, I require static libraries to always
carry their list of exported function names.
This is all pretty stop gap pending a good rework of TVM to supoprt the notion of artifacts
and, perhaps, build rules.
`tvm.analysis.roofline_analysis` adds estimated roofline performance to a
profiling report. The roofline model measures how close an operator gets
to best possible memory bandwidth or FLOP/s depending on whether it is
memory or compute bound. This computation uses the runtime of the
operator along with two numbers extracted from the TIR code: bytes of
memory touched and number of floating point operations. Because these
numbers are extracted from TIR, they may not be 100% accurate. The best
possible memory bandwidth and FLOP/s are measured by running small
programs that are memory and compute bound respectively.
For now, this function only works with llvm cpu targets, but it should
be possible to extend to GPU targets.
* method of creating of NDArray from external DLTensor was implemented
* set input without copying for DLTensor source
* code clean up
* update description and comments after review
Co-authored-by: Valery Chernov <valery.chernov@deelvin.com>
* [RUNTIME] Api to get number of runtime threads
Add `tvm::runtime::threading::NumThreads` and `tvm.runtime.num_threads`
as a way to get the number of threads in use by the TVM runtime.
* check if equal to hardware threads or hardware threads/2
Check that the function to be profiled is actually defined.
Check that the MetricCollector used actually can time the region
requested.
Default to using the module's entry_name instead of "main".
`runtime.TVMArrayAllocWithScope` predates the introduction of
ShapeTuple, and its use simplifies the `tvm.nd.empty` function. The
two modified locations are the only occurrences of the string
"TVMArrayAllocWithScope" in the repository, so no other call sites
should need to be updated.
Modifying the array view is needed for Hexagon targets, in order to
first call `tvm.nd.array` with the physical dimensions, then update the
shape to contain the logical dimensions.
* Add memory pools to Metadata classes.
* Move ShapeToJSON to utils.
* Track returned TensorType from AOTExecutorCodegen.
* Support calling Relay functions with Tuple.
* Expand supported TIR calling conventions to work with C++ runtime.
* Rename MetadataModule to ConstLoaderModule.
* Add runtime AOT executor module.
* Add AOT code-generation.
* Add a runtime Module to mux between .text Metadata and live Metadata.
* Move launch_param to namespace
* Add test of c++ AOT.
* Fix incongruity between kTvmRuntimeCrt constant
* Expand ExecutorCodegenMetadata to include AOT runtime metadata.
* commit cpp test
* Make Metadata compile under C.
* Ignore ephemeral metadata_module export_model_library_format.
* This module does not need to be exported, since it is merely a C++
wrapper around get_c_metadata, and get_metadata is not used in C.
* address manupa, kparszsyc, masahi comments.
* further address comments
* clang and python format
* Fix broken test
* Address lingering comments from masahi, kparszyzc
* set_input_with_index was implemented for VM
* clean code
* add getInputIndexFromName. add function descriptions. lint fix
* fix lint
* transfer comparison of parameter names number and assigned devices number to VMFunction constructor
* add GetVMFunctionWithName to Executable API
* clean code
* add SetInputWithName (set_input_with_name) to VM API
* join SetInputWithIndex and SetInputWithName to SetOneInputTensor (set_one_input) to VM API, the joined methods were removed
* fix lint
* some fixes after review
* add set_one_input method to python API of VirtualMachine
* pytests for set_input and set_one_input methods of VirtualMachine were implemented and checked
* CI restart
* construct simple model for pytests by relay instead of onnx tools (need for correct CI)
Co-authored-by: Valery Chernov <valery.chernov@deelvin.com>
* [FIX,AUTOTVM] Add backtraces to tuning errors
Collects tracebacks in LocalBuilder and LocalRunner and adds them to the
error messages.
* formatting
* correctly unpack traceback and exception
* add assert
* fix?
* one remaining measureresult
* formatting
* fixed
* introduce profile_all_alignments option
* add profile_all_alignment option to API
* wip
* fixed dynamic case
* black
* update gen_gemm too
* minor improvement
* fix
* all tests work
* add doc
* fixed for sm = 75 case
* fix typo
* remove unused import
* profile_all -> find_first_valid
* fix
* Allow export of C++ kernels using correct file extension
* [WIP] Set module_key=c for CSourceCrtMetadataModuleNode to temporarily fix failing tests
I realized that the module format `cc` is currently already used by the `CSourceCrtMetadataModuleNode` declared in `src/target/source/source_module.cc`.
This needs to be discussed first to decide if either the module_key should be changed or the test cases expecting the systemlib kernel (e.g. `default_lib0.c`) to have a `.c` extension.
* Update Makefiles used by tests/python/relay/aot/ to support C++ file extensions
AOT: Add c++ support to aot_test.mk
AOT: Add c++ support to corstone300.mk
* Add missing definition of GetFormat to cmsisnn and ethosn codegens (WIP)
* Resolve PR comments
* lint python/tvm/runtime/module.py
* fix EthosUModuleNode for CI
* Fix: detect empty module.format
* Add error message to assertion
* Lint python/tvm/runtime/module.py
* [PROFILING] Add ability to profile a single function_profiling
Add a new function `tvm.runtime.profiling.profile_function` which
collects performance metrics for a single function in an IRModule. For
example, collecting performance counters using `PAPIMetricCollector`.
This is helpful for optimizing kernels and schedules for a single
operator.
* fix docs
* configurable number of warmup iterations. avoid allocating when stopping collectors
* [Relay] Support large constants.
This allows constant tensors at or above a given byte limit to be marked as
'late bound' and saved/reloaded to a file independently of the overall
executable. Since the executable is often embedded in the data segment of
generated runtime Modules this avoids problems with external tools which can't
handle multi-gigabyte data segments.
[ACE-466 in OctoML JIRA]
* [checkpoint] fix latent bytecode/code bug
* [microNPU] Move the compilation to use Target Hooks.
This commits moves the current compilation flow
to use target hooks, so that the generated TIR
is provided to unified module to for unified
optimizations.
Change-Id: Ib3239a04ab201748e7f1b1ffa503cfe2aa7ccb7b
* [microNPU] Move the compilation to use Target Hooks.
*Fixing unpacked API tests
*Adding use_device_api target attr to example target hooks
Change-Id: I72c51caa57e9a0c2a538f40eb73939e28d4f112f
* [microNPU] Move the compilation to use Target Hooks.
* Modifed CLZ test case to support target hooks
* Modifed reference TIR for test to include allocate annotation
* TIR to CS translation tests are modified to run MakeUnpackedAPI
Change-Id: I3a3d28777a6995e7f2b8789e14c5cb0f280dc763
* [microNPU] Move the compilation to use Target Hooks.
* Added a missed documentation to changes in source module
* Skipping device api test for packed API as microNPU does not
support it.
Change-Id: I6da1adcf8fdd3f972ec9b37ff530ff673e93058c
* [microNPU] Move the compilation to use Target Hooks.
* fixed tvmc test use unpacked-api for microNPU compilation
Change-Id: Ib722d91ca3b3e4c6d13075ee0873acb86f487247
* [microNPU] Move the compilation to use Target Hooks.
* adjust target name.
Change-Id: I862957324440705fb6093939b97b1a00fa1d4b46
* [microNPU] follow up on using target hooks
* Fixed few typos and cleaned up as per suggestions
Change-Id: I2a744a4bc4015e1884dbef4165252aa13aa30b31
* [microNPU] follow up on using target hooks
Fixing some typos and change params to
const_dict as it seems more clearer
Change-Id: Ia36a4635a68f6490bcc3eeaa72eeeeaadb6aa7f6
* [microNPU] Move the compilation to use Target Hooks.
Fixing up lookup table tests to use new runtime module
import structure resulted from using target hooks.
Change-Id: I250aedef7cc73edad3812bb7e9aab013ed8bed5b
We replace use of the TECompiler::{Lower,LowerShapeFunc} methods from the VM's
compiler.cc with LowerTEPass. This clears the way for performing post-lowering
IRModule->IRModule transformations which combine Relay and TIR analysis. In particular,
it will allow us to use the PlanDevices pass to propagate memory scope constraints
across PrimFuncs.
We run LowerTEPass fairly early in the pipeline, which required quite a few passes
to become 'post-lowering friendly'. In particular, ManifestAlloc is now run after
rather than before lowering, and so must now work in a mixed Function/PrimFunc world.
The "vm.shape_func" operator has been removed since a) lowering has already generated
the necessary dynamic shape function, and b) the call to that function can be
represented by an 'ordinary' vm.invoke_tvm_op call.
We worked our way through the following glitches:
- Dynamic shape functions are now given their true type (rather than the type of
the primitive function they are paired with).
- Lowering was choosing definitional GlobalVars which were not pointer-equal to the
referential GlobalVars left behind in the rewritten Calls. We fixed that in
te_compiler.cc, though better would be to push GlobalVars deeper into the
lowering machinery.
- device_copy was rewritten to a call to @__copy without any definition. Though we
tried adding it as a global this (obviously in retrospect...) won't typecheck if
there are multiple device_copies in the program. Instead leave device_copy unchanged
during lowering and update each executor codegen to look for them specially.
- Calls to already-compiled BYOC functions were indistinguishable from calls
to (non-primitive) Relay functions. We move them into the call_lowered calling
convention, and leave behind a Function tagged with "ExternalSymbol". Better would
be a first-class representatn for externals in the IRModule but one step at a time.
- Functions with dynamic shapes tagged for BYOC compilation were not tracking their
connection to their dynamic shape function. We now use exactly the same attributes
as for non-BYOC primitives.
- VerilatorRuntime can legitimately be deleted before initialized.
- IRModule attributes must be preserved. In particular, since LowerTEPass can
be invoked more than once we need to be careful to preserve any existing external
modules and other attributes gatherd from an earlier LowerTEPass.
- GetUniqueName accounts for existing definitions in the module, but is not used
for external functions since their intended names are communicated to the codegen
toolchain via the already fixed "global_symbol" attribute.
This is a grab bag of fallout changes from switching the VM to use LoweTEPass
which can be easily split out of the main #9483 PR.
- AnnotateSpans can be used from C++ (though, unfortunately, it didn't help
me with debugging since spans are universally dropped in most passes).
- Can get a human readable dump of the VM's PackedFunc names and indexes for
debugging.
- If TVM_LOG_DEBUG defined then include types and ids of GlobalVars. I had
a lot of difficulty tracking down where duplicate GlobalVars for the same
name_hint were getting created and propagated.
- GetCallLoweredProps follows same API as GetDeviceCopy and GetOnDevice
where will return 'null' properties if call/expr is not of call_lowered
form. Mildly more convenient, though switching all the above to ICHECK
and push 'if (op == the relevant op)' into all use sites would also be just
fine.
- Misc VLOG improvements made while tracking down issues in #9483.
* Switch PlanDevices pass to be w.r.t. SEScopes instead of DLDeviceTypes.
CAUTION: Breaking VM executable serialization change. I needed a new 'virtual devices' array in the executable so that instructions can continue to refer to devices by a simple index yet the VM can respect both the device type and id for runtime devices.
Continuing from #9313, and as part of apache/tvm-rfcs#38, we switch PlanDevices to plan with respect to SEScopes instead of just DLDeviceTypes. Our ultimate goal is to be able to flow memory scopes between PrimFuncs by re-running PlanDevices after the LowerTE pass. This PR at least gets us to being able to flow the memory scopes, but the actual changes to PlanDevices to look inside PrimFuncs is still two PR's in the future.
However, we get two nice side effects right away:
- Since SEScopes contain Targets we can isolate all the device-to-target resolution machinery within PlanDevices (with the help of CompilationConfig). After PlanDevices has run we can retrieve the Target for any sub-expression directly from that sub-expression's SEScope. For now we retain the one-Target-per-DLDeviceType constraint since it baked into the public 'TargetMap' API, but the path to breaking that constraint is clearer.
- Device ids are now respected all the way from annotation to executor. Previously though we had a bit of plumbing using Devices the device_id therein was ignored or defaulted to zero.
The Python "on_device" annotation helpers still work w.r.t. devices. Thus though they now respect device ids, they do not allow the user to specify a Target or memory scope as supported by the underlying SEScope.
* [checkpoint] Revert emitter.py, must have run 'black .' by mistake.
* [checkpoint] Address PR comments
Also add back SplitArgs pass in build_module.cc which somehow got lost in the shuffle.
(try again -- flaky test_crt.py test_autotune?)
* [checkpoint] Fix after rebase on CallLowered.
* BUG: alloc_tensor offset and reshape shape should be on the CPU
The VM ManifestAlloc pass was allocating constants in a few places I
forgot to tag with on_device for the host/cpu. As a result the runtime
would (silently) do the x-device copy, which destroys perf.
To make this easier to spot in the future added a 'constants' property
to the VM Executable to dump the shape & device for all VM constants.
This is CORE-102 in OctoML JIRA.
* [checkpoint] Older compilers can't handle << overload
* [checkpoint] Woops, forgot requires_cuda
* [UnitTest][Flaky] In test_report_serialization, compare csv.
`str(report)` calls `ReportNode::AsTable()`, which includes aggregate
values. Otherwise negligible differences in the computed value can be
rounded differently after the round trip. This was first [noticed in
CI](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-9194/7/pipeline/#step-246-log-1217)
for an unrelated PR. Testing locally, this failure mode occurred 2
times out of 3000 trials.
Switching to `report.csv()` avoids this issue, as it does not include
aggregates.
* Switched back to using AsTable(), but with column sums disabled.
The .csv column headers are in arbitrary order, and do not test
whether the `device_metrics` field has been serialized/deserialized
correctly.
* Added explicit sorting of columns to Report::AsTable
* [PROFILING] Profiling over RPC
Allow for profiling over RPC by serializing the returned report before
sending it. Also remove collectors argument when profiling over rpc
because it cannot be serialized.
* lint
* fixes
* add comments
Add benchmarking that includes ovearhead of transfering inputs and
outputs to and from the device. This should give an accurate measurement
of the runtime a user would see when using the model. This is
accomplished by adding functions that run from inputs to return values
into the graph executor and the VM.
* [GRAPH EXECUTOR,VM] Add benchmarking function to graph executor and vm
This new benchmarking function is just a convenience function for
calling time_evaluator on the underlying module. Hopefully this should
make it easier for users to get good benchmarks of their code.
* formatting
* import order
* more test, more comments, more precision
* fix tests
* add seconds descriptions to doc
* [PROFILING] Use PAPI to collect hardware performance counters on CPU and CUDA
This PR adds an optional dependency on PAPI
(https://bitbucket.org/icl/papi/) in order to collect hardware
performance counters on CPU and CUDA. These performance counters include
data like total cycles, instructions executed, and cache misses. Users
can control which performance counters are collected by setting the
TVM_PAPI_${DEVICE}_METRICS environment variable to a semicolon separated
list of metrics.
* Update CMakeLists.txt
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
* move thread pool reset out of crt
* add docs
* comments
* formatting
* forgot one doc
* kDLGPU -> kDLCUDA
* Refactor API to more closely match pass instrument's.
* forgot files
* formatting
* more lint
* fix docs
* optional loading of papi metric collector in python
* more formatting
* fix check
* update docs and default value
* formatting
* addressing andrews comments
* fix docs
* address comments
* move shared initialization code into private function
* move most definitions from papi header to implementation file
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
* [Docs] Prevented docs/1 file from being generated.
Typo in tests/scripts/task_sphinx_precheck.sh caused $TVM_HOME/docs/1
file to be created with stderr output, rather than merged stderr and
stdout.
* [Docs] Corrected sphinx build warnings
- Previously, several warnings were generated by sphinx, but were
unintentionally suppressed. This PR resolves the sphinx warnings.
* [Docs] Corrected additional sphinx build warnings.
- Rebased on main and corrected warnings, now up to date as of commit
53e4c603.
* [Docs] Corrected additional sphinx build warnings
- Rebased on main and corrected warnings, now up to date as of commit
1f2ca068c.
* [Docs] Corrected additional sphinx build warnings
- Rebased on main and corrected warnings, now up to date as of commit
d0791d3db.
* [Docs] Ignore sphinx warnings from missing "git describe" and sckit-learn versions.
Co-authored-by: Eric Lunderberg <elunderberg@octoml.ai>
* [Codegen] Use "target.build.$TARGET_KIND" for all codegen functions.
- Removed special case for "micro_dev" target. Instead, register
BuildCHost as both "target.build.c" and "target.build.micro_dev".
- Renamed "target.build.build.aocl_sw_emu" to
"target.build.aocl_sw_emu". Appears to be a typo introduced in
#841725cc585
* [micro_dev] Removed references to non-existent micro_dev
device_api.micro_dev was removed in
745e542e4d, but several references still
remained.
Co-authored-by: Eric Lunderberg <elunderberg@octoml.ai>
* [PROFILER] Add CSV output to profiler
This patch changes the profiler output from a string to a Report object.
A Report can either output CSV or the usual human-readable table.
* no spaces after commas
* Update src/runtime/profiling.cc
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
* fix gcc
* fix test
* overall percent fix
* rename overall -> device_metrics
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
* [Profiling,VM] Profiling interface for VM and Graph Runtime
* lint
* fix test
* make profiling test actually run
* Try to better match the graph runtime function names to vm
* formatting
* DurationNode.value -> microseconds; PercentNode.value -> percent; make frame sorting optional.
* renaming for the tvmcontext -> device change
* formatting
* remove old vm profiler get_stat api
* fix tests
Mark Shields