This PR brings up the tirx namespace. We have been spliting out the
original tir namespace to include high-level component s_tir and this PR
updates the remaining low-level part as tirx namespace
This PR enables ruff pyupgrade (UP) rules with py310 target, auto-fixing
~5600 annotation modernizations (PEP 585 generics, PEP 604 unions,
deprecated typing imports).
Also removes from __future__ import annotations from ir/module.py and
rmsnorm.py, bumps requires-python to >=3.10, and removes absolute_import
aliases from topi/contrib files.
This PR cleans up the python API to make things more consistent
with existing python array api and torch.
Device update
- device_id => index, to be consistent with torch
- device_type => dlpack_device_type() returns int
- added type property same as torch.device
API updates:
- Move the convenient method like cpu() out into tvm runtime to keep device minimal
- tvm_ffi._init_api => tvm_ffi.init_ffi_api
- tvm_ffi.register_func => tvm_ffi.register_global_func
* [FFI][REFACTOR] Establish tvm_ffi as a standalone python module
This PR establishes tvm_ffi as a standalone python module.
The ffi is structured as a minimal pip module that can be
directly install by path or url.
examples/get_started provided a minimal example.
This is a major change as we are decoupling tvm_ffi as a
separate package, users need to install tvm_ffi separately.
Thanks to its minimal dependency, tvm_ffi can be easily installed
even just from the source by pip install ./ffi
This change would enable future improvement for library plugins
to have lightweight dependencies by just working on top of
the tvm_ffi, while the main compiler toolchain and runtime
can be layered on top.
* [FFI] Improve traceback setups
This PR improves traceback related setups
This PR phases out tvm._ffi redirections in favor of new FFI
new functions are now called via tvm.ffi.
We also enabled limited API support for python 3.12+
so the compiled binary can be forward compatible to future
python versions.
This refactor introduces a base Executable class and a `tvm.compile`
interface that can be used to compile both TIR and Relax programs.
`tvm.compile` will return an Executable object that can be used to call
either TIR or Relax functions.
* [REFACTOR] Phase out te.schedule python components
This PR phases out te.schedule python components.
te.compute is kept around for future usages.
tir.Schedule is a more modern version of the scheduling that we can use onwards.
Doing so also helps us to cleanup the testcases that relies on
explicit full build and execution. As we move future unit testcases
towards structural equality based unit tests.
* Simplify CI to focus on UT
The main rationale is that we should only have very few target
dependent UT in tests/python/codegen and possible
a new category in future for op-level integration if needed.
* Re-enable wasm
* fix lint
* remove hybrid,sparse autodoc and remove tests
---------
Co-authored-by: Siyuan Feng <hzfengsy@sjtu.edu.cn>
This PR starts the step 0 to phase out relay from the current
development main branch. This PR focuses on the python
components of relay, autotvm, auto_scheduler. To make the change
manageable, we will also do followup steps on te.Schedule and
c++ components in followup PRs.
To continue support community members who depends on
legacy flows, the [v0.19.0](https://github.com/apache/tvm/tree/v0.19.0)
branch will continue contain these components.
As noted in [discussion on phasing out legacy components](https://discuss.tvm.apache.org/t/phasing-out-legacy-components/17703/30),
this would help us to do two purposes:
- By removing outdated or redundant elements, we can significantly
reduce complexity and improve maintainability.
- Unify our focus: Concentrating our efforts on the new unity flow
will allow for more efficient development and innovation.
It is also a good opportunity for us to revisit and reduce CI time.
The past relay legacy flow contains a lot of end to end tests that
requires hardware resources to run and causing long CI time.
Moving onwards, we can focus more on unit-tests that focuses
on structural equality and runs within seconds, while be mindful
about tests that requires hardware resources (by restricting them
to specific folders and CI nightly in some cases).
---
Co-authored-by: Siyuan Feng <hzfengsy@sjtu.edu.cn>
Graph debug runtime to modifications to acommodate profiling through
BYOC external calls. Updated TVMC interface to add more formats while
profile dump. Added CLML helpers that can rebiuild CPP clml sources
from profile dumps. CLML runtime profiling is now controlled by runtime
profile flag.
* [Frontend][ArgParse] Compile with default(LLVM) target and build with BYOC(#17454)
It is a unique use-case to check the default target(LLVM), though TVM is built with BYOC(MRVL-ON)
The config of Codegen(BYOC) contains default values for configuration/options, it is extracted
during _generate_codegen_args. In command line processing, validate_target_args checks if there
are add-on options and it expects that particular target to be given explicitly in command line.
Here, it is test for default (LLVM) path only, hence validate_target_args need to ignore the codegen's
configuration for default target.
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* [Frontend][ArgParse] Compile with default(LLVM) target and build with BYOC(#17454)
It is a unique use-case to check the default target(LLVM), though TVM is built with BYOC(MRVL-ON)
The config of Codegen(BYOC) contains default values for configuration/options, it is extracted
during _generate_codegen_args. In command line processing, validate_target_args checks if there
are add-on options and it expects that particular target to be given explicitly in command line.
Here, it is test for default (LLVM) path only, hence validate_target_args need to ignore the codegen's
configuration for default target.
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* [Frontend][ArgParse] Compile with default(LLVM) target and build with BYOC(#17454)
It is a unique use-case to check the default target(LLVM), though TVM is built with BYOC(MRVL-ON)
The config of Codegen(BYOC) contains default values for configuration/optons, it is extracted
during _generate_codegen_args. In command line processing, validate_target_args checks if there are
add-on options and it expects that particular target to be given explicitly in command line.
Here, it is test for default (LLVM) path only, hence validate_target_args need to ignore the
codegen's configuration
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* [Frontend][ArgParse] Compile with default(LLVM) target and build with BYOC(#17454)
It is a unique use-case to check the default target(LLVM), though TVM is built with BYOC(MRVL-ON)
The config of Codegen(BYOC) contains default values for configuration/optons, it is extracted
during _generate_codegen_args. In command line processing, validate_target_args checks if there are
add-on options and it expects that particular target to be given explicitly in command line.
Here, it is test for default (LLVM) path only, hence validate_target_args need to ignore the
codegen's configuration
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* [Frontend][ArgParse] Compile with default(LLVM) target and build with BYOC(#17454)
It is a unique use-case to check the default target(LLVM), though TVM is built with BYOC(MRVL-ON)
The config of Codegen(BYOC) contains default values for configuration/optons, it is extracted
during _generate_codegen_args. In command line processing, validate_target_args checks if there are
add-on options and it expects that particular target to be given explicitly in command line.
Here, it is test for default (LLVM) path only, hence validate_target_args need to ignore the
codegen's configuration
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* [Frontend][ArgParse] Compile with default target(LLVM) when built USE_MRVL=ON(#17454)
This is a use-case of invoking TVMC with default target though it is built with MRVL_ON.
In command line processing, validate_target_args checks if there are add-on options
derived from the default arguments of codegen/BYOC and it expects that particular codegen
to be given explicitly in command line. However, certain codegen's can have default target alone,
in that case codegen optios are not extracted there by relaxing the validation
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* [Frontend][ArgParse] Compile with default target(LLVM) when built USE_MRVL=ON(#17454)
This is a use-case of invoking TVMC with default target though it is built with MRVL_ON.
In command line processing, validate_target_args checks if there are add-on options
derived from the default arguments of codegen/BYOC and it expects that particular codegen
to be given explicitly in command line. However, certain codegen's can have default target alone,
in that case codegen optios are not extracted there by relaxing the validation
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* [Frontend][ArgParse] Compile with default target(LLVM) when built USE_MRVL=ON(#17454)
This is a use-case of invoking TVMC with default target though it is built with MRVL_ON.
In command line processing, validate_target_args checks if there are add-on options
derived from the default arguments of codegen/BYOC and it expects that particular codegen
to be given explicitly in command line. However, certain codegen's can have default target alone,
in that case codegen optios are not extracted there by relaxing the validation
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
---------
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
Co-authored-by: M N Ganesan <muthusamynam@marvell.com>
* [Docs] Introduce Relax API and move legacy part to standalone page
As the TVM project evolves, the Unity strategy has been the recommended
way to use Apache TVM applications. Hence, we are pushing documentation
for the Relax API to the forefront and moving the legacy part to a
standalone page, which may be removed in the future.
* update for ci
* update for ci
* Revert "Revert "[FFI][RUNTIME] Introduce runtime boxed types for int/float/bool" (#17252)"
This reverts commit 11be832620.
* [FFI] Re-introduce the boxed primitive values
Initially introduced in https://github.com/apache/tvm/pull/16183,
these changes were reverted in
https://github.com/apache/tvm/pull/17252 due to performance
degredation in some Relax models. This could occur when a model
contained a large number of calls to `"vm.builtin.tuple_getitem"`,
which may occur when model weights are provided as a tuple.
This PR re-applies the changes from
https://github.com/apache/tvm/pull/16183, but with the performance
degredation resolved. The root cause was unnecessary type-checking
when converting from an untyped `tvm::ArrayNode*` to the typed
`tvm::Array<T>`, in the case where `T` is `ObjectRef`.
* Correct typo from T to U
* [Container] Support non-nullable types in Array::Map
Prior to this commit, the `Array::Map` member function could only be
applied to nullable object types. This was due to the internal use of
`U()` as the default value for initializing the output `ArrayNode`, where
`U` is the return type of the mapping function. This default
constructor is only available for nullable types, and would result in
a compile-time failure for non-nullable types.
This commit replaces `U()` with `ObjectRef()` in `Array::Map`,
removing this limitation. Since all items in the output array are
overwritten before returning to the calling scope, initializing the
output array with `ObjectRef()` does not violate type safety.
* [FFI] Separate runtime types from IR types for int/float/bool
Prior to this commit, `int`, `float`, and `bool` arguments from Python
were converted to `IntImm`, `FloatImm`, and `Bool`. These are
subtypes of `PrimExpr`, and should only be used at compile-time. By
automatically applying this conversion as part of the FFI, these types
are required to be present whenever a primitive is converted to a
`tvm::ObjectRef`.
This can become especially fragile for an end-user when storing
objects into a TVM container. Because TVM containers require all
contents to be `ObjectRef` subclasses, an automatic conversion may be
applied on storing into a container, resulting in an unexpected type
being retrieved from the container. For example, this currently
occurs in Relax when extracting a `R.Prim` from a `R.Tuple`.
This commit introduces a `Box<T>` type for storage of boxed primitives
at runtime, distinct from the IR types.
* Primitive arguments provided to a PackedFunc that requires an
`ObjectRef` will be converted to the corresponding boxed type.
(e.g. Passing a Python `int` to a C++ function accepting `ObjectRef`
produces a `Box<int64_t>`.
* Boxed primitives provided to a PackedFunc that requires an unboxed
primitive will be converted to the corresponding primitive.
* PackedFunc return values of `ObjectRef` are converted to the
corresponding primitive, if present. (e.g. If a `tuple_getitem`
with static return type `ObjectRef` returns a `Box<int64_t>`, it
will be unwrapped to a python `int`.)
Together, these three rules provide backwards compatibility for
existing PackedFunc definitions, while avoiding exposing the user to
any container-induced type conversions betweeen primitive types and
`ObjectRef`.
* Fix unit test failure after merge
* Fix breakage in new unit test
* [Frontend][ArgParse] Pass default values to target compiler(#13264)
BYOC Compiler's Config node defines the target compiler's
command line options, along with default values. This change
extract the default values from config node, while constructing
target options for codegen/target compiler.
Added test case for this feature as well.
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* [Frontend][ArgParse] Pass default values to target compiler(#13264)
BYOC Compiler's Config node defines the target compiler's
command line options, along with default values. This change
extract the default values from config node, while constructing
target options for codegen/target compiler.
Added test case for this feature as well.
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
* Lint Fix
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
---------
Signed-off-by: M N Ganesan <muthusamynam@marvell.com>
Co-authored-by: M N Ganesan <muthusamynam@marvell.com>
* [IR] Default to empty attributes, instead of NULL
Prior to this commit, the default `DictAttrs` for an `IRModule`,
`tir::PrimFunc`, `relax::Function`, and `relay::Function` was a null
value. At each callsite, the absence of a `DictAttrs` needed to be
treated as equivalent to an empty `DictAttrs`. In C++, this typically
was done using the `foo->GetAttr` helper function, but in Python it
needed to be checked explicitly. That is, every callsite needed to
check `if func.attrs is not None and attr_name in func.attrs`, rather
than only checking `if attr_name in func.attrs`.
Since most functions would have at least one attribute to specify the
global symbol, these bugs would often surface when working on
unrelated changes.
This commit changes the default attribute dictionary from
`NullValue<DictAttrs>()` to `DictAttrs()`. This avoids having two
separate representations of an object without any attributes, and
allows the `if attr_name in func.attrs` pattern in the Python API.
* Remove no-longer-needed checks on attrs being present
* Fix up unit tests
* More unit test fixes
* Undo erroneous find/replace
* A few more unit tests
* Provide `DictAttrs.get`
Prior to this commit, if a lowered `IRModule` does not contain any TIR
functions, `tvm.relax.build` provided an empty `tir_mod`, which caused
a segfault during TIR compilation. This could occur when
`tvm.relax.build` is called without an explicit target argument, for a
module that does not define any virtual devices.
This commit updates the `_filter_tir` utility function to return
`None` if there are no TIR functions, rather than an empty
`IRModule`. In addition, checks for an empty `IRModule` are added to
`tvm.build` and `TIRToRuntime`, so that a similar failure mode would
raise an exception rather than producing a segfault.
* [Unity] filter out non-GPU primfuncs in default_gpu_schedule
* Add relex heterogeneous e2e case
* Remove get_prim_func_device
* Update test cases
* Fix flake8
* fix lint
* Add test case for change of default_gpu_schedule
* fix comment
Added a new flag `--print-pass-times` for tvmc compile to provide debugging information for tvmc users using `PassTimingInstrument`. Also added a test to check the printing of timing results.
Lines from the initial Relay, which don't correspond to the relay.Call now are printed to the output with the --dump-offloads option enabled. Thus we get rid of the incomprehensible gaps in the line numbers of the initial relay, which occurred before.
---------
Co-authored-by: Sergey Smirnov <89378719+sergey-grovety@users.noreply.github.com>
Co-authored-by: Arina.Naumova <naumova@grovety.com>
This adds a `--tasks` flag to the `tvmc tune` command to filter the lists of tasks to be tuned. See examples below.
## Motivation
- As auto-tuning can be quite time consuming, it is often desirable to cut down the number of tuned tasks in a session.
- If the tuning session was canceled halfway through, it would be a bad idea to start from scratch. Instead continue with the last untuned task
- Some tasks have more impact on the model performance than others, thus we should be able to train some tasks longer than others
## Examples
1. Use `--task list` to show which tasks are available for tuning
```
$ tvmc tune toycar.tflite -o out.txt --task list
Available Tasks for tuning:
0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), 'int...
1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), 'int16...
2. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (128, 128), 'int...
3. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (128, 128), 'int16...
4. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (8, 128), 'int16...
5. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (8, 128), 'int16')...
6. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 8), 'int16'), ('TENSOR', (128, 8), 'int16')...
7. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 8), 'int16'), ('TENSOR', (128, 8), 'int16'), ...
8. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (640, 128), 'int...
9. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (640, 128), 'int16...
```
2. Filter the list of tasks to be tuned:
```
# Only tune a single task (index 5)
tvmc tune toycar.tflite -o out.txt --tasks 5
# Tunes tasks starting with index 6
tvmc tune toycar.tflite -o out.txt --tasks "6-"
# Tune tasks 1,4,5,6,8,9
tvmc tune toycar.tflite -o out.txt --tasks "1,4-6,8-"
```
## Tests
I added a basic unit test for the `filter_tasks` utility in `tests/python/driver/tvmc/test_autotuner.py`.
## Open Questions
- ~~While the (truncated) string representations of AutoTVM tasks are quite helpful to pick the correct tasks, using AutoScheduler the tasks can not really be distinguished from each other (only by index). Is there a way to get similar information from AutoScheduler tasks?~~
Added an option to tvmc and Ethos-U for printing to console or to the file which operators from the initial graph are offloaded to Ethos-U and which aren't. It forms line-by-line output of initial model IR, indicating which operations ported to Ethos-U.
Compiler option "--target-ethos-u-dump_npu_functions_coverage" has been replaced by more generic "--dump-offloads" with the same meaning.
## Usage
```
# output to console:
tvmc compile --target=ethos-u,cmsis-nn,c \
--dump-offloads=- \
........
# output to file:
tvmc compile --target=ethos-u,cmsis-nn,c \
--dump-offloads=<file path> \
........
```
## Example output:
...
Total number of operators and distribution by targets
Total: 211
target1: 198
target2: 10
generic: 3
'target1 <- target2.qnn_conv2d'
'target1 <- %0 = qnn.conv2d(%tfl.quantize, %v_param_1, ...'
'target1 <- %1 = nn.bias_add(%0, %v_param_2, axis=3);'
'target1 <- %2 = qnn.requantize(%1, meta[relay.Constant]...'
'target2 <- target2.reshape'
'target2 <- %3 = reshape(%2, newshape=[1, 1001]);'
'generic <- %4 = nn.pad(%3, -128f, pad_width=[[0, 0], [1, 1]...'
...
this aims to make the `--desired-layout` argument more powerful based on the previously merged changes from #14010 by introducing two new features:
1. Allow passing multiple arguments to `--desired-layout` instead of only one, to specify one layout per transformed operator specified in `--desired-layout-ops`. (Number of arguments has to bei either 1 or match the number of transformed operators)
2. Optionally, you can now specify a non-default kernel layout as follows: `NHWC:HWIO`
Example Usage: `tvmc compile … --desired-layout nn.max_pool2d qnn.conv2d --desired-layout-ops NCHW NHWC:HWIO`
I also added unit tests for the new use-cases.
### Known Limitations:
* It would make sense to specify individual kernel layouts for regular convolutions and depthwise ones. However since both are usually implemented as generalized `nn.conv2d`, we can not transform them individually. Are there any good workarounds for this?
* The arguments of `--desired-layouts` have previously been checked for validity during cmdline parsing (e.g. only NCHW and NHWC are allowed) which is not possible anymore. Should I add a regular expression for that?
Three logger related changes in this patch:
* Currently we don't set the output stream on the Python logger, so
it defaults to sys.stderr, which means we only get some logger output
when the command fails. So set the output stream to sys.stdout
* Currently we can add -v flag to anywhere in the command line for
tvmc compile, but only between tvmc and run/tune for run and tune.
Unify the behaviour such that we can add the flag anywhere on the
command line.
* Set the effective upper bound of -vs to 3 as 4 could result in
NOTSET which would not output anything.
Adds new command line options:
* `--mixed-precision` - Enable mixed precision conversion
* `--mixed-precision-ops` - List of operators to be converted to mixed precision
* `--mixed-precision-calculation-type` - Calculation precision type
* `--mixed-precision-acc-type` - Accumulator precision type
Additionally:
* `--desired-layout-ops` - The list of operators to be transformed with desired layout.
A simple tvmc tune command currently results in a huge wall of warnings
about target_host parameter being deprecated, even when the user hasn't
provided a target-host cmd line argument.
We can prevent that happening from just not providing the default
target-host to tvmc. Also, ensure that when the user does provide
target-host, we print the warning once, not 500 times.
Presently --help for vitis displays the target and option string,
it has no description. Eg: target vitis-ai dpu<class 'str'>
This can be made more meaningful by fetching the description from
the config node of the target. Eg: Vitis AI DPU identifier
Signed-off-by: MNGanesan <mnganesan@yahoo.co.uk>
* [TOOL][NATIVE] Android native appliction for deploy and run
This application helps as a reference for verifying and integration of
TVM compiled models on Android targets natively independent of RPC setup.
tvmc will be used to for compiling tuning and to run it before deployment.
This PR also covers
* Enabling clml for tvmc compilation tool.
* Graph runtime api "get_output_info" to return output tensor specification
similar to "get_input_into"
* This tool adds and enabled 3rdparty dependency "cnpy" to deal with npz files.
* Update apps/cpp_rtvm/README.md
Co-authored-by: Egor Churaev <egor.churaev@gmail.com>
* Update apps/cpp_rtvm/README.md
Co-authored-by: Egor Churaev <egor.churaev@gmail.com>
* * review comments.
* * proof reading
* Update apps/cpp_rtvm/README.md
Co-authored-by: Egor Churaev <egor.churaev@gmail.com>
* * review
Co-authored-by: Egor Churaev <egor.churaev@gmail.com>
* [TVMC] Global pass context for compile and tune
Comes as a followup from conversations in #13216. By making the pass
context a global value for both `compile` and `tune` commands, we can
ensure the pass context is exactly as the user expected and also
test components such as `convert_graph_layout` under a pass context
suitable for testing (e.g. add instruments). With this change, it
becomes the users responsibility to ensure the PassContext they
select is suitable for the passes that will be run. By default,
`opt_level` remains as 3 so current workflows that do not alter the pass
context from the command line / TVMC Python API should not be affected.
Change-Id: I7a601daf6fbe664f77bce1b45efeb7ca29f621b3
* fix vitis-ai test and typo
Change-Id: I04f5bd031ae4717825f42e373bcb0e1e2c1c9d90
* [ETHOSN] Consolidate target string usage
Removes support for a deprecated target string. The deprecation warning
has been around for a couple of releases now so it should be safe to
remove. The target to use moving forward is: `ethos-n -variant=n78 ...`
Refactored direct use of a driver stack target string in the testing
infrastructure to use the same string we expect users to provide. This
simplified some of the code in codegen and hopefully avoids confusion
in the future.
This commit ensures that constant folding is applied when a desired
layout is selected during compilation. It ensures that
`layout_transform` operations are removed where possible so that
pattern matching for BYOC backends can work effectively.
A test has been added to check this regression.
Hopefully fixes#12742, as the warning should only be printed when a user passes `target_host`, in the current case if the user passes `None` as `target_host` it'll be processed by `canon_target_map_and_host` which seems to always produce a `target_host` and thus triggering the warning despite the user doing nothing wrong.
* [TVMC] Run module once by default
Currently executing `tvmc run module.tar` will run the input model
twice. For benchmaking this is to be expected as the first run is used
to prime caches etc before taking a measurement. However, this seems a
bit unintuitive to have as default, especially when benchmarking is not
always intended. In this sense, this commit aims to amend the
number of runs for the default: `tvmc run module.tar` to a single run.
After inspection, this seems to be down to the use of the `.benchmark()`
method which runs (1 + repeat * number) executions in total. This means
that at least two runs are required (i.e. when repeat=1, number=1). It
also seems that it is only necessary to benchmark the model when
`--print-time` has been set from the CLI POV. From the python interface
POV, benchmarking is always run, but this may not always be necessary.
This commit makes use of the `.run()` method to singularly execute the
model by default. From the CLI this will be used when `--print-time` is
set to False whereas from the python interface this will be used when
`benchmark=False`. Otherwise, the `.benchmark()` method will be used
as before. Complementary to this change `repeat`, `number` and
`end_to_end` parameters are only used when either `--print-time` or
`benchmark` are set to True - and the documentation has been updated to
indicate this.
Change-Id: I18a38a9d430d660264f7fce5caf0779aa059fed3
* improve documentation with number of exectuions when benchmarking
Change-Id: Iecf557594420fcc9f3abcec5ce7d952db2c94271
Tianqi Chen