This PR simplifies Jenkins pytest execution around standard pytest-xdist
behavior.
- Runs each already-filtered CPU/GPU suite once with `-n auto`; the
broad suite keeps load-group scheduling because its order-sensitive
cases require it.
- Removes external sharding, wrapper/profile code, JUnit XML generation
and publication, the skipped-test XML consumer, obsolete suite naming,
and orphaned helpers.
- Retains one inert `task_clear_pytest.sh` entry point only because PR
jobs evaluate their Jenkinsfile from the trusted base branch before
checking out the PR; it performs no cleanup or reporting and can be
removed after this pipeline lands.
- Corrects stale broad-suite paths and explicit target guards, and
migrates a scalar stride test to the current `T.handle` pointer
semantics while preserving its negative lowering check.
- Prevents nested MetaSchedule/XGBoost unit tests from multiplying CPU
fanout without serializing the full suite.
- Builds only the `tvm_runtime` target for the secondary GPU
configuration and removes its unconsumed `gpu2` artifact upload.
The result reduces parallelism to one layer managed by pytest-xdist
while preserving GPU filtering and native failure visibility.
This pr adds a tutorial for user to have a quick understanding of how to
import models from our supporting frontends.
Besides, this pr also adds `absl::InitializeLog` and `trackable_obj`
warnings to the CI docs ignore list — these are emitted by TensorFlow's
C++ runtime during import and cannot be suppressed from Python.
This PR removes the root Makefile and inlines its remaining
functionality
into CI scripts and READMEs. Also removes legacy lint scripts that are
now handled by pre-commit.
## Changes
- **Delete root `Makefile`** — all targets either unused or inlined
- **Create `tests/scripts/task_jvm_build.sh`** — extracted JVM platform
detection + mvn build from Makefile
- **Update CI scripts**: `task_python_docs.sh` (cppdoc/javadoc),
`task_java_unittest.sh` (jvmpkg), `cppdocs.sh` (doxygen)
- **Update READMEs**: `docker/README.md`, `jvm/README.md`,
`apps/cpp_rpc/README.md` — replace `make` with direct commands
- **Remove legacy lint scripts** from `tests/lint/` (8 files) —
pre-commit handles all linting now
Subdirectory Makefiles (`docs/Makefile`, `web/Makefile`) are unchanged.
* [FFI][REFACTOR] Establish tvm_ffi as a standalone python module
This PR establishes tvm_ffi as a standalone python module.
The ffi is structured as a minimal pip module that can be
directly install by path or url.
examples/get_started provided a minimal example.
This is a major change as we are decoupling tvm_ffi as a
separate package, users need to install tvm_ffi separately.
Thanks to its minimal dependency, tvm_ffi can be easily installed
even just from the source by pip install ./ffi
This change would enable future improvement for library plugins
to have lightweight dependencies by just working on top of
the tvm_ffi, while the main compiler toolchain and runtime
can be layered on top.
* [FFI] Improve traceback setups
This PR improves traceback related setups
Historically we have been maintaining cython and ctypes for FFI.
One reason was cython was not that available then.
Cython have quite good availability now and also enjoys great
perf benefit. This PR changes the CMake to make cython as required
dependency of the python package and build cython by default.
It also removes ctypes testcases in favor of all cython tests.
If things go well, we can consider remove ctypes and focus on
cython support onward.
Add a script to download 3rd party embedded resources (CSS, JS, etc) to
local files. This ensures that the documentation can be viewed offline
and reduces external dependencies.
The script:
1. Detects external URLs in HTML/CSS files
2. Downloads the resources to _static/downloads
3. Updates the HTML/CSS files to use local copies
* use `20241105-030952-3e386fd3` for ci
* fix missing `language` kwarg in save_rst_example
* disable invalid-name check by pylint
* disable caffe frontend tests
* disable mxnet frontend tests
* fix stablehlo importer
* enable some stablehlo tests
* skip tests because mxnet raise AttributeError
* use stable sort for `np.argsort`
* fix `TypeError: arrays to stack must be passed as a "sequence" type such as list or tuple.`
* disable tests because of shape error
* remove `get_html_theme_path` because it's deprecated
* ignore warnings from sphinx
* disable oneflow frontend tests
* remove oneflow tutorial
* remove debug print
[CI] Update Docker images to tag 20220908-060034-62bdc91b1
Updates all Docker images to tag 20220908-060034-62bdc91b1, to
update TensorFlow/TFLite/Keras to 2.9, and cascaded dependencies
such as numpy. Updates ethos-u-vela to 3.4.0.
It also brings ONNX and PyTorch to ci_arm, to enable Integration
tests to be run in CI.
Standadises the minimum CMake version required in CI to be 3.18.4,
fixing apps/microtvm/zephyr_cmsisnn to require this version.
Finally, adds a new import error in the tutorials documentation
which doesn't affect the final result. The new warning added is
'absl:Found untraced functions such as _jit_compiled_convolution_op'
This implements an initial Target Parser which uses the same logic as
the CMSIS-NN compiler flags to update the features and keys of the `c`
and `llvm` `Target`s.
Refactoring of the CMSIS-NN logic will be in a separate patch.
* First draft of micro train tutorial
* unit test code
* Fix obvious formatting issues
* Linting
* Proof of concept showing that "Open in Colab" is possible
* Make test Python script more readable
* Fix formatting
* Ready for review
* Import pyserial only when needed
Changes from code review
Use official sphinx-gallery repo
Correctly specify version
Import pyserial only when necessary
* Add warning to ignored list
Try to avoid throwing warning
Fix linting, try verbosity filter
Try adding to ignore file
Remove fix attempts
* Grammar fixes
* Address code review comments
Include full git hashes
* Rerun tests
* Rerun again
* Deploy docs to tvm-site/asf-site on main
commit-id:59241556
* Use oauth
* testing code
commit-id:6cc27fce
Co-authored-by: driazati <driazati@users.noreply.github.com>
* Use /usr/bin/env bash in shebang
This makes scripts executable on system without /bin/bash (NixOS)
* Use `set -e` in script instead of `bash -e` in shebang
This adds `ci.py` to the docs to make it more clear how to easily build the docs locally. This also re-arranges CI following the merging of all CI steps to run concurrently since there's no need to run the Sphinx precheck during GPU unit tests. This still preserves it though in the docs step as a way to quickly bail out if there are formatting errors so the full tutorials don't get built.
Co-authored-by: driazati <driazati@users.noreply.github.com>
* Update doc building instructions and pin dependencies
This pins the dependencies for the docs and adds `pytest` as a
dependency which was missing when I built. Tested out the
requirements.txt with a fresh `ubuntu:focal` Docker image to verify that
the required depedencies work.
* Address comments, add Makefile for docs and add to instructions
* Use Python for running scripts
* Fix lint, add lint command
* Add option for cpu to only run the precheck, address comments
* Fix 'make doc' usage, add some -x's
* Fix bad condition on --cpu, add defaults for envs
* Fix another 'make doc'
* Fix for running on MacOS
Co-authored-by: driazati <driazati@users.noreply.github.com>
* Documentation Refactor - Stage 1
RFC: https://github.com/apache/tvm-rfcs/blob/main/rfcs/0027-formalize-documentation-organization.md
Tracking Issue: https://github.com/apache/tvm/issues/8987
Stage 1 of the documentation refactor reorganizes the docs structure,
moving files (without content changes) and adding new scaffolding to
generate the proper document tree.
It does not address naming, style, content, links, or other existing
content in documents that were moved. State 2 will address fixing these
issues with existing content.
Major changes include but are not limited to:
* Dividing the existing tutorials into two sections:
* Tutorials
* How Tos
* Moving all of the existing tutorials out of the `/tutorial`
directory and into the more general `/gallery` directory.
* Breaking up how-tos into individual sections for more
flexibility and more consistent rendering.
* Moving content into new classifications:
* `/docs/arch` for architecture guides
* `/docs/reference` for API guides and other reference material
* `/docs/topic` for topic specific guides such as microTVM and VTA
* Restructuring `/docs/dev`
* Adding a table of contents to the doc index
* Adding instructions on how to install using third-party tlcpack
* Documentation Refactor - Stage 2
RFC: https://github.com/apache/tvm-rfcs/blob/main/rfcs/0027-formalize-documentation-organization.md
Tracking Issue: https://github.com/apache/tvm/issues/8987
Stage 2 of the documentation refactor fixes naming and links
in the documentation to be consistent with the overall structure.
Major changes include:
* an update to how to contribute to docs.
* several updated index pages with title changes to match
the organization style and bring consistency to the sections
* expanded descriptions of some page collections
* fixed links
* Documentation Refactor - Stage 3
RFC: https://github.com/apache/tvm-rfcs/blob/main/rfcs/0027-formalize-documentation-organization.md
Tracking Issue: https://github.com/apache/tvm/issues/8987
Stage 3 of the documentation refactor adjusts CI for the new structure.
The CI build script takes into account the new gallery format. It
also prevents deleting existing documents, and takes advantage of the
`_staging` and `_build` directories to clean out previous builds.
* add a tutorial on auto-scheduling a network for cuda
* fix typo
* fix training time printing
* fix lint
* fix
* upload logs
* fix
* use weighted sum as the default objective function
* update ci logs
* fix the bug in kill_child_processes
* fix test
* address comments
* add early stopping in task scheduler & fix a stuck issue in measurement
* fix lint
* trigger CI
* fix early stopping
* WIP
WIP
* Add support for loading Python packed functions
* Flesh out Relay AST in Rust
* More tweeks for getting functions out
* Deploy Rust docs as part of build
* Add some more types
* Introduce NDArray 2.0
* Work on NDArray 2.0 before restoring tests
* Formatting and code fixes to get it to compile
* Add more Rust bindings
- Converts Conv2d attrs to use tvm::String, so that we can add Rust binding
- Uses Type for checked_type in Rust bindings
- Fix type key in Rust bindings
- Make data field contain NDArray in Rust bindings
* Clean up object ptr passing.
* WIP
* Add debugging for NDArray and fix all test cases
* Add breaking test
* Dispatch some todos
* Format
* Fix ndarray size and len
* Add BiasAddAttrs rust bindings
* Add DenseAttrs rust bindings
* Change to TVM string
* Add more Rust bindings
Add GlobalPool2DAttrs Rust binding
Add ExpandDimsAttrs Rust bindings
Add MaxPool2DAttrs rust bindings
* Fix some test attributes
* Improve the NDArray api
* Fix some more ndarray stuff
* Get the resnet demo kinda working
* Add SoftmaxAttrs Rust bindings
* Implement Hash and Eq for Relay Exprs
* Add underscore to unused function
* Fix broken ass resnet script
* Improve some ndarray conversions
* Make sure the build script runs correctly
* Clean up ResNet example tremedously
Expose C++ graph runtime via cleaner Rust API rewrite example.
* Add ASF header
* Format
* Format
* Format resnet rust python script
* Add type files and refactor span
* Format
* Format
* Change types from std::string to tvm::String in packed function
* Add ASF header
* Fix test w/ ndarray's API change
* Fix array test
* Fix anyhow import
* Put back some anyhow stuff
* Clean up
* Try and fix tests/scripts/task_rust.sh
* Disable ResNet for now
* Turn off building of Rust docs until we update CI
* Actually disable
Co-authored-by: Jared Roesch <jroesch@octoml.ai>
Co-authored-by: Gus Smith <guscomps@gmail.com>
* [RELAY] Fix the FoldConstant Regression for VTA
* [CI] Fix error guard that was missed in VTA.
This PR fixes an error guard during the documentation build step.
- Temporary disables VTA frontend tutorial due to
the regression of deploy_detection
* [CI] Add log check to the sphinx gallery docs
This PR add log check to sphinx gallery tutorials to prevent
the case when sphinx failed to capture the error in tutorials.
* Fix the status
* [WEB] Remove the old web runtime
* [WEB][RUNTIME] TVM WebAssembly Runtime
This PR introduces a brand new TVM web runtime based on the WASM standard API.
Main highlights:
- The new runtime is rewritten using the Typescript.
- The new runtime now directly interfaces with WebAssembly's standard API,
instead of relying on emscripten's API.
This change will make the js runtime more portable to runtime variants.
For example, we could also try to make it interface with the tvm's rust runtime implementation.
- System library can be provided through WASI
- We also build a hack to enable Emscripten to generate a WASI like
bundle for runtime environment on the Web.
- The wasm generation now uses the mainlin LLVM.
- Dynamic link(dlopen) is not used due to limitation of wasm,
instead we rely on the recent new RPC refactor to directly
restart a new session for each wasm binary sent to the RPC.
* Address review comments
* Skip tensorcore test
* [RFC] Pass pytest options globally.
In many places having a global pytest flag is useful . For me with the
build and test of tvm , I would like to be able to globally pass in
pytest options as part of development flow or CI flows where one would
like to measure other things regularly that need measurements including
pytest coverage data that I would like to experiment with across the stack.
This has been achieved with an additional setup-pytest-env.sh file in
tests/scripts rather than putting in something in every single task test
script and something I would like to avoid.
This now means the -v option to pytest is superfluous. I did consider
having a pytest.ini file but that doesn't allow me to pass any old
environment variable in and this seems to be the compromise.
* Improve other use case documentation
* Rationalize pytest environment.
* Remove the setting from docker/with_same_user.
* Take the opportunity to migrate common PYTHONPATH and
TVM_PATH into the common environment setting.
* Fixup vta fsim
* Be more explicit with common PYTHONPATH
* Fix python path for task_python_vta_fsim.sh properly
* Fix nit in documentation.
* [DOCS] Point docs to the ASF site.
We have migrated the main docs to the ASF site,
which will be periodically updated using the docs generated by the CI.
Points the docs to the ASF version.
* [CI] Improve the docs generation script
Tianqi Chen