## Motivation
Historically TVM ships a single monolithic `libtvm.so` that bundles both
the
runtime and the compiler/LLVM-heavy code paths. Deployment scenarios
that only
need the runtime end up paying the full compiler footprint (LLVM-static
dominates
the binary size), and the layout makes it awkward to install the project
under a
single Python package directory the way
`tvm_ffi`/`libinfo.load_lib_ctypes`
expects.
This PR splits the single shared library into two:
- `libtvm_runtime.so` — runtime-only symbols (loaded `RTLD_GLOBAL`).
- `libtvm_compiler.so` — compiler / LLVM / codegen, links
`libtvm_runtime.so`
publicly (loaded `RTLD_LOCAL`).
## Target restructure
- New CMake target `tvm_compiler` replaces the old `tvm` SHARED target.
- `tvm_compiler` depends on `tvm_runtime` via `target_link_libraries(...
PUBLIC tvm_runtime)`,
so anything that linked the old `tvm` now picks up the runtime
transitively.
- `tvm_libinfo_objs` (build-info TU) moved from `tvm_runtime` into
`tvm_compiler`
— it is compiler-side metadata and the runtime no longer needs it.
- All `target_link_libraries` / `target_compile_*` /
`set_target_properties` /
`tvm_ffi_add_apple_dsymutil` callsites have been rewired.
- The separate `libtvm_allvisible.so` target is **removed** (was only
consumed
by cpptests). Cpptests with private-symbol deps are deleted; remaining
cpptests now link directly against `libtvm_compiler.so` /
`libtvm_runtime.so`. `src/support/hexdump.cc` is folded into the header.
- `BUILD_DUMMY_LIBTVM` and the `BUILD_FOR_HEXAGON + USE_HEXAGON_GTEST`
cpp-test wiring are removed.
## Output and install layout
- All artifacts now go to `build/lib/` (was `build/`):
- `build/lib/libtvm_runtime.so`
- `build/lib/libtvm_compiler.so`
- Install layout is now `<package>/lib/` so
`tvm_ffi.libinfo.load_lib_ctypes`
with `package="tvm"` finds the libs in the wheel.
- CI Jenkins stash paths and `apps/hexagon_*` paths updated to the new
`build/lib/...` location.
## Python loader change
`python/tvm/base.py` now resolves the libs directly via a small
`package_lib_paths()` helper in `python/tvm/libinfo.py` (anchored on
`python/tvm/__file__`, returning the wheel `lib/`,
`<worktree>/build/lib`, and
`<worktree>/lib` candidates). Module-level `_LIB_RUNTIME`, `_LIB`, and
`_RUNTIME_ONLY` are set inline at import time:
- `libtvm_runtime.{so,dylib,dll}` loaded `RTLD_GLOBAL`.
- `libtvm_compiler.{so,dylib,dll}` loaded `RTLD_LOCAL`.
- `TVM_USE_RUNTIME_LIB` (parsed strictly: `1`/`true`/`yes`) selects
runtime-only at the loader level.
- When the compiler lib is absent, `_RUNTIME_ONLY` is set to True
automatically and `_LIB is _LIB_RUNTIME`.
## Non-obvious build-integration fixes
Three issues surfaced once both libs are loaded into the same process
and are
worth calling out:
1. **`fpA_intB_gemm` double-registration.** `fpA_intB_gemm_tvm` is an
OBJECT
library that registers a global `fastertransformer.gemm_fp16_int` at
static
init. Linking it into both `tvm_runtime` and `tvm_compiler` made the
registration run twice and trip the duplicate-registration check. Fix:
link
it (and the other runtime-only externals — `flash_attn`, NCCL, NVSHMEM,
RCCL) only into `tvm_runtime`. `tvm_compiler` picks them up via the
PUBLIC
`tvm_runtime` link.
2. **`-Wl,--no-as-needed` for minrpc.** `python/tvm/rpc/minrpc.py`
defaults
to `runtime="libtvm_runtime"` and passes `-Wl,--no-as-needed` so the
runtime static initializers actually run in the spawned minrpc binary
(without it, the linker drops the lib because no symbol is referenced
directly from the minrpc TU). minrpc does **not** link
`libtvm_compiler.so`.
3. **`testing.GetShape{Elem,Size}` moved to runtime.** Those two test
helpers
(the only `testing.*` symbols the minrpc test exercises) were registered
in
`src/support/ffi_testing.cc` (compiler-side). They are now registered in
`src/runtime/rpc/testing.cc` under `rpc.testing.GetShape{Elem,Size}` so
the minrpc server binary — runtime-only — can resolve them.
## Deprecations and breaking changes
- `BUILD_DUMMY_LIBTVM` is **removed** (option, libinfo entry, and CMake
wiring). Downstream consumers that built the dummy variant should link
`libtvm_runtime.so` directly.
- **Breaking change for downstream consumers** that read `libtvm.so` by
name:
there is no longer a `libtvm.so`. Replace with `libtvm_compiler.so`
(full)
or `libtvm_runtime.so` (runtime-only). The Vulkan device comment and a
few
test/CI comments have been updated accordingly.
- `libtvm_allvisible.so` is **removed**. Cpptests that depended on
private
out-of-line symbols have been deleted; the remaining cpp-test contract
is
documented as "public API or private header-only API only" (see
`tests/cpp/`).
- `tests/cpp-runtime/` (Hexagon + OpenCL backend tests) is **removed**
until
TVM moves to a plugin-mode backend architecture where each backend can
ship its own test harness with its own visibility scope.
## Tested
- `ninja` build: `build/lib/libtvm_runtime.so`,
`build/lib/libtvm_compiler.so`;
no `build/libtvm.so`, no `build/lib/libtvm_allvisible.so`.
`ldd build/lib/libtvm_compiler.so` links `libtvm_runtime.so`,
`libtvm_ffi.so`, `libfpA_intB_gemm.so`, `libflash_attn.so`.
- `ldd build/cpptest`: only `libtvm_compiler.so` + `libtvm_runtime.so` +
`libtvm_ffi.so` (no `libtvm_allvisible.so`).
- `./build/cpptest`: 144 / 144 tests pass across 29 suites.
- Smoke imports: full and `TVM_USE_RUNTIME_LIB=1` — both pass.
`TVM_USE_RUNTIME_LIB=0` correctly disables runtime-only mode (strict
parse).
- `tests/python/all-platform-minimal-test`: 75 passed, 77 skipped.
- `tests/python/runtime/`: 81 passed, 2 skipped (incl.
`test_rpc_return_remote_object` exercising the minrpc executable
end-to-end
via `rpc.testing.GetShape{Elem,Size}`).
- `tests/python/relax/test_vm_*.py`: 150 passed, 3 deselected
(`test_vm_multi_device.py` requires 3+ GPUs; host has 2 — env, not
regression),
2 xfailed.
- `tests/python/tirx-base/`: 273 passed, 2 skipped.
- `pre-commit` on edited files: green.
Closes#19443.
This PR fixes RPC tensor cleanup for tensors returned from remote calls.
When a remote function returns a `Tensor`, the RPC protocol sends both:
- the remote backing data pointer
- the remote tensor object handle used for deletion
Previously, `TensorFromRemoteOpaqueHandle` stored only the data pointer
and called
`FreeHandle(space_.data)` during local tensor destruction. That is
incorrect:
`FreeHandle` is meant for remote object handles, not raw data-space
pointers.
This could lead to invalid cleanup behavior and crashes during teardown
in RPC workflows, including the cross-compilation + RPC tutorial
scenario reported in #18923.
This change:
- stores the remote tensor object handle in `RemoteSpace`
- calls `FreeHandle(remote_tensor_handle)` during tensor destruction
- keeps cleanup fault-tolerant if the remote connection is already
closed
This PR cleans up the container redirections and headers
so the files directly points to new ones in ffi folder
- runtime/shape_tuple.h => ffi/container/shape.h
- for IntTuple alias, introduce runtime/int_tuple.h for now
- runtime/container/array.h => ffi/container/array.h
- runtime/container/map.h => ffi/container/map.h
- runtime/container/optional.h => ffi/optional.h
- runtime/container/string.h => ffi/string.h
- runtime/container/variant.h => ffi/container/variant.h
- runtime/container/tuple.h => ffi/container/tuple.h
We also introduce limited number of tvm::ffi classes into tvm namespace,
when they are commonly used and their is no ambiguity.
* [OPENCL][TEXTURE] Improved texture memory planning
Motivated form the fact that textures can be allocated over a clBuffer
object and the size of backing clBuffer can be computed based on
hardware image pitch alignment.
This optimizes the overall memory allocation on device and helps
greately the models with large memory requirements.
Improvised the graph memory planner to not differentiate buffer and
texture storage tokens and reuse them across. The texture pool in OpenCL
runtime is rebranded as memory pool that handles allocation for both
buffer and image objects.
NDArray to DeviceAPI interface is extended with AllocDataSpaceView and
FreeDataSpaceView. These new API's acommodates accessing same physical
memory as clBuffer / clImage objects.
Prior to this PR, each allocator is closely tied with a device.
To enable using a same allocator across different devices of the
same kind when needed, we lift the device to the allocator `Alloc`
interface.
* [VM] memory Manager moved up to runtime
Now graph runtime also uses the same memory manager
This acommodates a common memory manager with pooled and naive support.
As a follow up we can move the WorkspacePool to use this common memory manager.
* * update dependents with new file addition.
* * define memory_manager under new namespace
* * use ShapeTuple across vm executor and memory_manager
* * ShapeTuple across the Allocators
* * GetDataSize is moved to DeviceAPI and memory_manager uses this interface.
* * review comments
* * Make compiler happy with unused variables
* * lint
* Update src/runtime/memory/memory_manager.cc
Co-authored-by: Egor Churaev <egor.churaev@gmail.com>
* * allow multiple allocators to coexist for the same device.
Using available allocator instead of requested is leading to an unpexpected crash
---------
Co-authored-by: Egor Churaev <egor.churaev@gmail.com>
* [VM][OpenCL] Introduce textures allocation to VM memory manager
VM memory manager is extended to support allocation memory with
different memory scope. This functionality is fully implemented for
naive allocator. For pooled allocator is should be implemented in the
future.
* Fix lint
* Fix OpenCL tests
This test was causing cpptest to fail without reporting the
test as having failed. Looking back, this test doesn't really
make much sense as we are passing a file descriptor stating
that inference is running when it isn't. Therefore, removing
the test.
* [ETHOSN] Throw error message when inference fails
Previously the runtime would silently skip interence failures and return
random values as the result. This can make spotting inference failures
challenging. The runtime now throws a fatal error when inference did not
complete successfully along with an error message that gives some
details about the error that occurred.
Change-Id: Iadb6da04ad1c906e3ec49959eb3da0978295aebf
* Address comments
* clarify test file brief
* add test case for running status
* add driver stack reference to WaitStatus class
Change-Id: I792742892b761534904816135ae2ffcb3f028b2c
* link gtest to tvm runtime
* first test running!
* HexagonBuffer tests running in sim
* move to new tests directory
* use USE_HEXAGON_SDK
* add python frontend for Hexagon unit tests
* clean up after rebase
* isolate cmake changes to Hexagon
* add gtest init with arguments
* add hexagon sources only if building for Hexagon; remove workaround
* format & lint
* fix Hexagon build error
* remove x86 implementation and win32 code
* check if hexagon gtest path exists before linking
* make USE_HEXAGON_GTEST an optional cmake param
* turn on Hexagon gtest in Hexagon CI
* Hexagon unit tests should fail if run without proper gtest linkage
* add tvm option; move Hexagon tests to test/cpp-runtime/hexagon
* add libinfo
* trigger ci
* [Hexagon] Generalized HexagonBuffer::CopyTo/CopyFrom
This change operates on the allocation regions in a `HexagonBuffer`,
rather than referencing the managed allocation owned by a buffer,
handling copies between two sets of possibly discontiguous regions.
This will be necessary to handle discontiguous buffers that cannot be
statically planned at compile-time, such as user-initiated
allocations, within a shared memory pool.
Contiguous regions of memory are recognized and result in a single DMA
call.
* [Hexagon] Support both 1-d and 2-d VTCM allocations
Previously, all VTCM allocations were assumed to be 2-d buffers. This
commit extends `HexagonDeviceAPIv2::AllocVtcmWorkspace` to allow both
1-d and 2-d VTCM allocations. Matching the semantics used in
`CodeGenHexagon::CreateBufferPtr`, allocation of 1-d buffers returns a
`void*`, and allocation of 2-d buffers returns a `void**`.
Co-authored-by: Adam Straw <astraw@octoml.ai>
* [Hexagon] Distinguish between 1-d buffer and single-alloc 2-d buffer
Previously, HexagonBuffer represented 1-d buffers as 2-d buffers with
`nallocs==1`. Since this is used to determine the return type of the
data pointer exposed to the generated code, the ambiguity between
`shape=[N]` and `shape=[1,N]` must be avoided. This commit replaces
`HexagonBuffer::nallocs_` with `HexagonBuffer::ndim_`, avoiding this
ambiguity.
* [Hexagon] Treat "global" scope allocations as 1-d
This updates `HexagonDeviceAPIv2::AllocDataSpace` to follow the
semantics of `DeviceAPI::AllocDataSpace`, to avoid breaking caller
assumptions in `tvm.nd.array` or graph_executor/aot allocation.
* Updated C++ unit tests for HexagonBuffer
* Remove commented GetNumAllocs and unused GetBufferDimension
Co-authored-by: Adam Straw <astraw@octoml.ai>
* [Hexagon] Account for objects being smaller than the allocated space
In particular, graph executor will reuse allocated buffers for various
tensors. These tensors may be of various sizes as long as the buffer
is large enough to hold them.
* Trigger CI
* Add #include <algortihm> to hexagon_buffer.cc
* Update cpptest, plus fix one CHECK in hexagon_buffer
* Rename `offset` to `copied` for consistency
* [Relay] VLOG for finer grained control of hyper-detailed logging
I've been making very heavy use of DLOG and PrettyPrint to trace, understand and
debug Relay transforms. I've found myself deleting log statements to reduce the
output verbosity, only to have to re-add them a few days later. Better would be to
support leaving all those statements in place but have finer control over when they
are enabled.
This PR introduces a 'VLOG(level)' macro to that end. The log is ignored
unless TVM_LOG_DEBUG is enabled (both as #define and an environment var), and the
the current verbosity level is >= level. The current verbosity level can be set
globally and/or overridden per source file (see 'VerboseLoggingEnabled').
(Those familiar with the origin of the LOG and DLOG family will also recognize VLOG.)
I also introduce a 'VLOG_CONTEXT' macro which pushes a string onto an internal
per-thread stack. Each VLOG message includes that stack as its prefix, which is a
very handy way to keep track of the (often recursive) program context in which each
VLOG is executed.
I've rejigged some existing DLOGs to VLOGs to illustrate, but left most of them
alone for now. See the draft https://github.com/apache/tvm/pull/8788 for use in the
wild.
I noticed the DCHECK macros *disabled* instead enabled with TVM_LOG_DEBUG defined, so
fixed that.
I've also made changes to the Relay text printer to dump attributes in a human
readable format rather than the indirect but machine readable 'meta' representation.
This is gated by the show_meta_data_ flag, and I think this use is consistent with it's
original purpose.
* [checkpoint] lints
* [checkpoint] missing \n lint
Gotta get my docker setup going
* [checkpoint] woops, we don't support gmock.h
* [checkpoint] Address Hua Jiang's comments.
* [checkpoint] strlen not avail on all toolchains?
* [checkpoint] Rework TVM_LOG_DEBUG spec and parser
* [checkpoint] woops, forgot the static modifier on map
* [checkpoint] * -> DEFAULT for wildcard.
Andrew pointed out *=9 suggests foo/*.cc=9 would work but it is not supported.
* [checkpoint] constexpr length
* [checkpoint] length is not constexpr on all targets, reverting
* [checkpoint] minimize VLOG overhead when in legacy DLOG-only mode
Tianqi Chen