diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3e7f9bf7..f0611ee1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,11 +1,57 @@
# TensorRT OSS Release Changelog
+## [8.0.1](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/tensorrt-8.html#tensorrt-8) - 2021-07-02
+### Added
+- Added support for the following ONNX operators: `Celu`, `CumSum`, `EyeLike`, `GatherElements`, `GlobalLpPool`, `GreaterOrEqual`, `LessOrEqual`, `LpNormalization`, `LpPool`, `ReverseSequence`, and `SoftmaxCrossEntropyLoss` [details]().
+- Rehauled `Resize` ONNX operator, now fully supporting the following modes:
+ - Coordinate Transformation modes: `half_pixel`, `pytorch_half_pixel`, `tf_half_pixel_for_nn`, `asymmetric`, and `align_corners`.
+ - Modes: `nearest`, `linear`.
+ - Nearest Modes: `floor`, `ceil`, `round_prefer_floor`, `round_prefer_ceil`.
+- Added support for multi-input ONNX `ConvTranpose` operator.
+- Added support for 3D spatial dimensions in ONNX `InstanceNormalization`.
+- Added support for generic 2D padding in ONNX.
+- ONNX `QuantizeLinear` and `DequantizeLinear` operators leverage `IQuantizeLayer` and `IDequantizeLayer`.
+ - Added support for tensor scales.
+ - Added support for per-axis quantization.
+- Added `EfficientNMS_TRT`, `EfficientNMS_ONNX_TRT` plugins and experimental support for ONNX `NonMaxSuppression` operator.
+- Added `ScatterND` plugin.
+- Added TensorRT [QuickStart Guide](https://github.com/NVIDIA/TensorRT/tree/master/quickstart).
+- Added new samples: [engine_refit_onnx_bidaf](https://docs.nvidia.com/deeplearning/tensorrt/sample-support-guide/index.html#engine_refit_onnx_bidaf) builds an engine from ONNX BiDAF model and refits engine with new weights, [efficientdet](samples/python/efficientdet) and [efficientnet](samples/python/efficientnet) samples for demonstrating Object Detection using TensorRT.
+- Added support for Ubuntu20.04 and RedHat/CentOS 8.3.
+- Added Python 3.9 support.
+
+### Changed
+- Update Polygraphy to [v0.30.3](tools/Polygraphy/CHANGELOG.md#v0303-2021-06-25).
+- Update ONNX-GraphSurgeon to [v0.3.10](tools/onnx-graphsurgeon/CHANGELOG.md#v0310-2021-05-20).
+- Update Pytorch Quantization toolkit to v2.1.0.
+- Notable TensorRT API updates
+ - TensorRT now declares API’s with the `noexcept` keyword. All TensorRT classes that an application inherits from (such as IPluginV2) must guarantee that methods called by TensorRT do not throw uncaught exceptions, or the behavior is undefined.
+ - Destructors for classes with `destroy()` methods were previously protected. They are now public, enabling use of smart pointers for these classes. The `destroy()` methods are deprecated.
+- Moved `RefitMap` API from ONNX parser to core TensorRT.
+- Various bugfixes for plugins, samples and ONNX parser.
+- Port demoBERT to tensorflow2 and update UFF samples to leverage nvidia-tensorflow1 container.
+
+### Removed
+- `IPlugin` and `IPluginFactory` interfaces were deprecated in TensorRT 6.0 and have been removed in TensorRT 8.0. We recommend that you write new plugins or refactor existing ones to target the `IPluginV2DynamicExt` and `IPluginV2IOExt` interfaces. For more information, refer to [Migrating Plugins From TensorRT 6.x Or 7.x To TensorRT 8.x.x](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#migrating-plugins-6x-7x-to-8x).
+ - For plugins based on `IPluginV2DynamicExt` and `IPluginV2IOExt`, certain methods with legacy function signatures (derived from `IPluginV2` and `IPluginV2Ext` base classes) which were deprecated and marked for removal in TensorRT 8.0 will no longer be available.
+- Removed `samplePlugin` since it showcased IPluginExt interface, which is no longer supported in TensorRT 8.0.
+- Removed `sampleMovieLens` and `sampleMovieLensMPS`.
+- Removed Dockerfile for Ubuntu 16.04. TensorRT 8.0 debians for Ubuntu 16.04 require python 3.5 while minimum required python version for TensorRT OSS is 3.6.
+- Removed support for PowerPC builds, consistent with TensorRT GA releases.
+
+### Notes
+- We had deprecated the Caffe Parser and UFF Parser in TensorRT 7.0. They are still tested and functional in TensorRT 8.0, however, we plan to remove the support in a future release. Ensure you migrate your workflow to use `tf2onnx`, `keras2onnx` or [TensorFlow-TensorRT (TF-TRT)](https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html).
+- Refer to [TensorRT 8.0.1 GA Release Notes](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-801/release-notes/tensorrt-8.html#rel_8-0-1) for additional details
+
+
## [21.06](https://github.com/NVIDIA/TensorRT/releases/tag/21.06) - 2021-06-23
+### Added
+- Add switch for batch-agnostic mode in NMS plugin
+- Add missing model.py in `uff_custom_plugin` sample
### Changed
- Update to [Polygraphy v0.29.2](tools/Polygraphy/CHANGELOG.md#v0292-2021-04-30)
- Update to [ONNX-GraphSurgeon v0.3.9](tools/onnx-graphsurgeon/CHANGELOG.md#v039-2021-04-20)
-- Add missing model.py in `uff_custom_plugin` sample
- Fix numerical errors for float type in NMS/batchedNMS plugins
- Update demoBERT input dimensions to match Triton requirement [#1051](https://github.com/NVIDIA/TensorRT/pull/1051)
- Optimize TLT MaskRCNN plugins:
@@ -13,7 +59,6 @@
- Algorithms optimization for NMS kernels and ROIAlign kernel
- Fix invalid cuda config issue when bs is larger than 32
- Fix issues found on Jetson NANO
-- Add switch for batch-agnostic mode in NMS plugin
### Removed
- Removed fcplugin from demoBERT to improve latency
diff --git a/CMakeLists.txt b/CMakeLists.txt
index a18cbd01..ba3af18f 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -80,8 +80,8 @@ endif()
############################################################################################
# Dependencies
-set(DEFAULT_CUDA_VERSION 11.1)
-set(DEFAULT_CUDNN_VERSION 8.0)
+set(DEFAULT_CUDA_VERSION 11.3.1)
+set(DEFAULT_CUDNN_VERSION 8.2)
set(DEFAULT_PROTOBUF_VERSION 3.0.0)
set(DEFAULT_CUB_VERSION 1.8.0)
diff --git a/README.md b/README.md
index d30b525d..6bb3bb28 100644
--- a/README.md
+++ b/README.md
@@ -15,16 +15,16 @@ This repository contains the Open Source Software (OSS) components of NVIDIA Ten
To build the TensorRT-OSS components, you will first need the following software packages.
**TensorRT GA build**
-* [TensorRT](https://developer.nvidia.com/nvidia-tensorrt-download) v7.2.3.4
+* [TensorRT](https://developer.nvidia.com/nvidia-tensorrt-download) v8.0.1.6
**System Packages**
* [CUDA](https://developer.nvidia.com/cuda-toolkit)
* Recommended versions:
- * cuda-11.x + cuDNN-8.1
- * cuda-10.2 + cuDNN-8.1
+ * cuda-11.3.1 + cuDNN-8.2
+ * cuda-10.2 + cuDNN-8.2
* [GNU make](https://ftp.gnu.org/gnu/make/) >= v4.1
* [cmake](https://github.com/Kitware/CMake/releases) >= v3.13
-* [python]() >= v3.6.5
+* [python]() >= v3.6.9
* [pip](https://pypi.org/project/pip/#history) >= v19.0
* Essential utilities
* [git](https://git-scm.com/downloads), [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/), [wget](https://www.gnu.org/software/wget/faq.html#download), [zlib](https://zlib.net/)
@@ -34,16 +34,16 @@ To build the TensorRT-OSS components, you will first need the following software
* [Docker](https://docs.docker.com/install/) >= 19.03
* [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker)
* Toolchains and SDKs
- * (Cross compilation for Jetson platform) [NVIDIA JetPack](https://developer.nvidia.com/embedded/jetpack) >= 4.4
+ * (Cross compilation for Jetson platform) [NVIDIA JetPack](https://developer.nvidia.com/embedded/jetpack) >= 4.6 (July 2021)
* (For Windows builds) [Visual Studio](https://visualstudio.microsoft.com/vs/older-downloads/) 2017 Community or Enterprise edition
* (Cross compilation for QNX platform) [QNX Toolchain](https://blackberry.qnx.com/en)
* PyPI packages (for demo applications/tests)
- * [onnx](https://pypi.org/project/onnx/1.7.0/) 1.7.0
- * [onnxruntime](https://pypi.org/project/onnxruntime/1.6.0/) >= 1.6.0
- * [tensorflow-gpu](https://pypi.org/project/tensorflow/2.2.2/) >= 2.2.2
- * [Pillow](https://pypi.org/project/Pillow/8.1.2/) >= 8.1.2
- * [pycuda](https://pypi.org/project/pycuda/)
- * [numpy](https://pypi.org/project/numpy/)
+ * [onnx](https://pypi.org/project/onnx/) 1.8.0
+ * [onnxruntime](https://pypi.org/project/onnxruntime/) 1.7.0
+ * [tensorflow-gpu](https://pypi.org/project/tensorflow/) >= 2.4.1
+ * [Pillow](https://pypi.org/project/Pillow/) >= 8.1.2
+ * [pycuda](https://pypi.org/project/pycuda/) < 2020.1
+ * [numpy](https://pypi.org/project/numpy/) 1.21.0
* [pytest](https://pypi.org/project/pytest/)
* Code formatting tools (for contributors)
* [Clang-format](https://clang.llvm.org/docs/ClangFormat.html)
@@ -54,46 +54,38 @@ To build the TensorRT-OSS components, you will first need the following software
## Downloading TensorRT Build
1. #### Download TensorRT OSS
- **On Linux: Bash**
```bash
git clone -b master https://github.com/nvidia/TensorRT TensorRT
cd TensorRT
git submodule update --init --recursive
```
- **On Windows: Powershell**
- ```powershell
- git clone -b master https://github.com/nvidia/TensorRT TensorRT
- cd TensorRT
- git submodule update --init --recursive
- ```
-2. #### Specify the TensorRT Release build
+2. #### (Optional - if not using TensorRT container) Specify the TensorRT GA release build
- If using NVIDIA build containers, TensorRT is preinstalled under `/usr/lib/x86_64-linux-gnu`.
+ If using the TensorRT OSS build container, TensorRT libraries are preinstalled under `/usr/lib/x86_64-linux-gnu` and you may skip this step.
- Else download and extract the TensorRT build from [NVIDIA Developer Zone](https://developer.nvidia.com/nvidia-tensorrt-download).
+ Else download and extract the TensorRT GA build from [NVIDIA Developer Zone](https://developer.nvidia.com/nvidia-tensorrt-download).
- **Example: Ubuntu 18.04 on x86-64 with cuda-11.1**
+ **Example: Ubuntu 18.04 on x86-64 with cuda-11.3**
```bash
cd ~/Downloads
- tar -xvzf TensorRT-7.2.3.4.Ubuntu-18.04.x86_64-gnu.cuda-11.1.cudnn8.1.tar.gz
- export TRT_LIBPATH=`pwd`/TensorRT-7.2.3.4
+ tar -xvzf TensorRT-8.0.1.6.Ubuntu-18.04.x86_64-gnu.cuda-11.3.cudnn8.2.tar.gz
+ export TRT_LIBPATH=`pwd`/TensorRT-8.0.1.6
```
- **Example: Windows on x86-64 with cuda-11.0**
+ **Example: Windows on x86-64 with cuda-11.3**
```powershell
cd ~\Downloads
- Expand-Archive .\TensorRT-7.2.3.4.Windows10.x86_64.cuda-11.0.cudnn8.1.zip
- $Env:TRT_LIBPATH = '$(Get-Location)\TensorRT-7.2.3.4'
+ Expand-Archive .\TensorRT-8.0.1.6.Windows10.x86_64.cuda-11.3.cudnn8.2.zip
+ $Env:TRT_LIBPATH = '$(Get-Location)\TensorRT-8.0.1.6'
$Env:PATH += 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\'
```
-3. #### (Optional) JetPack SDK for Jetson builds
- Using the JetPack SDK manager, download the host components. Steps:
- 1. Download and launch the SDK manager. Login with your developer account.
+3. #### (Optional - for Jetson builds only) Download the JetPack SDK
+ 1. Download and launch the JetPack SDK manager. Login with your NVIDIA developer account.
2. Select the platform and target OS (example: Jetson AGX Xavier, `Linux Jetpack 4.4`), and click Continue.
3. Under `Download & Install Options` change the download folder and select `Download now, Install later`. Agree to the license terms and click Continue.
4. Move the extracted files into the `/docker/jetpack_files` folder.
@@ -101,42 +93,39 @@ To build the TensorRT-OSS components, you will first need the following software
## Setting Up The Build Environment
-For native builds, install the [prerequisite](#prerequisites) *System Packages*. Alternatively (recommended for non-Windows builds), install Docker and generate a build container as described below:
+For Linux platforms, we recommend that you generate a docker container for building TensorRT OSS as described below. For native builds, on Windows for example, please install the [prerequisite](#prerequisites) *System Packages*.
1. #### Generate the TensorRT-OSS build container.
- The TensorRT-OSS build container can be generated using the Dockerfiles and build script included with TensorRT-OSS. The build container is bundled with packages and environment required for building TensorRT OSS.
+ The TensorRT-OSS build container can be generated using the supplied Dockerfiles and build script. The build container is configured for building TensorRT OSS out-of-the-box.
- **Example: Ubuntu 18.04 on x86-64 with cuda-11.1**
+ **Example: Ubuntu 18.04 on x86-64 with cuda-11.3**
```bash
- ./docker/build.sh --file docker/ubuntu-18.04.Dockerfile --tag tensorrt-ubuntu-1804 --cuda 11.1
+ ./docker/build.sh --file docker/ubuntu-18.04.Dockerfile --tag tensorrt-ubuntu18.04-cuda11.3 --cuda 11.3.1
```
- **Example: Ubuntu 18.04 cross-compile for PowerPC with cuda-11.0**
+ **Example: CentOS/RedHat 8 on x86-64 with cuda-10.2**
```bash
- ./docker/build.sh --file docker/ubuntu-cross-ppc64le.Dockerfile --tag tensorrt-ubuntu-ppc --cuda 11.0
+ ./docker/build.sh --file docker/centos-8.Dockerfile --tag tensorrt-centos8-cuda10.2 --cuda 10.2
```
- **Example: CentOS/RedHat 7 on x86-64 with cuda-11.0**
+ **Example: Ubuntu 18.04 cross-compile for Jetson (aarch64) with cuda-10.2 (JetPack SDK)**
```bash
- ./docker/build.sh --file docker/centos-7.Dockerfile --tag tensorrt-centos --cuda 11.0
- ```
- **Example: Ubuntu 18.04 cross-compile for Jetson (arm64) with cuda-10.2 (JetPack SDK)**
- ```bash
- ./docker/build.sh --file docker/ubuntu-cross-aarch64.Dockerfile --tag tensorrt-cross-jetpack --cuda 10.2
+ ./docker/build.sh --file docker/ubuntu-cross-aarch64.Dockerfile --tag tensorrt-jetpack-cuda10.2 --cuda 10.2
```
2. #### Launch the TensorRT-OSS build container.
**Example: Ubuntu 18.04 build container**
```bash
- ./docker/launch.sh --tag tensorrt-ubuntu-1804 --gpus all
+ ./docker/launch.sh --tag tensorrt-ubuntu18.04-cuda11.3 --gpus all
```
> NOTE:
- 1. Use the tag corresponding to the build container.
- 2. To run TensorRT/CUDA programs in the build container, install [NVIDIA Container Toolkit](#prerequisites). Docker versions < 19.03 require `nvidia-docker2` and `--runtime=nvidia` flag for docker run commands. On versions >= 19.03, you need the `nvidia-container-toolkit` package and `--gpus all` flag.
- 3. sudo password for Ubuntu containers is 'nvidia'.
+ 1. Use the `--tag` corresponding to build container generated in Step 1.
+ 2. [NVIDIA Container Toolkit](#prerequisites) is required for GPU access (running TensorRT applications) inside the build container.
+ 3. `sudo` password for Ubuntu build containers is 'nvidia'.
+ 4. Specify port number using `--jupyter ` for launching Jupyter notebooks.
## Building TensorRT-OSS
* Generate Makefiles or VS project (Windows) and build.
- **Example: Linux (x86-64) build with default cuda-11.1**
+ **Example: Linux (x86-64) build with default cuda-11.3**
```bash
cd $TRT_OSSPATH
mkdir -p build && cd build
@@ -157,13 +146,6 @@ For native builds, install the [prerequisite](#prerequisites) *System Packages*.
cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DTRT_OUT_DIR=`pwd`/out -DCMAKE_TOOLCHAIN_FILE=$TRT_OSSPATH/cmake/toolchains/cmake_aarch64.toolchain -DCUDA_VERSION=10.2
make -j$(nproc)
```
- **Example: Cross-Compile for QNX with cuda-10.2**
- ```bash
- cd $TRT_OSSPATH
- mkdir -p build && cd build
- cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DTRT_OUT_DIR=`pwd`/out -DCMAKE_TOOLCHAIN_FILE=$TRT_OSSPATH/cmake/toolchains/cmake_qnx.toolchain -DCUDA_VERSION=10.2
- make -j$(nproc)
- ```
**Example: Windows (x86-64) build in Powershell**
```powershell
cd $Env:TRT_OSSPATH
@@ -172,16 +154,15 @@ For native builds, install the [prerequisite](#prerequisites) *System Packages*.
msbuild ALL_BUILD.vcxproj
```
> NOTE:
- 1. The default CUDA version used by CMake is 11.1. To override this, for example to 10.2, append `-DCUDA_VERSION=10.2` to the cmake command.
- 2. If samples fail to link on CentOS7, create this symbolic link: `ln -s $TRT_OUT_DIR/libnvinfer_plugin.so $TRT_OUT_DIR/libnvinfer_plugin.so.7`
+ 1. The default CUDA version used by CMake is 11.3.1. To override this, for example to 10.2, append `-DCUDA_VERSION=10.2` to the cmake command.
+ 2. If samples fail to link on CentOS7, create this symbolic link: `ln -s $TRT_OUT_DIR/libnvinfer_plugin.so $TRT_OUT_DIR/libnvinfer_plugin.so.8`
* Required CMake build arguments are:
- `TRT_LIB_DIR`: Path to the TensorRT installation directory containing libraries.
- `TRT_OUT_DIR`: Output directory where generated build artifacts will be copied.
* Optional CMake build arguments:
- `CMAKE_BUILD_TYPE`: Specify if binaries generated are for release or debug (contain debug symbols). Values consists of [`Release`] | `Debug`
- - `CUDA_VERISON`: The version of CUDA to target, for example [`11.1`].
- - `CUDNN_VERSION`: The version of cuDNN to target, for example [`8.0`].
- - `NVCR_SUFFIX`: Optional nvcr/cuda image suffix. Set to "-rc" for CUDA11 RC builds until general availability. Blank by default.
+ - `CUDA_VERISON`: The version of CUDA to target, for example [`11.3.1`].
+ - `CUDNN_VERSION`: The version of cuDNN to target, for example [`8.2`].
- `PROTOBUF_VERSION`: The version of Protobuf to use, for example [`3.0.0`]. Note: Changing this will not configure CMake to use a system version of Protobuf, it will configure CMake to download and try building that version.
- `CMAKE_TOOLCHAIN_FILE`: The path to a toolchain file for cross compilation.
- `BUILD_PARSERS`: Specify if the parsers should be built, for example [`ON`] | `OFF`. If turned OFF, CMake will try to find precompiled versions of the parser libraries to use in compiling samples. First in `${TRT_LIB_DIR}`, then on the system. If the build type is Debug, then it will prefer debug builds of the libraries before release versions if available.
@@ -199,13 +180,14 @@ For native builds, install the [prerequisite](#prerequisites) *System Packages*.
## TensorRT Resources
-* [TensorRT Homepage](https://developer.nvidia.com/tensorrt)
+* [TensorRT Developer Home](https://developer.nvidia.com/tensorrt)
+* [TensorRT QuickStart Guide](https://docs.nvidia.com/deeplearning/tensorrt/quick-start-guide/index.html)
* [TensorRT Developer Guide](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html)
* [TensorRT Sample Support Guide](https://docs.nvidia.com/deeplearning/tensorrt/sample-support-guide/index.html)
+* [TensorRT ONNX Tools](https://docs.nvidia.com/deeplearning/tensorrt/index.html#tools)
* [TensorRT Discussion Forums](https://devtalk.nvidia.com/default/board/304/tensorrt/)
-* [TensorRT Release Notes](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html).
+* [TensorRT Release Notes](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html)
## Known Issues
-#### TensorRT 7.2.3.4
* None
diff --git a/VERSION b/VERSION
index f9980270..fd49265d 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-7.2.3.4
+8.0.1.6
diff --git a/cmake/toolchains/cmake_aarch64.toolchain b/cmake/toolchains/cmake_aarch64.toolchain
index 7fbde2ff..2bb8ffa1 100644
--- a/cmake/toolchains/cmake_aarch64.toolchain
+++ b/cmake/toolchains/cmake_aarch64.toolchain
@@ -32,6 +32,8 @@ set(CMAKE_C_COMPILER_FORCED TRUE)
set(CMAKE_CXX_COMPILER_FORCED TRUE)
set(CUDA_ROOT /usr/local/cuda-${CUDA_VERSION}/targets/${CUDA_PLATFORM_ID} CACHE STRING "CUDA ROOT dir")
+set(CUDNN_ROOT_DIR /pdk_files/cudnn)
+set(BUILD_LIBRARY_ONLY 1)
set(CUDA_TOOLKIT_ROOT_DIR ${CUDA_ROOT})
set(CUDA_INCLUDE_DIRS ${CUDA_ROOT}/include)
@@ -39,7 +41,7 @@ set(CUDA_INCLUDE_DIRS ${CUDA_ROOT}/include)
set(RT_LIB /usr/aarch64-linux-gnu/lib/librt.so)
set(CMAKE_CUDA_HOST_COMPILER ${CMAKE_CXX_COMPILER} CACHE STRING "" FORCE)
-set(CMAKE_CUDA_FLAGS "-cudart none -I${CUDA_INCLUDE_DIRS} -Xcompiler=\"-fPIC ${CMAKE_CXX_FLAGS}\"" CACHE STRING "" FORCE)
+set(CMAKE_CUDA_FLAGS "-I${CUDA_INCLUDE_DIRS} -Xcompiler=\"-fPIC ${CMAKE_CXX_FLAGS}\"" CACHE STRING "" FORCE)
set(CMAKE_CUDA_COMPILER_FORCED TRUE)
set(CUDA_LIBS -L${CUDA_ROOT}/lib)
diff --git a/demo/BERT/README.md b/demo/BERT/README.md
old mode 100644
new mode 100755
index 07d5d49f..7e65eb09
--- a/demo/BERT/README.md
+++ b/demo/BERT/README.md
@@ -20,6 +20,11 @@ This subfolder of the BERT TensorFlow repository, tested and maintained by NVIDI
- [Accuracy](#accuracy)
* [Evaluating Post-Training-Quantization INT8 accuracy](#evaluating-ptq-post-training-quantization-int8-accuracy-using-the-squad-dataset)
* [Evaluating Quantization-Aware-Training INT8 accuracy](#evaluating-qat-quantization-aware-training-int8-accuracy-using-the-squad-dataset)
+- [Experimental](#experimental)
+ * [Variable sequence length](#variable-sequence-length)
+ * [Run command lines](#run-command-lines)
+ * [Sparsity with Quantization Aware Training](#sparsity-with-quantization-aware-training)
+ * [Megatron-LM for Question Answering](#megatron-lm-for-question-answering)
- [Performance](#performance)
* [Benchmarking](#benchmarking)
* [TensorRT inference benchmark](#tensorrt-inference-benchmark)
@@ -33,9 +38,6 @@ This subfolder of the BERT TensorFlow repository, tested and maintained by NVIDI
* [Inference performance: NVIDIA V100](#inference-performance-nvidia-v100-16gb)
* [BERT Base](#bert-base-2)
* [BERT Large](#bert-large-2)
-- [Experimental](#experimental)
- * [Variable sequence length](#variable-sequence-length)
- * [Run command lines](#run-command-lines)
## Model overview
@@ -78,9 +80,9 @@ The following software version configuration has been tested:
|Software|Version|
|--------|-------|
-|Python|3.6.9|
-|TensorRT|7.2.3.4|
-|CUDA|11.1.1|
+|Python|>=3.6.x|
+|TensorRT|8.0.1.6|
+|CUDA|11.3.1|
## Setup
@@ -93,9 +95,10 @@ This demo BERT application can be run within the TensorRT OSS build container. I
* [NGC CLI](https://ngc.nvidia.com/setup/installers/cli) - for downloading BERT checkpoints from NGC.
* PyPI Packages:
- * [pycuda](https://pypi.org/project/pycuda/) (tested 2019.1.2)
- * [onnx](https://pypi.org/project/onnx/1.7.0/) (tested 1.7.0)
- * [tensorflow](https://pypi.org/project/tensorflow/) >= 2.2
+ * [pycuda](https://pypi.org/project/pycuda/) (tested v2019.1.2)
+ * [onnx](https://pypi.org/project/onnx) (tested v1.8.1)
+ * [tensorflow](https://pypi.org/project/tensorflow/) (tested v2.4.1)
+ * [torch](https://pypi.org/project/torch/1.6.0/) (tested v1.8.1)
* NVIDIA [Volta](https://www.nvidia.com/en-us/data-center/volta-gpu-architecture/), [Turing](https://www.nvidia.com/en-us/geforce/turing/) or [Ampere](https://www.nvidia.com/en-us/data-center/nvidia-ampere-gpu-architecture/) based GPU.
@@ -133,7 +136,7 @@ This demo BERT application can be run within the TensorRT OSS build container. I
mkdir -p engines && python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/model.ckpt -o engines/bert_large_128.engine -b 1 -s 128 --fp16 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1
```
- This will build an engine with a maximum batch size of 1 (`-b 1`), and sequence length of 128 (`-s 128`) using mixed precision (`--fp16`) using the BERT Large SQuAD v2 FP16 Sequence Length 128 checkpoint (`-c /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1`).
+ This will build an engine with a maximum batch size of 1 (`-b 1`), and sequence length of 128 (`-s 128`) using mixed precision (`--fp16`) using the BERT Large SQuAD v2 FP16 Sequence Length 128 checkpoint (`-c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1`).
5. Run inference. Two options are provided for running the model.
@@ -161,7 +164,7 @@ This demo BERT application can be run within the TensorRT OSS build container. I
cmake .. -DPYTHON_EXECUTABLE=$(which python)
make -j
popd
- python3 inference_c.py -e /workspace/TensorRT/demo/BERT/engines/bert_large_128.engine --enable-graph -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt
+ python3 inference_c.py -e engines/bert_large_128.engine --enable-graph -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt
```
A separate C/C++ inference benchmark executable `perf` (compiled from `perf.cpp`) is provided to run inference benchmarks with CUDA Graph. The cmdline interface is the same as `perf.py` except for an extra `--enable_graph` option.
@@ -293,6 +296,106 @@ As mentioned in the [Quick Start Guide](#quick-start-guide), two options are pro
python3 inference.py -e engines/bert_large_384_int8mix.engine -s 384 -sq ./squad/dev-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -o ./predictions.json
python3 squad/evaluate-v1.1.py squad/dev-v1.1.json ./predictions.json 90
```
+## Experimental
+
+### Variable sequence length
+In our prior implementation, we used inputs padded to max length along with corresponding input masks to handle variable sequence length inputs in a batch. The padding results in some wasted computations which can be avoided by handling variable sequence length inputs natively. Now we have a new approach called the variable sequence length method. By concatenating each input id into a single long input id, and concatenating each input segment id into a single long segment id, TensorRT can know the exact starts and ends by providing an extra sequence length buffer that contains the start and end positions of each sequence. Now we can eliminate the wasted computation in the input paddings.
+
+Note this is an experimental feature because we only support Xavier+ GPUs, also there is neither FP32 support nor INT8 PTQ calibration.
+
+1. Download checkpoint for BERT Large FP16 SQuAD v1.1 model with sequence length of 384:
+ ```bash
+ bash scripts/download_model.sh pyt v1_1
+ ```
+
+2. Build an engine:
+
+ **FP16 engine**
+ ```bash
+ mkdir -p engines && python3 builder_varseqlen.py -x models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx -o engines/bert_varseq_fp16.engine -b 1 -s 64 --fp16 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt
+ ```
+
+ This will build and engine with a maximum batch size of 1 (`-b 1`) and sequence length of 64 (`-s 64`) using FP16 precision computation where possible (`--fp16`).
+
+
+ **INT8 engine**
+ ```bash
+ mkdir -p engines && python3 builder_varseqlen.py -x models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx -o engines/bert_varseq_int8.engine -b 1 -s 256 --int8 --fp16 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt
+ ```
+
+ This will build and engine with a maximum batch size of 1 (`-b 1`) and sequence length of 256 (`-s 256`) using INT8 precision computation where possible (`--int8`).
+
+3. Run inference
+
+ Evaluate the F1 score and exact match score using the squad dataset:
+
+ ```bash
+ python3 inference_varseqlen.py -e engines/bert_varseq_int8.engine -s 256 -sq ./squad/dev-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -o ./predictions.json
+ python3 squad/evaluate-v1.1.py squad/dev-v1.1.json ./predictions.json 90
+ ```
+
+ Run the quesion and answer mode:
+
+ ```bash
+ python3 inference_varseqlen.py -e engines/bert_varseq_int8.engine -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -s 256
+ ```
+
+4. Collect performance data
+
+ ```bash
+ python3 perf_varseqlen.py -e engines/bert_varseq_int8.engine -b 1 -s 256
+ ```
+
+ This will collect performance data run use batch size 1 (`-b 1`) and sequence length of 256 (`-s 256`).
+
+5. Collect performance data with CUDA graph enabled
+
+ We can use the same `inference_c.py` and `build/perf` to collect performance data with cuda graph enabled. The command line is the same as run without variable sequence length.
+
+### Sparsity with Quantization Aware Training
+
+Fine-grained 2:4 structured sparsity support introduced in NVIDIA Ampere GPUs can produce significant performance gains in BERT inference. The network is first trained using dense weights, then fine-grained structured pruning is applied, and finally the remaining non-zero weights are fine-tuned with additional training steps. This method results in virtually no loss in inferencing accuracy.
+
+Using INT8 precision with quantization scales obtained from Post-Training Quantization (PTQ) can produce additional performance gains, but may also result in accuracy loss. Alternatively, for PyTorch-trained models, NVIDIA [PyTorch-Quantization toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization) can be leveraged to perform quantized fine tuning (a.k.a. Quantization Aware Training or QAT) and generate the INT8 quantization scales as part of training. This generally results in higher accuracy compared to PTQ.
+
+To demonstrate the potential speedups from these optimizations in demoBERT, we provide the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) transformer model finetuned for SQuAD 2.0 task with sparsity and quantization.
+
+The sparse weights are generated by finetuning with INT8 Quantization Aware Training recipe. This feature can be used with the fixed or variable sequence length implementations by passing in `-sp` flag to demoBERT builder.
+
+#### Megatron-LM for Question Answering
+
+##### Example: Megatron-LM Large SQuAD v2.0 with sparse weights for sequence length 384
+
+**Build the TensorRT engine**:
+
+Options specified:
+* `--megatron` : assume Megatron style residuals instead of vanilla BERT.
+* `--pickle` : specify a pickle file containing the PyTorch statedict corresponding to fine-tuned Megatron model.
+* `-sp` : enable sparsity during engine optimization and treat the weights as sparse.
+* `--int8 --il` : enable int8 tactics/plugins with interleaving.
+
+```bash
+bash ./scripts/download_model.sh 384 v1_1 # BERT-large model checkpoint fine-tuned for SQuAD 1.1
+bash ./scripts/download_model.sh pyt megatron-large int8-qat sparse # Megatron-LM model weights
+export CKPT_PATH=models/fine-tuned/bert_pyt_statedict_megatron_sparse_int8qat_v21.03.0/bert_pyt_statedict_megatron_sparse_int8_qat
+mkdir -p engines && python3 builder_varseqlen.py -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -b 1 -s 384 -o engines/megatron_large_seqlen384_int8qat_sparse.engine --fp16 --int8 --strict -il --megatron --pickle $CKPT_PATH -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -sp
+```
+
+**Ask a question**:
+```bash
+python3 inference_varseqlen.py -e engines/megatron_large_seqlen384_int8qat_sparse.engine -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -s 256
+```
+
+**Evaluate F1 score**:
+```bash
+python3 inference_varseqlen.py -e engines/megatron_large_seqlen384_int8qat_sparse.engine -s 384 -sq ./squad/dev-v1.1.json -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -o ./predictions.json
+python3 squad/evaluate-v1.1.py squad/dev-v1.1.json ./predictions.json 90
+```
+Expected output:
+```
+&&&& PASSED TensorRT BERT Squad Accuracy matches reference.
+{"exact_match": 84.03973509933775, "f1": 90.88667129897755}
+```
## Performance
@@ -322,231 +425,4 @@ Also note that BERT Large engines, especially using mixed precision with large b
### Results
-The following sections provide details on how we achieved our performance and inference.
-
-#### Inference performance: NVIDIA A100 (40GB)
-
-Our results were obtained by running the `scripts/inference_benchmark.sh --gpu Ampere` script in the container generated by the TensorRT OSS Dockerfile on NVIDIA A100 with (1x A100 40G) GPUs.
-
-##### BERT Base
-
-| Sequence Length | Batch Size | INT8 Latency (ms) | | | FP16 Latency (ms) | | |
-|-----------------|------------|-----------------|-----------------|---------|-----------------|-----------------|---------|
-| | | 95th Percentile | 99th Percentile | Average | 95th Percentile | 99th Percentile | Average |
-| 128 | 1 | 0.77 | 0.77 | 0.77 | 0.78 | 0.80 | 0.78 |
-| 128 | 2 | 0.76 | 0.77 | 0.76 | 0.92 | 0.93 | 0.92 |
-| 128 | 4 | 0.93 | 1.18 | 0.93 | 1.19 | 1.51 | 1.19 |
-| 128 | 8 | 1.19 | 1.20 | 1.19 | 1.78 | 1.78 | 1.77 |
-| 128 | 12 | 1.57 | 1.57 | 1.56 | 2.07 | 2.08 | 2.05 |
-| 128 | 16 | 1.88 | 1.89 | 1.88 | 2.54 | 2.60 | 2.52 |
-| 128 | 24 | 2.65 | 2.65 | 2.64 | 3.65 | 3.70 | 3.61 |
-| 128 | 32 | 3.21 | 3.22 | 3.21 | 4.71 | 4.74 | 4.67 |
-| 128 | 64 | 5.69 | 5.70 | 5.64 | 8.87 | 8.96 | 8.81 |
-| 128 | 128 | 10.84 | 10.85 | 10.70 | 17.61 | 17.62 | 17.44 |
-| 384 | 1 | 1.34 | 1.35 | 1.34 | 1.46 | 1.46 | 1.45 |
-| 384 | 2 | 1.56 | 1.79 | 1.56 | 1.85 | 1.85 | 1.84 |
-| 384 | 4 | 2.02 | 2.03 | 2.02 | 2.46 | 2.46 | 2.45 |
-| 384 | 8 | 2.94 | 2.95 | 2.94 | 3.91 | 3.92 | 3.86 |
-| 384 | 12 | 4.07 | 4.07 | 4.06 | 5.54 | 5.55 | 5.47 |
-| 384 | 16 | 5.22 | 5.23 | 5.21 | 7.78 | 7.79 | 7.69 |
-| 384 | 24 | 7.42 | 7.42 | 7.37 | 10.75 | 10.76 | 10.63 |
-| 384 | 32 | 9.92 | 9.93 | 9.77 | 14.58 | 14.73 | 14.52 |
-| 384 | 64 | 18.74 | 18.78 | 18.61 | 28.66 | 28.70 | 28.39 |
-| 384 | 128 | 36.40 | 36.42 | 36.05 | 55.36 | 55.90 | 55.21 |
-
-##### BERT Large
-
-| Sequence Length | Batch Size | INT8 Latency (ms) | | | FP16 Latency (ms) | | |
-|-----------------|------------|-----------------|-----------------|---------|-----------------|-----------------|---------|
-| | | 95th Percentile | 99th Percentile | Average | 95th Percentile | 99th Percentile | Average |
-| 128 | 1 | 1.60 | 1.61 | 1.60 | 1.87 | 1.88 | 1.87 |
-| 128 | 2 | 1.94 | 1.95 | 1.94 | 2.36 | 2.37 | 2.35 |
-| 128 | 4 | 2.45 | 2.46 | 2.44 | 3.36 | 3.37 | 3.36 |
-| 128 | 8 | 3.82 | 3.83 | 3.79 | 4.98 | 5.00 | 4.95 |
-| 128 | 12 | 4.22 | 4.23 | 4.22 | 6.45 | 6.46 | 6.38 |
-| 128 | 16 | 5.75 | 5.75 | 5.74 | 8.50 | 8.53 | 8.43 |
-| 128 | 24 | 7.10 | 7.11 | 7.04 | 11.47 | 11.49 | 11.31 |
-| 128 | 32 | 9.61 | 9.61 | 9.51 | 15.49 | 15.50 | 15.25 |
-| 128 | 64 | 17.25 | 17.25 | 17.11 | 29.43 | 29.73 | 29.29 |
-| 128 | 128 | 33.25 | 33.58 | 33.05 | 56.98 | 57.17 | 56.68 |
-| 384 | 1 | 3.00 | 3.01 | 2.99 | 3.52 | 3.53 | 3.51 |
-| 384 | 2 | 3.71 | 3.72 | 3.71 | 4.97 | 4.99 | 4.97 |
-| 384 | 4 | 5.08 | 5.09 | 5.08 | 7.01 | 7.01 | 6.92 |
-| 384 | 8 | 9.04 | 9.05 | 9.04 | 12.71 | 12.72 | 12.67 |
-| 384 | 12 | 11.65 | 11.71 | 11.57 | 18.24 | 18.25 | 18.04 |
-| 384 | 16 | 15.63 | 15.63 | 15.49 | 24.24 | 24.28 | 23.94 |
-| 384 | 24 | 22.57 | 22.61 | 22.36 | 35.77 | 35.78 | 35.38 |
-| 384 | 32 | 29.66 | 29.66 | 29.33 | 47.09 | 47.11 | 46.81 |
-| 384 | 64 | 57.20 | 57.34 | 56.93 | 92.12 | 92.49 | 91.61 |
-| 384 | 128 | 112.00 | 112.42 | 111.24 | 180.61 | 181.02 | 179.56 |
-
-
-#### Inference performance: NVIDIA T4 (16GB)
-
-Our results were obtained by running the `scripts/inference_benchmark.sh --gpu Turing` script in the container generated by the TensorRT OSS Dockerfile on NVIDIA T4 with (1x T4 16G) GPUs.
-
-##### BERT Base
-
-| Sequence Length | Batch Size | INT8 Latency (ms) | | | FP16 Latency (ms) | | |
-|-----------------|------------|-----------------|-----------------|---------|-----------------|-----------------|---------|
-| | | 95th Percentile | 99th Percentile | Average | 95th Percentile | 99th Percentile | Average |
-| 128 | 1 | 1.67 | 1.67 | 1.66 | 1.82 | 1.96 | 1.76 |
-| 128 | 2 | 1.94 | 1.95 | 1.89 | 2.58 | 2.67 | 2.50 |
-| 128 | 4 | 2.73 | 2.80 | 2.64 | 4.30 | 4.34 | 4.17 |
-| 128 | 8 | 4.93 | 4.96 | 4.81 | 8.85 | 9.74 | 8.36 |
-| 128 | 12 | 6.85 | 7.05 | 6.70 | 12.83 | 13.19 | 12.34 |
-| 128 | 16 | 9.65 | 9.89 | 9.43 | 17.70 | 18.27 | 17.01 |
-| 128 | 24 | 15.04 | 15.70 | 14.68 | 27.00 | 27.87 | 26.50 |
-| 128 | 32 | 20.55 | 21.01 | 19.88 | 34.51 | 34.81 | 33.83 |
-| 128 | 64 | 40.48 | 41.29 | 39.87 | 67.84 | 68.57 | 67.03 |
-| 128 | 128 | 82.17 | 82.53 | 80.95 | 132.78 | 133.23 | 131.64 |
-| 384 | 1 | 2.75 | 2.78 | 2.67 | 3.73 | 3.79 | 3.63 |
-| 384 | 2 | 4.22 | 4.38 | 4.09 | 6.68 | 7.27 | 6.53 |
-| 384 | 4 | 7.87 | 8.07 | 7.75 | 13.22 | 13.50 | 12.83 |
-| 384 | 8 | 16.07 | 16.13 | 15.77 | 28.01 | 28.72 | 27.48 |
-| 384 | 12 | 23.87 | 24.15 | 23.53 | 40.96 | 41.51 | 39.39 |
-| 384 | 16 | 31.87 | 32.25 | 30.99 | 51.56 | 51.83 | 51.00 |
-| 384 | 24 | 48.14 | 48.33 | 47.22 | 82.06 | 82.56 | 80.13 |
-| 384 | 32 | 64.07 | 64.48 | 63.19 | 102.64 | 103.33 | 101.20 |
-| 384 | 64 | 129.58 | 130.37 | 125.79 | 215.79 | 216.38 | 213.87 |
-| 384 | 128 | 258.69 | 259.74 | 245.91 | 414.96 | 415.57 | 413.16 |
-
-##### BERT Large
-
-| Sequence Length | Batch Size | INT8 Latency (ms) | | | FP16 Latency (ms) | | |
-|-----------------|------------|-----------------|-----------------|---------|-----------------|-----------------|---------|
-| | | 95th Percentile | 99th Percentile | Average | 95th Percentile | 99th Percentile | Average |
-| 128 | 1 | 4.20 | 4.35 | 4.10 | 5.05 | 5.21 | 4.91 |
-| 128 | 2 | 5.41 | 5.70 | 5.30 | 7.99 | 8.31 | 7.79 |
-| 128 | 4 | 8.48 | 8.68 | 8.32 | 14.87 | 15.28 | 14.44 |
-| 128 | 8 | 15.20 | 15.22 | 14.91 | 29.66 | 30.20 | 28.97 |
-| 128 | 12 | 23.54 | 23.72 | 23.21 | 45.48 | 45.90 | 44.91 |
-| 128 | 16 | 31.04 | 31.38 | 30.46 | 62.06 | 62.61 | 60.27 |
-| 128 | 24 | 48.00 | 48.59 | 47.44 | 84.17 | 84.50 | 83.43 |
-| 128 | 32 | 64.41 | 64.77 | 63.54 | 113.60 | 113.98 | 112.32 |
-| 128 | 64 | 128.03 | 128.45 | 126.36 | 223.89 | 224.83 | 220.75 |
-| 128 | 128 | 246.96 | 247.80 | 245.00 | 441.52 | 442.26 | 439.65 |
-| 384 | 1 | 7.88 | 8.06 | 7.73 | 11.84 | 12.11 | 11.51 |
-| 384 | 2 | 13.00 | 13.18 | 12.80 | 23.59 | 24.13 | 23.12 |
-| 384 | 4 | 25.14 | 25.32 | 24.70 | 46.66 | 46.69 | 45.81 |
-| 384 | 8 | 50.14 | 50.65 | 49.41 | 86.74 | 87.47 | 85.40 |
-| 384 | 12 | 72.92 | 73.01 | 71.86 | 127.10 | 127.44 | 125.66 |
-| 384 | 16 | 97.00 | 97.26 | 95.47 | 169.41 | 169.93 | 167.55 |
-| 384 | 24 | 149.70 | 150.28 | 148.00 | 258.26 | 258.88 | 255.79 |
-| 384 | 32 | 192.74 | 193.85 | 190.59 | 339.87 | 340.55 | 337.86 |
-| 384 | 64 | 385.85 | 387.66 | 383.62 | 692.10 | 692.88 | 689.73 |
-| 384 | 128 | 780.95 | 781.81 | 778.82 | 1367.61 | 1368.85 | 1365.16 |
-
-
-#### Inference performance: NVIDIA V100 (16GB)
-
-Our results were obtained by running the `scripts/inference_benchmark.sh --gpu Volta` script in the container generated by the TensorRT OSS Dockerfile on NVIDIA V100 with (1x V100 16G) GPUs.
-
-##### BERT Base
-
-| Sequence Length | Batch Size | INT8 Latency (ms) | | | FP16 Latency (ms) | | |
-|-----------------|------------|-----------------|-----------------|---------|-----------------|-----------------|---------|
-| | | 95th Percentile | 99th Percentile | Average | 95th Percentile | 99th Percentile | Average |
-| 128 | 1 | 1.39 | 1.39 | 1.39 | 1.23 | 1.23 | 1.23 |
-| 128 | 2 | 1.76 | 1.76 | 1.75 | 1.49 | 1.49 | 1.48 |
-| 128 | 4 | 2.35 | 2.36 | 2.34 | 2.12 | 2.13 | 2.11 |
-| 128 | 8 | 3.69 | 3.7 | 3.65 | 3.35 | 3.36 | 3.32 |
-| 128 | 12 | 4.79 | 4.83 | 4.75 | 4.65 | 4.67 | 4.61 |
-| 128 | 16 | 6.7 | 6.72 | 6.64 | 6.3 | 6.35 | 6.25 |
-| 128 | 24 | 8.95 | 8.96 | 8.9 | 8.68 | 8.71 | 8.6 |
-| 128 | 32 | 14.74 | 14.77 | 14.59 | 14.16 | 14.18 | 14.06 |
-| 128 | 64 | 24.12 | 24.14 | 23.98 | 22.57 | 22.63 | 22.47 |
-| 128 | 128 | 45.59 | 45.65 | 45.53 | 43.45 | 43.51 | 43.25 |
-| 384 | 1 | 2.17 | 2.18 | 2.16 | 1.98 | 1.98 | 1.97 |
-| 384 | 2 | 3.4 | 3.42 | 3.38 | 3.11 | 3.11 | 3.08 |
-| 384 | 4 | 5.61 | 5.62 | 5.57 | 5.5 | 5.52 | 5.46 |
-| 384 | 8 | 10.58 | 10.63 | 10.49 | 10.26 | 10.29 | 10.17 |
-| 384 | 12 | 16.55 | 16.57 | 16.43 | 15.8 | 15.83 | 15.69 |
-| 384 | 16 | 21.15 | 21.19 | 21.04 | 20.09 | 20.12 | 19.94 |
-| 384 | 24 | 30.95 | 31 | 30.77 | 29.44 | 29.51 | 29.24 |
-| 384 | 32 | 47.94 | 48.03 | 47.66 | 47.97 | 48.05 | 47.56 |
-| 384 | 64 | 81.8 | 81.91 | 81.62 | 76.84 | 77.05 | 76.4 |
-| 384 | 128 | 159.87 | 160.06 | 159.47 | 151.4 | 151.61 | 150.85 |
-
-##### BERT Large
-
-| Sequence Length | Batch Size | INT8 Latency (ms) | | | FP16 Latency (ms) | | |
-|-----------------|------------|-----------------|-----------------|---------|-----------------|-----------------|---------|
-| | | 95th Percentile | 99th Percentile | Average | 95th Percentile | 99th Percentile | Average |
-| 128 | 1 | 3.43 | 3.44 | 3.42 | 3.06 | 3.07 | 3.05 |
-| 128 | 2 | 4.35 | 4.37 | 4.33 | 3.79 | 3.8 | 3.79 |
-| 128 | 4 | 6.8 | 6.83 | 6.74 | 6.02 | 6.05 | 5.98 |
-| 128 | 8 | 11 | 11.07 | 10.93 | 10.57 | 10.62 | 10.47 |
-| 128 | 12 | 16.28 | 16.31 | 16.15 | 15.06 | 15.1 | 14.96 |
-| 128 | 16 | 20.33 | 20.44 | 20.13 | 20.47 | 20.51 | 20.25 |
-| 128 | 24 | 30.63 | 30.66 | 30.33 | 28.65 | 28.8 | 28.48 |
-| 128 | 32 | 45.28 | 45.35 | 45.09 | 46.88 | 47.02 | 46.43 |
-| 128 | 64 | 75.33 | 75.57 | 74.82 | 71.88 | 71.97 | 71.47 |
-| 128 | 128 | 148.1 | 148.31 | 147.59 | 140.81 | 140.97 | 140.35 |
-| 384 | 1 | 6.16 | 6.17 | 6.12 | 5.7 | 5.72 | 5.66 |
-| 384 | 2 | 10.25 | 10.27 | 10.18 | 9.46 | 9.49 | 9.37 |
-| 384 | 4 | 18.44 | 18.5 | 18.27 | 17.22 | 17.28 | 17.09 |
-| 384 | 8 | 34.67 | 34.71 | 34.41 | 32.71 | 32.79 | 32.45 |
-| 384 | 12 | 49.04 | 49.13 | 48.79 | 47.53 | 47.77 | 47.27 |
-| 384 | 16 | 67.08 | 67.21 | 66.75 | 62.86 | 63.01 | 62.76 |
-| 384 | 24 | 94.22 | 94.39 | 94.04 | 92.08 | 92.2 | 91.86 |
-| 384 | 32 | 148.96 | 149.11 | 148.59 | 147.7 | 147.84 | 147.23 |
-| 384 | 64 | 245.91 | 246.09 | 244.67 | 240.16 | 240.43 | 239.07 |
-
-## Experimental
-### Variable sequence length
-In our prior implementation, we used inputs padded to max length along with corresponding input masks to handle variable sequence length inputs in a batch. The padding results in some wasted computations which can be avoided by handling variable sequence length inputs natively. Now we have a new approach called the variable sequence length method. By concatenating each input id into a single long input id, and concatenating each input segment id into a single long segment id, TensorRT can know the exact starts and ends by providing an extra sequence length buffer that contains the start and end positions of each sequence. Now we can eliminate the wasted computation in the input paddings.
-
-Note this is an experimental feature because we only support Xavier+ GPUs, also there is neither FP32 support nor INT8 PTQ calibration.
-
-#### Run command lines
-
-1. Download checkpoint for BERT Large FP16 SQuAD v1.1 model with sequence length of 384:
- ```bash
- bash scripts/download_model.sh pyt v1_1
- ```
-
-2. Build an engine:
-
- **FP16 engine**
- ```bash
- mkdir -p /workspace/TensorRT/demo/BERT/engines && python3 builder_varseqlen.py -x /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx -o /workspace/TensorRT/demo/BERT/engines/bert_varseq_fp16.engine -b 1 -s 64 --fp16 -c /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt
- ```
-
- This will build and engine with a maximum batch size of 1 (`-b 1`) and sequence length of 64 (`-s 64`) using FP16 precision computation where possible (`--fp16`).
-
-
- **INT8 engine**
- ```bash
- mkdir -p /workspace/TensorRT/demo/BERT/engines && python3 builder_varseqlen.py -x /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_pyt_onnx_large_qa_squad11_amp_fake_quant_v1/bert_large_v1_1_fake_quant.onnx -o /workspace/TensorRT/demo/BERT/engines/bert_varseq_int8.engine -b 1 -s 256 --int8 --fp16 -c /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1 -v /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt
- ```
-
- This will build and engine with a maximum batch size of 1 (`-b 1`) and sequence length of 256 (`-s 256`) using INT8 precision computation where possible (`--int8`).
-
-3. Run inference
-
- Evaluate the F1 score and exact match score using the squad dataset:
-
- ```bash
- python3 inference_varseqlen.py -e /workspace/TensorRT/demo/BERT/engines/bert_varseq_int8.engine -s 256 -sq ./squad/dev-v1.1.json -v /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -o ./predictions.json
- python3 squad/evaluate-v1.1.py squad/dev-v1.1.json ./predictions.json 90
- ```
-
- Run the quesion and answer mode:
-
- ```bash
- python3 inference_varseqlen.py -e /workspace/TensorRT/demo/BERT/engines/bert_varseq_int8.engine -p "TensorRT is a high performance deep learning inference platform that delivers low latency and high throughput for apps such as recommenders, speech and image/video on NVIDIA GPUs. It includes parsers to import models, and plugins to support novel ops and layers before applying optimizations for inference. Today NVIDIA is open-sourcing parsers and plugins in TensorRT so that the deep learning community can customize and extend these components to take advantage of powerful TensorRT optimizations for your apps." -q "What is TensorRT?" -v /workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_384_v19.03.1/vocab.txt -s 256
- ```
-
-3. Collect performance data
-
- ```bash
- python3 perf_varseqlen.py -e /workspace/TensorRT/demo/BERT/engines/bert_varseq_int8.engine -b 1 -s 256
- ```
-
- This will collect performance data run use batch size 1 (`-b 1`) and sequence length of 256 (`-s 256`).
-
-4. Collect performance data with CUDA graph enabled
-
- We can use the same `inference_c.py` and `build/perf` to collect performance data with cuda graph enabled. The command line is the same as run without variable sequence length.
-
+To be published soon.
diff --git a/demo/BERT/builder.py b/demo/BERT/builder.py
old mode 100644
new mode 100755
index 6408d748..eba4739c
--- a/demo/BERT/builder.py
+++ b/demo/BERT/builder.py
@@ -27,17 +27,13 @@ import time
import onnx
import pycuda.autoinit
-# Tensorflow v1 compatibility mode
-try:
- import tensorflow.compat.v1 as tf
- tf.disable_v2_behavior()
-except ImportError as err:
- sys.stderr.write("""Error: Failed to import tensorflow module ({})\n""".format(err))
- sys.exit()
-
# TensorRT
import tensorrt as trt
from helpers.calibrator import BertCalibrator as BertCalibrator
+from builder_utils import load_tf_weights, load_pytorch_weights_and_quant, load_onnx_weights_and_quant
+from builder_utils import WQKV, BQKV # Attention Keys
+from builder_utils import W_AOUT, B_AOUT, W_MID, B_MID, W_LOUT, B_LOUT # Transformer Keys
+from builder_utils import SQD_W, SQD_B # SQuAD Output Keys
"""
TensorRT Initialization
@@ -56,40 +52,8 @@ qkv2_plg_creator = plg_registry.get_plugin_creator("CustomQKVToContextPluginDyna
skln_plg_creator = plg_registry.get_plugin_creator("CustomSkipLayerNormPluginDynamic", "1", "")
fc_plg_creator = plg_registry.get_plugin_creator("CustomFCPluginDynamic", "1", "")
-"""
-Attentions Keys
-"""
-WQ = "self_query_kernel"
-BQ = "self_query_bias"
-WK = "self_key_kernel"
-BK = "self_key_bias"
-WV = "self_value_kernel"
-BV = "self_value_bias"
-WQKV = "self_qkv_kernel"
-BQKV = "self_qkv_bias"
-
-"""
-Transformer Keys
-"""
-W_AOUT = "attention_output_dense_kernel"
-B_AOUT = "attention_output_dense_bias"
-AOUT_LN_BETA = "attention_output_layernorm_beta"
-AOUT_LN_GAMMA = "attention_output_layernorm_gamma"
-W_MID = "intermediate_dense_kernel"
-B_MID = "intermediate_dense_bias"
-W_LOUT = "output_dense_kernel"
-B_LOUT = "output_dense_bias"
-LOUT_LN_BETA = "output_layernorm_beta"
-LOUT_LN_GAMMA = "output_layernorm_gamma"
-
-"""
-Squad Output Keys
-"""
-SQD_W = "squad_output_weights"
-SQD_B = "squad_output_bias"
-
class BertConfig:
- def __init__(self, bert_config_path, use_fp16, use_int8, use_strict, use_fc2_gemm, use_int8_skipln, use_int8_multihead, use_qat, timing_cache):
+ def __init__(self, bert_config_path, use_fp16, use_int8, use_strict, use_fc2_gemm, use_int8_skipln, use_int8_multihead, use_qat, use_sparsity, timing_cache):
with open(bert_config_path, "r") as f:
data = json.load(f)
self.num_attention_heads = data["num_attention_heads"]
@@ -105,6 +69,7 @@ class BertConfig:
self.use_int8_multihead = use_int8_multihead
self.is_calib_mode = False
self.use_qat = use_qat
+ self.use_sparsity = use_sparsity
self.timing_cache = timing_cache
def set_tensor_name(tensor, prefix, name):
@@ -139,7 +104,7 @@ def attention_layer_opt(prefix, config, init_dict, network, input_tensor, imask)
# FC_attention
if config.use_int8:
- mult_all = network.add_convolution(input_tensor, 3 * hidden_size, (1, 1), Wall, Ball)
+ mult_all = network.add_convolution_nd(input_tensor, 3 * hidden_size, (1, 1), Wall, Ball)
else:
mult_all = network.add_fully_connected(input_tensor, 3 * hidden_size, Wall, Ball)
@@ -245,7 +210,7 @@ def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, imas
B_aout = init_dict[prefix + B_AOUT]
if config.use_int8:
W_aout = init_dict[prefix + W_AOUT]
- attention_out_fc = network.add_convolution(attention_heads, hidden_size, (1, 1), W_aout, B_aout)
+ attention_out_fc = network.add_convolution_nd(attention_heads, hidden_size, (1, 1), W_aout, B_aout)
B_aout = None
if not config.use_int8_skipln:
@@ -268,7 +233,7 @@ def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, imas
B_mid = init_dict[prefix + B_MID]
W_mid = init_dict[prefix + W_MID]
if config.use_int8:
- mid_dense = network.add_convolution(attention_ln, config.intermediate_size, (1, 1), W_mid, B_mid)
+ mid_dense = network.add_convolution_nd(attention_ln, config.intermediate_size, (1, 1), W_mid, B_mid)
else:
mid_dense = network.add_fully_connected(attention_ln, config.intermediate_size, W_mid, B_mid)
@@ -305,7 +270,7 @@ def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, imas
B_lout = init_dict[prefix + B_LOUT]
if config.use_int8 and not config.use_fc2_gemm:
W_lout = init_dict[prefix + W_LOUT]
- out_dense = network.add_convolution(intermediate_act, hidden_size, (1, 1), W_lout, B_lout)
+ out_dense = network.add_convolution_nd(intermediate_act, hidden_size, (1, 1), W_lout, B_lout)
B_lout = None
if not config.use_int8_skipln:
@@ -359,176 +324,6 @@ def squad_output(prefix, config, init_dict, network, input_tensor):
set_output_name(OUT, prefix, "squad_logits")
return OUT
-def load_tf_weights(inputbase, config):
- """
- Load the weights from the tensorflow checkpoint
- """
- weights_dict = dict()
-
- try:
- reader = tf.train.NewCheckpointReader(inputbase)
- tensor_dict = reader.get_variable_to_shape_map()
-
- # There might be training-related variables in the checkpoint that can be discarded
- param_names = [key for key in sorted(tensor_dict) if "adam" not in key and "global_step" not in key and "pooler" not in key]
- count = len(param_names)
- TRT_LOGGER.log(TRT_LOGGER.INFO, "Found {:} entries in weight map".format(count))
-
- for pn in param_names:
- toks = pn.lower().split("/")
- if "encoder" in pn:
- assert ("layer" in pn)
- l = (re.findall("\d+", pn))[0]
- outname = "l{}_".format(l) + "_".join(toks[3:])
- else:
- outname = "_".join(toks)
-
- tensor = reader.get_tensor(pn)
- shape = tensor.shape
- if pn.find("kernel") != -1:
- weights_dict[outname + "_notrans"] = trt.Weights(np.ascontiguousarray(tensor).flatten())
-
- TRT_LOGGER.log(TRT_LOGGER.VERBOSE, "Transposing {}\n".format(np))
- tensor = np.transpose(tensor)
-
- shape = tensor.shape
- flat_tensor = tensor.flatten()
- shape_str = "{} ".format(len(shape)) + " ".join([str(d) for d in shape])
- weights_dict[outname] = trt.Weights(flat_tensor)
-
- TRT_LOGGER.log(TRT_LOGGER.VERBOSE, "Original name: {:}, TensorRT name: {:}, shape: {:}".format(pn, outname, shape_str))
-
- N = config.num_attention_heads
- H = config.head_size
-
- additional_dict = dict()
- for key, value in weights_dict.items():
- pos = key.find(BQ)
- if pos != -1:
- hidden_size = value.size
- prefix = key[:pos]
-
- Bq_ = value
- Bk_ = weights_dict[prefix + BK]
- Bv_ = weights_dict[prefix + BV]
- Wq_ = weights_dict[prefix + WQ]
- Wk_ = weights_dict[prefix + WK]
- Wv_ = weights_dict[prefix + WV]
-
- mat_size = hidden_size * hidden_size
- wcount = 3 * mat_size
- Wall = np.zeros(wcount, np.float32)
- bcount = 3 * hidden_size
- Ball = np.zeros(bcount, np.float32)
- Wall[0:mat_size] = Wq_.numpy()[0:mat_size]
- Wall[mat_size:2*mat_size] = Wk_.numpy()[0:mat_size]
- Wall[2*mat_size:3*mat_size] = Wv_.numpy()[0:mat_size]
- Ball[0:hidden_size] = Bq_.numpy()[0:hidden_size]
- Ball[hidden_size:2*hidden_size] = Bk_.numpy()[0:hidden_size]
- Ball[2*hidden_size:3*hidden_size] = Bv_.numpy()[0:hidden_size]
-
- Wall = np.ascontiguousarray(Wall.reshape((3, N, H, N, H)).transpose((1, 0, 2, 3, 4)), dtype=np.float32)
- Ball = np.ascontiguousarray(Ball.reshape((3, N, H)).transpose((1, 0, 2)), dtype=np.float32)
-
- additional_dict[prefix + WQKV] = trt.Weights(Wall)
- additional_dict[prefix + BQKV] = trt.Weights(Ball)
-
- additional_dict[prefix + WQKV + "_notrans"] = trt.Weights(Wall.T)
-
- except Exception as error:
- TRT_LOGGER.log(TRT_LOGGER.ERROR, str(error))
-
- weights_dict.update(additional_dict)
- return weights_dict
-
-def onnx_to_trt_name(onnx_name):
- """
- Converting variables in the onnx checkpoint to names corresponding to the naming convention used in the TF version, expected by the builder
- """
- onnx_name = onnx_name.lower()
- toks = [t.strip('_') for t in onnx_name.split('.')]
- if toks[0] == 'bert': #embeddings or encoder
- if toks[1] == 'encoder': #transformer
-
- if toks[-2] == 'layernorm': #bias->beta, weight->gamma
- toks[-1] = 'beta' if toks[-1] == 'bias' else 'gamma'
- elif (toks[-2] == 'dense' or toks[-2] in {'key', 'value', 'query'}) and toks[-1] == 'weight':
- toks[-1] = 'kernel'
- elif (toks[-3] == 'dense' or toks[-3] in {'key', 'value', 'query'}) and toks[-1] == 'amax':
- if toks[-2] == 'weight_quantizer':
- toks[-2] = 'kernel'
- elif toks[-2] == 'input_quantizer':
- toks[-2] = 'input'
-
- if 'final_input_quantizer' not in toks[2]:
- toks = toks[3:]
- toks[0] = 'l{}'.format(int(toks[0]))
- else:
- if toks[-2] == 'layernorm': #bias->beta, weight->gamma
- toks[-1] = 'beta' if toks[-1] == 'bias' else 'gamma'
- else: #embeddings: drop "_weight" suffix
- if toks[-1] == 'amax':
- toks[-2] = 'amax'
- toks = toks[:-1]
- elif 'qa' in onnx_name:
- name = 'cls_squad_output_bias' if toks[-1] == 'bias' else 'cls_squad_output_weights'
- return name
- else:
- print("Encountered unknown case:", onnx_name)
- assert(False)
- parsed = '_'.join(toks)
- return parsed
-
-def load_onnx_weights_and_quant(path, config):
- """
- Load the weights from the onnx checkpoint
- """
- N = config.num_attention_heads
- H = config.head_size
- hidden_size = config.hidden_size
-
- model = onnx.load(path)
- weights = model.graph.initializer
- tensor_dict = dict([(onnx_to_trt_name(w.name), np.frombuffer(w.raw_data, np.float32).reshape(w.dims)) for w in weights])
-
- weights_dict = dict()
- for outname, tensor in tensor_dict.items():
- if outname.find("_amax") != -1:
- weights_dict[outname] = tensor
- elif outname.find(BQ) != -1:
- prefix = outname[:outname.find(BQ)]
-
- Wqkv = np.zeros((3, hidden_size, hidden_size), np.float32)
- Bqkv = np.zeros((3, hidden_size), np.float32)
-
- Wqkv[0,:,:] = tensor_dict[prefix + WQ]
- Wqkv[1,:,:] = tensor_dict[prefix + WK]
- Wqkv[2,:,:] = tensor_dict[prefix + WV]
- Bqkv[0,:] = tensor
- Bqkv[1,:] = tensor_dict[prefix + BK]
- Bqkv[2,:] = tensor_dict[prefix + BV]
-
- Wqkv = np.ascontiguousarray(Wqkv.reshape((3, N, H, N, H)).transpose((1,0,2,3,4)))
- Bqkv = np.ascontiguousarray(Bqkv.reshape((3, N, H)).transpose((1,0,2)))
-
- weights_dict[prefix + WQKV] = trt.Weights(Wqkv)
- weights_dict[prefix + BQKV] = trt.Weights(Bqkv)
- weights_dict[prefix + WQKV + "_notrans"] = trt.Weights(Wqkv.T)
-
- elif outname.find(BK) != -1 or outname.find(BV) != -1 or outname.find(WQ) != -1 or outname.find(WK) != -1 or outname.find(WV) != -1:
- pass
- else:
- flat_tensor = np.ascontiguousarray(tensor).flatten()
- weights_dict[outname] = trt.Weights(flat_tensor)
-
- if outname.find("kernel") != -1:
- tensor = np.transpose(tensor)
- weights_dict[outname + "_notrans"] = trt.Weights(np.ascontiguousarray(tensor).flatten())
-
-
- TRT_LOGGER.log(TRT_LOGGER.INFO, "Found {:} entries in weight map".format(len(weights_dict)))
- return weights_dict
-
def emb_layernorm(builder, network, config, weights_dict, builder_config, sequence_lengths, batch_sizes):
# int8 only support some of the sequence length, we dynamic on sequence length is not allowed.
input_ids = network.add_input(name="input_ids", dtype=trt.int32, shape=(-1, -1 if len(sequence_lengths) > 1 else sequence_lengths[0]))
@@ -600,6 +395,10 @@ def build_engine(batch_sizes, workspace_size, sequence_lengths, config, weights_
builder_config.int8_calibrator = calibrator
if config.use_strict:
builder_config.set_flag(trt.BuilderFlag.STRICT_TYPES)
+
+ if config.use_sparsity:
+ TRT_LOGGER.log(TRT_LOGGER.INFO, "Setting sparsity flag on builder_config.")
+ builder_config.set_flag(trt.BuilderFlag.SPARSE_WEIGHTS)
# speed up the engine build for trt major version >= 8
# 1. disable cudnn tactic
@@ -678,6 +477,7 @@ def main():
parser.add_argument("-m", "--ckpt", required=False,
help="The checkpoint file basename, e.g.: basename(model.ckpt-766908.data-00000-of-00001) is model.ckpt-766908")
parser.add_argument("-x", "--onnx", required=False, help="The ONNX model file path.")
+ parser.add_argument("-pt", "--pytorch", required=False, help="The PyTorch checkpoint file path.")
parser.add_argument("-o", "--output", required=True, default="bert_base_384.engine", help="The bert engine file, ex bert.engine")
parser.add_argument("-b", "--batch-size", default=[], action="append", help="Batch size(s) to optimize for. The engine will be usable with any batch size below this, but may not be optimal for smaller sizes. Can be specified multiple times to optimize for more than one batch size.", type=int)
parser.add_argument("-s", "--sequence-length", default=[], action="append", help="Sequence length of the BERT model", type=int)
@@ -694,6 +494,7 @@ def main():
parser.add_argument("-g", "--force-fc2-gemm", action="store_true", help="Force use gemm to implement FC2 layer", required=False)
parser.add_argument("-iln", "--force-int8-skipln", action="store_true", help="Run skip layernorm with INT8 (FP32 or FP16 by default) inputs and output", required=False)
parser.add_argument("-imh", "--force-int8-multihead", action="store_true", help="Run multi-head attention with INT8 (FP32 or FP16 by default) input and output", required=False)
+ parser.add_argument("-sp", "--sparse", action="store_true", help="Indicates that model is sparse", required=False)
parser.add_argument("-tcf", "--timing-cache-file", help="Path to tensorrt build timeing cache file, only available for tensorrt 8.0 and later", required=False)
args, _ = parser.parse_known_args()
@@ -709,7 +510,7 @@ def main():
bert_config_path = os.path.join(args.config_dir, "bert_config.json")
TRT_LOGGER.log(TRT_LOGGER.INFO, "Using configuration file: {:}".format(bert_config_path))
- config = BertConfig(bert_config_path, args.fp16, args.int8, args.strict, args.force_fc2_gemm, args.force_int8_skipln, args.force_int8_multihead, args.int8 and args.onnx != None, args.timing_cache_file)
+ config = BertConfig(bert_config_path, args.fp16, args.int8, args.strict, args.force_fc2_gemm, args.force_int8_skipln, args.force_int8_multihead, args.int8 and args.onnx != None, args.sparse, args.timing_cache_file)
if args.calib_path != None:
calib_cache = args.calib_path
@@ -718,6 +519,8 @@ def main():
if args.onnx != None:
weights_dict = load_onnx_weights_and_quant(args.onnx, config)
+ elif args.pytorch != None:
+ weights_dict = load_pytorch_weights_and_quant(args.pytorch, config)
elif args.ckpt != None:
weights_dict = load_tf_weights(args.ckpt, config)
generate_calibration_cache(args.sequence_length, args.workspace_size, config, weights_dict, args.squad_json, args.vocab_file, calib_cache, args.calib_num)
diff --git a/demo/BERT/builder_utils.py b/demo/BERT/builder_utils.py
new file mode 100644
index 00000000..897fbc92
--- /dev/null
+++ b/demo/BERT/builder_utils.py
@@ -0,0 +1,298 @@
+import re
+import pickle
+
+import numpy as np
+import onnx
+import torch
+import tensorrt as trt
+
+try:
+ import tensorflow.compat.v1 as tf
+ tf.disable_v2_behavior()
+except ImportError as err:
+ sys.stderr.write("""Error: Failed to import tensorflow module ({})\n""".format(err))
+ sys.exit()
+
+TRT_LOGGER = trt.Logger(trt.Logger.INFO)
+
+"""
+Attentions Keys
+"""
+WQ = "self_query_kernel"
+BQ = "self_query_bias"
+WK = "self_key_kernel"
+BK = "self_key_bias"
+WV = "self_value_kernel"
+BV = "self_value_bias"
+WQKV = "self_qkv_kernel"
+BQKV = "self_qkv_bias"
+
+"""
+Transformer Keys
+"""
+W_AOUT = "attention_output_dense_kernel"
+B_AOUT = "attention_output_dense_bias"
+AOUT_LN_BETA = "attention_output_layernorm_beta"
+AOUT_LN_GAMMA = "attention_output_layernorm_gamma"
+W_MID = "intermediate_dense_kernel"
+B_MID = "intermediate_dense_bias"
+W_LOUT = "output_dense_kernel"
+B_LOUT = "output_dense_bias"
+LOUT_LN_BETA = "output_layernorm_beta"
+LOUT_LN_GAMMA = "output_layernorm_gamma"
+
+"""
+Squad Output Keys
+"""
+SQD_W = "squad_output_weights"
+SQD_B = "squad_output_bias"
+
+
+def load_tf_weights(inputbase, config):
+ """
+ Load the weights from the tensorflow checkpoint
+ """
+ weights_dict = dict()
+
+ try:
+ reader = tf.train.NewCheckpointReader(inputbase)
+ tensor_dict = reader.get_variable_to_shape_map()
+
+ # There might be training-related variables in the checkpoint that can be discarded
+ param_names = [key for key in sorted(tensor_dict) if "adam" not in key and "global_step" not in key and "pooler" not in key]
+ count = len(param_names)
+ TRT_LOGGER.log(TRT_LOGGER.INFO, "Found {:} entries in weight map".format(count))
+
+ for pn in param_names:
+ toks = pn.lower().split("/")
+ if "encoder" in pn:
+ assert ("layer" in pn)
+ l = (re.findall("\d+", pn))[0]
+ outname = "l{}_".format(l) + "_".join(toks[3:])
+ else:
+ outname = "_".join(toks)
+
+ tensor = reader.get_tensor(pn)
+ shape = tensor.shape
+ if pn.find("kernel") != -1:
+ weights_dict[outname + "_notrans"] = trt.Weights(np.ascontiguousarray(tensor).flatten())
+
+ TRT_LOGGER.log(TRT_LOGGER.VERBOSE, "Transposing {}\n".format(np))
+ tensor = np.transpose(tensor)
+
+ shape = tensor.shape
+ flat_tensor = tensor.flatten()
+ shape_str = "{} ".format(len(shape)) + " ".join([str(d) for d in shape])
+ weights_dict[outname] = trt.Weights(flat_tensor)
+
+ TRT_LOGGER.log(TRT_LOGGER.VERBOSE, "Original name: {:}, TensorRT name: {:}, shape: {:}".format(pn, outname, shape_str))
+
+ N = config.num_attention_heads
+ H = config.head_size
+
+ additional_dict = dict()
+ for key, value in weights_dict.items():
+ pos = key.find(BQ)
+ if pos != -1:
+ hidden_size = value.size
+ prefix = key[:pos]
+
+ Bq_ = value
+ Bk_ = weights_dict[prefix + BK]
+ Bv_ = weights_dict[prefix + BV]
+ Wq_ = weights_dict[prefix + WQ]
+ Wk_ = weights_dict[prefix + WK]
+ Wv_ = weights_dict[prefix + WV]
+
+ mat_size = hidden_size * hidden_size
+ wcount = 3 * mat_size
+ Wall = np.zeros(wcount, np.float32)
+ bcount = 3 * hidden_size
+ Ball = np.zeros(bcount, np.float32)
+ Wall[0:mat_size] = Wq_.numpy()[0:mat_size]
+ Wall[mat_size:2*mat_size] = Wk_.numpy()[0:mat_size]
+ Wall[2*mat_size:3*mat_size] = Wv_.numpy()[0:mat_size]
+ Ball[0:hidden_size] = Bq_.numpy()[0:hidden_size]
+ Ball[hidden_size:2*hidden_size] = Bk_.numpy()[0:hidden_size]
+ Ball[2*hidden_size:3*hidden_size] = Bv_.numpy()[0:hidden_size]
+
+ if config.use_int8 and getattr(config, 'interleaved', False):
+ Wall = np.ascontiguousarray(Wall.reshape((3, N, H, N, H)), dtype=np.float32)
+ Ball = np.ascontiguousarray(Ball.reshape((3, N, H)), dtype=np.float32)
+ else:
+ Wall = np.ascontiguousarray(Wall.reshape((3, N, H, N, H)).transpose((1, 0, 2, 3, 4)), dtype=np.float32)
+ Ball = np.ascontiguousarray(Ball.reshape((3, N, H)).transpose((1, 0, 2)), dtype=np.float32)
+
+ additional_dict[prefix + WQKV] = trt.Weights(Wall)
+ additional_dict[prefix + BQKV] = trt.Weights(Ball)
+
+ additional_dict[prefix + WQKV + "_notrans"] = trt.Weights(Wall.T)
+
+ except Exception as error:
+ TRT_LOGGER.log(TRT_LOGGER.ERROR, str(error))
+
+ weights_dict.update(additional_dict)
+ return weights_dict
+
+def onnx_to_trt_name(onnx_name):
+ """
+ Converting variables in the onnx checkpoint to names corresponding to the naming convention used in the TF version, expected by the builder
+ """
+ qkv_strings = {'key', 'value', 'query', 'query_key_value'}
+ onnx_name = onnx_name.lower()
+ toks = [t.strip('_') for t in onnx_name.split('.')]
+ if toks[0] == 'bert': #embeddings or encoder
+ if toks[1] == 'encoder': #transformer
+ # Token conversions for sparse checkpoints
+ if toks[-2] == 'dense_act':
+ toks[-2] = 'dense'
+ elif toks[-3] == 'dense_act':
+ if toks[-2] == 'input_quantizer':
+ toks[-2] = 'input'
+ elif toks[-2] == 'weight_quantizer':
+ toks[-2] = 'kernel'
+ toks[-3] = 'dense'
+ elif toks[-2].startswith('matmul'):
+ toks[-2] = {
+ 'matmul_q_quantizer': 'qv_a_input_quantizer',
+ 'matmul_k_quantizer': 'qv_b_input_quantizer',
+ 'matmul_v_quantizer': 'av_b_input_quantizer',
+ 'matmul_a_quantizer': 'av_a_input_quantizer',
+ }[toks[-2].replace('input_', '')]
+
+ # Token conversions for all checkpoints
+ if toks[-2] == 'layernorm': #bias->beta, weight->gamma
+ toks[-1] = 'beta' if toks[-1] == 'bias' else 'gamma'
+ elif (toks[-2] == 'dense' or toks[-2] in qkv_strings) and toks[-1] == 'weight':
+ toks[-1] = 'kernel'
+ elif (toks[-3] == 'dense' or toks[-3] in qkv_strings) and toks[-1] == 'amax':
+ if toks[-2] == 'weight_quantizer':
+ toks[-2] = 'kernel'
+ elif toks[-2] == 'input_quantizer':
+ toks[-2] = 'input'
+
+ if 'final_input_quantizer' not in toks[2]:
+ ind = toks.index('layers')+1 if 'layers' in toks else 3
+ toks = toks[ind:]
+ toks[0] = 'l{}'.format(int(toks[0]))
+ else:
+ if toks[-2] == 'layernorm': #bias->beta, weight->gamma
+ toks[-1] = 'beta' if toks[-1] == 'bias' else 'gamma'
+ else: #embeddings: drop "_weight" suffix
+ if toks[-1] == 'amax':
+ toks[-2] = 'amax'
+ toks = toks[:-1]
+ elif 'qa' in onnx_name:
+ name = 'cls_squad_output_bias' if toks[-1] == 'bias' else 'cls_squad_output_weights'
+ return name
+ else:
+ print("Encountered unknown case:", onnx_name)
+ assert(False)
+ parsed = '_'.join(toks)
+ return parsed
+
+def get_onnx_weight_dict(tensor_dict, config):
+ N = config.num_attention_heads
+ H = config.head_size
+ hidden_size = config.hidden_size
+
+ weights_dict = dict()
+ for outname, tensor in tensor_dict.items():
+ if outname.find("_amax") != -1:
+ weights_dict[outname] = tensor
+ elif outname.find(BQ) != -1:
+ prefix = outname[:outname.find(BQ)]
+
+ Wqkv = np.zeros((3, hidden_size, hidden_size), np.float32)
+ Bqkv = np.zeros((3, hidden_size), np.float32)
+
+ Wqkv[0,:,:] = tensor_dict[prefix + WQ]
+ Wqkv[1,:,:] = tensor_dict[prefix + WK]
+ Wqkv[2,:,:] = tensor_dict[prefix + WV]
+ Bqkv[0,:] = tensor
+ Bqkv[1,:] = tensor_dict[prefix + BK]
+ Bqkv[2,:] = tensor_dict[prefix + BV]
+
+ if config.use_int8 and getattr(config, 'interleaved', False):
+ Wqkv = np.ascontiguousarray(Wqkv.reshape((3, N, H, N, H)))
+ Bqkv = np.ascontiguousarray(Bqkv.reshape((3, N, H)))
+ else:
+ Wqkv = np.ascontiguousarray(Wqkv.reshape((3, N, H, N, H)).transpose((1,0,2,3,4)))
+ Bqkv = np.ascontiguousarray(Bqkv.reshape((3, N, H)).transpose((1,0,2)))
+
+ weights_dict[prefix + WQKV] = trt.Weights(Wqkv)
+ weights_dict[prefix + BQKV] = trt.Weights(Bqkv)
+ weights_dict[prefix + WQKV + "_notrans"] = trt.Weights(Wqkv.T)
+
+ elif outname.find(BK) != -1 or outname.find(BV) != -1 or outname.find(WQ) != -1 or outname.find(WK) != -1 or outname.find(WV) != -1:
+ pass
+ else:
+ flat_tensor = np.ascontiguousarray(tensor).flatten()
+ weights_dict[outname] = trt.Weights(flat_tensor)
+
+ if outname.find("kernel") != -1:
+ tensor = np.transpose(tensor)
+ weights_dict[outname + "_notrans"] = trt.Weights(np.ascontiguousarray(tensor).flatten())
+
+ TRT_LOGGER.log(TRT_LOGGER.INFO, "Found {:} entries in weight map".format(len(weights_dict)))
+ return weights_dict
+
+def load_onnx_weights_and_quant(path, config):
+ """
+ Load the weights from the onnx checkpoint
+ """
+ model = onnx.load(path)
+ weights = model.graph.initializer
+ tensor_dict = dict((onnx_to_trt_name(w.name), np.frombuffer(w.raw_data, np.int8).reshape(w.dims))
+ if w.name.split('_')[-1] == 'mask' else
+ (onnx_to_trt_name(w.name), np.frombuffer(w.raw_data, np.float32).reshape(w.dims))
+ for w in weights)
+ return get_onnx_weight_dict(tensor_dict, config)
+
+def load_pytorch_weights_and_quant(path, config):
+ """
+ Load the weights from the pytorch checkpoint
+ """
+ state_dict = torch.load(path, map_location='cpu')["model"]
+ tensor_dict = {onnx_to_trt_name(name):val.numpy() for name, val in state_dict.items()}
+ return get_onnx_weight_dict(tensor_dict, config)
+
+def load_megatron_pickle_weights(path, config):
+ N = config.num_attention_heads
+ H = config.head_size
+
+ with open(path, 'rb') as f:
+ tensor_dict = pickle.load(f)
+
+ weight_dict = {}
+ for name, tensor in tensor_dict.items():
+ if 'scale' in name:
+ continue
+
+ name = (onnx_to_trt_name(name)
+ .replace('embedding_', 'embeddings_')
+ .replace('tokentype_', 'token_type_')
+ .replace('_av', '_self_av')
+ .replace('_qv', '_self_qv')
+ .replace('query_key_value', 'self_qkv'))
+
+ if name.endswith('self_qkv_kernel'):
+ tensor = np.ascontiguousarray(tensor.reshape((3, N, H, N, H))).astype(np.float32)
+ weight_dict[name] = trt.Weights(tensor)
+ elif name.endswith('self_qkv_bias'):
+ tensor = np.ascontiguousarray(tensor.reshape((3, N, H))).astype(np.float32)
+ weight_dict[name] = trt.Weights(tensor)
+ elif name == 'l{}_output_layernorm_output_quantizer_amax'.format(config.num_hidden_layers-1):
+ weight_dict['bert_encoder_final_input_quantizer_amax'] = tensor
+ elif name.endswith('_amax'):
+ weight_dict[name] = tensor
+ if name.endswith('_qkv_input_amax'):
+ weight_dict[name.replace('_qkv_input_amax', '_query_input_amax')] = tensor
+ weight_dict[name.replace('_qkv_input_amax', '_key_input_amax')] = tensor
+ weight_dict[name.replace('_qkv_input_amax', '_value_input_amax')] = tensor
+ else:
+ flat_tensor = np.ascontiguousarray(tensor).flatten().astype(np.float32)
+ weight_dict[name] = trt.Weights(flat_tensor)
+
+ TRT_LOGGER.log(TRT_LOGGER.INFO, "Found {:} entries in weight map".format(len(weight_dict)))
+ return weight_dict
diff --git a/demo/BERT/builder_varseqlen.py b/demo/BERT/builder_varseqlen.py
old mode 100644
new mode 100755
index 4e88399e..be8e9b37
--- a/demo/BERT/builder_varseqlen.py
+++ b/demo/BERT/builder_varseqlen.py
@@ -29,12 +29,10 @@ import pycuda.autoinit
# TensorRT
import tensorrt as trt
-
-try:
- from tensorflow.python import pywrap_tensorflow as pyTF
-except ImportError as err:
- sys.stderr.write("""Error: Failed to import tensorflow module ({})\n""".format(err))
- sys.exit()
+from builder_utils import load_tf_weights, load_pytorch_weights_and_quant, load_onnx_weights_and_quant, load_megatron_pickle_weights
+from builder_utils import WQKV, BQKV # Attention Keys
+from builder_utils import W_AOUT, B_AOUT, W_MID, B_MID, W_LOUT, B_LOUT # Transformer Keys
+from builder_utils import SQD_W, SQD_B # SQuAD Output Keys
"""
TensorRT Initialization
@@ -55,40 +53,12 @@ skln_plg_creator2 = plg_registry.get_plugin_creator("CustomSkipLayerNormPluginDy
mha_plg_creator3 = plg_registry.get_plugin_creator("CustomQKVToContextPluginDynamic", "3", "")
skln_plg_creator3 = plg_registry.get_plugin_creator("CustomSkipLayerNormPluginDynamic", "3", "")
-"""
-Attentions Keys
-"""
-WQ = "self_query_kernel"
-BQ = "self_query_bias"
-WK = "self_key_kernel"
-BK = "self_key_bias"
-WV = "self_value_kernel"
-BV = "self_value_bias"
-WQKV = "self_qkv_kernel"
-BQKV = "self_qkv_bias"
-
-"""
-Transformer Keys
-"""
-W_AOUT = "attention_output_dense_kernel"
-B_AOUT = "attention_output_dense_bias"
-AOUT_LN_BETA = "attention_output_layernorm_beta"
-AOUT_LN_GAMMA = "attention_output_layernorm_gamma"
-W_MID = "intermediate_dense_kernel"
-B_MID = "intermediate_dense_bias"
-W_LOUT = "output_dense_kernel"
-B_LOUT = "output_dense_bias"
-LOUT_LN_BETA = "output_layernorm_beta"
-LOUT_LN_GAMMA = "output_layernorm_gamma"
-
-"""
-Squad Output Keys
-"""
-SQD_W = "squad_output_weights"
-SQD_B = "squad_output_bias"
+# Megatron Plugins
+emln_plg_creator3 = plg_registry.get_plugin_creator("CustomEmbLayerNormPluginDynamic", "3", "")
+skln_plg_creator4 = plg_registry.get_plugin_creator("CustomSkipLayerNormPluginDynamic", "4", "")
class BertConfig:
- def __init__(self, bert_config_path, use_fp16, use_int8, use_qat, interleaved, timing_cache):
+ def __init__(self, bert_config_path, use_fp16, use_int8, use_qat, interleaved, timing_cache, use_sparsity, use_megatron):
with open(bert_config_path, "r") as f:
data = json.load(f)
self.num_attention_heads = data["num_attention_heads"]
@@ -101,6 +71,8 @@ class BertConfig:
self.use_qat = use_qat
self.interleaved = interleaved
self.timing_cache = timing_cache
+ self.use_sparsity = use_sparsity
+ self.use_megatron = use_megatron
def get_trt_dtype(self):
dtype = trt.float32
@@ -132,7 +104,7 @@ def attention_layer_opt(prefix, config, init_dict, network, input_tensor, mask_i
# FC_attention
if config.use_int8:
- mult_all = network.add_convolution(input_tensor, 3 * hidden_size, (1, 1), Wall, Ball)
+ mult_all = network.add_convolution_nd(input_tensor, 3 * hidden_size, (1, 1), Wall, Ball)
else:
mult_all = network.add_fully_connected(input_tensor, 3 * hidden_size, Wall, Ball)
@@ -182,7 +154,7 @@ def attention_layer_opt(prefix, config, init_dict, network, input_tensor, mask_i
set_output_name(qkv2ctx, prefix, "context_layer")
return qkv2ctx
-def skipln(prefix, config, init_dict, network, input_tensor, skip):
+def skipln(prefix, config, init_dict, network, input_tensor, skip, is_last_skipln=False):
"""
Add the skip layer
"""
@@ -198,7 +170,8 @@ def skipln(prefix, config, init_dict, network, input_tensor, skip):
if config.use_int8 and config.interleaved:
pfc = trt.PluginFieldCollection([pf_beta, pf_gamma])
- skipln_plug = skln_plg_creator3.create_plugin("skipln", pfc)
+ creator = skln_plg_creator3 if not config.use_megatron or is_last_skipln else skln_plg_creator4
+ skipln_plug = creator.create_plugin("skipln", pfc)
else:
pfc = trt.PluginFieldCollection([pf_ld, pf_beta, pf_gamma, pf_type])
skipln_plug = skln_plg_creator2.create_plugin("skipln", pfc)
@@ -207,7 +180,7 @@ def skipln(prefix, config, init_dict, network, input_tensor, skip):
layer = network.add_plugin_v2(skipln_inputs, skipln_plug)
return layer
-def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, mask_idx, cu_seqlens, max_seqlen):
+def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, residual, mask_idx, cu_seqlens, max_seqlen):
"""
Add the transformer layer
"""
@@ -226,14 +199,20 @@ def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, mask
B_aout = init_dict[prefix + B_AOUT]
W_aout = init_dict[prefix + W_AOUT]
if config.use_int8:
- attention_out_fc = network.add_convolution(attention_heads, hidden_size, (1, 1), W_aout, B_aout)
+ attention_out_fc = network.add_convolution_nd(attention_heads, hidden_size, (1, 1), W_aout, B_aout)
else:
attention_out_fc = network.add_fully_connected(attention_heads, hidden_size, W_aout, B_aout)
if config.use_int8 and config.use_qat:
dr_fc_aout = init_dict[prefix + 'attention_output_add_local_input_quantizer_amax']
set_output_range(attention_out_fc, dr_fc_aout)
- skiplayer = skipln(prefix + "attention_output_layernorm_", config, init_dict, network, attention_out_fc.get_output(0), input_tensor)
+ if config.use_megatron:
+ dr_skln1_res_in = init_dict[prefix + "attention_output_add_residual_input_quantizer_amax"]
+ residual.set_dynamic_range(-dr_skln1_res_in, dr_skln1_res_in)
+ skip = residual
+ else:
+ skip = input_tensor
+ skiplayer = skipln(prefix + "attention_output_layernorm_", config, init_dict, network, attention_out_fc.get_output(0), skip)
attention_ln = skiplayer.get_output(0)
if config.use_qat:
dr_skln1 = init_dict[prefix + 'intermediate_dense_input_amax']
@@ -243,27 +222,11 @@ def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, mask
B_mid = init_dict[prefix + B_MID]
W_mid = init_dict[prefix + W_MID]
if config.use_int8:
- mid_dense = network.add_convolution(attention_ln, config.intermediate_size, (1, 1), W_mid, B_mid)
+ mid_dense = network.add_convolution_nd(attention_ln, config.intermediate_size, (1, 1), W_mid, B_mid)
else:
mid_dense = network.add_fully_connected(attention_ln, config.intermediate_size, W_mid, B_mid)
- mid_dense_out = mid_dense.get_output(0)
- POW = network.add_constant((1,) * len(mid_dense_out.shape), trt.Weights(np.ascontiguousarray([3.0], dtype=np.float32)))
- MULTIPLY = network.add_constant((1,) * len(mid_dense_out.shape), trt.Weights(np.ascontiguousarray([0.044715], dtype=np.float32)))
- SQRT = network.add_constant((1,) * len(mid_dense_out.shape), trt.Weights((np.ascontiguousarray([0.79788456080286535587989211986876], dtype=np.float32))))
- ONE = network.add_constant((1,) * len(mid_dense_out.shape), trt.Weights((np.ascontiguousarray([1.0], dtype=np.float32))))
- HALF = network.add_constant((1,) * len(mid_dense_out.shape), trt.Weights((np.ascontiguousarray([0.5], dtype=np.float32))))
- X_pow = network.add_elementwise(mid_dense_out, POW.get_output(0), trt.ElementWiseOperation.POW)
- X_pow_t = X_pow.get_output(0)
- X_mul = network.add_elementwise(X_pow_t, MULTIPLY.get_output(0), trt.ElementWiseOperation.PROD)
- X_add = network.add_elementwise(mid_dense_out, X_mul.get_output(0), trt.ElementWiseOperation.SUM)
- X_sqrt = network.add_elementwise(X_add.get_output(0), SQRT.get_output(0), trt.ElementWiseOperation.PROD)
- X_sqrt_tensor = X_sqrt.get_output(0)
- X_tanh = network.add_activation(X_sqrt_tensor, trt.ActivationType.TANH)
- X_tanh_tensor = X_tanh.get_output(0)
- X_one = network.add_elementwise(X_tanh_tensor, ONE.get_output(0), trt.ElementWiseOperation.SUM)
- CDF = network.add_elementwise(X_one.get_output(0), HALF.get_output(0), trt.ElementWiseOperation.PROD)
- gelu_layer = network.add_elementwise(CDF.get_output(0), mid_dense_out, trt.ElementWiseOperation.PROD)
+ gelu_layer = add_gelu(network, mid_dense.get_output(0))
intermediate_act = gelu_layer.get_output(0)
set_tensor_name(intermediate_act, prefix, "gelu")
@@ -281,7 +244,7 @@ def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, mask
W_lout = init_dict[prefix + W_LOUT]
if config.use_int8:
- out_dense = network.add_convolution(intermediate_act, hidden_size, (1, 1), W_lout, B_lout)
+ out_dense = network.add_convolution_nd(intermediate_act, hidden_size, (1, 1), W_lout, B_lout)
else:
out_dense = network.add_fully_connected(intermediate_act, hidden_size, W_lout, B_lout)
if config.use_int8 and config.use_qat:
@@ -289,25 +252,79 @@ def transformer_layer_opt(prefix, config, init_dict, network, input_tensor, mask
set_output_range(out_dense, dr_fc_out)
set_output_name(out_dense, prefix + "output_", "dense")
- out_layer = skipln(prefix + "output_layernorm_", config, init_dict, network, out_dense.get_output(0), attention_ln)
+ if config.use_megatron:
+ dr_skln2_res_in = init_dict[prefix + 'output_add_residual_input_quantizer_amax']
+ set_output_range(skiplayer, dr_skln2_res_in, out_idx=1)
+ skip = skiplayer.get_output(1)
+ else:
+ skip = attention_ln
+
+ is_last_skipln = prefix.startswith('l{}'.format(config.num_hidden_layers-1))
+ out_layer = skipln(prefix + "output_layernorm_", config, init_dict, network, out_dense.get_output(0), skip, is_last_skipln)
set_output_name(out_layer, prefix + "output_", "reshape")
return out_layer
-def bert_model(config, init_dict, network, input_tensor, mask_idx, cu_seqlens, max_seqlen):
+def add_gelu(network, input_tensor):
+ """
+ Adds elementwise GELU, and will trigger FC+GELU fusion in TRT
+ """
+ shape = (1, ) * len(input_tensor.shape)
+ POW = network.add_constant(shape, trt.Weights(np.ascontiguousarray([3.0], dtype=np.float32)))
+ MULTIPLY = network.add_constant(shape, trt.Weights(np.ascontiguousarray([0.044715], dtype=np.float32)))
+ SQRT = network.add_constant(shape, trt.Weights((np.ascontiguousarray([0.79788456080286535587989211986876], dtype=np.float32))))
+ ONE = network.add_constant(shape, trt.Weights((np.ascontiguousarray([1.0], dtype=np.float32))))
+ HALF = network.add_constant(shape, trt.Weights((np.ascontiguousarray([0.5], dtype=np.float32))))
+ X_pow = network.add_elementwise(input_tensor, POW.get_output(0), trt.ElementWiseOperation.POW)
+ X_pow_t = X_pow.get_output(0)
+ X_mul = network.add_elementwise(X_pow_t, MULTIPLY.get_output(0), trt.ElementWiseOperation.PROD)
+ X_add = network.add_elementwise(input_tensor, X_mul.get_output(0), trt.ElementWiseOperation.SUM)
+ X_sqrt = network.add_elementwise(X_add.get_output(0), SQRT.get_output(0), trt.ElementWiseOperation.PROD)
+ X_sqrt_tensor = X_sqrt.get_output(0)
+ X_tanh = network.add_activation(X_sqrt_tensor, trt.ActivationType.TANH)
+ X_tanh_tensor = X_tanh.get_output(0)
+ X_one = network.add_elementwise(X_tanh_tensor, ONE.get_output(0), trt.ElementWiseOperation.SUM)
+ CDF = network.add_elementwise(X_one.get_output(0), HALF.get_output(0), trt.ElementWiseOperation.PROD)
+ gelu_layer = network.add_elementwise(CDF.get_output(0), input_tensor, trt.ElementWiseOperation.PROD)
+
+ # enable elementwise fusing for int8 && fp16
+ POW.precision = trt.DataType.FLOAT
+ MULTIPLY.precision = trt.DataType.FLOAT
+ SQRT.precision = trt.DataType.FLOAT
+ ONE.precision = trt.DataType.FLOAT
+ HALF.precision = trt.DataType.FLOAT
+ X_pow.precision = trt.DataType.FLOAT
+ X_mul.precision = trt.DataType.FLOAT
+ X_add.precision = trt.DataType.FLOAT
+ X_sqrt.precision = trt.DataType.FLOAT
+ X_tanh.precision = trt.DataType.FLOAT
+ X_one.precision = trt.DataType.FLOAT
+ CDF.precision = trt.DataType.FLOAT
+ gelu_layer.precision = trt.DataType.FLOAT
+ return gelu_layer
+
+
+def bert_model(config, init_dict, network, input_tensor, residual, mask_idx, cu_seqlens, max_seqlen):
"""
Create the bert model
"""
prev_input = input_tensor
for layer in range(0, config.num_hidden_layers):
ss = "l{}_".format(layer)
- out_layer = transformer_layer_opt(ss, config, init_dict, network, prev_input, mask_idx, cu_seqlens, max_seqlen)
+ out_layer = transformer_layer_opt(ss, config, init_dict, network, prev_input, residual, mask_idx, cu_seqlens, max_seqlen)
prev_input = out_layer.get_output(0)
+ # Skip reading residual from final layer
+ if config.use_megatron and (layer != config.num_hidden_layers - 1):
+ residual = out_layer.get_output(1)
if config.use_qat:
dr_out = init_dict["bert_encoder_final_input_quantizer_amax"]
set_output_range(out_layer, dr_out)
- return prev_input
+
+ squad_logits = squad_output("cls_", config, init_dict, network, prev_input)
+ squad_logits_out = squad_logits.get_output(0)
+ network.mark_output(squad_logits_out)
+
def squad_output(prefix, config, init_dict, network, input_tensor):
"""
@@ -319,7 +336,7 @@ def squad_output(prefix, config, init_dict, network, input_tensor):
B_out = init_dict[prefix + SQD_B]
if config.use_int8:
- dense = network.add_convolution(input_tensor, 2, (1, 1), W_out, B_out)
+ dense = network.add_convolution_nd(input_tensor, 2, (1, 1), W_out, B_out)
else:
dense = network.add_fully_connected(input_tensor, 2, W_out, B_out)
@@ -331,184 +348,6 @@ def squad_output(prefix, config, init_dict, network, input_tensor):
set_output_name(OUT, prefix, "squad_logits")
return OUT
-def load_tf_weights(inputbase, config):
- """
- Load the weights from the tensorflow checkpoint
- """
- weights_dict = dict()
-
- try:
- reader = pyTF.NewCheckpointReader(inputbase)
- tensor_dict = reader.get_variable_to_shape_map()
-
- # There might be training-related variables in the checkpoint that can be discarded
- param_names = [key for key in sorted(tensor_dict) if "adam" not in key and "global_step" not in key and "pooler" not in key]
- count = len(param_names)
- TRT_LOGGER.log(TRT_LOGGER.INFO, "Found {:} entries in weight map".format(count))
-
- for pn in param_names:
- toks = pn.lower().split("/")
- if "encoder" in pn:
- assert ("layer" in pn)
- l = (re.findall("\d+", pn))[0]
- outname = "l{}_".format(l) + "_".join(toks[3:])
- else:
- outname = "_".join(toks)
-
- tensor = reader.get_tensor(pn)
- shape = tensor.shape
- if pn.find("kernel") != -1:
- weights_dict[outname + "_notrans"] = trt.Weights(np.ascontiguousarray(tensor).flatten())
-
- TRT_LOGGER.log(TRT_LOGGER.VERBOSE, "Transposing {}\n".format(np))
- tensor = np.transpose(tensor)
-
- shape = tensor.shape
- flat_tensor = tensor.flatten()
- shape_str = "{} ".format(len(shape)) + " ".join([str(d) for d in shape])
- weights_dict[outname] = trt.Weights(flat_tensor)
-
- TRT_LOGGER.log(TRT_LOGGER.VERBOSE, "Original name: {:}, TensorRT name: {:}, shape: {:}".format(pn, outname, shape_str))
-
- N = config.num_attention_heads
- H = config.head_size
-
- additional_dict = dict()
- for key, value in weights_dict.items():
- pos = key.find(BQ)
- if pos != -1:
- hidden_size = value.size
- prefix = key[:pos]
-
- Bq_ = value
- Bk_ = weights_dict[prefix + BK]
- Bv_ = weights_dict[prefix + BV]
- Wq_ = weights_dict[prefix + WQ]
- Wk_ = weights_dict[prefix + WK]
- Wv_ = weights_dict[prefix + WV]
-
- mat_size = hidden_size * hidden_size
- wcount = 3 * mat_size
- Wall = np.zeros(wcount, np.float32)
- bcount = 3 * hidden_size
- Ball = np.zeros(bcount, np.float32)
- Wall[0:mat_size] = Wq_.numpy()[0:mat_size]
- Wall[mat_size:2*mat_size] = Wk_.numpy()[0:mat_size]
- Wall[2*mat_size:3*mat_size] = Wv_.numpy()[0:mat_size]
- Ball[0:hidden_size] = Bq_.numpy()[0:hidden_size]
- Ball[hidden_size:2*hidden_size] = Bk_.numpy()[0:hidden_size]
- Ball[2*hidden_size:3*hidden_size] = Bv_.numpy()[0:hidden_size]
-
- if config.use_int8 and config.interleaved:
- Wall = np.ascontiguousarray(Wall.reshape((3, N, H, N, H)), dtype=np.float32)
- Ball = np.ascontiguousarray(Ball.reshape((3, N, H)), dtype=np.float32)
- else:
- Wall = np.ascontiguousarray(Wall.reshape((3, N, H, N, H)).transpose((1, 0, 2, 3, 4)), dtype=np.float32)
- Ball = np.ascontiguousarray(Ball.reshape((3, N, H)).transpose((1, 0, 2)), dtype=np.float32)
-
- additional_dict[prefix + WQKV] = trt.Weights(Wall)
- additional_dict[prefix + BQKV] = trt.Weights(Ball)
-
- additional_dict[prefix + WQKV + "_notrans"] = trt.Weights(Wall.T)
-
- except Exception as error:
- TRT_LOGGER.log(TRT_LOGGER.ERROR, str(error))
-
- weights_dict.update(additional_dict)
- return weights_dict
-
-def onnx_to_trt_name(onnx_name):
- """
- Converting variables in the onnx checkpoint to names corresponding to the naming convention used in the TF version, expected by the builder
- """
- onnx_name = onnx_name.lower()
- toks = [t.strip('_') for t in onnx_name.split('.')]
- if toks[0] == 'bert': #embeddings or encoder
- if toks[1] == 'encoder': #transformer
-
- if toks[-2] == 'layernorm': #bias->beta, weight->gamma
- toks[-1] = 'beta' if toks[-1] == 'bias' else 'gamma'
- elif (toks[-2] == 'dense' or toks[-2] in {'key', 'value', 'query'}) and toks[-1] == 'weight':
- toks[-1] = 'kernel'
- elif (toks[-3] == 'dense' or toks[-3] in {'key', 'value', 'query'}) and toks[-1] == 'amax':
- if toks[-2] == 'weight_quantizer':
- toks[-2] = 'kernel'
- elif toks[-2] == 'input_quantizer':
- toks[-2] = 'input'
-
- if 'final_input_quantizer' not in toks[2]:
- toks = toks[3:]
- toks[0] = 'l{}'.format(int(toks[0]))
- else:
- if toks[-2] == 'layernorm': #bias->beta, weight->gamma
- toks[-1] = 'beta' if toks[-1] == 'bias' else 'gamma'
- else: #embeddings: drop "_weight" suffix
- if toks[-1] == 'amax':
- toks[-2] = 'amax'
- toks = toks[:-1]
- elif 'qa' in onnx_name:
- name = 'cls_squad_output_bias' if toks[-1] == 'bias' else 'cls_squad_output_weights'
- return name
- else:
- print("Encountered unknown case:", onnx_name)
- assert(False)
- parsed = '_'.join(toks)
- return parsed
-
-def load_onnx_weights_and_quant(path, config):
- """
- Load the weights from the onnx checkpoint
- """
- N = config.num_attention_heads
- H = config.head_size
- hidden_size = config.hidden_size
-
- model = onnx.load(path)
- weights = model.graph.initializer
- tensor_dict = dict([(onnx_to_trt_name(w.name), np.frombuffer(w.raw_data, np.float32).reshape(w.dims)) for w in weights])
-
- weights_dict = dict()
- for outname, tensor in tensor_dict.items():
- if outname.find("_amax") != -1:
- weights_dict[outname] = tensor
- elif outname.find(BQ) != -1:
- prefix = outname[:outname.find(BQ)]
-
- Wqkv = np.zeros((3, hidden_size, hidden_size), np.float32)
- Bqkv = np.zeros((3, hidden_size), np.float32)
-
- Wqkv[0,:,:] = tensor_dict[prefix + WQ]
- Wqkv[1,:,:] = tensor_dict[prefix + WK]
- Wqkv[2,:,:] = tensor_dict[prefix + WV]
- Bqkv[0,:] = tensor
- Bqkv[1,:] = tensor_dict[prefix + BK]
- Bqkv[2,:] = tensor_dict[prefix + BV]
-
- if config.use_int8 and config.interleaved:
- Wqkv = np.ascontiguousarray(Wqkv.reshape((3, N, H, N, H)))
- Bqkv = np.ascontiguousarray(Bqkv.reshape((3, N, H)))
- else:
- Wqkv = np.ascontiguousarray(Wqkv.reshape((3, N, H, N, H)).transpose((1,0,2,3,4)))
- Bqkv = np.ascontiguousarray(Bqkv.reshape((3, N, H)).transpose((1,0,2)))
-
- weights_dict[prefix + WQKV] = trt.Weights(Wqkv)
- weights_dict[prefix + BQKV] = trt.Weights(Bqkv)
- weights_dict[prefix + WQKV + "_notrans"] = trt.Weights(Wqkv.T)
-
- elif outname.find(BK) != -1 or outname.find(BV) != -1 or outname.find(WQ) != -1 or outname.find(WK) != -1 or outname.find(WV) != -1:
- pass
- else:
- flat_tensor = np.ascontiguousarray(tensor).flatten()
- weights_dict[outname] = trt.Weights(flat_tensor)
-
- if outname.find("kernel") != -1:
- tensor = np.transpose(tensor)
- weights_dict[outname + "_notrans"] = trt.Weights(np.ascontiguousarray(tensor).flatten())
-
-
- TRT_LOGGER.log(TRT_LOGGER.INFO, "Found {:} entries in weight map".format(len(weights_dict)))
- return weights_dict
-
def emb_layernorm(builder, network, config, weights_dict, builder_config, max_sequence_length, max_batch_size):
input_ids = network.add_input(name="input_ids", dtype=trt.int32, shape=(-1,))
segment_ids = network.add_input(name="segment_ids", dtype=trt.int32, shape=(-1,))
@@ -530,18 +369,22 @@ def emb_layernorm(builder, network, config, weights_dict, builder_config, max_se
wwordemb = trt.PluginField("bert_embeddings_word_embeddings", weights_dict["bert_embeddings_word_embeddings"].numpy(), trt.PluginFieldType.FLOAT32)
wtokemb = trt.PluginField("bert_embeddings_token_type_embeddings", weights_dict["bert_embeddings_token_type_embeddings"].numpy(), trt.PluginFieldType.FLOAT32)
wposemb = trt.PluginField("bert_embeddings_position_embeddings", weights_dict["bert_embeddings_position_embeddings"].numpy(), trt.PluginFieldType.FLOAT32)
-
output_fp16 = trt.PluginField("output_fp16", np.array([1 if config.use_fp16 or config.use_int8 else 0]).astype(np.int32), trt.PluginFieldType.INT32)
pfc = trt.PluginFieldCollection([wbeta, wgamma, wwordemb, wtokemb, wposemb, output_fp16])
- fn = emln_plg_creator2.create_plugin("embeddings", pfc)
+ fn = (emln_plg_creator3 if config.use_megatron else emln_plg_creator2).create_plugin("embeddings", pfc)
inputs = [input_ids, segment_ids, cu_seqlens, max_seqlen]
emb_layer = network.add_plugin_v2(inputs, fn)
if config.use_int8 and config.use_qat:
dr_input = weights_dict['l0_attention_self_query_input_amax']
- set_output_range(emb_layer, dr_input)
+ set_output_range(emb_layer, dr_input, out_idx=0)
+
+ if config.use_megatron:
+ dr_skln1_res_in = weights_dict['l0_attention_output_add_residual_input_quantizer_amax']
+ set_output_range(emb_layer, dr_skln1_res_in, out_idx=1)
+
set_output_name(emb_layer, "embeddings_", "output")
return emb_layer, cu_seqlens, max_seqlen
@@ -572,6 +415,9 @@ def build_engine(batch_size, workspace_size, sequence_length, config, weights_di
cache = builder_config.create_timing_cache(b"")
builder_config.set_timing_cache(cache, ignore_mismatch = False)
+ if config.use_sparsity:
+ TRT_LOGGER.log(TRT_LOGGER.INFO, "Setting sparsity flag on builder_config.")
+ builder_config.set_flag(trt.BuilderFlag.SPARSE_WEIGHTS)
# Create the network
emb_layer, cu_seqlens, max_seqlen = emb_layernorm(builder, network, config, weights_dict, builder_config, sequence_length, batch_size)
@@ -583,12 +429,20 @@ def build_engine(batch_size, workspace_size, sequence_length, config, weights_di
mask_idx = None
else:
mask_idx = emb_layer.get_output(1)
+
+ if config.use_megatron: # megatron currently only supports int8 and interleaved
+ shuffler = network.add_shuffle(emb_layer.get_output(1))
+ shuffler.second_transpose = (2, 1, 0, 3)
+ residual = shuffler.get_output(0)
- bert_out = bert_model(config, weights_dict, network, embeddings, mask_idx, cu_seqlens, max_seqlen)
+ dr_emb = weights_dict['l0_attention_self_query_input_amax']
+ embeddings.set_dynamic_range(-dr_emb, dr_emb)
+ dr_skln1_res_in = weights_dict['l0_attention_output_add_residual_input_quantizer_amax']
+ residual.set_dynamic_range(-dr_skln1_res_in, dr_skln1_res_in)
+ else:
+ residual = None
- squad_logits = squad_output("cls_", config, weights_dict, network, bert_out)
- squad_logits_out = squad_logits.get_output(0)
- network.mark_output(squad_logits_out)
+ bert_model(config, weights_dict, network, embeddings, residual, mask_idx, cu_seqlens, max_seqlen)
build_start_time = time.time()
engine = builder.build_engine(network, builder_config)
@@ -611,6 +465,8 @@ def main():
parser.add_argument("-m", "--ckpt", required=False,
help="The checkpoint file basename, e.g.: basename(model.ckpt-766908.data-00000-of-00001) is model.ckpt-766908")
parser.add_argument("-x", "--onnx", required=False, help="The ONNX model file path.")
+ parser.add_argument("-pt", "--pytorch", required=False, help="The PyTorch checkpoint file path.")
+ parser.add_argument("-pkl", "--pickle", required=False, help="The Pickle weights dictionary file path for the Megatron variant of BERT.")
parser.add_argument("-o", "--output", required=True, default="bert_base_384.engine", help="The bert engine file, ex bert.engine")
parser.add_argument("-b", "--max-batch-size", default=1, help="Max batch size. The engine will be usable with any input with (batch-size * sequence-length) below (max-batch-size * max-sequence-length).", type=int)
parser.add_argument("-s", "--max-sequence-length", default=128, help="Max sequence length of the BERT model. The engine will be usable with any input with (batch-size * sequence-length) below (max-batch-size * max-sequence-length).", type=int)
@@ -625,17 +481,25 @@ def main():
parser.add_argument("-p", "--calib-path", help="calibration cache path", required=False)
parser.add_argument("-il", "--interleaved", action="store_true", help="use interleaved format, only valid in INT8 precision", required=False)
parser.add_argument("-tcf", "--timing-cache-file", help="Path to tensorrt build timeing cache file, only available for tensorrt 8.0 and later", required=False)
+ parser.add_argument("-sp", "--sparse", action="store_true", help="Indicates that model is sparse", required=False)
+ parser.add_argument("--megatron", action="store_true", help="Indicates that model is the Megatron-style architecture", required=False)
args, _ = parser.parse_known_args()
cc = pycuda.autoinit.device.compute_capability()
if cc[0] * 10 + cc[1] < 72:
raise RuntimeError("This variable-length BERT demo only support Xavier+ GPU.")
+
+ if args.megatron:
+ if not (args.interleaved and args.int8):
+ raise RuntimeError("Megatron BERT currently only supports int8 and interleaved.")
+ if not args.pickle:
+ raise RuntimeError("Megatron BERT currently only supports loading a pickle weights dictionary.")
bert_config_path = os.path.join(args.config_dir, "bert_config.json")
TRT_LOGGER.log(TRT_LOGGER.INFO, "Using configuration file: {:}".format(bert_config_path))
- config = BertConfig(bert_config_path, args.fp16, args.int8, args.int8 and args.onnx != None, args.interleaved, args.timing_cache_file)
+ config = BertConfig(bert_config_path, args.fp16, args.int8, args.int8 and (args.onnx or args.pytorch or args.pickle), args.interleaved, args.timing_cache_file, args.sparse, args.megatron)
if args.calib_path != None:
calib_cache = args.calib_path
@@ -644,10 +508,16 @@ def main():
if args.onnx != None:
weights_dict = load_onnx_weights_and_quant(args.onnx, config)
+ elif args.pytorch != None:
+ weights_dict = load_pytorch_weights_and_quant(args.pytorch, config)
elif args.ckpt != None:
weights_dict = load_tf_weights(args.ckpt, config)
+ elif args.pickle != None:
+ weights_dict = load_megatron_pickle_weights(args.pickle, config)
else:
- raise RuntimeError("You need either specify TF checkpoint using option --ckpt or ONNX using option --onnx to build TRT BERT model.")
+ raise RuntimeError("You need either specify TF checkpoint using option --ckpt, ONNX using option --onnx, "
+ "PyTorch using option --pytorch, or Pickle weight dictionary using option --pickle "
+ "to build TRT BERT model.")
with build_engine(args.max_batch_size, args.workspace_size, args.max_sequence_length, config, weights_dict, args.squad_json, args.vocab_file, calib_cache, args.calib_num) as engine:
TRT_LOGGER.log(TRT_LOGGER.VERBOSE, "Serializing Engine...")
diff --git a/demo/BERT/infer_c/bert_infer.h b/demo/BERT/infer_c/bert_infer.h
index f50776b9..f4fba2f8 100644
--- a/demo/BERT/infer_c/bert_infer.h
+++ b/demo/BERT/infer_c/bert_infer.h
@@ -318,7 +318,7 @@ struct BertInference
gLogInfo << "\tAverage Time: " << avgTime << " ms\n";
gLogInfo << "\t95th Percentile Time: " << percentile95 << " ms\n";
gLogInfo << "\t99th Percentile Time: " << percentile99 << " ms\n";
- gLogInfo << "\tThroughtput: " << throughput << " sentences/s\n";
+ gLogInfo << "\tThroughput: " << throughput << " sentences/s\n";
}
~BertInference()
diff --git a/demo/BERT/infer_c/logging.h b/demo/BERT/infer_c/logging.h
index c89e827a..d4d51099 100644
--- a/demo/BERT/infer_c/logging.h
+++ b/demo/BERT/infer_c/logging.h
@@ -66,7 +66,7 @@ public:
}
private:
- void log(Severity severity, const char* msg) override
+ void log(Severity severity, const char* msg) noexcept override
{
report(severity, msg) << "\n";
}
diff --git a/demo/BERT/inference.py b/demo/BERT/inference.py
index ab4a7123..96c9e812 100644
--- a/demo/BERT/inference.py
+++ b/demo/BERT/inference.py
@@ -76,6 +76,9 @@ def parse_args():
parser.add_argument('--n-best-size',
help='Total number of n-best predictions to generate in the nbest_predictions.json output file',
default=20, type=int)
+ parser.add_argument('--doc-stride',
+ help='When splitting up a long document into chunks, what stride to take between chunks',
+ default=128, type=int)
args, _ = parser.parse_known_args()
return args
@@ -106,7 +109,7 @@ if __name__ == '__main__':
tokenizer = tokenization.FullTokenizer(vocab_file=args.vocab_file, do_lower_case=True)
# When splitting up a long document into chunks, how much stride to take between chunks.
- doc_stride = 128
+ doc_stride = args.doc_stride
# The maximum total input sequence length after WordPiece tokenization.
# Sequences longer than this will be truncated, and sequences shorter
max_seq_length = args.sequence_length
diff --git a/demo/BERT/inference_varseqlen.py b/demo/BERT/inference_varseqlen.py
index a1d61428..a8bf3a8c 100644
--- a/demo/BERT/inference_varseqlen.py
+++ b/demo/BERT/inference_varseqlen.py
@@ -75,6 +75,9 @@ def parse_args():
parser.add_argument('--n-best-size',
help='Total number of n-best predictions to generate in the nbest_predictions.json output file',
default=20, type=int)
+ parser.add_argument('--doc-stride',
+ help='When splitting up a long document into chunks, what stride to take between chunks',
+ default=128, type=int)
args, _ = parser.parse_known_args()
return args
@@ -105,7 +108,7 @@ if __name__ == '__main__':
tokenizer = tokenization.FullTokenizer(vocab_file=args.vocab_file, do_lower_case=True)
# When splitting up a long document into chunks, how much stride to take between chunks.
- doc_stride = 128
+ doc_stride = args.doc_stride
# The maximum total input sequence length after WordPiece tokenization.
# Sequences longer than this will be truncated, and sequences shorter
max_seq_length = args.sequence_length
diff --git a/demo/BERT/perf.py b/demo/BERT/perf.py
index 976afea3..4851de8a 100644
--- a/demo/BERT/perf.py
+++ b/demo/BERT/perf.py
@@ -78,8 +78,9 @@ def main():
bench_times = {}
+ stream = cuda.Stream()
for idx, batch_size in enumerate(sorted(args.batch_size)):
- context.active_optimization_profile = idx
+ context.set_optimization_profile_async(idx, stream.handle)
# Each profile has unique bindings
binding_idx_offset = idx * num_binding_per_profile
@@ -99,7 +100,6 @@ def main():
total_time = 0
start = cuda.Event()
end = cuda.Event()
- stream = cuda.Stream()
# Warmup
for _ in range(args.warm_up_runs):
diff --git a/demo/BERT/scripts/download_model.sh b/demo/BERT/scripts/download_model.sh
index 5339df9f..54ceaf44 100755
--- a/demo/BERT/scripts/download_model.sh
+++ b/demo/BERT/scripts/download_model.sh
@@ -20,17 +20,21 @@ SQUAD='2'
MODEL='large'
SEQ_LEN='128'
FW='tf'
+WTYPE='dense'
+PREC='fp16'
while test $# -gt 0
do
case "$1" in
- -h) echo "Usage: sh download_model.sh [tf|pyt] [base|large] [128|384] [v2|v1_1]"
+ -h) echo "Usage: sh download_model.sh [tf|pyt] [base|large|megatron-large] [128|384] [v2|v1_1] [sparse] [int8-qat]"
exit 0
;;
base) MODEL='base'
;;
large) MODEL='large'
;;
+ megatron-large) MODEL='megatron'
+ ;;
128) SEQ_LEN='128'
;;
384) SEQ_LEN='384'
@@ -43,6 +47,10 @@ do
;;
pyt) FW='pyt'
;;
+ int8-qat) PREC='int8qat'
+ ;;
+ sparse) WTYPE='sparse'
+ ;;
*) echo "Invalid argument $1...exiting"
exit 0
;;
@@ -56,12 +64,14 @@ pushd models/fine-tuned
# Download the BERT fine-tuned model
echo "Downloading BERT-${FW} ${MODEL} checkpoints for sequence length ${SEQ_LEN} and fine-tuned for SQuAD ${SQUAD}."
-
if [ "${FW}" = 'tf' ]; then
CKPT=bert_${FW}_ckpt_${MODEL}_qa_squad${SQUAD}_amp_${SEQ_LEN}
CKPT_VERSION=19.03.1
elif [ "${FW}" = 'pyt' ]; then
- if [ "${MODEL}" != 'large' ] || [ "${SQUAD}" != '11' ]; then
+ if [ "${MODEL}" == 'megatron' ]; then
+ CKPT=bert_${FW}_statedict_megatron_${WTYPE}_${PREC}
+ CKPT_VERSION=21.03.0
+ elif [ "${MODEL}" != 'large' ] || [ "${SQUAD}" != '11' ]; then
echo "ERROR: Only BERT-large checkpoint fine-tuned for SQuAD v1.1 available in the QAT (PyTorch) workflow."
else
CKPT=bert_${FW}_onnx_${MODEL}_qa_squad${SQUAD}_amp_fake_quant
@@ -78,4 +88,5 @@ if [ -n "$CKPT" ]; then
ngc registry model download-version nvidia/${CKPT}:${CKPT_VERSION}
fi
fi
+
popd
diff --git a/demo/BERT/squad/evaluate-v2.0.py b/demo/BERT/squad/evaluate-v2.0.py
new file mode 100644
index 00000000..d034da91
--- /dev/null
+++ b/demo/BERT/squad/evaluate-v2.0.py
@@ -0,0 +1,295 @@
+#!/usr/bin/env python3
+#
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Obtained from https://rajpurkar.github.io/SQuAD-explorer/
+
+"""Official evaluation script for SQuAD version 2.0.
+
+In addition to basic functionality, we also compute additional statistics and
+plot precision-recall curves if an additional na_prob.json file is provided.
+This file is expected to map question ID's to the model's predicted probability
+that a question is unanswerable.
+"""
+import argparse
+import collections
+import json
+import numpy as np
+import os
+import re
+import string
+import sys
+
+OPTS = None
+
+def parse_args():
+ parser = argparse.ArgumentParser('Official evaluation script for SQuAD version 2.0.')
+ parser.add_argument('data_file', metavar='data.json', help='Input data JSON file.')
+ parser.add_argument('pred_file', metavar='pred.json', help='Model predictions.')
+ parser.add_argument('--out-file', '-o', metavar='eval.json',
+ help='Write accuracy metrics to file (default is stdout).')
+ parser.add_argument('--na-prob-file', '-n', metavar='na_prob.json',
+ help='Model estimates of probability of no answer.')
+ parser.add_argument('--na-prob-thresh', '-t', type=float, default=1.0,
+ help='Predict "" if no-answer probability exceeds this (default = 1.0).')
+ parser.add_argument('--out-image-dir', '-p', metavar='out_images', default=None,
+ help='Save precision-recall curves to directory.')
+ parser.add_argument('--verbose', '-v', action='store_true')
+ if len(sys.argv) == 1:
+ parser.print_help()
+ sys.exit(1)
+ return parser.parse_args()
+
+def make_qid_to_has_ans(dataset):
+ qid_to_has_ans = {}
+ for article in dataset:
+ for p in article['paragraphs']:
+ for qa in p['qas']:
+ qid_to_has_ans[qa['id']] = bool(qa['answers'])
+ return qid_to_has_ans
+
+def normalize_answer(s):
+ """Lower text and remove punctuation, articles and extra whitespace."""
+ def remove_articles(text):
+ regex = re.compile(r'\b(a|an|the)\b', re.UNICODE)
+ return re.sub(regex, ' ', text)
+ def white_space_fix(text):
+ return ' '.join(text.split())
+ def remove_punc(text):
+ exclude = set(string.punctuation)
+ return ''.join(ch for ch in text if ch not in exclude)
+ def lower(text):
+ return text.lower()
+ return white_space_fix(remove_articles(remove_punc(lower(s))))
+
+def get_tokens(s):
+ if not s: return []
+ return normalize_answer(s).split()
+
+def compute_exact(a_gold, a_pred):
+ return int(normalize_answer(a_gold) == normalize_answer(a_pred))
+
+def compute_f1(a_gold, a_pred):
+ gold_toks = get_tokens(a_gold)
+ pred_toks = get_tokens(a_pred)
+ common = collections.Counter(gold_toks) & collections.Counter(pred_toks)
+ num_same = sum(common.values())
+ if len(gold_toks) == 0 or len(pred_toks) == 0:
+ # If either is no-answer, then F1 is 1 if they agree, 0 otherwise
+ return int(gold_toks == pred_toks)
+ if num_same == 0:
+ return 0
+ precision = 1.0 * num_same / len(pred_toks)
+ recall = 1.0 * num_same / len(gold_toks)
+ f1 = (2 * precision * recall) / (precision + recall)
+ return f1
+
+def get_raw_scores(dataset, preds):
+ exact_scores = {}
+ f1_scores = {}
+ for article in dataset:
+ for p in article['paragraphs']:
+ for qa in p['qas']:
+ qid = qa['id']
+ gold_answers = [a['text'] for a in qa['answers']
+ if normalize_answer(a['text'])]
+ if not gold_answers:
+ # For unanswerable questions, only correct answer is empty string
+ gold_answers = ['']
+ if qid not in preds:
+ print('Missing prediction for %s' % qid)
+ continue
+ a_pred = preds[qid]
+ # Take max over all gold answers
+ exact_scores[qid] = max(compute_exact(a, a_pred) for a in gold_answers)
+ f1_scores[qid] = max(compute_f1(a, a_pred) for a in gold_answers)
+ return exact_scores, f1_scores
+
+def apply_no_ans_threshold(scores, na_probs, qid_to_has_ans, na_prob_thresh):
+ new_scores = {}
+ for qid, s in scores.items():
+ pred_na = na_probs[qid] > na_prob_thresh
+ if pred_na:
+ new_scores[qid] = float(not qid_to_has_ans[qid])
+ else:
+ new_scores[qid] = s
+ return new_scores
+
+def make_eval_dict(exact_scores, f1_scores, qid_list=None):
+ if not qid_list:
+ total = len(exact_scores)
+ return collections.OrderedDict([
+ ('exact', 100.0 * sum(exact_scores.values()) / total),
+ ('f1', 100.0 * sum(f1_scores.values()) / total),
+ ('total', total),
+ ])
+ else:
+ total = len(qid_list)
+ return collections.OrderedDict([
+ ('exact', 100.0 * sum(exact_scores[k] for k in qid_list) / total),
+ ('f1', 100.0 * sum(f1_scores[k] for k in qid_list) / total),
+ ('total', total),
+ ])
+
+def merge_eval(main_eval, new_eval, prefix):
+ for k in new_eval:
+ main_eval['%s_%s' % (prefix, k)] = new_eval[k]
+
+def plot_pr_curve(precisions, recalls, out_image, title):
+ plt.step(recalls, precisions, color='b', alpha=0.2, where='post')
+ plt.fill_between(recalls, precisions, step='post', alpha=0.2, color='b')
+ plt.xlabel('Recall')
+ plt.ylabel('Precision')
+ plt.xlim([0.0, 1.05])
+ plt.ylim([0.0, 1.05])
+ plt.title(title)
+ plt.savefig(out_image)
+ plt.clf()
+
+def make_precision_recall_eval(scores, na_probs, num_true_pos, qid_to_has_ans,
+ out_image=None, title=None):
+ qid_list = sorted(na_probs, key=lambda k: na_probs[k])
+ true_pos = 0.0
+ cur_p = 1.0
+ cur_r = 0.0
+ precisions = [1.0]
+ recalls = [0.0]
+ avg_prec = 0.0
+ for i, qid in enumerate(qid_list):
+ if qid_to_has_ans[qid]:
+ true_pos += scores[qid]
+ cur_p = true_pos / float(i+1)
+ cur_r = true_pos / float(num_true_pos)
+ if i == len(qid_list) - 1 or na_probs[qid] != na_probs[qid_list[i+1]]:
+ # i.e., if we can put a threshold after this point
+ avg_prec += cur_p * (cur_r - recalls[-1])
+ precisions.append(cur_p)
+ recalls.append(cur_r)
+ if out_image:
+ plot_pr_curve(precisions, recalls, out_image, title)
+ return {'ap': 100.0 * avg_prec}
+
+def run_precision_recall_analysis(main_eval, exact_raw, f1_raw, na_probs,
+ qid_to_has_ans, out_image_dir):
+ if out_image_dir and not os.path.exists(out_image_dir):
+ os.makedirs(out_image_dir)
+ num_true_pos = sum(1 for v in qid_to_has_ans.values() if v)
+ if num_true_pos == 0:
+ return
+ pr_exact = make_precision_recall_eval(
+ exact_raw, na_probs, num_true_pos, qid_to_has_ans,
+ out_image=os.path.join(out_image_dir, 'pr_exact.png'),
+ title='Precision-Recall curve for Exact Match score')
+ pr_f1 = make_precision_recall_eval(
+ f1_raw, na_probs, num_true_pos, qid_to_has_ans,
+ out_image=os.path.join(out_image_dir, 'pr_f1.png'),
+ title='Precision-Recall curve for F1 score')
+ oracle_scores = {k: float(v) for k, v in qid_to_has_ans.items()}
+ pr_oracle = make_precision_recall_eval(
+ oracle_scores, na_probs, num_true_pos, qid_to_has_ans,
+ out_image=os.path.join(out_image_dir, 'pr_oracle.png'),
+ title='Oracle Precision-Recall curve (binary task of HasAns vs. NoAns)')
+ merge_eval(main_eval, pr_exact, 'pr_exact')
+ merge_eval(main_eval, pr_f1, 'pr_f1')
+ merge_eval(main_eval, pr_oracle, 'pr_oracle')
+
+def histogram_na_prob(na_probs, qid_list, image_dir, name):
+ if not qid_list:
+ return
+ x = [na_probs[k] for k in qid_list]
+ weights = np.ones_like(x) / float(len(x))
+ plt.hist(x, weights=weights, bins=20, range=(0.0, 1.0))
+ plt.xlabel('Model probability of no-answer')
+ plt.ylabel('Proportion of dataset')
+ plt.title('Histogram of no-answer probability: %s' % name)
+ plt.savefig(os.path.join(image_dir, 'na_prob_hist_%s.png' % name))
+ plt.clf()
+
+def find_best_thresh(preds, scores, na_probs, qid_to_has_ans):
+ num_no_ans = sum(1 for k in qid_to_has_ans if not qid_to_has_ans[k])
+ cur_score = num_no_ans
+ best_score = cur_score
+ best_thresh = 0.0
+ qid_list = sorted(na_probs, key=lambda k: na_probs[k])
+ for i, qid in enumerate(qid_list):
+ if qid not in scores: continue
+ if qid_to_has_ans[qid]:
+ diff = scores[qid]
+ else:
+ if preds[qid]:
+ diff = -1
+ else:
+ diff = 0
+ cur_score += diff
+ if cur_score > best_score:
+ best_score = cur_score
+ best_thresh = na_probs[qid]
+ return 100.0 * best_score / len(scores), best_thresh
+
+def find_all_best_thresh(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans):
+ best_exact, exact_thresh = find_best_thresh(preds, exact_raw, na_probs, qid_to_has_ans)
+ best_f1, f1_thresh = find_best_thresh(preds, f1_raw, na_probs, qid_to_has_ans)
+ main_eval['best_exact'] = best_exact
+ main_eval['best_exact_thresh'] = exact_thresh
+ main_eval['best_f1'] = best_f1
+ main_eval['best_f1_thresh'] = f1_thresh
+
+def main():
+ with open(OPTS.data_file) as f:
+ dataset_json = json.load(f)
+ dataset = dataset_json['data']
+ with open(OPTS.pred_file) as f:
+ preds = json.load(f)
+ if OPTS.na_prob_file:
+ with open(OPTS.na_prob_file) as f:
+ na_probs = json.load(f)
+ else:
+ na_probs = {k: 0.0 for k in preds}
+ qid_to_has_ans = make_qid_to_has_ans(dataset) # maps qid to True/False
+ has_ans_qids = [k for k, v in qid_to_has_ans.items() if v]
+ no_ans_qids = [k for k, v in qid_to_has_ans.items() if not v]
+ exact_raw, f1_raw = get_raw_scores(dataset, preds)
+ exact_thresh = apply_no_ans_threshold(exact_raw, na_probs, qid_to_has_ans,
+ OPTS.na_prob_thresh)
+ f1_thresh = apply_no_ans_threshold(f1_raw, na_probs, qid_to_has_ans,
+ OPTS.na_prob_thresh)
+ out_eval = make_eval_dict(exact_thresh, f1_thresh)
+ if has_ans_qids:
+ has_ans_eval = make_eval_dict(exact_thresh, f1_thresh, qid_list=has_ans_qids)
+ merge_eval(out_eval, has_ans_eval, 'HasAns')
+ if no_ans_qids:
+ no_ans_eval = make_eval_dict(exact_thresh, f1_thresh, qid_list=no_ans_qids)
+ merge_eval(out_eval, no_ans_eval, 'NoAns')
+ if OPTS.na_prob_file:
+ find_all_best_thresh(out_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans)
+ if OPTS.na_prob_file and OPTS.out_image_dir:
+ run_precision_recall_analysis(out_eval, exact_raw, f1_raw, na_probs,
+ qid_to_has_ans, OPTS.out_image_dir)
+ histogram_na_prob(na_probs, has_ans_qids, OPTS.out_image_dir, 'hasAns')
+ histogram_na_prob(na_probs, no_ans_qids, OPTS.out_image_dir, 'noAns')
+ if OPTS.out_file:
+ with open(OPTS.out_file, 'w') as f:
+ json.dump(out_eval, f)
+ else:
+ print(json.dumps(out_eval, indent=2))
+
+if __name__ == '__main__':
+ OPTS = parse_args()
+ if OPTS.out_image_dir:
+ import matplotlib
+ matplotlib.use('Agg')
+ import matplotlib.pyplot as plt
+ main()
diff --git a/demo/Tacotron2/README.md b/demo/Tacotron2/README.md
index d7f5cbeb..2dd36e15 100644
--- a/demo/Tacotron2/README.md
+++ b/demo/Tacotron2/README.md
@@ -48,33 +48,38 @@ Software version configuration tested for the instructions that follow:
```bash
mkdir -p output
- python3 exports/export_tacotron2_onnx.py --tacotron2 checkpoints/tacotron2_pyt_ckpt_amp_v19.09.0/nvidia_tacotron2pyt_fp16_20190427 -o output/ --fp16
+ python tensorrt/convert_tacotron22onnx.py --tacotron2 ./checkpoints/nvidia_tacotron2pyt_fp16_20190427 -o output/ --fp16
```
- Export WaveGlow to ONNX IR:
+ Convert WaveGlow to ONNX IR:
```bash
- python3 exports/export_waveglow_onnx.py --waveglow checkpoints/waveglow_ckpt_amp_256_v19.10.0/nvidia_waveglow256pyt_fp16 --wn-channels 256 -o output/ --fp16
- ```
+ python tensorrt/convert_waveglow2onnx.py --waveglow ./checkpoints/nvidia_waveglow256pyt_fp16 --config-file config.json --wn-channels 256 -o output/ --fp16
+ ```
After running the above commands, there should be four new ONNX files in `./output/` directory:
- `encoder.onnx`, `decoder_iter.onnx`, `postnet.onnx`, and `waveglow.onnx`.
+ `encoder.onnx`, `decoder_iter.onnx`, `postnet.onnx`, and `waveglow.onnx`. If TensorRT 8.0+ is being used and the `--no-loop` option is not specified, `decoder.onnx` will also be created.
6. Export the ONNX IRs to TensorRT engines with fp16 mode enabled:
- ```bash
- python3 trt/export_onnx2trt.py --encoder output/encoder.onnx --decoder output/decoder_iter.onnx --postnet output/postnet.onnx --waveglow output/waveglow.onnx -o output/ --fp16
+ ```bash
+ python tensorrt/convert_onnx2trt.py --encoder output/encoder.onnx --decoder output/decoder_iter.onnx --postnet output/postnet.onnx --waveglow output/waveglow.onnx -o output/ --fp16
```
After running the command, there should be four new engine files in `./output/` directory:
`encoder_fp16.engine`, `decoder_iter_fp16.engine`, `postnet_fp16.engine`, and `waveglow_fp16.engine`.
+ For TensorRT 8.0+, use `output/decoder.onnx` for the decoder, which will output `decoder_with_outer_loop_fp16.engine`.
+
7. Run TTS inference pipeline with fp16:
+
```bash
- python3 trt/inference_trt.py -i phrases/phrase.txt --encoder output/encoder_fp16.engine --decoder output/decoder_iter_fp16.engine --postnet output/postnet_fp16.engine --waveglow output/waveglow_fp16.engine -o output/ --fp16
+ python tensorrt/inference_trt.py -i phrases/phrase.txt --encoder output/encoder_fp16.engine --decoder output/decoder_iter_fp16.engine --postnet output/postnet_fp16.engine --waveglow output/waveglow_fp16.engine -o output/ --fp16
```
+ For TensorRT 8.0+, use `decoder_with_outer_loop_fp16.engine` for the decoder.
+
## Performance
### Benchmarking
diff --git a/demo/Tacotron2/common/utils.py b/demo/Tacotron2/common/utils.py
index 3ed68cdf..27f8e311 100644
--- a/demo/Tacotron2/common/utils.py
+++ b/demo/Tacotron2/common/utils.py
@@ -19,6 +19,22 @@ from scipy.io.wavfile import read
import torch
import os
+import argparse
+import json
+
+class ParseFromConfigFile(argparse.Action):
+
+ def __init__(self, option_strings, type, dest, help=None, required=False):
+ super(ParseFromConfigFile, self).__init__(option_strings=option_strings, type=type, dest=dest, help=help, required=required)
+
+ def __call__(self, parser, namespace, values, option_string):
+ with open(values, 'r') as f:
+ data = json.load(f)
+
+ for group in data.keys():
+ for k,v in data[group].items():
+ underscore_k = k.replace('-', '_')
+ setattr(namespace, underscore_k, v)
def get_mask_from_lengths(lengths):
max_len = torch.max(lengths).item()
diff --git a/demo/Tacotron2/tensorrt/convert_onnx2trt.py b/demo/Tacotron2/tensorrt/convert_onnx2trt.py
new file mode 100644
index 00000000..f742f4e8
--- /dev/null
+++ b/demo/Tacotron2/tensorrt/convert_onnx2trt.py
@@ -0,0 +1,168 @@
+#
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import argparse
+import onnx
+import pycuda.autoinit
+import pycuda.driver as cuda
+import sys
+import tensorrt as trt
+from os.path import join
+
+from trt_utils import build_engine, parse_dynamic_size
+
+def parse_args(parser):
+ """
+ Parse commandline arguments.
+ """
+ parser.add_argument('-o', '--output', required=True,
+ help='output folder to save audio (file per phrase)')
+ parser.add_argument('--encoder', type=str, default="",
+ help='full path to the Encoder ONNX')
+ parser.add_argument('--decoder', type=str, default="",
+ help='full path to the Decoder or DecoderIter ONNX.')
+ parser.add_argument('--postnet', type=str, default="",
+ help='full path to the Postnet ONNX')
+ parser.add_argument('--waveglow', type=str, default="",
+ help='full path to the WaveGlow ONNX')
+ parser.add_argument('--encoder_out', type=str,
+ help='Filename of the exported encoder engine')
+ parser.add_argument('--decoder_out', type=str,
+ help='Filename of the exported decoder engine')
+ parser.add_argument('--postnet_out', type=str,
+ help='Filename of the exported postnet engine')
+ parser.add_argument('--waveglow_out', type=str,
+ help='Filename of the exported waveglow engine')
+ parser.add_argument('--fp16', action='store_true',
+ help='inference with FP16')
+ parser.add_argument('-bs', '--batch-size', type=str, default="1",
+ help='One or three comma separated integers specifying the batch size. Specify "min,opt,max" for dynamic shape')
+ parser.add_argument('--mel-size', type=str, default="32,768,1664",
+ help='One or three comma separated integers specifying the mels size for waveglow.')
+ parser.add_argument('--z-size', type=str, default="1024,24576,53248",
+ help='One or three comma separated integers specifying the z size for waveglow.')
+ parser.add_argument('--loop', dest='loop', action='store_true',
+ help='Includes the outer decoder loop in the ONNX model. Enabled by default and only supported on TensorRT 8.0 or later.')
+ parser.add_argument('--no-loop', dest='loop', action='store_false',
+ help='Excludes outer decoder loop from decoder ONNX model. Default behavior and necessary for TensorRT 7.2 or earlier.')
+ parser.set_defaults(loop=int(trt.__version__[0]) >= 8)
+
+ return parser
+
+
+def main():
+
+ parser = argparse.ArgumentParser(
+ description='Export from ONNX to TensorRT for Tacotron 2 and WaveGlow')
+ parser = parse_args(parser)
+ args = parser.parse_args()
+
+ precision = "fp16" if args.fp16 else "fp32"
+ encoder_path = join(args.output, args.encoder_out if args.encoder_out else f"encoder_{precision}.engine")
+ decoder_path = join(args.output, args.decoder_out if args.decoder_out else f"decoder_with_outer_loop_{precision}.engine" if args.loop else f"decoder_iter_{precision}.engine")
+ postnet_path = join(args.output, args.postnet_out if args.postnet_out else f"postnet_{precision}.engine")
+ waveglow_path = join(args.output, args.waveglow_out if args.waveglow_out else f"waveglow_{precision}.engine")
+
+ bs_min, bs_opt, bs_max = parse_dynamic_size(args.batch_size)
+ mel_min, mel_opt, mel_max = parse_dynamic_size(args.mel_size)
+ z_min, z_opt, z_max = parse_dynamic_size(args.z_size)
+
+ # Encoder
+ shapes=[{"name": "sequences", "min": (bs_min,4), "opt": (bs_opt,128), "max": (bs_max,256)},
+ {"name": "sequence_lengths", "min": (bs_min,), "opt": (bs_opt,), "max": (bs_max,)}]
+ if args.encoder != "":
+ print("Building Encoder ...")
+ encoder_engine = build_engine(args.encoder, shapes=shapes, fp16=args.fp16)
+ if encoder_engine is not None:
+ with open(encoder_path, 'wb') as f:
+ f.write(encoder_engine.serialize())
+ else:
+ print("Failed to build engine from", args.encoder)
+ sys.exit(1)
+
+ if args.loop:
+ # Decoder
+ shapes=[{"name": "decoder_input_0", "min": (bs_min,80), "opt": (bs_opt,80), "max": (bs_max,80)},
+ {"name": "attention_hidden_0", "min": (bs_min,1024), "opt": (bs_opt,1024), "max": (bs_max,1024)},
+ {"name": "attention_cell_0", "min": (bs_min,1024), "opt": (bs_opt,1024), "max": (bs_max,1024)},
+ {"name": "decoder_hidden_0", "min": (bs_min,1024), "opt": (bs_opt,1024), "max": (bs_max,1024)},
+ {"name": "decoder_cell_0", "min": (bs_min,1024), "opt": (bs_opt,1024), "max": (bs_max,1024)},
+ {"name": "attention_weights_0", "min": (bs_min,4), "opt": (bs_opt,128), "max": (bs_max,256)},
+ {"name": "attention_weights_cum_0", "min": (bs_min,4), "opt": (bs_opt,128), "max": (bs_max,256)},
+ {"name": "attention_context_0", "min": (bs_min,512), "opt": (bs_opt,512), "max": (bs_max,512)},
+ {"name": "memory", "min": (bs_min,4,512), "opt": (bs_opt,128,512), "max": (bs_max,256,512)},
+ {"name": "processed_memory", "min": (bs_min,4,128), "opt": (bs_opt,128,128), "max": (bs_max,256,128)},
+ {"name": "mask", "min": (bs_min,4), "opt": (bs_opt,128), "max": (bs_max,256)}]
+ if args.decoder != "":
+ print("Building Decoder with loop...")
+ decoder_engine = build_engine(args.decoder, shapes=shapes, fp16=args.fp16)
+ if decoder_engine is not None:
+ with open(decoder_path, 'wb') as f:
+ f.write(decoder_engine.serialize())
+ else:
+ print("Failed to build engine from", args.decoder)
+ sys.exit(1)
+ else:
+ # DecoderIter
+ shapes=[{"name": "decoder_input", "min": (bs_min,80), "opt": (bs_opt,80), "max": (bs_max,80)},
+ {"name": "attention_hidden", "min": (bs_min,1024), "opt": (bs_opt,1024), "max": (bs_max,1024)},
+ {"name": "attention_cell", "min": (bs_min,1024), "opt": (bs_opt,1024), "max": (bs_max,1024)},
+ {"name": "decoder_hidden", "min": (bs_min,1024), "opt": (bs_opt,1024), "max": (bs_max,1024)},
+ {"name": "decoder_cell", "min": (bs_min,1024), "opt": (bs_opt,1024), "max": (bs_max,1024)},
+ {"name": "attention_weights", "min": (bs_min,4), "opt": (bs_opt,128), "max": (bs_max,256)},
+ {"name": "attention_weights_cum", "min": (bs_min,4), "opt": (bs_opt,128), "max": (bs_max,256)},
+ {"name": "attention_context", "min": (bs_min,512), "opt": (bs_opt,512), "max": (bs_max,512)},
+ {"name": "memory", "min": (bs_min,4,512), "opt": (bs_opt,128,512), "max": (bs_max,256,512)},
+ {"name": "processed_memory", "min": (bs_min,4,128), "opt": (bs_opt,128,128), "max": (bs_max,256,128)},
+ {"name": "mask", "min": (bs_min,4), "opt": (bs_opt,128), "max": (bs_max,256)}]
+ if args.decoder != "":
+ print("Building Decoder ...")
+ decoder_iter_engine = build_engine(args.decoder, shapes=shapes, fp16=args.fp16)
+ if decoder_iter_engine is not None:
+ with open(decoder_path, 'wb') as f:
+ f.write(decoder_iter_engine.serialize())
+ else:
+ print("Failed to build engine from", args.decoder)
+ sys.exit(1)
+
+ # Postnet
+ shapes=[{"name": "mel_outputs", "min": (bs_min,80,32), "opt": (bs_opt,80,768), "max": (bs_max,80,1664)}]
+ if args.postnet != "":
+ print("Building Postnet ...")
+ postnet_engine = build_engine(args.postnet, shapes=shapes, fp16=args.fp16)
+ if postnet_engine is not None:
+ with open(postnet_path, 'wb') as f:
+ f.write(postnet_engine.serialize())
+ else:
+ print("Failed to build engine from", args.postnet)
+ sys.exit(1)
+
+ # WaveGlow
+ shapes=[{"name": "mel", "min": (bs_min,80,mel_min,1), "opt": (bs_opt,80,mel_opt,1), "max": (bs_max,80,mel_max,1)},
+ {"name": "z", "min": (bs_min,8,z_min,1), "opt": (bs_opt,8,z_opt,1), "max": (bs_max,8,z_max,1)}]
+ if args.waveglow != "":
+ print("Building WaveGlow ...")
+ waveglow_engine = build_engine(args.waveglow, shapes=shapes, fp16=args.fp16)
+ if waveglow_engine is not None:
+ with open(waveglow_path, 'wb') as f:
+ f.write(waveglow_engine.serialize())
+ else:
+ print("Failed to build engine from", args.waveglow)
+ sys.exit(1)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/demo/Tacotron2/exports/export_tacotron2_onnx.py b/demo/Tacotron2/tensorrt/convert_tacotron22onnx.py
similarity index 86%
rename from demo/Tacotron2/exports/export_tacotron2_onnx.py
rename to demo/Tacotron2/tensorrt/convert_tacotron22onnx.py
index 00f0bb21..9ca79392 100644
--- a/demo/Tacotron2/exports/export_tacotron2_onnx.py
+++ b/demo/Tacotron2/tensorrt/convert_tacotron22onnx.py
@@ -18,24 +18,39 @@ import torch
from torch import nn
from torch.nn import functional as F
import argparse
+import tensorrt
import sys
-sys.path.append('./')
+import os
+from pathlib import Path
+sys.path.append(str(Path(__file__).parents[1]))
import models
from inference import checkpoint_from_distributed, unwrap_distributed, load_and_setup_model, prepare_input_sequence
from common.utils import to_gpu, get_mask_from_lengths
+torch.backends.cudnn.enabled = True
def parse_args(parser):
"""
Parse commandline arguments.
"""
- parser.add_argument('--tacotron2', type=str,
+ parser.add_argument('--tacotron2', type=str, required=True,
help='Full path to the Tacotron2 model checkpoint file')
parser.add_argument('-o', '--output', type=str, required=True,
- help='Directory for the exported Tacotron 2 ONNX model')
+ help='Directory for the exported Tacotron2 ONNX models')
+ parser.add_argument('-e', '--encoder', type=str, required=False, default="encoder.onnx",
+ help='Filename for exported encoder ONNX model')
+ parser.add_argument('-d', '--decoder', type=str, required=False, default="decoder_iter.onnx",
+ help='Filename for exported decoder ONNX model')
+ parser.add_argument('-p', '--postnet', type=str, required=False, default="postnet.onnx",
+ help='Filename for exported postnet ONNX model')
parser.add_argument('--fp16', action='store_true',
help='Export with half precision to ONNX')
+ parser.add_argument('--loop', dest='loop', action='store_true',
+ help='Includes the outer decoder loop in the ONNX model. Enabled by default and only supported on TensorRT 8.0 or later.')
+ parser.add_argument('--no-loop', dest='loop', action='store_false',
+ help='Excludes outer decoder loop from decoder ONNX model. Default behavior and necessary for TensorRT 7.2 or earlier.')
+ parser.set_defaults(loop=int(tensorrt.__version__[0]) >= 8)
return parser
@@ -211,7 +226,8 @@ def test_inference(encoder, decoder_iter, postnet):
decoder_iter.eval()
postnet.eval()
- from trt.inference_trt import init_decoder_inputs
+ sys.path.append('./tensorrt')
+ from inference_trt import init_decoder_inputs
texts = ["Hello World, good day."]
sequences, sequence_lengths = prepare_input_sequence(texts)
@@ -284,6 +300,10 @@ def main():
parser = parse_args(parser)
args, _ = parser.parse_known_args()
+ args.encoder = os.path.join(args.output, args.encoder)
+ args.decoder = os.path.join(args.output, args.decoder)
+ args.postnet = os.path.join(args.output, args.postnet)
+
tacotron2 = load_and_setup_model('Tacotron2', parser, args.tacotron2,
fp16_run=args.fp16, cpu_run=False)
@@ -291,7 +311,7 @@ def main():
sequences = torch.randint(low=0, high=148, size=(1,50),
dtype=torch.long).cuda()
- sequence_lengths = torch.IntTensor([sequences.size(1)]).cuda().long()
+ sequence_lengths = torch.IntTensor([sequences.size(1)])
dummy_input = (sequences, sequence_lengths)
encoder = Encoder(tacotron2)
@@ -299,21 +319,23 @@ def main():
with torch.no_grad():
encoder(*dummy_input)
- torch.onnx.export(encoder, dummy_input, args.output+"/"+"encoder.onnx",
+ torch.onnx.export(encoder, dummy_input, args.encoder,
opset_version=opset_version,
do_constant_folding=True,
input_names=["sequences", "sequence_lengths"],
output_names=["memory", "processed_memory", "lens"],
dynamic_axes={"sequences": {0: "batch_size", 1: "text_seq"},
+ "sequence_lengths": {0: "batch_size"},
"memory": {0: "batch_size", 1: "mem_seq"},
- "processed_memory": {0: "batch_size", 1: "mem_seq"}
+ "processed_memory": {0: "batch_size", 1: "mem_seq"},
+ "lens": {0: "batch_size"}
})
decoder_iter = DecoderIter(tacotron2)
memory = torch.randn((1,sequence_lengths[0],512)).cuda() #encoder_outputs
if args.fp16:
memory = memory.half()
- memory_lengths = sequence_lengths
+ memory_lengths = sequence_lengths.cuda()
# initialize decoder states for dummy_input
decoder_input = tacotron2.decoder.get_go_frame(memory)
mask = get_mask_from_lengths(memory_lengths)
@@ -342,7 +364,7 @@ def main():
with torch.no_grad():
decoder_iter(*dummy_input)
- torch.onnx.export(decoder_iter, dummy_input, args.output+"/"+"decoder_iter.onnx",
+ torch.onnx.export(decoder_iter, dummy_input, args.decoder,
opset_version=opset_version,
do_constant_folding=True,
input_names=["decoder_input",
@@ -365,33 +387,25 @@ def main():
"out_attention_weights",
"out_attention_weights_cum",
"out_attention_context"],
- dynamic_axes={"decoder_input" : {0: "batch_size"},
- "attention_hidden" : {0: "batch_size"},
- "attention_cell" : {0: "batch_size"},
- "decoder_hidden" : {0: "batch_size"},
- "decoder_cell" : {0: "batch_size"},
- "attention_weights" : {0: "batch_size", 1: "seq_len"},
+ dynamic_axes={"attention_weights" : {0: "batch_size", 1: "seq_len"},
"attention_weights_cum" : {0: "batch_size", 1: "seq_len"},
- "attention_context" : {0: "batch_size"},
"memory" : {0: "batch_size", 1: "seq_len"},
"processed_memory" : {0: "batch_size", 1: "seq_len"},
"mask" : {0: "batch_size", 1: "seq_len"},
- "decoder_output" : {0: "batch_size"},
- "gate_prediction" : {0: "batch_size"},
- "out_attention_hidden" : {0: "batch_size"},
- "out_attention_cell" : {0: "batch_size"},
- "out_decoder_hidden" : {0: "batch_size"},
- "out_decoder_cell" : {0: "batch_size"},
"out_attention_weights" : {0: "batch_size", 1: "seq_len"},
- "out_attention_weights_cum" : {0: "batch_size", 1: "seq_len"},
- "out_attention_context" : {0: "batch_size"}
+ "out_attention_weights_cum" : {0: "batch_size", 1: "seq_len"}
})
+ if args.loop:
+ from generate_decoder import insert_decoder_loop
+ decoder_dir = os.path.dirname(os.path.abspath(args.decoder))
+ insert_decoder_loop(args.decoder, decoder_dir, os.path.basename(args.decoder).replace("_iter", ""), args.fp16)
+
postnet = Postnet(tacotron2)
dummy_input = torch.randn((1,80,620)).cuda()
if args.fp16:
dummy_input = dummy_input.half()
- torch.onnx.export(postnet, dummy_input, args.output+"/"+"postnet.onnx",
+ torch.onnx.export(postnet, dummy_input, args.postnet,
opset_version=opset_version,
do_constant_folding=True,
input_names=["mel_outputs"],
@@ -399,8 +413,5 @@ def main():
dynamic_axes={"mel_outputs": {0: "batch_size", 2: "mel_seq"},
"mel_outputs_postnet": {0: "batch_size", 2: "mel_seq"}})
- mel = test_inference(encoder, decoder_iter, postnet)
- torch.save(mel, "mel.pt")
-
if __name__ == '__main__':
main()
diff --git a/demo/Tacotron2/exports/export_waveglow_onnx.py b/demo/Tacotron2/tensorrt/convert_waveglow2onnx.py
similarity index 68%
rename from demo/Tacotron2/exports/export_waveglow_onnx.py
rename to demo/Tacotron2/tensorrt/convert_waveglow2onnx.py
index 3433514c..91d84f5b 100644
--- a/demo/Tacotron2/exports/export_waveglow_onnx.py
+++ b/demo/Tacotron2/tensorrt/convert_waveglow2onnx.py
@@ -14,30 +14,15 @@
# limitations under the License.
#
-import types
import torch
import argparse
-
+import os
import sys
-sys.path.append('./')
-
-from inference import checkpoint_from_distributed, unwrap_distributed, load_and_setup_model
-
-def parse_args(parser):
- """
- Parse commandline arguments.
- """
- parser.add_argument('--waveglow', type=str, required=True,
- help='Full path to the WaveGlow model checkpoint file')
- parser.add_argument('-o', '--output', type=str, required=True,
- help='Directory for the exported WaveGlow ONNX model')
- parser.add_argument('--fp16', action='store_true',
- help='Inference with Automatic Mixed Precision')
- parser.add_argument('-s', '--sigma-infer', default=0.6, type=float,
- help='Standard deviation of the Gaussian distribution')
-
- return parser
+from pathlib import Path
+sys.path.append(str(Path(__file__).parents[1]))
+from common.utils import ParseFromConfigFile
+from inference import load_and_setup_model
def convert_convinv_1d_to_2d(convinv):
"""
@@ -105,52 +90,22 @@ def convert_1d_to_2d_(glow):
glow.cuda()
+def parse_args(parser):
+ """
+ Parse commandline arguments.
+ """
+ parser.add_argument('--waveglow', type=str, required=True,
+ help='full path to the WaveGlow model checkpoint file')
+ parser.add_argument('-o', '--output', type=str, required=True,
+ help='Directory or file name for the exported WaveGlow ONNX model')
+ parser.add_argument('--fp16', action='store_true',
+ help='inference with AMP')
+ parser.add_argument('-s', '--sigma-infer', default=0.6, type=float)
-def infer_onnx(self, spect, z, sigma=0.9):
+ parser.add_argument('--config-file', action=ParseFromConfigFile,
+ type=str, help='Path to configuration file')
- spect = self.upsample(spect)
- # trim conv artifacts. maybe pad spec to kernel multiple
- time_cutoff = self.upsample.kernel_size[0] - self.upsample.stride[0]
- spect = spect[:, :, :-time_cutoff]
-
- length_spect_group = spect.size(2)//8
- mel_dim = 80
- batch_size = spect.size(0)
-
- spect = torch.squeeze(spect, 3)
- spect = spect.view((batch_size, mel_dim, length_spect_group, self.n_group))
- spect = spect.permute(0, 2, 1, 3)
- spect = spect.contiguous()
- spect = spect.view((batch_size, length_spect_group, self.n_group*mel_dim))
- spect = spect.permute(0, 2, 1)
- spect = torch.unsqueeze(spect, 3)
- spect = spect.contiguous()
-
- audio = z[:, :self.n_remaining_channels, :, :]
- z = z[:, self.n_remaining_channels:self.n_group, :, :]
- audio = sigma*audio
-
- for k in reversed(range(self.n_flows)):
- n_half = int(audio.size(1) / 2)
- audio_0 = audio[:, :n_half, :, :]
- audio_1 = audio[:, n_half:(n_half+n_half), :, :]
-
- output = self.WN[k]((audio_0, spect))
- s = output[:, n_half:(n_half+n_half), :, :]
- b = output[:, :n_half, :, :]
- audio_1 = (audio_1 - b) / torch.exp(s)
- audio = torch.cat([audio_0, audio_1], 1)
-
- audio = self.convinv[k](audio)
-
- if k % self.n_early_every == 0 and k > 0:
- audio = torch.cat((z[:, :self.n_early_size, :, :], audio), 1)
- z = z[:, self.n_early_size:self.n_group, :, :]
-
- audio = torch.squeeze(audio, 3)
- audio = audio.permute(0,2,1).contiguous().view(batch_size, (length_spect_group * self.n_group))
-
- return audio
+ return parser
def export_onnx(parser, args):
@@ -173,18 +128,23 @@ def export_onnx(parser, args):
# run inference to force calculation of inverses
waveglow.infer(mel, sigma=args.sigma_infer)
- # export to ONNX
convert_1d_to_2d_(waveglow)
+ mel = mel.unsqueeze(3)
+
+ # export to ONNX
if args.fp16:
waveglow = waveglow.half()
- fType = types.MethodType
- waveglow.forward = fType(infer_onnx, waveglow)
+ waveglow.forward = waveglow.infer_onnx
- mel = mel.unsqueeze(3)
- opset_version = 10
+ opset_version = 11
- torch.onnx.export(waveglow, (mel, z), args.output+"/"+"waveglow.onnx",
+ if os.path.isdir(args.output):
+ output_path = os.path.join(args.output, "waveglow.onnx")
+ else:
+ output_path = args.output
+
+ torch.onnx.export(waveglow, (mel, z), output_path,
opset_version=opset_version,
do_constant_folding=True,
input_names=["mel", "z"],
@@ -195,7 +155,6 @@ def export_onnx(parser, args):
def main():
-
parser = argparse.ArgumentParser(
description='PyTorch Tacotron 2 Inference')
parser = parse_args(parser)
diff --git a/demo/Tacotron2/tensorrt/generate_decoder.py b/demo/Tacotron2/tensorrt/generate_decoder.py
new file mode 100644
index 00000000..7e7a25b0
--- /dev/null
+++ b/demo/Tacotron2/tensorrt/generate_decoder.py
@@ -0,0 +1,211 @@
+#
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import onnx_graphsurgeon as gs
+import onnx
+import sys
+import os
+import numpy as np
+import argparse
+
+def insert_decoder_loop(decoder_iter_onnx_path, output_dir, decoder_out_name, fp16):
+ float_prec = np.float16 if fp16 else np.float32
+
+ # Modify loop body so that it has 2+N inputs: (iteration_num, condition, loop carried dependencies...)
+ # and 1+N+K outputs: (condition, loop carried dependencies..., scan_outputs...)
+
+ # In this case, the loop carried dependencies include the following IN ORDER
+ # - decoder_output/decoder_input
+ # - attention_hidden
+ # - attention_cell
+ # - decoder_hidden
+ # - decoder_cell
+ # - attention_weights
+ # - attention_weights_cum
+ # - attention_context
+ # - not_finished (bool tensor, initialized to all True)
+ # - mel_lengths
+
+ # The following are NOT loop carried dependencies (they remain constant through the loop), and must be moved to be inputs outside of the loop body
+ # - memory
+ # - processed_memory
+ # - mask
+
+ # The scan outputs are
+ # - mel_outputs (which scans across decoder_output)
+ # - gate_outputs (scans across gate_prediction)
+ # - alignments (scans across attention_weights)
+
+
+ loop_body = gs.import_onnx(onnx.load(decoder_iter_onnx_path))
+ loop_tensors = loop_body.tensors()
+
+ iteration_num = gs.Variable("iteration_num", dtype=np.int64, shape=())
+ cond_in = gs.Variable("cond_in", dtype=bool, shape=())
+ cond_out = gs.Variable("cond_out", dtype=bool, shape=())
+ not_finished_in = gs.Variable("not_finished_in", shape=('batch_size', 1), dtype=bool)
+ not_finished_out = gs.Variable("not_finished_out", shape=('batch_size', 1), dtype=bool)
+ mel_lengths_in = gs.Variable("mel_lengths_in", shape=('batch_size', 1), dtype=np.int32)
+ mel_lengths_out = gs.Variable("mel_lengths_out", shape=('batch_size', 1), dtype=np.int32)
+
+
+ # Set loop body inputs in the correct order
+ loop_body.inputs = [iteration_num, cond_in, loop_tensors["decoder_input"], loop_tensors["attention_hidden"], loop_tensors["attention_cell"], loop_tensors["decoder_hidden"], loop_tensors["decoder_cell"], loop_tensors["attention_weights"], loop_tensors["attention_weights_cum"], loop_tensors["attention_context"], not_finished_in, mel_lengths_in]
+
+ # Set loop body outputs in the correct order
+ loop_body.outputs = [cond_out, loop_tensors["decoder_output"], loop_tensors["out_attention_hidden"], loop_tensors["out_attention_cell"], loop_tensors["out_decoder_hidden"], loop_tensors["out_decoder_cell"], loop_tensors["out_attention_weights"], loop_tensors["out_attention_weights_cum"], loop_tensors["out_attention_context"], not_finished_out, mel_lengths_out, loop_tensors["decoder_output"], loop_tensors["gate_prediction"], loop_tensors["out_attention_weights"]]
+
+ # The loop stop condition is given by the following lines in PyTorch
+ # dec = torch.le(torch.sigmoid(decoder_outputs[8]), gate_threshold).to(torch.int32).squeeze(1)
+ # not_finished = not_finished*dec
+ # if torch.sum(not_finished) == 0:
+ # break
+
+ # To compute cond_out, we can essentially follow the same steps. Using Less instead of Greater+Not for now
+
+ gate_threshold = gs.Constant("gate_threshold", np.array([0.5], dtype=float_prec))
+ gate_sigmoid = gs.Variable("gate_sigmoid", dtype=float_prec, shape=())
+ sigmoid = loop_body.nodes.append(gs.Node(op="Sigmoid", inputs=[loop_tensors["gate_prediction"]], outputs=[gate_sigmoid]))
+
+ leq_output = gs.Variable("leq_output", dtype=bool)
+ leq = loop_body.nodes.append(gs.Node(op="Less", inputs=[gate_sigmoid, gate_threshold], outputs=[leq_output]))
+
+ loop_body.nodes.append(gs.Node(op="And", inputs=[not_finished_in, leq_output], outputs=[not_finished_out]))
+
+ cast_output = gs.Variable("cast_output", dtype=np.int32)
+ loop_body.nodes.append(gs.Node(op="Cast", inputs=[not_finished_out], outputs=[cast_output], attrs={"to": 6})) # int32
+
+ reduce_output = gs.Variable("reduce_output", dtype=np.int32)
+ loop_body.nodes.append( gs.Node(op="ReduceSum", inputs=[cast_output], outputs=[reduce_output], attrs={"axes": [0], "keepdims": 0}))
+
+ unsqueezed_cond_out = gs.Variable("unsqueezed_cond_out", dtype=bool)
+ loop_body.nodes.append(gs.Node(op="Equal", inputs=[reduce_output, gs.Constant("zero", np.array(0, dtype=np.int32))], outputs=[unsqueezed_cond_out]))
+
+ squeezed_cond_out = gs.Variable("squeezed_cond_out", dtype=bool)
+ loop_body.nodes.append(gs.Node(op="Squeeze", inputs=[unsqueezed_cond_out], outputs=[squeezed_cond_out], attrs={"axes": [0]}))
+
+ loop_body.nodes.append(gs.Node(op="Not", inputs=[squeezed_cond_out], outputs=[cond_out]))
+
+ # Compute mel_lengths
+ # from PyTorch: mel_lengths += not_finished
+
+ loop_body.nodes.append(gs.Node(op="Add", inputs=[mel_lengths_in, cast_output], outputs=[mel_lengths_out]))
+
+ memory = gs.Variable("memory", dtype=float_prec, shape=('batch_size', 'seq_len', 512))
+ processed_memory = gs.Variable("processed_memory", dtype=float_prec, shape=('batch_size', 'seq_len', 128))
+ mask = gs.Variable("mask", dtype=bool, shape=('batch_size', 'seq_len'))
+
+ loop_body.toposort()
+ onnx.save(gs.export_onnx(loop_body), os.path.join(output_dir, "loop_body_{prec}.onnx".format(prec="fp16" if float_prec == np.float16 else "fp32")))
+
+ # Create outer graph
+
+ # Inputs to outer graph are the following (suffixed with _0 to signify initial states)
+ # - decoder_input_0
+ # - attention_hidden_0
+ # - attention_cell_0
+ # - decoder_hidden_0
+ # - decoder_cell_0
+ # - attention_weights_0
+ # - attention_weights_cum_0
+ # - attention_context_0
+ # - memory
+ # - processed_memory
+ # - mask
+
+ # Outputs are the following
+ # - mel_outputs
+ # - mel_lengths
+
+ # Note: alignments and gate_outputs are scan outputs, but don't seem to be used later in the PyTorch implementation. For now, we will make them intermediate tensors that are not outputted
+
+ graph = gs.Graph()
+
+ decoder_input_0 = gs.Variable("decoder_input_0", dtype=float_prec, shape=('batch_size', 80))
+ attention_hidden_0 = gs.Variable("attention_hidden_0", dtype=float_prec, shape=('batch_size', 1024))
+ attention_cell_0 = gs.Variable("attention_cell_0", dtype=float_prec, shape=('batch_size', 1024))
+ decoder_hidden_0 = gs.Variable("decoder_hidden_0", dtype=float_prec, shape=('batch_size', 1024))
+ decoder_cell_0 = gs.Variable("decoder_cell_0", dtype=float_prec, shape=('batch_size', 1024))
+ attention_weights_0 = gs.Variable("attention_weights_0", dtype=float_prec, shape=('batch_size', 'seq_len'))
+ attention_weights_cum_0 = gs.Variable("attention_weights_cum_0", dtype=float_prec, shape=('batch_size', 'seq_len'))
+ attention_context_0 = gs.Variable("attention_context_0", dtype=float_prec, shape=('batch_size', 512))
+ not_finished_0 = gs.Variable("not_finished_0", dtype=bool)
+ mel_lengths_0 = gs.Variable("mel_lengths_0", dtype=np.int32)
+
+ # For not_finished, we need to generate a tensor of shape (batch_size) that is all 1s
+ # We can use the ONNX ConstantOfShape op to do this
+ not_finished_shape = gs.Variable("not_finished_shape", dtype=np.int64)
+ reduced = gs.Variable("reduced", dtype=float_prec)
+ graph.nodes.append(gs.Node(op="ReduceSum", inputs=[decoder_input_0], outputs=[reduced], attrs={"axes":[1], "keepdims": 1}))
+ graph.nodes.append(gs.Node(op="Shape", inputs=[reduced], outputs=[not_finished_shape]))
+ before_cast = gs.Variable("before_cast", dtype=np.int32)
+ graph.nodes.append(gs.Node(op="ConstantOfShape", inputs=[not_finished_shape], outputs=[before_cast], attrs={"value":gs.Constant("one", np.array([1], dtype=np.int32))}))
+ graph.nodes.append(gs.Node(op="Cast", inputs=[before_cast], outputs=[not_finished_0], attrs={"to": 9}))
+
+ # Same thing for mel_lengths, but we need all 0s
+ graph.nodes.append(gs.Node(op="ConstantOfShape", inputs=[not_finished_shape], outputs=[mel_lengths_0], attrs={"value":gs.Constant("zero", np.array([0], dtype=np.int32))}))
+
+ # Loop carried dependecies at the end of the loop
+ decoder_input_t = gs.Variable("decoder_input_t", dtype=float_prec, shape=('batch_size', 80))
+ attention_hidden_t = gs.Variable("attention_hidden_t", dtype=float_prec, shape=('batch_size', 1024))
+ attention_cell_t = gs.Variable("attention_cell_t", dtype=float_prec, shape=('batch_size', 1024))
+ decoder_hidden_t = gs.Variable("decoder_hidden_t", dtype=float_prec, shape=('batch_size', 1024))
+ decoder_cell_t = gs.Variable("decoder_cell_t", dtype=float_prec, shape=('batch_size', 1024))
+ attention_weights_t = gs.Variable("attention_weights_t", dtype=float_prec, shape=('batch_size', 'seq_len'))
+ attention_weights_cum_t = gs.Variable("attention_weights_cum_t", dtype=float_prec, shape=('batch_size', 'seq_len'))
+ attention_context_t = gs.Variable("attention_context_t", dtype=float_prec, shape=('batch_size', 512))
+ not_finished_t = gs.Variable("not_finished_t", dtype=bool)
+ mel_lengths_t = gs.Variable("mel_lengths_t", dtype=np.int32, shape=('batch_size', 1))
+
+ # Scan outputs
+ mel_outputs_raw = gs.Variable("mel_outputs_raw", dtype=float_prec, shape=(-1, 'batch_size', 80))
+ gate_outputs = gs.Variable("gate_outputs", dtype=float_prec, shape=(-1, 'batch_size', 1))
+ alignments = gs.Variable("alignments", dtype=float_prec, shape=(-1, 1, 'seq_len'))
+
+ mel_outputs = gs.Variable("mel_outputs", dtype=float_prec, shape=('batch_size', 80, -1))
+
+ graph.inputs = [decoder_input_0, attention_hidden_0, attention_cell_0, decoder_hidden_0, decoder_cell_0, attention_weights_0, attention_weights_cum_0, attention_context_0, memory, processed_memory, mask]
+ graph.outputs = [mel_outputs, mel_lengths_t]
+
+ trip_count = gs.Constant("trip_count", np.array(0, dtype=np.int64)) # In ONNX, this is an optional parameter, but I don't think ONNX-GS supports optional inputs. To fix this, after we export the ONNX ModelProto from GS, we replace this input with ""
+ initial_cond = gs.Constant("initial_cond", np.array(True, dtype=bool))
+ loop_inputs = [trip_count, initial_cond, decoder_input_0, attention_hidden_0, attention_cell_0, decoder_hidden_0, decoder_cell_0, attention_weights_0, attention_weights_cum_0, attention_context_0, not_finished_0, mel_lengths_0]
+ loop_outputs = [decoder_input_t, attention_hidden_t, attention_cell_t, decoder_hidden_t, decoder_cell_t, attention_weights_t, attention_weights_cum_t, attention_context_t, not_finished_t, mel_lengths_t, mel_outputs_raw, gate_outputs, alignments]
+ decoder_loop = gs.Node(op="Loop", name="decoder_loop", inputs=loop_inputs, outputs=loop_outputs, attrs={"body": loop_body})
+ graph.nodes.append(decoder_loop)
+
+ graph.nodes.append(gs.Node(op="Transpose", inputs=[mel_outputs_raw], outputs=[mel_outputs], attrs={"perm": [1, 2, 0]})) # Output needs to have loop dimension as inner-most dim
+
+ graph.toposort()
+ exported_graph = gs.export_onnx(graph)
+ [x for x in exported_graph.graph.node if x.name == "decoder_loop"][0].input[0] = "" # Remove trip count input
+
+ onnx.save(exported_graph, os.path.join(output_dir, decoder_out_name))
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument('model_path', type=str,
+ help='path to original decoder_iter ONNX model')
+ parser.add_argument('-o', '--output_dir', type=str, default='.', help='Output directory')
+ parser.add_argument('--decoder_out', type=str, help='Filename of the exported decoder with outer loop')
+ parser.add_argument('--fp16', action='store_true')
+
+ args = parser.parse_args()
+
+ if args.decoder_out == None:
+ args.decoder_out = "decoder_with_outer_loop_{}.onnx".format("fp16" if args.fp16 else "fp32")
+
+ insert_decoder_loop(args.model_path, args.output_dir, args.decoder_out, args.fp16)
\ No newline at end of file
diff --git a/demo/Tacotron2/trt/inference_trt.py b/demo/Tacotron2/tensorrt/inference_trt.py
similarity index 59%
rename from demo/Tacotron2/trt/inference_trt.py
rename to demo/Tacotron2/tensorrt/inference_trt.py
index c05dbfcc..04200664 100644
--- a/demo/Tacotron2/trt/inference_trt.py
+++ b/demo/Tacotron2/tensorrt/inference_trt.py
@@ -20,16 +20,18 @@ from scipy.io.wavfile import write
import time
import torch
import argparse
-import sys
+import os.path as path
-sys.path.append('./')
+import sys
+from pathlib import Path
+sys.path.append(str(Path(__file__).parents[1]))
from common.utils import to_gpu, get_mask_from_lengths
from tacotron2.text import text_to_sequence
from inference import MeasureTime, prepare_input_sequence, load_and_setup_model
import dllogger as DLLogger
from dllogger import StdOutBackend, JSONStreamBackend, Verbosity
-from trt.trt_utils import load_engine, run_trt_engine
+from trt_utils import load_engine, run_trt_engine
from waveglow.denoiser import Denoiser
@@ -38,30 +40,37 @@ def parse_args(parser):
Parse commandline arguments.
"""
parser.add_argument('-i', '--input', type=str, required=True,
- help='Full path to the input text (phareses separated by new line)')
+ help='full path to the input text (phareses separated by new line)')
parser.add_argument('-o', '--output', required=True,
- help='Output folder to save audio (file per phrase)')
+ help='output folder to save audio (file per phrase)')
parser.add_argument('--encoder', type=str, required=True,
- help='Full path to the Encoder engine')
+ help='full path to the Encoder engine')
parser.add_argument('--decoder', type=str, required=True,
- help='Full path to the DecoderIter engine')
+ help='full path to the DecoderIter engine')
parser.add_argument('--postnet', type=str, required=True,
- help='Full path to the Postnet engine')
+ help='full path to the Postnet engine')
parser.add_argument('--waveglow', type=str, required=True,
- help='Full path to the WaveGlow engine')
+ help='full path to the WaveGlow engine')
parser.add_argument('--waveglow-ckpt', type=str, default="",
- help='Full path to the WaveGlow model checkpoint file')
+ help='full path to the WaveGlow model checkpoint file')
parser.add_argument('--log-file', type=str, default='nvlog.json',
help='Filename for logging')
- parser.add_argument('-d', '--denoising-strength', default=0.01, type=float,
- help='Denoising strength for removing model bias')
+ parser.add_argument('-d', '--denoising-strength', default=0.01, type=float)
parser.add_argument('-sr', '--sampling-rate', default=22050, type=int,
help='Sampling rate')
parser.add_argument('--stft-hop-length', type=int, default=256,
help='STFT hop length for estimating audio length from mel size')
parser.add_argument('--fp16', action='store_true',
- help='Inference with FP16 precision')
-
+ help='inference with FP16')
+ parser.add_argument('--loop', dest='loop', action='store_true',
+ help='Includes the outer decoder loop in the ONNX model. Enabled by default and only supported on TensorRT 8.0 or later.')
+ parser.add_argument('--no-loop', dest='loop', action='store_false',
+ help='Excludes outer decoder loop from decoder ONNX model. Default behavior and necessary for TensorRT 7.2 or earlier.')
+ parser.set_defaults(loop=int(trt.__version__[0]) >= 8)
+ parser.add_argument('--waveglow-onnxruntime', action='store_true',
+ help='Specify this option to use ONNX runtime instead of TRT for running Waveglow')
+ parser.add_argument('--decoder-onnxruntime', action='store_true',
+ help='Specify this option to use ONNX runtime instead of TRT for running the TT2 Decoder with loop. When using this option, pass the decoder ONNX model to the --decoder argument')
return parser
@@ -174,16 +183,19 @@ def swap_inputs_outputs(decoder_inputs, decoder_outputs):
def infer_tacotron2_trt(encoder, decoder_iter, postnet,
encoder_context, decoder_context, postnet_context,
- sequences, sequence_lengths, measurements, fp16):
+ sequences, sequence_lengths, measurements, fp16, loop):
- memory = torch.zeros((len(sequence_lengths), sequence_lengths[0], 512)).cuda()
+ batch_size = len(sequence_lengths)
+ max_sequence_len = sequence_lengths[0]
+ memory = torch.zeros((batch_size, max_sequence_len, 512)).cuda()
if fp16:
memory = memory.half()
device = memory.device
dtype = memory.dtype
- processed_memory = torch.zeros((len(sequence_lengths),sequence_lengths[0],128), device=device, dtype=dtype)
+ processed_memory = torch.zeros((batch_size, max_sequence_len, 128), device=device, dtype=dtype)
lens = torch.zeros_like(sequence_lengths)
+ print(f"batch_size: {batch_size}, max sequence length: {max_sequence_len}")
encoder_tensors = {
"inputs" :
@@ -195,50 +207,101 @@ def infer_tacotron2_trt(encoder, decoder_iter, postnet,
print("Running Tacotron2 Encoder")
with MeasureTime(measurements, "tacotron2_encoder_time"):
run_trt_engine(encoder_context, encoder, encoder_tensors)
-
+ max_decoder_steps = 1024
device = memory.device
mel_lengths = torch.zeros([memory.size(0)], dtype=torch.int32, device = device)
not_finished = torch.ones([memory.size(0)], dtype=torch.int32, device = device)
- mel_outputs, gate_outputs, alignments = (torch.zeros(1, device = device), torch.zeros(1, device = device), torch.zeros(1, device = device))
+ mel_outputs = torch.ones((batch_size, 80, max_decoder_steps), device = device, dtype=dtype).cuda()
gate_threshold = 0.5
- max_decoder_steps = 1664
first_iter = True
decoder_inputs = init_decoder_inputs(memory, processed_memory, sequence_lengths)
decoder_outputs = init_decoder_outputs(memory, sequence_lengths)
- print("Running Tacotron2 Decoder")
- measurements_decoder = {}
- while True:
- decoder_tensors = init_decoder_tensors(decoder_inputs, decoder_outputs)
- with MeasureTime(measurements_decoder, "step"):
- run_trt_engine(decoder_context, decoder_iter, decoder_tensors)
+ if loop:
+ if decoder_context is None:
+ print("Running Tacotron2 Decoder with loop with ONNX-RT")
+ decoder_inputs_onnxrt = [x.cpu().numpy().copy() for x in decoder_inputs]
+ import onnx
+ import onnxruntime
+ sess = onnxruntime.InferenceSession(decoder_iter)
- if first_iter:
- mel_outputs = torch.unsqueeze(decoder_outputs[7], 2)
- gate_outputs = torch.unsqueeze(decoder_outputs[8], 2)
- alignments = torch.unsqueeze(decoder_outputs[4], 2)
- measurements['tacotron2_decoder_time'] = measurements_decoder['step']
- first_iter = False
- else:
- mel_outputs = torch.cat((mel_outputs, torch.unsqueeze(decoder_outputs[7], 2)), 2)
- gate_outputs = torch.cat((gate_outputs, torch.unsqueeze(decoder_outputs[8], 2)), 2)
- alignments = torch.cat((alignments, torch.unsqueeze(decoder_outputs[4], 2)), 2)
- measurements['tacotron2_decoder_time'] += measurements_decoder['step']
+ with MeasureTime(measurements, "tacotron2_decoder_time"):
+ result = sess.run(["mel_outputs", "mel_lengths_t"], {
+ 'decoder_input_0': decoder_inputs_onnxrt[0],
+ 'attention_hidden_0': decoder_inputs_onnxrt[1],
+ 'attention_cell_0': decoder_inputs_onnxrt[2],
+ 'decoder_hidden_0': decoder_inputs_onnxrt[3],
+ 'decoder_cell_0': decoder_inputs_onnxrt[4],
+ 'attention_weights_0': decoder_inputs_onnxrt[5],
+ 'attention_weights_cum_0': decoder_inputs_onnxrt[6],
+ 'attention_context_0': decoder_inputs_onnxrt[7],
+ 'memory': decoder_inputs_onnxrt[8],
+ 'processed_memory': decoder_inputs_onnxrt[9],
+ 'mask': decoder_inputs_onnxrt[10]
+ })
- dec = torch.le(torch.sigmoid(decoder_outputs[8]), gate_threshold).to(torch.int32).squeeze(1)
- not_finished = not_finished*dec
- mel_lengths += not_finished
+ mel_outputs = torch.tensor(result[0], device=device)
+ mel_lengths = torch.tensor(result[1], device=device)
+ else:
+ print("Running Tacotron2 Decoder with loop")
+ decoder_tensors = {
+ "inputs" :
+ {
+ 'decoder_input_0': decoder_inputs[0],
+ 'attention_hidden_0': decoder_inputs[1],
+ 'attention_cell_0': decoder_inputs[2],
+ 'decoder_hidden_0': decoder_inputs[3],
+ 'decoder_cell_0': decoder_inputs[4],
+ 'attention_weights_0': decoder_inputs[5],
+ 'attention_weights_cum_0': decoder_inputs[6],
+ 'attention_context_0': decoder_inputs[7],
+ 'memory': decoder_inputs[8],
+ 'processed_memory': decoder_inputs[9],
+ 'mask': decoder_inputs[10]
+ },
+ "outputs" :
+ {'mel_outputs': mel_outputs, 'mel_lengths_t': mel_lengths}
+ }
- if torch.sum(not_finished) == 0:
- print("Stopping after",mel_outputs.size(2),"decoder steps")
- break
- if mel_outputs.size(2) == max_decoder_steps:
- print("Warning! Reached max decoder steps")
- break
+ with MeasureTime(measurements, "tacotron2_decoder_time"):
+ run_trt_engine(decoder_context, decoder_iter, decoder_tensors)
+ mel_outputs = mel_outputs[:,:,:torch.max(mel_lengths)]
- decoder_inputs, decoder_outputs = swap_inputs_outputs(decoder_inputs, decoder_outputs)
+ else:
+ print("Running Tacotron2 Decoder")
+ measurements_decoder = {}
+ while True:
+ decoder_tensors = init_decoder_tensors(decoder_inputs, decoder_outputs)
+ with MeasureTime(measurements_decoder, "step"):
+ run_trt_engine(decoder_context, decoder_iter, decoder_tensors)
+ if first_iter:
+ mel_outputs = torch.unsqueeze(decoder_outputs[7], 2)
+ gate_outputs = torch.unsqueeze(decoder_outputs[8], 2)
+ alignments = torch.unsqueeze(decoder_outputs[4], 2)
+ measurements['tacotron2_decoder_time'] = measurements_decoder['step']
+ first_iter = False
+ else:
+ mel_outputs = torch.cat((mel_outputs, torch.unsqueeze(decoder_outputs[7], 2)), 2)
+ gate_outputs = torch.cat((gate_outputs, torch.unsqueeze(decoder_outputs[8], 2)), 2)
+ alignments = torch.cat((alignments, torch.unsqueeze(decoder_outputs[4], 2)), 2)
+ measurements['tacotron2_decoder_time'] += measurements_decoder['step']
+
+ dec = torch.le(torch.sigmoid(decoder_outputs[8]), gate_threshold).to(torch.int32).squeeze(1)
+ not_finished = not_finished*dec
+ mel_lengths += not_finished
+
+ if torch.sum(not_finished) == 0:
+ print("Stopping after",mel_outputs.size(2),"decoder steps")
+ break
+ if mel_outputs.size(2) == max_decoder_steps:
+ print("Warning! Reached max decoder steps")
+ break
+
+ decoder_inputs, decoder_outputs = swap_inputs_outputs(decoder_inputs, decoder_outputs)
+
+ mel_outputs = mel_outputs.clone().detach()
mel_outputs_postnet = torch.zeros_like(mel_outputs, device=device, dtype=dtype)
postnet_tensors = {
@@ -255,6 +318,7 @@ def infer_tacotron2_trt(encoder, decoder_iter, postnet,
return mel_outputs_postnet, mel_lengths
+
def infer_waveglow_trt(waveglow, waveglow_context, mel, measurements, fp16):
mel_size = mel.size(2)
@@ -264,9 +328,10 @@ def infer_waveglow_trt(waveglow, waveglow_context, mel, measurements, fp16):
z_size = mel_size*stride
z_size = z_size//n_group
z = torch.randn(batch_size, n_group, z_size).cuda()
+ audios = torch.zeros(batch_size, mel_size*stride).cuda()
+
mel = mel.unsqueeze(3)
z = z.unsqueeze(3)
- audios = torch.zeros(batch_size, mel_size*stride).cuda()
if fp16:
z = z.half()
@@ -274,17 +339,49 @@ def infer_waveglow_trt(waveglow, waveglow_context, mel, measurements, fp16):
audios = audios.half()
waveglow_tensors = {
- "inputs" :
- {'mel': mel, 'z': z},
- "outputs" :
- {'audio': audios}
+ "inputs" : {'mel': mel, 'z': z},
+ "outputs" : {'audio': audios}
}
- print("Running WaveGlow")
+
+ print("Running WaveGlow with TensorRT")
with MeasureTime(measurements, "waveglow_time"):
run_trt_engine(waveglow_context, waveglow, waveglow_tensors)
return audios
+def infer_waveglow_onnx(waveglow_path, mel, measurements, fp16):
+ import onnx
+ import onnxruntime
+ sess = onnxruntime.InferenceSession(waveglow_path)
+
+ device=mel.device
+ mel_size = mel.size(2)
+ batch_size = mel.size(0)
+ stride = 256
+ n_group = 8
+ z_size = mel_size*stride
+ z_size = z_size//n_group
+ z = torch.randn(batch_size, n_group, z_size).cuda()
+
+ mel = mel.unsqueeze(3)
+ z = z.unsqueeze(3)
+
+ if fp16:
+ z = z.half()
+ mel = mel.half()
+
+ mel = mel.cpu().numpy().copy()
+ z = z.cpu().numpy().copy()
+
+ print("Running WaveGlow with ONNX Runtime")
+ with MeasureTime(measurements, "waveglow_time"):
+ result = sess.run(["audio"], {
+ 'mel': mel,
+ 'z': z
+ })
+ audios = torch.tensor(result[0], device=device)
+ return audios
+
def main():
parser = argparse.ArgumentParser(
@@ -297,27 +394,35 @@ def main():
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
encoder = load_engine(args.encoder, TRT_LOGGER)
- decoder_iter = load_engine(args.decoder, TRT_LOGGER)
postnet = load_engine(args.postnet, TRT_LOGGER)
- waveglow = load_engine(args.waveglow, TRT_LOGGER)
if args.waveglow_ckpt != "":
# setup denoiser using WaveGlow PyTorch checkpoint
waveglow_ckpt = load_and_setup_model('WaveGlow', parser, args.waveglow_ckpt,
True, forward_is_infer=True)
denoiser = Denoiser(waveglow_ckpt).cuda()
- # after initialization, we don't need WaveGlow PyTorch checkpoint anymore - deleting
+ # after initialization, we don't need WaveGlow PyTorch checkpoint
+ # anymore - deleting
del waveglow_ckpt
torch.cuda.empty_cache()
# create TRT contexts for each engine
encoder_context = encoder.create_execution_context()
- decoder_context = decoder_iter.create_execution_context()
+ decoder_context = None
+ if not args.decoder_onnxruntime:
+ decoder_iter = load_engine(args.decoder, TRT_LOGGER)
+ decoder_context = decoder_iter.create_execution_context()
+ else:
+ decoder_iter = args.decoder
postnet_context = postnet.create_execution_context()
- waveglow_context = waveglow.create_execution_context()
+
+ waveglow_context = None
+ if not args.waveglow_onnxruntime:
+ waveglow = load_engine(args.waveglow, TRT_LOGGER)
+ waveglow_context = waveglow.create_execution_context()
DLLogger.init(backends=[JSONStreamBackend(Verbosity.DEFAULT,
- args.output+'/'+args.log_file),
+ path.join(args.output, args.log_file)),
StdOutBackend(Verbosity.VERBOSE)])
texts = []
@@ -333,14 +438,22 @@ def main():
sequences, sequence_lengths = prepare_input_sequence(texts)
sequences = sequences.to(torch.int32)
sequence_lengths = sequence_lengths.to(torch.int32)
+
with MeasureTime(measurements, "latency"):
mel, mel_lengths = infer_tacotron2_trt(encoder, decoder_iter, postnet,
encoder_context, decoder_context, postnet_context,
- sequences, sequence_lengths, measurements, args.fp16)
- audios = infer_waveglow_trt(waveglow, waveglow_context, mel, measurements, args.fp16)
+ sequences, sequence_lengths, measurements, args.fp16, args.loop)
+ audios = infer_waveglow_onnx(args.waveglow, mel, measurements, args.fp16) if args.waveglow_onnxruntime else \
+ infer_waveglow_trt(waveglow, waveglow_context, mel, measurements, args.fp16)
- with encoder_context, decoder_context, postnet_context, waveglow_context:
+ with encoder_context, postnet_context:
pass
+
+ if decoder_context is not None:
+ with decoder_context: pass
+
+ if waveglow_context is not None:
+ with waveglow_context: pass
audios = audios.float()
if args.waveglow_ckpt != "":
@@ -350,7 +463,7 @@ def main():
for i, audio in enumerate(audios):
audio = audio[:mel_lengths[i]*args.stft_hop_length]
audio = audio/torch.max(torch.abs(audio))
- audio_path = args.output + "audio_"+str(i)+"_trt.wav"
+ audio_path = path.join(args.output, f"audio_{i}_trt.wav")
write(audio_path, args.sampling_rate, audio.cpu().numpy())
@@ -367,8 +480,9 @@ def main():
prec = "fp16" if args.fp16 else "fp32"
latency = measurements['latency']
throughput = audios.size(1)/latency
- log_data = "1,"+str(sequence_lengths[0].item())+","+prec+","+str(latency)+","+str(throughput)+","+str(mel_lengths[0].item())+"\n"
- with open("log_bs1_"+prec+".log", 'a') as f:
+ log_data = f"1,{sequence_lengths[0].item()},{prec},{latency},{throughput},{mel_lengths[0].item()}\n"
+ log_file = path.join(args.output, f"log_bs1_{prec}.log")
+ with open(log_file, 'a') as f:
f.write(log_data)
if __name__ == "__main__":
diff --git a/demo/Tacotron2/trt/run_latency_tests_trt.sh b/demo/Tacotron2/tensorrt/run_latency_tests_trt.sh
similarity index 68%
rename from demo/Tacotron2/trt/run_latency_tests_trt.sh
rename to demo/Tacotron2/tensorrt/run_latency_tests_trt.sh
index 2040d804..07dfd704 100644
--- a/demo/Tacotron2/trt/run_latency_tests_trt.sh
+++ b/demo/Tacotron2/tensorrt/run_latency_tests_trt.sh
@@ -14,4 +14,4 @@
# limitations under the License.
#
-bash test_infer.sh --test trt/test_infer_trt.py -bs 1 -il 128 --fp16 --num-iters 1003 --encoder ./output/encoder_fp16.engine --decoder ./output/decoder_iter_fp16.engine --postnet ./output/postnet_fp16.engine --waveglow ./output/waveglow_fp16.engine --wn-channels 256
+bash test_infer.sh --test tensorrt/test_infer_trt.py -bs 1 -il 128 --fp16 --num-iters 1003 --encoder ./output/encoder_fp16.engine --decoder ./output/decoder_with_outer_loop_fp16.engine --postnet ./output/postnet_fp16.engine --waveglow ./output/waveglow_fp16.engine --wn-channels 256
diff --git a/demo/Tacotron2/trt/test_infer_trt.py b/demo/Tacotron2/tensorrt/test_infer_trt.py
similarity index 84%
rename from demo/Tacotron2/trt/test_infer_trt.py
rename to demo/Tacotron2/tensorrt/test_infer_trt.py
index e78d1737..05031bbe 100644
--- a/demo/Tacotron2/trt/test_infer_trt.py
+++ b/demo/Tacotron2/tensorrt/test_infer_trt.py
@@ -23,38 +23,37 @@ import argparse
import numpy as np
from scipy.io.wavfile import write
-from inference import checkpoint_from_distributed, unwrap_distributed, MeasureTime, prepare_input_sequence
+from inference import checkpoint_from_distributed, unwrap_distributed, MeasureTime, prepare_input_sequence, load_and_setup_model
from inference_trt import infer_tacotron2_trt, infer_waveglow_trt
-from trt.trt_utils import load_engine
+from trt_utils import load_engine
import tensorrt as trt
import time
import dllogger as DLLogger
from dllogger import StdOutBackend, JSONStreamBackend, Verbosity
-from apex import amp
+# from apex import amp
def parse_args(parser):
"""
Parse commandline arguments.
"""
parser.add_argument('--encoder', type=str, required=True,
- help='Full path to the Encoder engine')
+ help='full path to the Encoder engine')
parser.add_argument('--decoder', type=str, required=True,
- help='Full path to the DecoderIter engine')
+ help='full path to the DecoderIter engine')
parser.add_argument('--postnet', type=str, required=True,
- help='Full path to the Postnet engine')
+ help='full path to the Postnet engine')
parser.add_argument('--waveglow', type=str, required=True,
- help='Full path to the WaveGlow engine')
+ help='full path to the WaveGlow engine')
parser.add_argument('--waveglow-ckpt', type=str, default="",
- help='Full path to the WaveGlow model checkpoint file')
- parser.add_argument('-s', '--sigma-infer', default=0.6, type=float,
- help='Standard deviation of the Gaussian distribution')
+ help='full path to the WaveGlow model checkpoint file')
+ parser.add_argument('-s', '--sigma-infer', default=0.6, type=float)
parser.add_argument('-sr', '--sampling-rate', default=22050, type=int,
help='Sampling rate')
parser.add_argument('--fp16', action='store_true',
- help='Inference with FP16 precision')
+ help='inference with FP16')
parser.add_argument('--log-file', type=str, default='nvlog.json',
help='Filename for logging')
parser.add_argument('--stft-hop-length', type=int, default=256,
@@ -69,34 +68,6 @@ def parse_args(parser):
return parser
-def load_and_setup_model(model_name, parser, checkpoint, amp_run, to_cuda=True):
- model_parser = models.parse_model_args(model_name, parser, add_help=False)
- model_args, _ = model_parser.parse_known_args()
-
- model_config = models.get_model_config(model_name, model_args)
- model = models.get_model(model_name, model_config, to_cuda=to_cuda)
-
- if checkpoint is not None:
- if to_cuda:
- state_dict = torch.load(checkpoint)['state_dict']
- else:
- state_dict = torch.load(checkpoint,map_location='cpu')['state_dict']
- if checkpoint_from_distributed(state_dict):
- state_dict = unwrap_distributed(state_dict)
-
- model.load_state_dict(state_dict)
-
- if model_name == "WaveGlow":
- model = model.remove_weightnorm(model)
-
- model.eval()
-
- if amp_run:
- model, _ = amp.initialize(model, [], opt_level="O3")
-
- return model
-
-
def print_stats(measurements_all):
print(np.mean(measurements_all['latency'][1:]),
@@ -127,7 +98,7 @@ def print_stats(measurements_all):
print("Throughput average (samples/sec) = {:.4f}".format(np.mean(throughput)))
print("Preprocessing average (seconds) = {:.4f}".format(np.mean(preprocessing)))
print("Postprocessing average (seconds) = {:.4f}".format(np.mean(postprocessing)))
- print("Number of mels per audio average = {}".format(np.mean(num_mels_per_audio)))
+ print("Number of mels per audio average = {}".format(np.mean(num_mels_per_audio))) #
print("Latency average (seconds) = {:.4f}".format(np.mean(latency)))
print("Latency std (seconds) = {:.4f}".format(np.std(latency)))
print("Latency cl 50 (seconds) = {:.4f}".format(cf_50))
@@ -180,8 +151,11 @@ def main():
if args.waveglow_ckpt != "":
# setup denoiser using WaveGlow PyTorch checkpoint
- waveglow_ckpt = load_and_setup_model('WaveGlow', parser, args.waveglow_ckpt,
- True, forward_is_infer=True)
+ waveglow_ckpt = load_and_setup_model('WaveGlow', parser,
+ args.waveglow_ckpt,
+ fp16_run=args.fp16,
+ cpu_run=False,
+ forward_is_infer=True)
denoiser = Denoiser(waveglow_ckpt).cuda()
# after initialization, we don't need WaveGlow PyTorch checkpoint
# anymore - deleting
@@ -215,11 +189,11 @@ def main():
with MeasureTime(measurements, "tacotron2_latency"):
mel, mel_lengths = infer_tacotron2_trt(encoder, decoder_iter, postnet,
encoder_context, decoder_context, postnet_context,
- sequences_padded, input_lengths, measurements, args.fp16)
+ sequences_padded, input_lengths, measurements, args.fp16, True)
with MeasureTime(measurements, "waveglow_latency"):
audios = infer_waveglow_trt(waveglow, waveglow_context, mel, measurements, args.fp16)
-
+
num_mels = mel.size(0)*mel.size(2)
num_samples = audios.size(0)*audios.size(1)
diff --git a/demo/Tacotron2/trt/trt_utils.py b/demo/Tacotron2/tensorrt/trt_utils.py
similarity index 88%
rename from demo/Tacotron2/trt/trt_utils.py
rename to demo/Tacotron2/tensorrt/trt_utils.py
index b25a447f..91de22cb 100644
--- a/demo/Tacotron2/trt/trt_utils.py
+++ b/demo/Tacotron2/tensorrt/trt_utils.py
@@ -16,9 +16,24 @@
import tensorrt as trt
+# For a single dimension this will return the min, opt, and max size when given
+# input of either one or three (comma delimited) values
+# dim="1" or dim=1 returns (1, 1, 1)
+# dim="1,4,5" returns (1, 4, 5)
+def parse_dynamic_size(dim):
+ split = str(dim).split(',')
+ assert len(split) in (1,3) , "Dynamic size input must be either 1 or 3 comma-separated integers"
+ ints = [int(i) for i in split]
+
+ if len(ints) == 1:
+ ints *= 3
+
+ assert ints[0] <= ints[1] <= ints[2]
+ return tuple(ints)
+
def is_dimension_dynamic(dim):
- return dim is None or dim == -1
+ return dim is None or dim <= 0
def is_shape_dynamic(shape):
@@ -92,7 +107,6 @@ def engine_info(engine_filepath):
def build_engine(model_file, shapes, max_ws=512*1024*1024, fp16=False):
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(TRT_LOGGER)
- builder.fp16_mode = fp16
config = builder.create_builder_config()
config.max_workspace_size = max_ws
diff --git a/demo/Tacotron2/trt/export_onnx2trt.py b/demo/Tacotron2/trt/export_onnx2trt.py
deleted file mode 100644
index 7ab5f581..00000000
--- a/demo/Tacotron2/trt/export_onnx2trt.py
+++ /dev/null
@@ -1,119 +0,0 @@
-#
-# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-import pycuda.driver as cuda
-import pycuda.autoinit
-import tensorrt as trt
-import onnx
-import argparse
-
-import sys
-sys.path.append('./')
-
-from trt.trt_utils import build_engine
-
-def parse_args(parser):
- """
- Parse commandline arguments.
- """
- parser.add_argument('-o', '--output', required=True,
- help='output folder to save audio (file per phrase)')
- parser.add_argument('--encoder', type=str, default="",
- help='full path to the Encoder ONNX')
- parser.add_argument('--decoder', type=str, default="",
- help='full path to the DecoderIter ONNX')
- parser.add_argument('--postnet', type=str, default="",
- help='full path to the Postnet ONNX')
- parser.add_argument('--waveglow', type=str, default="",
- help='full path to the WaveGlow ONNX')
- parser.add_argument('--fp16', action='store_true',
- help='inference with FP16')
-
- return parser
-
-
-def main():
-
- parser = argparse.ArgumentParser(
- description='Export from ONNX to TensorRT for Tacotron 2 and WaveGlow')
- parser = parse_args(parser)
- args = parser.parse_args()
-
- engine_prec = "_fp16" if args.fp16 else "_fp32"
-
- # Encoder
- shapes=[{"name": "sequences", "min": (1,4), "opt": (1,128), "max": (1,256)},
- {"name": "sequence_lengths", "min": (1,), "opt": (1,), "max": (1,)}]
- if args.encoder != "":
- print("Building Encoder ...")
- encoder_engine = build_engine(args.encoder, shapes=shapes, fp16=args.fp16)
- if encoder_engine is not None:
- with open(args.output+"/"+"encoder"+engine_prec+".engine", 'wb') as f:
- f.write(encoder_engine.serialize())
- else:
- print("Failed to build engine from", args.encoder)
- sys.exit()
-
- # DecoderIter
- shapes=[{"name": "decoder_input", "min": (1,80), "opt": (1,80), "max": (1,80)},
- {"name": "attention_hidden", "min": (1,1024), "opt": (1,1024), "max": (1,1024)},
- {"name": "attention_cell", "min": (1,1024), "opt": (1,1024), "max": (1,1024)},
- {"name": "decoder_hidden", "min": (1,1024), "opt": (1,1024), "max": (1,1024)},
- {"name": "decoder_cell", "min": (1,1024), "opt": (1,1024), "max": (1,1024)},
- {"name": "attention_weights", "min": (1,4), "opt": (1,128), "max": (1,256)},
- {"name": "attention_weights_cum", "min": (1,4), "opt": (1,128), "max": (1,256)},
- {"name": "attention_context", "min": (1,512), "opt": (1,512), "max": (1,512)},
- {"name": "memory", "min": (1,4,512), "opt": (1,128,512), "max": (1,256,512)},
- {"name": "processed_memory", "min": (1,4,128), "opt": (1,128,128), "max": (1,256,128)},
- {"name": "mask", "min": (1,4), "opt": (1,128), "max": (1,256)}]
- if args.decoder != "":
- print("Building Decoder ...")
- decoder_iter_engine = build_engine(args.decoder, shapes=shapes, fp16=args.fp16)
- if decoder_iter_engine is not None:
- with open(args.output+"/"+"decoder_iter"+engine_prec+".engine", 'wb') as f:
- f.write(decoder_iter_engine.serialize())
- else:
- print("Failed to build engine from", args.decoder)
- sys.exit()
-
- # Postnet
- shapes=[{"name": "mel_outputs", "min": (1,80,32), "opt": (1,80,768), "max": (1,80,1664)}]
- if args.postnet != "":
- print("Building Postnet ...")
- postnet_engine = build_engine(args.postnet, shapes=shapes, fp16=args.fp16)
- if postnet_engine is not None:
- with open(args.output+"/"+"postnet"+engine_prec+".engine", 'wb') as f:
- f.write(postnet_engine.serialize())
- else:
- print("Failed to build engine from", args.postnet)
- sys.exit()
-
- # WaveGlow
- shapes=[{"name": "mel", "min": (1,80,32,1), "opt": (1,80,768,1), "max": (1,80,1664,1)},
- {"name": "z", "min": (1,8,1024,1), "opt": (1,8,24576,1), "max": (1,8,53248,1)}]
- if args.waveglow != "":
- print("Building WaveGlow ...")
- waveglow_engine = build_engine(args.waveglow, shapes=shapes, fp16=args.fp16)
- if waveglow_engine is not None:
- with open(args.output+"/"+"waveglow"+engine_prec+".engine", 'wb') as f:
- f.write(waveglow_engine.serialize())
- else:
- print("Failed to build engine from", args.waveglow)
- sys.exit()
-
-
-if __name__ == '__main__':
- main()
diff --git a/docker/build.sh b/docker/build.sh
index 0799365c..d87e7c03 100755
--- a/docker/build.sh
+++ b/docker/build.sh
@@ -17,7 +17,7 @@
arg_dockerfile=docker/ubuntu-18.04.Dockerfile
arg_imagename=tensorrt-ubuntu
-arg_cudaversion=11.1
+arg_cudaversion=11.3.1
arg_help=0
while [[ "$#" -gt 0 ]]; do case $1 in
diff --git a/docker/centos-7.Dockerfile b/docker/centos-7.Dockerfile
index 83f02945..6aaff682 100644
--- a/docker/centos-7.Dockerfile
+++ b/docker/centos-7.Dockerfile
@@ -12,19 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-ARG CUDA_VERSION=11.1
+ARG CUDA_VERSION=11.3.1
ARG OS_VERSION=7
FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-centos${OS_VERSION}
LABEL maintainer="NVIDIA CORPORATION"
-ENV TRT_VERSION 7.2.3.4
+ENV TRT_VERSION 8.0.1.6
SHELL ["/bin/bash", "-c"]
# Setup user account
ARG uid=1000
ARG gid=1000
-RUN groupadd -r -f -g ${gid} trtuser && useradd -r -u ${uid} -g ${gid} -ms /bin/bash trtuser
+RUN groupadd -r -f -g ${gid} trtuser && useradd -o -r -u ${uid} -g ${gid} -ms /bin/bash trtuser
RUN usermod -aG wheel trtuser
RUN echo 'trtuser:nvidia' | chpasswd
RUN mkdir -p /workspace && chown trtuser /workspace
@@ -44,23 +44,22 @@ RUN yum -y install \
sudo
# Install python3
-RUN cd /tmp &&\
- curl -O https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz &&\
- tar -xzf Python-3.8.3.tgz && cd Python-3.8.3 &&\
- ./configure --enable-optimizations && make altinstall &&\
- rm -rf /tmp/Python-3.8.3
+RUN yum install -y python36 python3-devel
# Install TensorRT
-RUN cd /tmp &&\
- wget https://developer.download.nvidia.com/compute/machine-learning/repos/rhel7/x86_64/nvidia-machine-learning-repo-rhel7-1.0.0-1.x86_64.rpm &&\
- rpm -Uvh nvidia-machine-learning-repo-*.rpm
-RUN yum install -y libnvinfer7 libnvparsers7 libnvinfer-plugin7 libnvonnxparsers7 libnvinfer-devel libnvparsers-devel libnvinfer-plugin-devel python3-libnvinfer
+RUN v="${TRT_VERSION%.*}-1.cuda${CUDA_VERSION%.*}" &&\
+ yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo &&\
+ yum -y install libnvinfer8-${v} libnvparsers8-${v} libnvonnxparsers8-${v} libnvinfer-plugin8-${v} \
+ libnvinfer-devel-${v} libnvparsers-devel-${v} libnvonnxparsers-devel-${v} libnvinfer-plugin-devel-${v} \
+ python3-libnvinfer-${v}
# Install PyPI packages
RUN pip3 install --upgrade pip
RUN pip3 install setuptools>=41.0.0
+RUN pip3 install numpy
COPY requirements.txt /tmp/requirements.txt
RUN pip3 install -r /tmp/requirements.txt
+RUN pip3 install jupyter jupyterlab
# Install Cmake
RUN cd /tmp && \
@@ -72,6 +71,8 @@ RUN cd /tmp && \
# Download NGC client
RUN cd /usr/local/bin && wget https://ngc.nvidia.com/downloads/ngccli_cat_linux.zip && unzip ngccli_cat_linux.zip && chmod u+x ngc && rm ngccli_cat_linux.zip ngc.md5 && echo "no-apikey\nascii\n" | ngc config set
+RUN rm /usr/bin/python && ln -s /usr/bin/python3 /usr/bin/python
+
# Set environment and working directory
ENV TRT_LIBPATH /usr/lib/x86_64-linux-gnu
ENV TRT_OSSPATH /workspace/TensorRT
diff --git a/docker/centos-8.Dockerfile b/docker/centos-8.Dockerfile
new file mode 100644
index 00000000..277a233d
--- /dev/null
+++ b/docker/centos-8.Dockerfile
@@ -0,0 +1,83 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG CUDA_VERSION=11.3.1
+ARG OS_VERSION=8
+
+FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-centos${OS_VERSION}
+LABEL maintainer="NVIDIA CORPORATION"
+
+ENV TRT_VERSION 8.0.1.6
+SHELL ["/bin/bash", "-c"]
+
+# Setup user account
+ARG uid=1000
+ARG gid=1000
+RUN groupadd -r -f -g ${gid} trtuser && useradd -o -r -u ${uid} -g ${gid} -ms /bin/bash trtuser
+RUN usermod -aG wheel trtuser
+RUN echo 'trtuser:nvidia' | chpasswd
+RUN mkdir -p /workspace && chown trtuser /workspace
+
+# Install requried packages
+RUN yum -y groupinstall "Development Tools"
+RUN yum -y install \
+ openssl-devel \
+ bzip2-devel \
+ libffi-devel \
+ zlib-devel \
+ wget \
+ perl-core \
+ git \
+ pkg-config \
+ unzip \
+ sudo
+
+# Install python3
+RUN yum install -y python3-devel
+
+# Install TensorRT
+RUN v="${TRT_VERSION%.*}-1.cuda${CUDA_VERSION%.*}" &&\
+ dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo &&\
+ yum -y install libnvinfer8-${v} libnvparsers8-${v} libnvonnxparsers8-${v} libnvinfer-plugin8-${v} \
+ libnvinfer-devel-${v} libnvparsers-devel-${v} libnvonnxparsers-devel-${v} libnvinfer-plugin-devel-${v} \
+ python3-libnvinfer-${v}
+
+# Install PyPI packages
+RUN pip3 install --upgrade pip
+RUN pip3 install setuptools>=41.0.0
+RUN pip3 install numpy
+COPY requirements.txt /tmp/requirements.txt
+RUN pip3 install -r /tmp/requirements.txt
+RUN pip3 install jupyter jupyterlab
+
+# Install Cmake
+RUN cd /tmp && \
+ wget https://github.com/Kitware/CMake/releases/download/v3.14.4/cmake-3.14.4-Linux-x86_64.sh && \
+ chmod +x cmake-3.14.4-Linux-x86_64.sh && \
+ ./cmake-3.14.4-Linux-x86_64.sh --prefix=/usr/local --exclude-subdir --skip-license && \
+ rm ./cmake-3.14.4-Linux-x86_64.sh
+
+# Download NGC client
+RUN cd /usr/local/bin && wget https://ngc.nvidia.com/downloads/ngccli_cat_linux.zip && unzip ngccli_cat_linux.zip && chmod u+x ngc && rm ngccli_cat_linux.zip ngc.md5 && echo "no-apikey\nascii\n" | ngc config set
+
+RUN ln -s /usr/bin/python3 /usr/bin/python
+
+# Set environment and working directory
+ENV TRT_LIBPATH /usr/lib/x86_64-linux-gnu
+ENV TRT_OSSPATH /workspace/TensorRT
+ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${TRT_OSSPATH}/build/out:${TRT_LIBPATH}"
+WORKDIR /workspace
+
+USER trtuser
+RUN ["/bin/bash"]
diff --git a/docker/launch.sh b/docker/launch.sh
index ccfdf057..551b2038 100755
--- a/docker/launch.sh
+++ b/docker/launch.sh
@@ -17,11 +17,13 @@
arg_tag=tensorrt-ubuntu20.04
arg_gpus=all
+arg_jupyter=0
arg_help=0
while [[ "$#" -gt 0 ]]; do case $1 in
--tag) arg_tag="$2"; shift;;
--gpus) arg_gpus="$2"; shift;;
+ --jupyter) arg_jupyter="$2"; shift;;
-h|--help) arg_help=1;;
*) echo "Unknown parameter passed: $1"; echo "For help type: $0 --help"; exit 1;
esac; shift; done
@@ -31,6 +33,7 @@ if [ "$arg_help" -eq "1" ]; then
echo " --help or -h : Print this help menu."
echo " --tag : Image name for generated container."
echo " --gpus : Number of GPUs visible in container. Set 'none' to disable, and 'all' to make all visible."
+ echo " --jupyter : Launch Jupyter notebook using the specified port number."
exit;
fi
@@ -39,8 +42,16 @@ if [ "$arg_gpus" != "none" ]; then
extra_args="$extra_args --gpus $arg_gpus"
fi
+if [ "$arg_jupyter" -ne "0" ]; then
+ extra_args+=" -p $arg_jupyter:$arg_jupyter"
+fi
+
docker_args="$extra_args -v ${PWD}:/workspace/TensorRT --rm -it $arg_tag:latest"
+if [ "$arg_jupyter" -ne "0" ]; then
+ docker_args+=" jupyter-lab --port=$arg_jupyter --no-browser --ip 0.0.0.0 --allow-root"
+fi
+
echo "Launching container:"
echo "> docker run $docker_args"
docker run $docker_args
diff --git a/docker/patch/centos-python-ssl.patch b/docker/patch/centos-python-ssl.patch
deleted file mode 100644
index 2a7619be..00000000
--- a/docker/patch/centos-python-ssl.patch
+++ /dev/null
@@ -1,4 +0,0 @@
-SSL=/usr/local/ssl
-_ssl _ssl.c \
- -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
- -L$(SSL)/lib -lssl -lcrypto
diff --git a/docker/ubuntu-18.04.Dockerfile b/docker/ubuntu-18.04.Dockerfile
index c36d4fa2..994111e1 100644
--- a/docker/ubuntu-18.04.Dockerfile
+++ b/docker/ubuntu-18.04.Dockerfile
@@ -12,13 +12,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-ARG CUDA_VERSION=11.1
+ARG CUDA_VERSION=11.3.1
ARG OS_VERSION=18.04
FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu${OS_VERSION}
LABEL maintainer="NVIDIA CORPORATION"
-ENV TRT_VERSION 7.2.3.4
+ENV TRT_VERSION 8.0.1.6
SHELL ["/bin/bash", "-c"]
# Setup user account
@@ -62,18 +62,21 @@ RUN apt-get install -y --no-install-recommends \
ln -s /usr/bin/pip3 pip;
# Install TensorRT
-RUN cd /tmp &&\
- wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb &&\
- dpkg -i nvidia-machine-learning-repo-*.deb && apt-get update
RUN v="${TRT_VERSION%.*}-1+cuda${CUDA_VERSION%.*}" &&\
- apt-get install -y libnvinfer7=${v} libnvinfer-plugin7=${v} libnvparsers7=${v} libnvonnxparsers7=${v} libnvinfer-dev=${v} libnvinfer-plugin-dev=${v} libnvparsers-dev=${v} python3-libnvinfer=${v} &&\
- apt-mark hold libnvinfer7 libnvinfer-plugin7 libnvparsers7 libnvonnxparsers7 libnvinfer-dev libnvinfer-plugin-dev libnvparsers-dev python3-libnvinfer
+ apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub &&\
+ apt-get update &&\
+ sudo apt-get install libnvinfer8=${v} libnvonnxparsers8=${v} libnvparsers8=${v} libnvinfer-plugin8=${v} \
+ libnvinfer-dev=${v} libnvonnxparsers-dev=${v} libnvparsers-dev=${v} libnvinfer-plugin-dev=${v} \
+ python3-libnvinfer=${v}
# Install PyPI packages
RUN pip3 install --upgrade pip
RUN pip3 install setuptools>=41.0.0
COPY requirements.txt /tmp/requirements.txt
RUN pip3 install -r /tmp/requirements.txt
+RUN pip3 install jupyter jupyterlab
+# Workaround to remove numpy installed with tensorflow
+RUN pip3 install --upgrade numpy
# Install Cmake
RUN cd /tmp && \
diff --git a/docker/ubuntu-16.04.Dockerfile b/docker/ubuntu-20.04.Dockerfile
similarity index 70%
rename from docker/ubuntu-16.04.Dockerfile
rename to docker/ubuntu-20.04.Dockerfile
index c9564f47..ba09e236 100644
--- a/docker/ubuntu-16.04.Dockerfile
+++ b/docker/ubuntu-20.04.Dockerfile
@@ -12,13 +12,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-ARG CUDA_VERSION=11.1
-ARG OS_VERSION=16.04
+ARG CUDA_VERSION=11.3.1
+ARG OS_VERSION=20.04
FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu${OS_VERSION}
LABEL maintainer="NVIDIA CORPORATION"
-ENV TRT_VERSION 7.2.2.3
+ENV TRT_VERSION 8.0.1.6
SHELL ["/bin/bash", "-c"]
# Setup user account
@@ -29,6 +29,9 @@ RUN usermod -aG sudo trtuser
RUN echo 'trtuser:nvidia' | chpasswd
RUN mkdir -p /workspace && chown trtuser /workspace
+# Required to build Ubuntu 20.04 without user prompts with DLFW container
+ENV DEBIAN_FRONTEND=noninteractive
+
# Install requried libraries
RUN apt-get update && apt-get install -y software-properties-common
RUN add-apt-repository ppa:ubuntu-toolchain-r/test
@@ -52,27 +55,31 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential
# Install python3
-RUN add-apt-repository ppa:deadsnakes/ppa && apt-get update &&\
- apt-get remove -y python3 python && apt-get autoremove -y &&\
- apt-get install -y python3.6 python3.6-dev &&\
- cd /tmp && wget https://bootstrap.pypa.io/get-pip.py && python3.6 get-pip.py &&\
- python3.6 -m pip install wheel &&\
- ln -s /usr/bin/python3.6 /usr/bin/python3 &&\
- ln -s /usr/bin/python3.6 /usr/bin/python
+RUN apt-get install -y --no-install-recommends \
+ python3 \
+ python3-pip \
+ python3-dev \
+ python3-wheel &&\
+ cd /usr/local/bin &&\
+ ln -s /usr/bin/python3 python &&\
+ ln -s /usr/bin/pip3 pip;
# Install TensorRT
-RUN cd /tmp &&\
- wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb &&\
- dpkg -i nvidia-machine-learning-repo-*.deb && apt-get update
RUN v="${TRT_VERSION%.*}-1+cuda${CUDA_VERSION%.*}" &&\
- apt-get install -y libnvinfer7=${v} libnvinfer-plugin7=${v} libnvparsers7=${v} libnvonnxparsers7=${v} libnvinfer-dev=${v} libnvinfer-plugin-dev=${v} libnvparsers-dev=${v} python3-libnvinfer=${v} &&\
- apt-mark hold libnvinfer7 libnvinfer-plugin7 libnvparsers7 libnvonnxparsers7 libnvinfer-dev libnvinfer-plugin-dev libnvparsers-dev python3-libnvinfer
+ apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub &&\
+ apt-get update &&\
+ sudo apt-get install libnvinfer8=${v} libnvonnxparsers8=${v} libnvparsers8=${v} libnvinfer-plugin8=${v} \
+ libnvinfer-dev=${v} libnvonnxparsers-dev=${v} libnvparsers-dev=${v} libnvinfer-plugin-dev=${v} \
+ python3-libnvinfer=${v}
# Install PyPI packages
RUN pip3 install --upgrade pip
RUN pip3 install setuptools>=41.0.0
COPY requirements.txt /tmp/requirements.txt
RUN pip3 install -r /tmp/requirements.txt
+RUN pip3 install jupyter jupyterlab
+# Workaround to remove numpy installed with tensorflow
+RUN pip3 install --upgrade numpy
# Install Cmake
RUN cd /tmp && \
diff --git a/docker/ubuntu-cross-aarch64.Dockerfile b/docker/ubuntu-cross-aarch64.Dockerfile
index d478aa9a..a4fe68f9 100644
--- a/docker/ubuntu-cross-aarch64.Dockerfile
+++ b/docker/ubuntu-cross-aarch64.Dockerfile
@@ -14,10 +14,12 @@
ARG CUDA_VERSION=10.2
ARG OS_VERSION=18.04
-FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu${OS_VERSION}
+FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu${OS_VERSION}
LABEL maintainer="NVIDIA CORPORATION"
+ENV TRT_VERSION 8.0.1.6
+
ARG uid=1000
ARG gid=1000
RUN groupadd -r -f -g ${gid} trtuser && useradd -o -r -u ${uid} -g ${gid} -ms /bin/bash trtuser
@@ -94,15 +96,18 @@ RUN dpkg -x /pdk_files/libcudnn[7-8]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/cudn
&& ln -s /pdk_files/cudnn/usr/include/aarch64-linux-gnu/cudnn_version_v[7-9].h /usr/include/cudnn_version.h
# Unpack libnvinfer
-RUN dpkg -x /pdk_files/libnvinfer[0-7]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
+RUN dpkg -x /pdk_files/libnvinfer[0-8]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
&& dpkg -x /pdk_files/libnvinfer-dev_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
- && dpkg -x /pdk_files/libnvparsers[6-7]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
+ && dpkg -x /pdk_files/libnvparsers[6-8]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
&& dpkg -x /pdk_files/libnvparsers-dev_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
- && dpkg -x /pdk_files/libnvinfer-plugin[6-7]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
+ && dpkg -x /pdk_files/libnvinfer-plugin[6-8]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
&& dpkg -x /pdk_files/libnvinfer-plugin-dev_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
- && dpkg -x /pdk_files/libnvonnxparsers[6-7]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
+ && dpkg -x /pdk_files/libnvonnxparsers[6-8]_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt \
&& dpkg -x /pdk_files/libnvonnxparsers-dev_*-1+cuda10.[0-9]_arm64.deb /pdk_files/tensorrt
+# Clean up debs
+RUN rm -rf /pdk_files/*.deb
+
# create stub libraries
RUN cd /pdk_files/tensorrt \
&& ln -s usr/include/aarch64-linux-gnu include \
diff --git a/docker/ubuntu-cross-ppc64le.Dockerfile b/docker/ubuntu-cross-ppc64le.Dockerfile
deleted file mode 100755
index 9771a959..00000000
--- a/docker/ubuntu-cross-ppc64le.Dockerfile
+++ /dev/null
@@ -1,109 +0,0 @@
-# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-ARG CUDA_VERSION=11.0
-ARG OS_VERSION=18.04
-FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu${OS_VERSION}
-
-LABEL maintainer="NVIDIA CORPORATION"
-
-ARG uid=1000
-ARG gid=1000
-RUN groupadd -r -f -g ${gid} trtuser && useradd -o -r -u ${uid} -g ${gid} -ms /bin/bash trtuser
-RUN usermod -aG sudo trtuser
-RUN echo 'trtuser:nvidia' | chpasswd
-RUN mkdir -p /workspace && chown -R trtuser:trtuser /workspace
-
-# Install requried libraries
-RUN apt-get update && apt-get install -y software-properties-common
-RUN add-apt-repository ppa:ubuntu-toolchain-r/test
-RUN apt-get update && apt-get install -y --no-install-recommends \
- libcurl4-openssl-dev \
- wget \
- zlib1g-dev \
- git \
- pkg-config \
- python3 \
- python3-pip \
- python3-dev \
- python3-setuptools \
- python3-wheel \
- sudo \
- ssh \
- pbzip2 \
- pv \
- bzip2 \
- unzip \
- g++-powerpc64le-linux-gnu \
- libc6-powerpc-cross
-
-RUN cd /usr/local/bin &&\
- ln -s /usr/bin/python3 python &&\
- ln -s /usr/bin/pip3 pip
-
-RUN cd /tmp && \
- wget https://github.com/Kitware/CMake/releases/download/v3.14.4/cmake-3.14.4-Linux-x86_64.sh && \
- chmod +x cmake-3.14.4-Linux-x86_64.sh && \
- ./cmake-3.14.4-Linux-x86_64.sh --prefix=/usr/local --exclude-subdir --skip-license && \
- rm ./cmake-3.14.4-Linux-x86_64.sh
-# Download ppc Cudnn, Cublas, Cudart, RT, nvprof
-# TODO Remove once packages are added to cuda cross compiler
-RUN wget http://cuda-repo/release-candidates/Libraries/cuDNN/v8.0/8.0.2.5_20200617_28575977/11.0.x-r445/Installer/Ubuntu18_04-ppc64le/libcudnn8_8.0.2.5-1+cuda11.0_ppc64el.deb && \
- wget http://cuda-repo/release-candidates/Libraries/cuDNN/v8.0/8.0.2.5_20200617_28575977/11.0.x-r445/Installer/Ubuntu18_04-ppc64le/libcudnn8-dev_8.0.2.5-1+cuda11.0_ppc64el.deb && \
- wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/libcublas-dev-11-0_11.0.0.191-1_ppc64el.deb && \
- wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/libcublas-11-0_11.0.0.191-1_ppc64el.deb && \
- wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-cudart-11-0_11.0.171-1_ppc64el.deb && \
- wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-cudart-dev-11-0_11.0.171-1_ppc64el.deb && \
- wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-nvrtc-11-0_11.0.167-1_ppc64el.deb && \
- wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-nvrtc-dev-11-0_11.0.167-1_ppc64el.deb && \
- wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-nvcc-11-0_11.0.167-1_ppc64el.deb && \
- wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-nvprof-11-0_11.0.167-1_ppc64el.deb
-
-# Unpack Cublas
-RUN dpkg -x libcublas-11-0_11.0.0.191-1_ppc64el.deb cublas && \
- dpkg -x libcublas-dev-11-0_11.0.0.191-1_ppc64el.deb cublas && \
- cp -r cublas/* /
-
-# Unpack Cudart
-RUN dpkg -x cuda-cudart-11-0_11.0.171-1_ppc64el.deb cudart && \
- dpkg -x cuda-cudart-dev-11-0_11.0.171-1_ppc64el.deb cudart && \
- cp -r cudart/* /
-
-# Unpack RT
-RUN dpkg -x cuda-nvrtc-11-0_11.0.167-1_ppc64el.deb rt && \
- dpkg -x cuda-nvrtc-dev-11-0_11.0.167-1_ppc64el.deb rt && \
- cp -r rt/* /
-
-# Unpack Cudnn
-RUN dpkg -x libcudnn8_8.0.2.5-1+cuda11.0_ppc64el.deb cudnn && \
- dpkg -x libcudnn8-dev_8.0.2.5-1+cuda11.0_ppc64el.deb cudnn && \
- cp -r cudnn/* /
-
-# Unpack NVCC, and copy headers
-RUN dpkg -x cuda-nvcc-11-0_11.0.167-1_ppc64el.deb nvcc && \
- cp -r nvcc/usr/local/cuda-11.0/targets/ppc64le-linux/include/* /usr/local/cuda-11.0/targets/ppc64le-linux/include/
-
-# Install nvprof
-RUN dpkg -x cuda-nvprof-11-0_11.0.167-1_ppc64el.deb prof && \
- cp -r prof/* /
-
-# Clean up temporary files
-RUN rm -rf cublas cudart rt prof nvcc
-RUN rm *.deb
-
-WORKDIR /workspace
-ENV TRT_LIBPATH /usr/lib/x86_64-linux-gnu
-ENV TRT_OSSPATH /workspace/TensorRT
-USER trtuser
-RUN ["/bin/bash"]
diff --git a/include/NvCaffeParser.h b/include/NvCaffeParser.h
index 520aeb20..495bf414 100644
--- a/include/NvCaffeParser.h
+++ b/include/NvCaffeParser.h
@@ -53,7 +53,7 @@ public:
//!
//! \return ITensor* corresponding to the queried name. If no such ITensor exists, then nullptr is returned.
//!
- virtual nvinfer1::ITensor* find(const char* name) const TRTNOEXCEPT = 0;
+ virtual nvinfer1::ITensor* find(const char* name) const noexcept = 0;
protected:
virtual ~IBlobNameToTensor() {}
@@ -71,62 +71,16 @@ protected:
class IBinaryProtoBlob
{
public:
- virtual const void* getData() TRTNOEXCEPT = 0;
- virtual nvinfer1::DimsNCHW getDimensions() TRTNOEXCEPT = 0;
- virtual nvinfer1::DataType getDataType() TRTNOEXCEPT = 0;
- virtual void destroy() TRTNOEXCEPT = 0;
-
-protected:
- virtual ~IBinaryProtoBlob() {}
-};
-
-//!
-//! \class IPluginFactory
-//!
-//! \brief Plugin factory used to configure plugins.
-//!
-class IPluginFactory
-{
-public:
+ virtual const void* getData() noexcept = 0;
+ virtual nvinfer1::Dims4 getDimensions() noexcept = 0;
+ virtual nvinfer1::DataType getDataType() noexcept = 0;
//!
- //! \brief A user implemented function that determines if a layer configuration is provided by an IPlugin.
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
//!
- //! \param layerName Name of the layer which the user wishes to validate.
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
//!
- virtual bool isPlugin(const char* layerName) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Creates a plugin.
- //!
- //! \param layerName Name of layer associated with the plugin.
- //! \param weights Weights used for the layer.
- //! \param nbWeights Number of weights.
- //!
- virtual nvinfer1::IPlugin* createPlugin(
- const char* layerName, const nvinfer1::Weights* weights, int32_t nbWeights) TRTNOEXCEPT = 0;
-
- virtual ~IPluginFactory() {}
-};
-
-//!
-//! \class IPluginFactoryExt
-//!
-//! \brief Plugin factory used to configure plugins with added support for TRT versioning.
-//!
-class IPluginFactoryExt : public IPluginFactory
-{
-public:
- virtual int32_t getVersion() const TRTNOEXCEPT
- {
- return NV_TENSORRT_VERSION;
- }
-
- //!
- //! \brief A user implemented function that determines if a layer configuration is provided by an IPluginExt.
- //!
- //! \param layerName Name of the layer which the user wishes to validate.
- //!
- virtual bool isPluginExt(const char* layerName) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED virtual void destroy() noexcept = 0;
+ virtual ~IBinaryProtoBlob() noexcept = default;
};
//!
@@ -142,7 +96,7 @@ public:
//!
//! \param layerName Name of the layer which the user wishes to validate.
//!
- virtual bool isPluginV2(const char* layerName) TRTNOEXCEPT = 0;
+ virtual bool isPluginV2(const char* layerName) noexcept = 0;
//!
//! \brief Creates a plugin.
@@ -153,9 +107,10 @@ public:
//! \param libNamespace Library Namespace associated with the plugin object
//!
virtual nvinfer1::IPluginV2* createPlugin(const char* layerName, const nvinfer1::Weights* weights,
- int32_t nbWeights, const char* libNamespace = "") TRTNOEXCEPT = 0;
+ int32_t nbWeights, const char* libNamespace = "") noexcept
+ = 0;
- virtual ~IPluginFactoryV2() {}
+ virtual ~IPluginFactoryV2() noexcept = default;
};
//!
//! \class ICaffeParser
@@ -182,13 +137,12 @@ public:
//!
//! \see nvcaffeparser1::IBlobNameToTensor
//!
- virtual const IBlobNameToTensor* parse(const char* deploy,
- const char* model,
- nvinfer1::INetworkDefinition& network,
- nvinfer1::DataType weightType) TRTNOEXCEPT = 0;
+ virtual const IBlobNameToTensor* parse(const char* deploy, const char* model, nvinfer1::INetworkDefinition& network,
+ nvinfer1::DataType weightType) noexcept
+ = 0;
//!
- //! \brief Parse a deploy prototxt a binaryproto Caffe model from memory buffers to extract
+ //! \brief Parse a deploy prototxt and a binaryproto Caffe model from memory buffers to extract
//! network definition and weights associated with the network, respectively.
//!
//! \param deployBuffer The plain text deploy prototxt used to define the network definition.
@@ -202,12 +156,10 @@ public:
//!
//! \see nvcaffeparser1::IBlobNameToTensor
//!
- virtual const IBlobNameToTensor* parseBuffers(const char* deployBuffer,
- std::size_t deployLength,
- const char* modelBuffer,
- std::size_t modelLength,
- nvinfer1::INetworkDefinition& network,
- nvinfer1::DataType weightType) TRTNOEXCEPT = 0;
+ virtual const IBlobNameToTensor* parseBuffers(const char* deployBuffer, std::size_t deployLength,
+ const char* modelBuffer, std::size_t modelLength, nvinfer1::INetworkDefinition& network,
+ nvinfer1::DataType weightType) noexcept
+ = 0;
//!
//! \brief Parse and extract data stored in binaryproto file.
@@ -221,7 +173,7 @@ public:
//!
//! \see nvcaffeparser1::IBinaryProtoBlob
//!
- virtual IBinaryProtoBlob* parseBinaryProto(const char* fileName) TRTNOEXCEPT = 0;
+ virtual IBinaryProtoBlob* parseBinaryProto(const char* fileName) noexcept = 0;
//!
//! \brief Set buffer size for the parsing and storage of the learned model.
@@ -230,41 +182,30 @@ public:
//!
//! \note Default size is 2^30 bytes.
//!
- virtual void setProtobufBufferSize(size_t size) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the IPluginFactory used to create the user defined plugins.
- //!
- //! \param factory Pointer to an instance of the user implmentation of IPluginFactory.
- //!
- virtual void setPluginFactory(IPluginFactory* factory) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the IPluginFactoryExt used to create the user defined pluginExts.
- //!
- //! \param factory Pointer to an instance of the user implmentation of IPluginFactoryExt.
- //!
- virtual void setPluginFactoryExt(IPluginFactoryExt* factory) TRTNOEXCEPT = 0;
+ virtual void setProtobufBufferSize(size_t size) noexcept = 0;
//!
//! \brief Destroy this ICaffeParser object.
//!
- virtual void destroy() TRTNOEXCEPT = 0;
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
+ //!
+ TRT_DEPRECATED virtual void destroy() noexcept = 0;
//!
//! \brief Set the IPluginFactoryV2 used to create the user defined pluginV2 objects.
//!
- //! \param factory Pointer to an instance of the user implmentation of IPluginFactoryV2.
+ //! \param factory Pointer to an instance of the user implementation of IPluginFactoryV2.
//!
- virtual void setPluginFactoryV2(IPluginFactoryV2* factory) TRTNOEXCEPT = 0;
+ virtual void setPluginFactoryV2(IPluginFactoryV2* factory) noexcept = 0;
//!
//! \brief Set the namespace used to lookup and create plugins in the network.
//!
- virtual void setPluginNamespace(const char* libNamespace) TRTNOEXCEPT = 0;
+ virtual void setPluginNamespace(const char* libNamespace) noexcept = 0;
-protected:
- virtual ~ICaffeParser() {}
+ virtual ~ICaffeParser() noexcept = default;
public:
//!
@@ -275,23 +216,25 @@ public:
//! recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if
//! a recorder has been registered.
//!
+ //! If an error recorder is not set, messages will be sent to the global log stream.
+ //!
//! \param recorder The error recorder to register with this interface.
//!
- //! \see getErrorRecorder
+ //! \see getErrorRecorder()
//!
- virtual void setErrorRecorder(nvinfer1::IErrorRecorder* recorder) TRTNOEXCEPT = 0;
+ virtual void setErrorRecorder(nvinfer1::IErrorRecorder* recorder) noexcept = 0;
//!
//! \brief get the ErrorRecorder assigned to this interface.
//!
- //! Retrieves the assigned error recorder object for the given class. A default error recorder does not exist,
- //! so a nullptr will be returned if setErrorRecorder has not been called.
+ //! Retrieves the assigned error recorder object for the given class. A
+ //! nullptr will be returned if setErrorRecorder has not been called.
//!
//! \return A pointer to the IErrorRecorder object that has been registered.
//!
- //! \see setErrorRecorder
+ //! \see setErrorRecorder()
//!
- virtual nvinfer1::IErrorRecorder* getErrorRecorder() const TRTNOEXCEPT = 0;
+ virtual nvinfer1::IErrorRecorder* getErrorRecorder() const noexcept = 0;
};
//!
@@ -301,19 +244,22 @@ public:
//!
//! \see nvcaffeparser1::ICaffeParser
//!
-TENSORRTAPI ICaffeParser* createCaffeParser() TRTNOEXCEPT;
+//! \deprecated ICaffeParser will be removed in TensorRT 9.0. Plan to migrate your workflow to
+//! use nvonnxparser::IParser for deployment.
+//!
+TENSORRTAPI ICaffeParser* createCaffeParser() noexcept;
//!
//! \brief Shuts down protocol buffers library.
//!
//! \note No part of the protocol buffers library can be used after this function is called.
//!
-TENSORRTAPI void shutdownProtobufLibrary() TRTNOEXCEPT;
+TENSORRTAPI void shutdownProtobufLibrary() noexcept;
} // namespace nvcaffeparser1
//!
//! Internal C entry point for creating ICaffeParser.
//! @private
//!
-extern "C" TENSORRTAPI void* createNvCaffeParser_INTERNAL();
+extern "C" TENSORRTAPI void* createNvCaffeParser_INTERNAL() noexcept;
#endif
diff --git a/include/NvInfer.h b/include/NvInfer.h
index bbbd24f1..ca722190 100644
--- a/include/NvInfer.h
+++ b/include/NvInfer.h
@@ -17,6 +17,7 @@
#ifndef NV_INFER_H
#define NV_INFER_H
+#include "NvInferLegacyDims.h"
#include "NvInferRuntime.h"
//!
@@ -45,382 +46,6 @@
namespace nvinfer1
{
-//!
-//! \class Dims2
-//! \brief Descriptor for two-dimensional data.
-//!
-class Dims2 : public Dims
-{
-public:
- //!
- //! \brief Construct an empty Dims2 object.
- //!
- Dims2()
- {
- nbDims = 2;
- d[0] = d[1] = 0;
- }
-
- //!
- //! \brief Construct a Dims2 from 2 elements.
- //!
- //! \param d0 The first element.
- //! \param d1 The second element.
- //!
- Dims2(int32_t d0, int32_t d1)
- {
- nbDims = 2;
- d[0] = d0;
- d[1] = d1;
- }
-};
-
-//!
-//! \class DimsHW
-//! \brief Descriptor for two-dimensional spatial data.
-//!
-class DimsHW : public Dims2
-{
-public:
- //!
- //! \brief Construct an empty DimsHW object.
- //!
- DimsHW()
- : Dims2()
- {
- type[0] = type[1] = DimensionType::kSPATIAL;
- }
-
- //!
- //! \brief Construct a DimsHW given height and width.
- //!
- //! \param Height the height of the data
- //! \param Width the width of the data
- //!
- DimsHW(int32_t height, int32_t width)
- : Dims2(height, width)
- {
- type[0] = type[1] = DimensionType::kSPATIAL;
- }
-
- //!
- //! \brief Get the height.
- //!
- //! \return The height.
- //!
- int32_t& h()
- {
- return d[0];
- }
-
- //!
- //! \brief Get the height.
- //!
- //! \return The height.
- //!
- int32_t h() const
- {
- return d[0];
- }
-
- //!
- //! \brief Get the width.
- //!
- //! \return The width.
- //!
- int32_t& w()
- {
- return d[1];
- }
-
- //!
- //! \brief Get the width.
- //!
- //! \return The width.
- //!
- int32_t w() const
- {
- return d[1];
- }
-};
-
-//!
-//! \class Dims3
-//! \brief Descriptor for three-dimensional data.
-//!
-class Dims3 : public Dims
-{
-public:
- //!
- //! \brief Construct an empty Dims3 object.
- //!
- Dims3()
- {
- nbDims = 3;
- d[0] = d[1] = d[2] = 0;
- }
-
- //!
- //! \brief Construct a Dims3 from 3 elements.
- //!
- //! \param d0 The first element.
- //! \param d1 The second element.
- //! \param d2 The third element.
- //!
- Dims3(int32_t d0, int32_t d1, int32_t d2)
- {
- nbDims = 3;
- d[0] = d0;
- d[1] = d1;
- d[2] = d2;
- }
-};
-
-//!
-//! \class DimsCHW
-//! \brief Descriptor for data with one channel dimension and two spatial dimensions.
-//!
-//! \deprecated DimsCHW will be removed in TensorRT 8.0, use Dims3 instead.
-//!
-class TRT_DEPRECATED DimsCHW : public Dims3
-{
-public:
- //!
- //! \brief Construct an empty DimsCHW object.
- //!
- DimsCHW()
- : Dims3()
- {
- type[0] = DimensionType::kCHANNEL;
- type[1] = type[2] = DimensionType::kSPATIAL;
- }
-
- //!
- //! \brief Construct a DimsCHW given channel count, height and width.
- //!
- //! \param channels The channel count.
- //! \param height The height of the data.
- //! \param width The width of the data.
- //!
- DimsCHW(int32_t channels, int32_t height, int32_t width)
- : Dims3(channels, height, width)
- {
- type[0] = DimensionType::kCHANNEL;
- type[1] = type[2] = DimensionType::kSPATIAL;
- }
-
- //!
- //! \brief Get the channel count.
- //!
- //! \return The channel count.
- //!
- int32_t& c()
- {
- return d[0];
- }
-
- //!
- //! \brief Get the channel count.
- //!
- //! \return The channel count.
- //!
- int32_t c() const
- {
- return d[0];
- }
-
- //!
- //! \brief Get the height.
- //!
- //! \return The height.
- //!
- int32_t& h()
- {
- return d[1];
- }
-
- //!
- //! \brief Get the height.
- //!
- //! \return The height.
- //!
- int32_t h() const
- {
- return d[1];
- }
-
- //!
- //! \brief Get the width.
- //!
- //! \return The width.
- //!
- int32_t& w()
- {
- return d[2];
- }
-
- //!
- //! \brief Get the width.
- //!
- //! \return The width.
- //!
- int32_t w() const
- {
- return d[2];
- }
-};
-
-//!
-//! \class Dims4
-//! \brief Descriptor for four-dimensional data.
-//!
-class Dims4 : public Dims
-{
-public:
- //!
- //! \brief Construct an empty Dims2 object.
- //!
- Dims4()
- {
- nbDims = 4;
- d[0] = d[1] = d[2] = d[3] = 0;
- }
-
- //!
- //! \brief Construct a Dims4 from 4 elements.
- //!
- //! \param d0 The first element.
- //! \param d1 The second element.
- //! \param d2 The third element.
- //! \param d3 The fourth element.
- //!
- Dims4(int32_t d0, int32_t d1, int32_t d2, int32_t d3)
- {
- nbDims = 4;
- d[0] = d0;
- d[1] = d1;
- d[2] = d2;
- d[3] = d3;
- }
-};
-
-//!
-//! \class DimsNCHW
-//! \brief Descriptor for data with one index dimension, one channel dimension and two spatial dimensions.
-//!
-//! \deprecated DimsNCHW will be removed in TensorRT 8.0, use Dims4 instead.
-//!
-class TRT_DEPRECATED DimsNCHW : public Dims4
-{
-public:
- //!
- //! \brief Construct an empty DimsNCHW object.
- //!
- DimsNCHW()
- : Dims4()
- {
- type[0] = DimensionType::kINDEX;
- type[1] = DimensionType::kCHANNEL;
- type[2] = type[3] = DimensionType::kSPATIAL;
- }
-
- //!
- //! \brief Construct a DimsNCHW given batch size, channel count, height and width.
- //!
- //! \param batchSize The batch size (commonly denoted N).
- //! \param channels The channel count.
- //! \param height The height of the data.
- //! \param width The width of the data.
- //!
- DimsNCHW(int32_t batchSize, int32_t channels, int32_t height, int32_t width)
- : Dims4(batchSize, channels, height, width)
- {
- type[0] = DimensionType::kINDEX;
- type[1] = DimensionType::kCHANNEL;
- type[2] = type[3] = DimensionType::kSPATIAL;
- }
-
- //!
- //! \brief Get the index count.
- //!
- //! \return The index count.
- //!
- int32_t& n()
- {
- return d[0];
- }
-
- //!
- //! \brief Get the index count.
- //!
- //! \return The index count.
- //!
- int32_t n() const
- {
- return d[0];
- }
-
- //!
- //! \brief Get the channel count.
- //!
- //! \return The channel count.
- //!
- int32_t& c()
- {
- return d[1];
- }
-
- //!
- //! \brief Get the channel count.
- //!
- //! \return The channel count.
- //!
- int32_t c() const
- {
- return d[1];
- }
-
- //!
- //! \brief Get the height.
- //!
- //! \return The height.
- //!
- int32_t& h()
- {
- return d[2];
- }
-
- //!
- //! \brief Get the height.
- //!
- //! \return The height.
- //!
- int32_t h() const
- {
- return d[2];
- }
-
- //!
- //! \brief Get the width.
- //!
- //! \return The width.
- //!
- int32_t& w()
- {
- return d[3];
- }
-
- //!
- //! \brief Get the width.
- //!
- //! \return The width.
- //!
- int32_t w() const
- {
- return d[3];
- }
-};
-
//!
//! \enum LayerType
//!
@@ -441,38 +66,78 @@ enum class LayerType : int32_t
kCONCATENATION = 8, //!< Concatenation layer.
kELEMENTWISE = 9, //!< Elementwise layer.
kPLUGIN = 10, //!< Plugin layer.
- kRNN = 11, //!< RNN layer.
- kUNARY = 12, //!< UnaryOp operation Layer.
- kPADDING = 13, //!< Padding layer.
- kSHUFFLE = 14, //!< Shuffle layer.
- kREDUCE = 15, //!< Reduce layer.
- kTOPK = 16, //!< TopK layer.
- kGATHER = 17, //!< Gather layer.
- kMATRIX_MULTIPLY = 18, //!< Matrix multiply layer.
- kRAGGED_SOFTMAX = 19, //!< Ragged softmax layer.
- kCONSTANT = 20, //!< Constant layer.
- kRNN_V2 = 21, //!< RNNv2 layer.
- kIDENTITY = 22, //!< Identity layer.
- kPLUGIN_V2 = 23, //!< PluginV2 layer.
- kSLICE = 24, //!< Slice layer.
- kSHAPE = 25, //!< Shape layer.
- kPARAMETRIC_RELU = 26, //!< Parametric ReLU layer.
- kRESIZE = 27, //!< Resize Layer.
- kTRIP_LIMIT = 28, //!< Loop Trip limit layer
- kRECURRENCE = 29, //!< Loop Recurrence layer
- kITERATOR = 30, //!< Loop Iterator layer
- kLOOP_OUTPUT = 31, //!< Loop output layer
- kSELECT = 32, //!< Select layer.
- kFILL = 33 //!< Fill layer
+ kUNARY = 11, //!< UnaryOp operation Layer.
+ kPADDING = 12, //!< Padding layer.
+ kSHUFFLE = 13, //!< Shuffle layer.
+ kREDUCE = 14, //!< Reduce layer.
+ kTOPK = 15, //!< TopK layer.
+ kGATHER = 16, //!< Gather layer.
+ kMATRIX_MULTIPLY = 17, //!< Matrix multiply layer.
+ kRAGGED_SOFTMAX = 18, //!< Ragged softmax layer.
+ kCONSTANT = 19, //!< Constant layer.
+ kRNN_V2 = 20, //!< RNNv2 layer.
+ kIDENTITY = 21, //!< Identity layer.
+ kPLUGIN_V2 = 22, //!< PluginV2 layer.
+ kSLICE = 23, //!< Slice layer.
+ kSHAPE = 24, //!< Shape layer.
+ kPARAMETRIC_RELU = 25, //!< Parametric ReLU layer.
+ kRESIZE = 26, //!< Resize Layer.
+ kTRIP_LIMIT = 27, //!< Loop Trip limit layer
+ kRECURRENCE = 28, //!< Loop Recurrence layer
+ kITERATOR = 29, //!< Loop Iterator layer
+ kLOOP_OUTPUT = 30, //!< Loop output layer
+ kSELECT = 31, //!< Select layer.
+ kFILL = 32, //!< Fill layer
+ kQUANTIZE = 33, //!< Quantize layer
+ kDEQUANTIZE = 34, //!< Dequantize layer
};
//! Maximum number of elements in LayerType enum. \see LayerType
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
- return 34;
+ return 35;
}
+//!
+//! \brief It is capable of representing one or more TensorFormat by binary OR
+//! operations, e.g., 1U << TensorFormat::kCHW4 | 1U << TensorFormat::kCHW32.
+//!
+//! \see ITensor::getAllowedFormats(), ITensor::setAllowedFormats(),
+//!
+using TensorFormats = uint32_t;
+
+//!
+//! \enum ActivationType
+//!
+//! \brief Enumerates the types of activation to perform in an activation layer.
+//!
+enum class ActivationType : int32_t
+{
+ kRELU = 0, //!< Rectified linear activation.
+ kSIGMOID = 1, //!< Sigmoid activation.
+ kTANH = 2, //!< TanH activation.
+ kLEAKY_RELU = 3, //!< LeakyRelu activation: x>=0 ? x : alpha * x.
+ kELU = 4, //!< Elu activation: x>=0 ? x : alpha * (exp(x) - 1).
+ kSELU = 5, //!< Selu activation: x>0 ? beta * x : beta * (alpha*exp(x) - alpha)
+ kSOFTSIGN = 6, //!< Softsign activation: x / (1+|x|)
+ kSOFTPLUS = 7, //!< Parametric softplus activation: alpha*log(exp(beta*x)+1)
+ kCLIP = 8, //!< Clip activation: max(alpha, min(beta, x))
+ kHARD_SIGMOID = 9, //!< Hard sigmoid activation: max(0, min(1, alpha*x+beta))
+ kSCALED_TANH = 10, //!< Scaled tanh activation: alpha*tanh(beta*x)
+ kTHRESHOLDED_RELU = 11 //!< Thresholded ReLU activation: x>alpha ? x : 0
+};
+
+namespace impl
+{
+//! Maximum number of elements in ActivationType enum. \see ActivationType
+template <>
+struct EnumMaxImpl
+{
+ static constexpr int32_t kVALUE = 12;
+};
+} // namespace impl
+
//!
//! \class ITensor
//!
@@ -486,7 +151,7 @@ constexpr inline int32_t EnumMax()
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class ITensor
+class ITensor : public INoCopy
{
public:
//!
@@ -501,7 +166,10 @@ public:
//!
//! \see getName()
//!
- virtual void setName(const char* name) TRTNOEXCEPT = 0;
+ void setName(const char* name) noexcept
+ {
+ mImpl->setName(name);
+ }
//!
//! \brief Get the tensor name.
@@ -510,14 +178,17 @@ public:
//!
//! \see setName()
//!
- virtual const char* getName() const TRTNOEXCEPT = 0;
+ const char* getName() const noexcept
+ {
+ return mImpl->getName();
+ }
//!
//! \brief Set the dimensions of a tensor.
//!
- //! For a network input the name is assigned by the application. For a network output it is computed based on
- //! the layer parameters and the inputs to the layer. If a tensor size or a parameter is modified in the network,
- //! the dimensions of all dependent tensors will be recomputed.
+ //! For a network input, the dimensions are assigned by the application. For a network output, the dimensions are
+ //! computed based on the layer parameters and the inputs to the layer. If a tensor size or a parameter is modified
+ //! in the network, the dimensions of all dependent tensors will be recomputed.
//!
//! This call is only legal for network input tensors, since the dimensions of layer output tensors are inferred
//! based on layer inputs and parameters.
@@ -526,7 +197,10 @@ public:
//!
//! \see getDimensions()
//!
- virtual void setDimensions(Dims dimensions) TRTNOEXCEPT = 0; // only valid for input tensors
+ void setDimensions(Dims dimensions) noexcept
+ {
+ mImpl->setDimensions(dimensions);
+ }
//!
//! \brief Get the dimensions of a tensor.
@@ -536,7 +210,10 @@ public:
//! \warning getDimensions() returns a -1 for dimensions that are derived from a wildcard dimension.
//! \see setDimensions()
//!
- virtual Dims getDimensions() const TRTNOEXCEPT = 0;
+ Dims getDimensions() const noexcept
+ {
+ return mImpl->getDimensions();
+ }
//!
//! \brief Set the data type of a tensor.
@@ -548,7 +225,10 @@ public:
//!
//! \see getType()
//!
- virtual void setType(DataType type) TRTNOEXCEPT = 0;
+ void setType(DataType type) noexcept
+ {
+ mImpl->setType(type);
+ }
//!
//! \brief Get the data type of a tensor.
@@ -557,7 +237,10 @@ public:
//!
//! \see setType()
//!
- virtual DataType getType() const TRTNOEXCEPT = 0;
+ DataType getType() const noexcept
+ {
+ return mImpl->getType();
+ }
//!
//! \brief Set dynamic range for the tensor
@@ -569,32 +252,27 @@ public:
//!
//! Requires that min and max be finite, and min <= max.
//!
- virtual bool setDynamicRange(float min, float max) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get dynamic range for the tensor
- //!
- //! \return maximal absolute value of the dynamic range, -1.0f if no dynamic range is set.
- //!
- //! \deprecated This interface is superseded by getDynamicRangeMin and getDynamicRangeMax and will be removed in
- //! TensorRT 8.0.
- //!
- TRT_DEPRECATED virtual float getDynamicRange() const TRTNOEXCEPT = 0;
+ bool setDynamicRange(float min, float max) noexcept
+ {
+ return mImpl->setDynamicRange(min, max);
+ }
//!
//! \brief Whether the tensor is a network input.
//!
- virtual bool isNetworkInput() const TRTNOEXCEPT = 0;
+ bool isNetworkInput() const noexcept
+ {
+ return mImpl->isNetworkInput();
+ }
//!
//! \brief Whether the tensor is a network output.
//!
- virtual bool isNetworkOutput() const TRTNOEXCEPT = 0;
+ bool isNetworkOutput() const noexcept
+ {
+ return mImpl->isNetworkOutput();
+ }
-protected:
- virtual ~ITensor() {}
-
-public:
//!
//! \brief Set whether to enable broadcast of tensor across the batch.
//!
@@ -612,7 +290,10 @@ public:
//!
//! \see getBroadcastAcrossBatch()
//!
- virtual void setBroadcastAcrossBatch(bool broadcastAcrossBatch) TRTNOEXCEPT = 0;
+ void setBroadcastAcrossBatch(bool broadcastAcrossBatch) noexcept
+ {
+ mImpl->setBroadcastAcrossBatch(broadcastAcrossBatch);
+ }
//!
//! \brief Check if tensor is broadcast across the batch.
@@ -625,14 +306,20 @@ public:
//!
//! \see setBroadcastAcrossBatch()
//!
- virtual bool getBroadcastAcrossBatch() const TRTNOEXCEPT = 0;
+ bool getBroadcastAcrossBatch() const noexcept
+ {
+ return mImpl->getBroadcastAcrossBatch();
+ }
//!
//! \brief Get the storage location of a tensor.
//! \return The location of tensor data.
//! \see setLocation()
//!
- virtual TensorLocation getLocation() const TRTNOEXCEPT = 0;
+ TensorLocation getLocation() const noexcept
+ {
+ return mImpl->getLocation();
+ }
//!
//! \brief Set the storage location of a tensor
@@ -644,33 +331,48 @@ public:
//!
//! \see getLocation()
//!
- virtual void setLocation(TensorLocation location) TRTNOEXCEPT = 0;
+ void setLocation(TensorLocation location) noexcept
+ {
+ mImpl->setLocation(location);
+ }
//!
//! \brief Query whether dynamic range is set.
//!
//! \return True if dynamic range is set, false otherwise.
//!
- virtual bool dynamicRangeIsSet() const TRTNOEXCEPT = 0;
+ bool dynamicRangeIsSet() const noexcept
+ {
+ return mImpl->dynamicRangeIsSet();
+ }
//!
//! \brief Undo effect of setDynamicRange.
//!
- virtual void resetDynamicRange() TRTNOEXCEPT = 0;
+ void resetDynamicRange() noexcept
+ {
+ mImpl->resetDynamicRange();
+ }
//!
//! \brief Get minimum of dynamic range.
//!
//! \return Minimum of dynamic range, or quiet NaN if range was not set.
//!
- virtual float getDynamicRangeMin() const TRTNOEXCEPT = 0;
+ float getDynamicRangeMin() const noexcept
+ {
+ return mImpl->getDynamicRangeMin();
+ }
//!
//! \brief Get maximum of dynamic range.
//!
//! \return Maximum of dynamic range, or quiet NaN if range was not set.
//!
- virtual float getDynamicRangeMax() const TRTNOEXCEPT = 0;
+ float getDynamicRangeMax() const noexcept
+ {
+ return mImpl->getDynamicRangeMax();
+ }
//!
//! \brief Set allowed formats for this tensor. By default all formats are allowed.
@@ -686,7 +388,10 @@ public:
//! \see ITensor::getAllowedFormats()
//! \see TensorFormats
//!
- virtual void setAllowedFormats(TensorFormats formats) TRTNOEXCEPT = 0;
+ void setAllowedFormats(TensorFormats formats) noexcept
+ {
+ mImpl->setAllowedFormats(formats);
+ }
//!
//! \brief Get a bitmask of TensorFormat values that the tensor supports.
@@ -696,7 +401,10 @@ public:
//!
//! \see ITensor::setAllowedFormats()
//!
- virtual TensorFormats getAllowedFormats() const TRTNOEXCEPT = 0;
+ TensorFormats getAllowedFormats() const noexcept
+ {
+ return mImpl->getAllowedFormats();
+ }
//!
//! \brief Whether the tensor is a shape tensor.
@@ -727,7 +435,10 @@ public:
//!
//! \see INetworkDefinition::markOutputForShapes(), ICudaEngine::isShapeBinding()
//!
- virtual bool isShapeTensor() const TRTNOEXCEPT = 0;
+ bool isShapeTensor() const noexcept
+ {
+ return mImpl->isShapeTensor();
+ }
//!
//! \brief Whether the tensor is an execution tensor.
@@ -747,7 +458,14 @@ public:
//! In that case, only its dimensions need to be set at runtime and a nullptr
//! can be passed instead of a pointer to its contents.
//!
- virtual bool isExecutionTensor() const TRTNOEXCEPT = 0;
+ bool isExecutionTensor() const noexcept
+ {
+ return mImpl->isExecutionTensor();
+ }
+
+protected:
+ apiv::VTensor* mImpl;
+ virtual ~ITensor() noexcept = default;
};
//!
@@ -757,7 +475,7 @@ public:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class ILayer
+class ILayer : public INoCopy
{
public:
//!
@@ -765,7 +483,10 @@ public:
//!
//! \see LayerType
//!
- virtual LayerType getType() const TRTNOEXCEPT = 0;
+ LayerType getType() const noexcept
+ {
+ return mLayer->getType();
+ }
//!
//! \brief Set the name of a layer.
@@ -774,7 +495,10 @@ public:
//!
//! \see getName()
//!
- virtual void setName(const char* name) TRTNOEXCEPT = 0;
+ void setName(const char* name) noexcept
+ {
+ mLayer->setName(name);
+ }
//!
//! \brief Return the name of a layer.
@@ -782,12 +506,18 @@ public:
//! \see setName()
//!
- virtual const char* getName() const TRTNOEXCEPT = 0;
+ const char* getName() const noexcept
+ {
+ return mLayer->getName();
+ }
//!
//! \brief Get the number of inputs of a layer.
//!
- virtual int32_t getNbInputs() const TRTNOEXCEPT = 0;
+ int32_t getNbInputs() const noexcept
+ {
+ return mLayer->getNbInputs();
+ }
//!
//! \brief Get the layer input corresponding to the given index.
@@ -795,38 +525,48 @@ public:
//! \param index The index of the input tensor.
//!
//! \return The input tensor, or nullptr if the index is out of range or the tensor is optional
- //! (\ref ISliceLayer, \ref IRNNLayer and \ref IRNNv2Layer).
+ //! (\ref ISliceLayer and \ref IRNNv2Layer).
//!
- virtual ITensor* getInput(int32_t index) const TRTNOEXCEPT = 0;
+ ITensor* getInput(int32_t index) const noexcept
+ {
+ return mLayer->getInput(index);
+ }
//!
//! \brief Get the number of outputs of a layer.
//!
- virtual int32_t getNbOutputs() const TRTNOEXCEPT = 0;
+ int32_t getNbOutputs() const noexcept
+ {
+ return mLayer->getNbOutputs();
+ }
//!
//! \brief Get the layer output corresponding to the given index.
//!
//! \return The indexed output tensor, or nullptr if the index is out of range or the tensor is optional
- //! (\ref IRNNLayer and \ref IRNNv2Layer).
+ //! (\ref IRNNv2Layer).
//!
- virtual ITensor* getOutput(int32_t index) const TRTNOEXCEPT = 0;
+ ITensor* getOutput(int32_t index) const noexcept
+ {
+ return mLayer->getOutput(index);
+ }
//!
- //! \brief Replace an input of this layer with a specific tensor
- //!
- //! \param index the index of the input to modify.
- //! \param tensor the new input tensor
- //
- //! Except for IShuffleLayer, ISliceLayer, IResizeLayer and ILoopOutputLayer, this method cannot change the number
- //! of inputs to a layer. The index argument must be less than the value of getNbInputs().
- //!
- //! See overloaded setInput() comments for the layers special behavior.
+ //! \brief Replace an input of this layer with a specific tensor.
//!
//! \param index the index of the input to modify.
//! \param tensor the new input tensor
//!
- virtual void setInput(int32_t index, ITensor& tensor) TRTNOEXCEPT = 0;
+ //! Except for IFillLayer, ILoopOutputLayer, IResizeLayer, IShuffleLayer, and ISliceLayer,
+ //! this method cannot change the number of inputs to a layer. The index argument must be
+ //! less than the value of getNbInputs().
+ //!
+ //! See comments for overloads of setInput() for layers with special behavior.
+ //!
+ void setInput(int32_t index, ITensor& tensor) noexcept
+ {
+ return mLayer->setInput(index, tensor);
+ }
//!
//! \brief Set the computational precision of this layer
@@ -839,11 +579,14 @@ public:
//! computational precision and layer input type based on performance considerations and the flags specified to the
//! builder.
//!
- //! \param precision the computational precision.
+ //! \param dataType the computational precision.
//!
//! \see getPrecision() precisionIsSet() resetPrecision()
-
- virtual void setPrecision(DataType dataType) TRTNOEXCEPT = 0;
+ //!
+ void setPrecision(DataType dataType) noexcept
+ {
+ mLayer->setPrecision(dataType);
+ }
//!
//! \brief get the computational precision of this layer
@@ -851,8 +594,11 @@ public:
//! \return the computational precision
//!
//! \see setPrecision() precisionIsSet() resetPrecision()
-
- virtual DataType getPrecision() const TRTNOEXCEPT = 0;
+ //!
+ DataType getPrecision() const noexcept
+ {
+ return mLayer->getPrecision();
+ }
//!
//! \brief whether the computational precision has been set for this layer
@@ -860,15 +606,21 @@ public:
//! \return whether the computational precision has been explicitly set
//!
//! \see setPrecision() getPrecision() resetPrecision()
-
- virtual bool precisionIsSet() const TRTNOEXCEPT = 0;
+ //!
+ bool precisionIsSet() const noexcept
+ {
+ return mLayer->precisionIsSet();
+ }
//!
//! \brief reset the computational precision for this layer
//!
//! \see setPrecision() getPrecision() precisionIsSet()
-
- virtual void resetPrecision() TRTNOEXCEPT = 0;
+ //!
+ void resetPrecision() noexcept
+ {
+ mLayer->resetPrecision();
+ }
//!
//! \brief Set the output type of this layer
@@ -895,8 +647,11 @@ public:
//! \param dataType the type of the output
//!
//! \see getOutputType() outputTypeIsSet() resetOutputType()
-
- virtual void setOutputType(int32_t index, DataType dataType) TRTNOEXCEPT = 0;
+ //!
+ void setOutputType(int32_t index, DataType dataType) noexcept
+ {
+ mLayer->setOutputType(index, dataType);
+ }
//!
//! \brief get the output type of this layer
@@ -906,8 +661,11 @@ public:
//! unless the output type is inherently DataType::kINT32.
//!
//! \see getOutputType() outputTypeIsSet() resetOutputType()
-
- virtual DataType getOutputType(int32_t index) const TRTNOEXCEPT = 0;
+ //!
+ DataType getOutputType(int32_t index) const noexcept
+ {
+ return mLayer->getOutputType(index);
+ }
//!
//! \brief whether the output type has been set for this layer
@@ -916,8 +674,11 @@ public:
//! \return whether the output type has been explicitly set
//!
//! \see setOutputType() getOutputType() resetOutputType()
-
- virtual bool outputTypeIsSet(int32_t index) const TRTNOEXCEPT = 0;
+ //!
+ bool outputTypeIsSet(int32_t index) const noexcept
+ {
+ return mLayer->outputTypeIsSet(index);
+ }
//!
//! \brief reset the output type for this layer
@@ -925,11 +686,15 @@ public:
//! \param index the index of the output
//!
//! \see setOutputType() getOutputType() outputTypeIsSet()
-
- virtual void resetOutputType(int32_t index) TRTNOEXCEPT = 0;
+ //!
+ void resetOutputType(int32_t index) noexcept
+ {
+ return mLayer->resetOutputType(index);
+ }
protected:
- virtual ~ILayer() {}
+ virtual ~ILayer() noexcept = default;
+ apiv::VLayer* mLayer;
};
//!
@@ -1164,12 +929,15 @@ enum class PaddingMode : int32_t
kCAFFE_ROUND_UP = 5 //!< Use CAFFE padding, rounding output size up, uses prePadding value.
};
+namespace impl
+{
//! Maximum number of elements in PaddingMode enum. \see PaddingMode
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 6;
-}
+ static constexpr int32_t kVALUE = 6;
+};
+} // namespace impl
//!
//! \class IConvolutionLayer
@@ -1195,7 +963,10 @@ public:
//!
//! \deprecated Superseded by setKernelSizeNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setKernelSize(DimsHW kernelSize) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setKernelSize(DimsHW kernelSize) noexcept
+ {
+ mImpl->setKernelSize(kernelSize);
+ }
//!
//! \brief Get the HW kernel size of the convolution.
@@ -1204,7 +975,10 @@ public:
//!
//! \deprecated Superseded by getKernelSizeNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getKernelSize() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getKernelSize() const noexcept
+ {
+ return mImpl->getKernelSize();
+ }
//!
//! \brief Set the number of output maps for the convolution.
@@ -1213,14 +987,20 @@ public:
//!
//! \see getNbOutputMaps()
//!
- virtual void setNbOutputMaps(int32_t nbOutputMaps) TRTNOEXCEPT = 0;
+ void setNbOutputMaps(int32_t nbOutputMaps) noexcept
+ {
+ mImpl->setNbOutputMaps(nbOutputMaps);
+ }
//!
//! \brief Get the number of output maps for the convolution.
//!
//! \see setNbOutputMaps()
//!
- virtual int32_t getNbOutputMaps() const TRTNOEXCEPT = 0;
+ int32_t getNbOutputMaps() const noexcept
+ {
+ return mImpl->getNbOutputMaps();
+ }
//!
//! \brief Get the stride of the convolution.
@@ -1233,14 +1013,20 @@ public:
//!
//! \deprecated Superseded by setStrideNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setStride(DimsHW stride) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setStride(DimsHW stride) noexcept
+ {
+ mImpl->setStride(stride);
+ }
//!
//! \brief Get the stride of the convolution.
//!
//! \deprecated Superseded by getStrideNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getStride() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getStride() const noexcept
+ {
+ return mImpl->getStride();
+ }
//!
//! \brief Set the padding of the convolution.
@@ -1257,7 +1043,10 @@ public:
//!
//! \deprecated Superseded by setPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setPadding(DimsHW padding) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setPadding(DimsHW padding) noexcept
+ {
+ return mImpl->setPadding(padding);
+ }
//!
//! \brief Get the padding of the convolution. If the padding is asymmetric, the pre-padding is returned.
@@ -1266,7 +1055,10 @@ public:
//!
//! \deprecated Superseded by getPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getPadding() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getPadding() const noexcept
+ {
+ return mImpl->getPadding();
+ }
//!
//! \brief Set the number of groups for a convolution.
@@ -1283,14 +1075,20 @@ public:
//!
//! \see getNbGroups()
//!
- virtual void setNbGroups(int32_t nbGroups) TRTNOEXCEPT = 0;
+ void setNbGroups(int32_t nbGroups) noexcept
+ {
+ mImpl->setNbGroups(nbGroups);
+ }
//!
//! \brief Get the number of groups of the convolution.
//!
//! \see setNbGroups()
//!
- virtual int32_t getNbGroups() const TRTNOEXCEPT = 0;
+ int32_t getNbGroups() const noexcept
+ {
+ return mImpl->getNbGroups();
+ }
//!
//! \brief Set the kernel weights for the convolution.
@@ -1301,14 +1099,20 @@ public:
//!
//! \see getKernelWeights()
//!
- virtual void setKernelWeights(Weights weights) TRTNOEXCEPT = 0;
+ void setKernelWeights(Weights weights) noexcept
+ {
+ mImpl->setKernelWeights(weights);
+ }
//!
//! \brief Get the kernel weights of the convolution.
//!
//! \see setKernelWeights()
//!
- virtual Weights getKernelWeights() const TRTNOEXCEPT = 0;
+ Weights getKernelWeights() const noexcept
+ {
+ return mImpl->getKernelWeights();
+ }
//!
//! \brief Set the bias weights for the convolution.
@@ -1320,14 +1124,20 @@ public:
//!
//! \see getBiasWeights()
//!
- virtual void setBiasWeights(Weights weights) TRTNOEXCEPT = 0;
+ void setBiasWeights(Weights weights) noexcept
+ {
+ mImpl->setBiasWeights(weights);
+ }
//!
//! \brief Get the bias weights for the convolution.
//!
//! \see setBiasWeights()
//!
- virtual Weights getBiasWeights() const TRTNOEXCEPT = 0;
+ Weights getBiasWeights() const noexcept
+ {
+ return mImpl->getBiasWeights();
+ }
//!
//! \brief Set the dilation for a convolution.
@@ -1340,7 +1150,10 @@ public:
//!
//! \deprecated Superseded by setDilationNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setDilation(DimsHW dilation) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setDilation(DimsHW dilation) noexcept
+ {
+ return mImpl->setDilation(dilation);
+ }
//!
//! \brief Get the dilation for a convolution.
@@ -1349,12 +1162,11 @@ public:
//!
//! \deprecated Superseded by getDilationNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getDilation() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getDilation() const noexcept
+ {
+ return mImpl->getDilation();
+ }
-protected:
- virtual ~IConvolutionLayer() {}
-
-public:
//!
//! \brief Set the multi-dimension pre-padding of the convolution.
//!
@@ -1367,14 +1179,20 @@ public:
//!
//! \see getPrePadding()
//!
- virtual void setPrePadding(Dims padding) TRTNOEXCEPT = 0;
+ void setPrePadding(Dims padding) noexcept
+ {
+ mImpl->setPrePadding(padding);
+ }
//!
//! \brief Get the pre-padding.
//!
//! \see setPrePadding()
//!
- virtual Dims getPrePadding() const TRTNOEXCEPT = 0;
+ Dims getPrePadding() const noexcept
+ {
+ return mImpl->getPrePadding();
+ }
//!
//! \brief Set the multi-dimension post-padding of the convolution.
@@ -1388,14 +1206,20 @@ public:
//!
//! \see getPostPadding()
//!
- virtual void setPostPadding(Dims padding) TRTNOEXCEPT = 0;
+ void setPostPadding(Dims padding) noexcept
+ {
+ mImpl->setPostPadding(padding);
+ }
//!
//! \brief Get the post-padding.
//!
//! \see setPostPadding()
//!
- virtual Dims getPostPadding() const TRTNOEXCEPT = 0;
+ Dims getPostPadding() const noexcept
+ {
+ return mImpl->getPostPadding();
+ }
//!
//! \brief Set the padding mode.
@@ -1406,7 +1230,10 @@ public:
//!
//! \see getPaddingMode()
//!
- virtual void setPaddingMode(PaddingMode paddingMode) TRTNOEXCEPT = 0;
+ void setPaddingMode(PaddingMode paddingMode) noexcept
+ {
+ mImpl->setPaddingMode(paddingMode);
+ }
//!
//! \brief Get the padding mode.
@@ -1415,7 +1242,10 @@ public:
//!
//! \see setPaddingMode()
//!
- virtual PaddingMode getPaddingMode() const TRTNOEXCEPT = 0;
+ PaddingMode getPaddingMode() const noexcept
+ {
+ return mImpl->getPaddingMode();
+ }
//!
//! \brief Set the multi-dimension kernel size of the convolution.
@@ -1425,32 +1255,45 @@ public:
//!
//! \see getKernelSizeNd()
//!
- virtual void setKernelSizeNd(Dims kernelSize) TRTNOEXCEPT = 0;
+ void setKernelSizeNd(Dims kernelSize) noexcept
+ {
+ mImpl->setKernelSizeNd(kernelSize);
+ }
//!
//! \brief Get the multi-dimension kernel size of the convolution.
//!
//! \see setKernelSizeNd()
//!
- virtual Dims getKernelSizeNd() const TRTNOEXCEPT = 0;
+ Dims getKernelSizeNd() const noexcept
+ {
+ return mImpl->getKernelSizeNd();
+ }
//!
//! \brief Set the multi-dimension stride of the convolution.
//!
//! Default: (1, 1, ..., 1)
//!
- //! If executing this layer on DLA, only support 2D stride, both height and width of stride must be in the range [1,8].
+ //! If executing this layer on DLA, only support 2D stride, both height and width of stride must be in the range
+ //! [1,8].
//!
//! \see getStrideNd() setStride() getStride()
//!
- virtual void setStrideNd(Dims stride) TRTNOEXCEPT = 0;
+ void setStrideNd(Dims stride) noexcept
+ {
+ mImpl->setStrideNd(stride);
+ }
//!
//! \brief Get the multi-dimension stride of the convolution.
//!
//! \see setStrideNd()
//!
- virtual Dims getStrideNd() const TRTNOEXCEPT = 0;
+ Dims getStrideNd() const noexcept
+ {
+ return mImpl->getStrideNd();
+ }
//!
//! \brief Set the multi-dimension padding of the convolution.
@@ -1465,7 +1308,10 @@ public:
//!
//! \see getPaddingNd() setPadding() getPadding()
//!
- virtual void setPaddingNd(Dims padding) TRTNOEXCEPT = 0;
+ void setPaddingNd(Dims padding) noexcept
+ {
+ mImpl->setPaddingNd(padding);
+ }
//!
//! \brief Get the multi-dimension padding of the convolution.
@@ -1474,7 +1320,10 @@ public:
//!
//! \see setPaddingNd()
//!
- virtual Dims getPaddingNd() const TRTNOEXCEPT = 0;
+ Dims getPaddingNd() const noexcept
+ {
+ return mImpl->getPaddingNd();
+ }
//!
//! \brief Set the multi-dimension dilation of the convolution.
@@ -1485,14 +1334,20 @@ public:
//!
//! \see getDilation()
//!
- virtual void setDilationNd(Dims dilation) TRTNOEXCEPT = 0;
+ void setDilationNd(Dims dilation) noexcept
+ {
+ mImpl->setDilationNd(dilation);
+ }
//!
//! \brief Get the multi-dimension dilation of the convolution.
//!
//! \see setDilation()
//!
- virtual Dims getDilationNd() const TRTNOEXCEPT = 0;
+ Dims getDilationNd() const noexcept
+ {
+ return mImpl->getDilationNd();
+ }
//!
//! \brief Append or replace an input of this layer with a specific tensor
@@ -1512,7 +1367,11 @@ public:
//! - 1: The kernel weights tensor (a constant tensor).
//!
//! If this function is called with a value greater than 0, then the function getNbInputs() changes
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ using ILayer::setInput;
+
+protected:
+ virtual ~IConvolutionLayer() noexcept = default;
+ apiv::VConvolutionLayer* mImpl;
};
//! \class IFullyConnectedLayer
@@ -1554,28 +1413,40 @@ public:
//!
//! \see getNbOutputChannels()
//!
- virtual void setNbOutputChannels(int32_t nbOutputs) TRTNOEXCEPT = 0;
+ void setNbOutputChannels(int32_t nbOutputs) noexcept
+ {
+ mImpl->setNbOutputChannels(nbOutputs);
+ }
//!
//! \brief Get the number of output channels `K` from the fully connected layer.
//!
//! \see setNbOutputChannels()
//!
- virtual int32_t getNbOutputChannels() const TRTNOEXCEPT = 0;
+ int32_t getNbOutputChannels() const noexcept
+ {
+ return mImpl->getNbOutputChannels();
+ }
//!
//! \brief Set the kernel weights, given as a `KxC` matrix in row-major order.
//!
//! \see getKernelWeights()
//!
- virtual void setKernelWeights(Weights weights) TRTNOEXCEPT = 0;
+ void setKernelWeights(Weights weights) noexcept
+ {
+ mImpl->setKernelWeights(weights);
+ }
//!
//! \brief Get the kernel weights.
//!
//! \see setKernelWeights()
//!
- virtual Weights getKernelWeights() const TRTNOEXCEPT = 0;
+ Weights getKernelWeights() const noexcept
+ {
+ return mImpl->getKernelWeights();
+ }
//!
//! \brief Set the bias weights.
@@ -1584,19 +1455,21 @@ public:
//!
//! \see getBiasWeightsWeights()
//!
- virtual void setBiasWeights(Weights weights) TRTNOEXCEPT = 0;
+ void setBiasWeights(Weights weights) noexcept
+ {
+ mImpl->setBiasWeights(weights);
+ }
//!
//! \brief Get the bias weights.
//!
//! \see setBiasWeightsWeights()
//!
- virtual Weights getBiasWeights() const TRTNOEXCEPT = 0;
+ Weights getBiasWeights() const noexcept
+ {
+ return mImpl->getBiasWeights();
+ }
-protected:
- virtual ~IFullyConnectedLayer() {}
-
-public:
//!
//! \brief Append or replace an input of this layer with a specific tensor
//!
@@ -1613,7 +1486,11 @@ public:
//! - 1: The kernel weights tensor (a constant tensor).
//!
//! If this function is called with a value greater than 0, then the function getNbInputs() changes
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ using ILayer::setInput;
+
+protected:
+ virtual ~IFullyConnectedLayer() noexcept = default;
+ apiv::VFullyConnectedLayer* mImpl;
};
//!
@@ -1637,18 +1514,21 @@ public:
//!
//! \see getActivationType(), ActivationType
//!
- virtual void setActivationType(ActivationType type) TRTNOEXCEPT = 0;
+ void setActivationType(ActivationType type) noexcept
+ {
+ mImpl->setActivationType(type);
+ }
//!
//! \brief Get the type of activation to be performed.
//!
//! \see setActivationType(), ActivationType
//!
- virtual ActivationType getActivationType() const TRTNOEXCEPT = 0;
+ ActivationType getActivationType() const noexcept
+ {
+ return mImpl->getActivationType();
+ }
-protected:
- virtual ~IActivationLayer() {}
-public:
//!
//! \brief Set the alpha parameter (must be finite).
//!
@@ -1659,7 +1539,10 @@ public:
//! It is ignored by the other activations.
//!
//! \see getAlpha(), setBeta()
- virtual void setAlpha(float alpha) TRTNOEXCEPT = 0;
+ void setAlpha(float alpha) noexcept
+ {
+ mImpl->setAlpha(alpha);
+ }
//!
//! \brief Set the beta parameter (must be finite).
@@ -1670,19 +1553,32 @@ public:
//! It is ignored by the other activations.
//!
//! \see getBeta(), setAlpha()
- virtual void setBeta(float beta) TRTNOEXCEPT = 0;
+ void setBeta(float beta) noexcept
+ {
+ mImpl->setBeta(beta);
+ }
//!
//! \brief Get the alpha parameter.
//!
//! \see getBeta(), setAlpha()
- virtual float getAlpha() const TRTNOEXCEPT = 0;
+ float getAlpha() const noexcept
+ {
+ return mImpl->getAlpha();
+ }
//!
//! \brief Get the beta parameter.
//!
//! \see getAlpha(), setBeta()
- virtual float getBeta() const TRTNOEXCEPT = 0;
+ float getBeta() const noexcept
+ {
+ return mImpl->getBeta();
+ }
+
+protected:
+ virtual ~IActivationLayer() noexcept = default;
+ apiv::VActivationLayer* mImpl;
};
//!
@@ -1697,12 +1593,15 @@ enum class PoolingType : int32_t
kMAX_AVERAGE_BLEND = 2 // Blending between max and average pooling: (1-blendFactor)*maxPool + blendFactor*avgPool
};
+namespace impl
+{
//! Maximum number of elements in PoolingType enum. \see PoolingType
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 3;
-}
+ static constexpr int32_t kVALUE = 3;
+};
+} // namespace impl
//! \class IPoolingLayer
//!
@@ -1725,14 +1624,20 @@ public:
//!
//! \see getPoolingType(), PoolingType
//!
- virtual void setPoolingType(PoolingType type) TRTNOEXCEPT = 0;
+ void setPoolingType(PoolingType type) noexcept
+ {
+ mImpl->setPoolingType(type);
+ }
//!
//! \brief Get the type of activation to be performed.
//!
//! \see setPoolingType(), PoolingType
//!
- virtual PoolingType getPoolingType() const TRTNOEXCEPT = 0;
+ PoolingType getPoolingType() const noexcept
+ {
+ return mImpl->getPoolingType();
+ }
//!
//! \brief Set the window size for pooling.
@@ -1743,7 +1648,10 @@ public:
//!
//! \deprecated Superseded by setWindowSizeNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setWindowSize(DimsHW windowSize) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setWindowSize(DimsHW windowSize) noexcept
+ {
+ mImpl->setWindowSize(windowSize);
+ }
//!
//! \brief Get the window size for pooling.
@@ -1752,7 +1660,10 @@ public:
//!
//! \deprecated Superseded by getWindowSizeNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getWindowSize() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getWindowSize() const noexcept
+ {
+ return mImpl->getWindowSize();
+ }
//!
//! \brief Set the stride for pooling.
@@ -1765,7 +1676,10 @@ public:
//!
//! \deprecated Superseded by setStrideNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setStride(DimsHW stride) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setStride(DimsHW stride) noexcept
+ {
+ mImpl->setStride(stride);
+ }
//!
//! \brief Get the stride for pooling.
@@ -1774,7 +1688,10 @@ public:
//!
//! \deprecated Superseded by getStrideNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getStride() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getStride() const noexcept
+ {
+ return mImpl->getStride();
+ }
//!
//! \brief Set the padding for pooling.
@@ -1787,7 +1704,10 @@ public:
//!
//! \deprecated Superseded by setPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setPadding(DimsHW padding) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setPadding(DimsHW padding) noexcept
+ {
+ mImpl->setPadding(padding);
+ }
//!
//! \brief Get the padding for pooling.
@@ -1798,7 +1718,10 @@ public:
//!
//! \deprecated Superseded by getPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getPadding() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getPadding() const noexcept
+ {
+ return mImpl->getPadding();
+ }
//!
//! \brief Set the blending factor for the max_average_blend mode:
@@ -1810,7 +1733,10 @@ public:
//!
//! \see getBlendFactor()
//!
- virtual void setBlendFactor(float blendFactor) TRTNOEXCEPT = 0;
+ void setBlendFactor(float blendFactor) noexcept
+ {
+ mImpl->setBlendFactor(blendFactor);
+ }
//!
//! \brief Get the blending factor for the max_average_blend mode:
@@ -1820,35 +1746,39 @@ public:
//!
//! \see setBlendFactor()
//!
- virtual float getBlendFactor() const TRTNOEXCEPT = 0;
+ float getBlendFactor() const noexcept
+ {
+ return mImpl->getBlendFactor();
+ }
//!
//! \brief Set whether average pooling uses as a denominator the overlap area between the window
//! and the unpadded input.
//! If this is not set, the denominator is the overlap between the pooling window and the padded input.
//!
- //! If executing this layer on the DLA, only inclusive padding is supported.
- //!
//! Default: true
//!
- //! If executing this layer on the DLA, this is ignored as the DLA does not support exclusive padding.
+ //! \note DLA supports only inclusive padding, and thus when executing this layer on DLA, this must be explicitly
+ //! set to false.
//!
//! \see getAverageCountExcludesPadding()
//!
- virtual void setAverageCountExcludesPadding(bool exclusive) TRTNOEXCEPT = 0;
+ void setAverageCountExcludesPadding(bool exclusive) noexcept
+ {
+ mImpl->setAverageCountExcludesPadding(exclusive);
+ }
//!
- //! \brief Get whether exclusive pooling uses as a denominator the overlap area betwen the window
+ //! \brief Get whether average pooling uses as a denominator the overlap area between the window
//! and the unpadded input.
//!
//! \see setAverageCountExcludesPadding()
//!
- virtual bool getAverageCountExcludesPadding() const TRTNOEXCEPT = 0;
+ bool getAverageCountExcludesPadding() const noexcept
+ {
+ return mImpl->getAverageCountExcludesPadding();
+ }
-protected:
- virtual ~IPoolingLayer() {}
-
-public:
//!
//! \brief Set the multi-dimension pre-padding for pooling.
//!
@@ -1857,18 +1787,25 @@ public:
//!
//! Default: (0, 0, ..., 0)
//!
- //! If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,7].
+ //! If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range
+ //! [0,7].
//!
//! \see getPrePadding()
//!
- virtual void setPrePadding(Dims padding) TRTNOEXCEPT = 0;
+ void setPrePadding(Dims padding) noexcept
+ {
+ mImpl->setPrePadding(padding);
+ }
//!
//! \brief Get the pre-padding.
//!
//! \see setPrePadding()
//!
- virtual Dims getPrePadding() const TRTNOEXCEPT = 0;
+ Dims getPrePadding() const noexcept
+ {
+ return mImpl->getPrePadding();
+ }
//!
//! \brief Set the multi-dimension post-padding for pooling.
@@ -1878,18 +1815,25 @@ public:
//!
//! Default: (0, 0, ..., 0)
//!
- //! If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,7].
+ //! If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range
+ //! [0,7].
//!
//! \see getPostPadding()
//!
- virtual void setPostPadding(Dims padding) TRTNOEXCEPT = 0;
+ void setPostPadding(Dims padding) noexcept
+ {
+ mImpl->setPostPadding(padding);
+ }
//!
//! \brief Get the padding.
//!
- //! \see setPadding()
+ //! \see setPostPadding()
//!
- virtual Dims getPostPadding() const TRTNOEXCEPT = 0;
+ Dims getPostPadding() const noexcept
+ {
+ return mImpl->getPostPadding();
+ }
//!
//! \brief Set the padding mode.
@@ -1899,7 +1843,10 @@ public:
//! Default: kEXPLICIT_ROUND_DOWN
//!
//! \see getPaddingMode()
- virtual void setPaddingMode(PaddingMode paddingMode) TRTNOEXCEPT = 0;
+ void setPaddingMode(PaddingMode paddingMode) noexcept
+ {
+ mImpl->setPaddingMode(paddingMode);
+ }
//!
//! \brief Get the padding mode.
@@ -1907,41 +1854,58 @@ public:
//! Default: kEXPLICIT_ROUND_DOWN
//!
//! \see setPaddingMode()
- virtual PaddingMode getPaddingMode() const TRTNOEXCEPT = 0;
+ PaddingMode getPaddingMode() const noexcept
+ {
+ return mImpl->getPaddingMode();
+ }
//!
//! \brief Set the multi-dimension window size for pooling.
//!
- //! If executing this layer on DLA, only support 2D window size, both height and width of window size must be in the range [1,8].
+ //! If executing this layer on DLA, only support 2D window size, both height and width of window size must be in the
+ //! range [1,8].
//!
//! \see getWindowSizeNd() setWindowSize() getWindowSize()
//!
- virtual void setWindowSizeNd(Dims windowSize) TRTNOEXCEPT = 0;
+ void setWindowSizeNd(Dims windowSize) noexcept
+ {
+ mImpl->setWindowSizeNd(windowSize);
+ }
//!
//! \brief Get the multi-dimension window size for pooling.
//!
//! \see setWindowSizeNd()
//!
- virtual Dims getWindowSizeNd() const TRTNOEXCEPT = 0;
+ Dims getWindowSizeNd() const noexcept
+ {
+ return mImpl->getWindowSizeNd();
+ }
//!
//! \brief Set the multi-dimension stride for pooling.
//!
//! Default: (1, 1, ..., 1)
//!
- //! If executing this layer on DLA, only support 2D stride, both height and width of stride must be in the range [1,16].
+ //! If executing this layer on DLA, only support 2D stride, both height and width of stride must be in the range
+ //! [1,16].
//!
//! \see getStrideNd() setStride() getStride()
//!
- virtual void setStrideNd(Dims stride) TRTNOEXCEPT = 0;
+ void setStrideNd(Dims stride) noexcept
+ {
+ mImpl->setStrideNd(stride);
+ }
//!
//! \brief Get the multi-dimension stride for pooling.
//!
//! \see setStrideNd()
//!
- virtual Dims getStrideNd() const TRTNOEXCEPT = 0;
+ Dims getStrideNd() const noexcept
+ {
+ return mImpl->getStrideNd();
+ }
//!
//! \brief Set the multi-dimension padding for pooling.
@@ -1952,11 +1916,15 @@ public:
//!
//! Default: (0, 0, ..., 0)
//!
- //! If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,7].
+ //! If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range
+ //! [0,7].
//!
//! \see getPaddingNd() setPadding() getPadding()
//!
- virtual void setPaddingNd(Dims padding) TRTNOEXCEPT = 0;
+ void setPaddingNd(Dims padding) noexcept
+ {
+ mImpl->setPaddingNd(padding);
+ }
//!
//! \brief Get the multi-dimension padding for pooling.
@@ -1965,7 +1933,14 @@ public:
//!
//! \see setPaddingNd()
//!
- virtual Dims getPaddingNd() const TRTNOEXCEPT = 0;
+ Dims getPaddingNd() const noexcept
+ {
+ return mImpl->getPaddingNd();
+ }
+
+protected:
+ virtual ~IPoolingLayer() noexcept = default;
+ apiv::VPoolingLayer* mImpl;
};
//!
@@ -1989,14 +1964,20 @@ public:
//!
//! \see setWindowStride()
//!
- virtual void setWindowSize(int32_t windowSize) TRTNOEXCEPT = 0;
+ void setWindowSize(int32_t windowSize) noexcept
+ {
+ mImpl->setWindowSize(windowSize);
+ }
//!
//! \brief Get the LRN window size.
//!
//! \see getWindowStride()
//!
- virtual int32_t getWindowSize() const TRTNOEXCEPT = 0;
+ int32_t getWindowSize() const noexcept
+ {
+ return mImpl->getWindowSize();
+ }
//!
//! \brief Set the LRN alpha value.
@@ -2004,14 +1985,20 @@ public:
//! The valid range is [-1e20, 1e20].
//! \see getAlpha()
//!
- virtual void setAlpha(float alpha) TRTNOEXCEPT = 0;
+ void setAlpha(float alpha) noexcept
+ {
+ mImpl->setAlpha(alpha);
+ }
//!
//! \brief Get the LRN alpha value.
//!
//! \see setAlpha()
//!
- virtual float getAlpha() const TRTNOEXCEPT = 0;
+ float getAlpha() const noexcept
+ {
+ return mImpl->getAlpha();
+ }
//!
//! \brief Set the LRN beta value.
@@ -2019,14 +2006,20 @@ public:
//! The valid range is [0.01, 1e5f].
//! \see getBeta()
//!
- virtual void setBeta(float beta) TRTNOEXCEPT = 0;
+ void setBeta(float beta) noexcept
+ {
+ mImpl->setBeta(beta);
+ }
//!
//! \brief Get the LRN beta value.
//!
//! \see setBeta()
//!
- virtual float getBeta() const TRTNOEXCEPT = 0;
+ float getBeta() const noexcept
+ {
+ return mImpl->getBeta();
+ }
//!
//! \brief Set the LRN K value.
@@ -2034,17 +2027,24 @@ public:
//! The valid range is [1e-5, 1e10].
//! \see getK()
//!
- virtual void setK(float k) TRTNOEXCEPT = 0;
+ void setK(float k) noexcept
+ {
+ mImpl->setK(k);
+ }
//!
//! \brief Get the LRN K value.
//!
//! \see setK()
//!
- virtual float getK() const TRTNOEXCEPT = 0;
+ float getK() const noexcept
+ {
+ return mImpl->getK();
+ }
protected:
- virtual ~ILRNLayer() {}
+ virtual ~ILRNLayer() noexcept = default;
+ apiv::VLRNLayer* mImpl;
};
//!
@@ -2061,7 +2061,7 @@ enum class ScaleMode : int32_t
//! Maximum number of elements in ScaleMode enum. \see ScaleMode
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 3;
}
@@ -2082,7 +2082,11 @@ constexpr inline int32_t EnumMax()
//!
//! The output size is the same as the input size.
//!
-//! \note The input tensor for this layer is required to have a minimum of 3 dimensions.
+//! \note The input tensor for this layer is required to have a minimum of 3 dimensions in implicit batch mode
+//! and a minimum of 4 dimensions in explicit batch mode.
+//!
+//! A scale layer may be used as an INT8 quantization node in a graph, if the output is constrained to INT8 and
+//! the input to FP32. Quantization rounds ties to even, and clamps to [-128, 127].
//!
//! \see ScaleMode
//!
@@ -2096,74 +2100,120 @@ public:
//!
//! \see getMode()
//!
- virtual void setMode(ScaleMode mode) TRTNOEXCEPT = 0;
+ void setMode(ScaleMode mode) noexcept
+ {
+ mImpl->setMode(mode);
+ }
//!
//! \brief Get the scale mode.
//!
//! \see setMode()
//!
- virtual ScaleMode getMode() const TRTNOEXCEPT = 0;
+ ScaleMode getMode() const noexcept
+ {
+ return mImpl->getMode();
+ }
//!
//! \brief Set the shift value.
//!
//! \see getShift()
//!
- virtual void setShift(Weights shift) TRTNOEXCEPT = 0;
+ void setShift(Weights shift) noexcept
+ {
+ mImpl->setShift(shift);
+ }
//!
//! \brief Get the shift value.
//!
//! \see setShift()
//!
- virtual Weights getShift() const TRTNOEXCEPT = 0;
+ Weights getShift() const noexcept
+ {
+ return mImpl->getShift();
+ }
//!
//! \brief Set the scale value.
//!
//! \see getScale()
//!
- virtual void setScale(Weights scale) TRTNOEXCEPT = 0;
+ void setScale(Weights scale) noexcept
+ {
+ mImpl->setScale(scale);
+ }
//!
//! \brief Get the scale value.
//!
//! \see setScale()
//!
- virtual Weights getScale() const TRTNOEXCEPT = 0;
+ Weights getScale() const noexcept
+ {
+ return mImpl->getScale();
+ }
//!
//! \brief Set the power value.
//!
//! \see getPower()
//!
- virtual void setPower(Weights power) TRTNOEXCEPT = 0;
+ void setPower(Weights power) noexcept
+ {
+ mImpl->setPower(power);
+ }
//!
//! \brief Get the power value.
//!
//! \see setPower()
//!
- virtual Weights getPower() const TRTNOEXCEPT = 0;
+ Weights getPower() const noexcept
+ {
+ return mImpl->getPower();
+ }
-protected:
- virtual ~IScaleLayer() {}
-
-public:
//!
//! \brief Get the channel axis.
//!
- //! \return channelAxis parameter passed to addScaleNd()
+ //! \return channelAxis parameter passed to addScaleNd() or set by setChannelAxis()
//!
- //! The value is the index of the channel axis in the input tensor's dimensions. All dimensions
- //! after the channel axis are assumed to be spatial dimensions, and the only spatial dimensions
- //! in the tensor. The number of spatial dimensions is thus getDimensions().nbDims - channelAxis - 1.
- //! Supported numbers of spatial dimensions are 2 and 3 for 2d and 3d scale layers respectively.
+ //! The value is the index of the channel axis in the input tensor's dimensions.
+ //! Scaling happens along the channel axis when ScaleMode::kCHANNEL is enabled.
//!
//! \see addScaleNd()
//!
- virtual int32_t getChannelAxis() const TRTNOEXCEPT = 0;
+ int32_t getChannelAxis() const noexcept
+ {
+ return mImpl->getChannelAxis();
+ }
+
+ //!
+ //! \brief Set the channel axis.
+ //!
+ //! The value is the index of the channel axis in the input tensor's dimensions.
+ //!
+ //! For ScaleMode::kCHANNEL, there can be distinct scale, shift, and power weights for each channel coordinate.
+ //! For ScaleMode::kELEMENTWISE, there can be distinct scale, shift, and power weights for each combination of
+ //! coordinates from the channel axis and axes after it.
+ //!
+ //! For example, suppose the input tensor has dimensions [10,20,30,40] and the channel axis is 1.
+ //! Let [n,c,h,w] denote an input coordinate.
+ //! For ScaleMode::kCHANNEL, the scale, shift, and power weights are indexed by c.
+ //! For ScaleMode::kELEMENTWISE, the scale, shift, and power weights are indexed by [c,h,w].
+ //!
+ //! \see addScaleNd()
+ //!
+ void setChannelAxis(int32_t channelAxis) noexcept
+ {
+ mImpl->setChannelAxis(channelAxis);
+ }
+
+protected:
+ virtual ~IScaleLayer() noexcept = default;
+ apiv::VScaleLayer* mImpl;
};
//!
@@ -2179,8 +2229,6 @@ public:
//!
class ISoftMaxLayer : public ILayer
{
-protected:
- virtual ~ISoftMaxLayer() {}
public:
//!
//! \brief Set the axis along which softmax is computed. Currently, only one axis can be set.
@@ -2209,16 +2257,27 @@ public:
//! set bit 3 with explicit batch mode.
//!
//! \param axes The axis along which softmax is computed.
- //! Here axes is a bitmap. For example, when doing softmax along axis 0, bit 0 is set to 1, axes = 1 << axis = 1.
+ //! Here axes is a bitmap. For example, when doing softmax along axis 0, bit 0 is set to 1, axes = 1 << axis
+ //! = 1.
//!
- virtual void setAxes(uint32_t axes) TRTNOEXCEPT = 0;
+ void setAxes(uint32_t axes) noexcept
+ {
+ mImpl->setAxes(axes);
+ }
//!
//! \brief Get the axis along which softmax occurs.
//!
//! \see setAxes()
//!
- virtual uint32_t getAxes() const TRTNOEXCEPT = 0;
+ uint32_t getAxes() const noexcept
+ {
+ return mImpl->getAxes();
+ }
+
+protected:
+ virtual ~ISoftMaxLayer() noexcept = default;
+ apiv::VSoftMaxLayer* mImpl;
};
//!
@@ -2226,17 +2285,15 @@ public:
//!
//! \brief A concatenation layer in a network definition.
//!
-//! The output channel size is the sum of the channel sizes of the inputs.
-//! The other output sizes are the same as the other input sizes,
-//! which must all match.
+//! The output dimension along the concatenation axis is the sum of the corresponding input dimensions.
+//! Every other output dimension is the same as the corresponding dimension of the inputs.
+//!
+//! \warning All tensors must have the same dimensions except along the concatenation axis.
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
class IConcatenationLayer : public ILayer
{
-protected:
- virtual ~IConcatenationLayer() {}
-
public:
//!
//! \brief Set the axis along which concatenation occurs.
@@ -2248,14 +2305,24 @@ public:
//!
//! \param axis The axis along which concatenation occurs.
//!
- virtual void setAxis(int32_t axis) TRTNOEXCEPT = 0;
+ void setAxis(int32_t axis) noexcept
+ {
+ mImpl->setAxis(axis);
+ }
//!
//! \brief Get the axis along which concatenation occurs.
//!
//! \see setAxis()
//!
- virtual int32_t getAxis() const TRTNOEXCEPT = 0;
+ int32_t getAxis() const noexcept
+ {
+ return mImpl->getAxis();
+ }
+
+protected:
+ virtual ~IConcatenationLayer() noexcept = default;
+ apiv::VConcatenationLayer* mImpl;
};
//!
@@ -2263,8 +2330,6 @@ public:
//!
//! \brief A deconvolution layer in a network definition.
//!
-//! The output size is defined using the formula set by INetworkDefinition::setDeconvolutionOutputDimensionsFormula().
-//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
class IDeconvolutionLayer : public ILayer
@@ -2281,7 +2346,10 @@ public:
//!
//! \deprecated Superseded by setKernelSizeNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setKernelSize(DimsHW kernelSize) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setKernelSize(DimsHW kernelSize) noexcept
+ {
+ mImpl->setKernelSize(kernelSize);
+ }
//!
//! \brief Get the HW kernel size of the deconvolution.
@@ -2290,7 +2358,10 @@ public:
//!
//! \deprecated Superseded by getKernelSizeNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getKernelSize() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getKernelSize() const noexcept
+ {
+ return mImpl->getKernelSize();
+ }
//!
//! \brief Set the number of output feature maps for the deconvolution.
@@ -2299,14 +2370,20 @@ public:
//!
//! \see getNbOutputMaps()
//!
- virtual void setNbOutputMaps(int32_t nbOutputMaps) TRTNOEXCEPT = 0;
+ void setNbOutputMaps(int32_t nbOutputMaps) noexcept
+ {
+ mImpl->setNbOutputMaps(nbOutputMaps);
+ }
//!
//! \brief Get the number of output feature maps for the deconvolution.
//!
//! \see setNbOutputMaps()
//!
- virtual int32_t getNbOutputMaps() const TRTNOEXCEPT = 0;
+ int32_t getNbOutputMaps() const noexcept
+ {
+ return mImpl->getNbOutputMaps();
+ }
//!
//! \brief Get the stride of the deconvolution.
@@ -2319,7 +2396,10 @@ public:
//!
//! \deprecated Superseded by setStrideNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setStride(DimsHW stride) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setStride(DimsHW stride) noexcept
+ {
+ mImpl->setStride(stride);
+ }
//!
//! \brief Get the stride of the deconvolution.
@@ -2328,7 +2408,10 @@ public:
//!
//! \deprecated Superseded by getStrideNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getStride() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getStride() const noexcept
+ {
+ return mImpl->getStride();
+ }
//!
//! \brief Set the padding of the deconvolution.
@@ -2345,7 +2428,10 @@ public:
//!
//! \deprecated Superseded by setPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setPadding(DimsHW padding) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setPadding(DimsHW padding) noexcept
+ {
+ mImpl->setPadding(padding);
+ }
//!
//! \brief Get the padding of the deconvolution.
@@ -2356,7 +2442,10 @@ public:
//!
//! \deprecated Superseded by getPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getPadding() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getPadding() const noexcept
+ {
+ return mImpl->getPadding();
+ }
//!
//! \brief Set the number of groups for a deconvolution.
@@ -2373,14 +2462,20 @@ public:
//!
//! \see getNbGroups()
//!
- virtual void setNbGroups(int32_t nbGroups) TRTNOEXCEPT = 0;
+ void setNbGroups(int32_t nbGroups) noexcept
+ {
+ mImpl->setNbGroups(nbGroups);
+ }
//!
//! \brief Get the number of groups for a deconvolution.
//!
//! \see setNbGroups()
//!
- virtual int32_t getNbGroups() const TRTNOEXCEPT = 0;
+ int32_t getNbGroups() const noexcept
+ {
+ return mImpl->getNbGroups();
+ }
//!
//! \brief Set the kernel weights for the deconvolution.
@@ -2391,14 +2486,20 @@ public:
//!
//! \see getWeights()
//!
- virtual void setKernelWeights(Weights weights) TRTNOEXCEPT = 0;
+ void setKernelWeights(Weights weights) noexcept
+ {
+ mImpl->setKernelWeights(weights);
+ }
//!
//! \brief Get the kernel weights for the deconvolution.
//!
//! \see setNbGroups()
//!
- virtual Weights getKernelWeights() const TRTNOEXCEPT = 0;
+ Weights getKernelWeights() const noexcept
+ {
+ return mImpl->getKernelWeights();
+ }
//!
//! \brief Set the bias weights for the deconvolution.
@@ -2410,19 +2511,21 @@ public:
//!
//! \see getBiasWeights()
//!
- virtual void setBiasWeights(Weights weights) TRTNOEXCEPT = 0;
+ void setBiasWeights(Weights weights) noexcept
+ {
+ mImpl->setBiasWeights(weights);
+ }
//!
//! \brief Get the bias weights for the deconvolution.
//!
//! \see getBiasWeights()
//!
- virtual Weights getBiasWeights() const TRTNOEXCEPT = 0;
+ Weights getBiasWeights() const noexcept
+ {
+ return mImpl->getBiasWeights();
+ }
-protected:
- virtual ~IDeconvolutionLayer() {}
-
-public:
//!
//! \brief Set the multi-dimension pre-padding of the deconvolution.
//!
@@ -2436,14 +2539,20 @@ public:
//!
//! \see getPrePadding()
//!
- virtual void setPrePadding(Dims padding) TRTNOEXCEPT = 0;
+ void setPrePadding(Dims padding) noexcept
+ {
+ mImpl->setPrePadding(padding);
+ }
//!
//! \brief Get the pre-padding.
//!
//! \see setPrePadding()
//!
- virtual Dims getPrePadding() const TRTNOEXCEPT = 0;
+ Dims getPrePadding() const noexcept
+ {
+ return mImpl->getPrePadding();
+ }
//!
//! \brief Set the multi-dimension post-padding of the deconvolution.
@@ -2458,14 +2567,20 @@ public:
//!
//! \see getPostPadding()
//!
- virtual void setPostPadding(Dims padding) TRTNOEXCEPT = 0;
+ void setPostPadding(Dims padding) noexcept
+ {
+ mImpl->setPostPadding(padding);
+ }
//!
//! \brief Get the padding.
//!
- //! \see setPadding()
+ //! \see setPostPadding()
//!
- virtual Dims getPostPadding() const TRTNOEXCEPT = 0;
+ Dims getPostPadding() const noexcept
+ {
+ return mImpl->getPostPadding();
+ }
//!
//! \brief Set the padding mode.
@@ -2476,7 +2591,10 @@ public:
//!
//! \see getPaddingMode()
//!
- virtual void setPaddingMode(PaddingMode paddingMode) TRTNOEXCEPT = 0;
+ void setPaddingMode(PaddingMode paddingMode) noexcept
+ {
+ mImpl->setPaddingMode(paddingMode);
+ }
//!
//! \brief Get the padding mode.
@@ -2485,7 +2603,10 @@ public:
//!
//! \see setPaddingMode()
//!
- virtual PaddingMode getPaddingMode() const TRTNOEXCEPT = 0;
+ PaddingMode getPaddingMode() const noexcept
+ {
+ return mImpl->getPaddingMode();
+ }
//!
//! \brief Set the multi-dimension kernel size of the deconvolution.
@@ -2495,14 +2616,20 @@ public:
//!
//! \see getKernelSizeNd() setKernelSize() getKernelSize()
//!
- virtual void setKernelSizeNd(Dims kernelSize) TRTNOEXCEPT = 0;
+ void setKernelSizeNd(Dims kernelSize) noexcept
+ {
+ mImpl->setKernelSizeNd(kernelSize);
+ }
//!
//! \brief Get the multi-dimension kernel size of the deconvolution.
//!
//! \see setKernelSizeNd()
//!
- virtual Dims getKernelSizeNd() const TRTNOEXCEPT = 0;
+ Dims getKernelSizeNd() const noexcept
+ {
+ return mImpl->getKernelSizeNd();
+ }
//!
//! \brief Set the multi-dimension stride of the deconvolution.
@@ -2514,14 +2641,20 @@ public:
//!
//! \see getStrideNd() setStride() getStride()
//!
- virtual void setStrideNd(Dims stride) TRTNOEXCEPT = 0;
+ void setStrideNd(Dims stride) noexcept
+ {
+ mImpl->setStrideNd(stride);
+ }
//!
//! \brief Get the multi-dimension stride of the deconvolution.
//!
//! \see setStrideNd()
//!
- virtual Dims getStrideNd() const TRTNOEXCEPT = 0;
+ Dims getStrideNd() const noexcept
+ {
+ return mImpl->getStrideNd();
+ }
//!
//! \brief Set the multi-dimension padding of the deconvolution.
@@ -2536,7 +2669,10 @@ public:
//!
//! \see getPaddingNd() setPadding() getPadding()
//!
- virtual void setPaddingNd(Dims padding) TRTNOEXCEPT = 0;
+ void setPaddingNd(Dims padding) noexcept
+ {
+ mImpl->setPaddingNd(padding);
+ }
//!
//! \brief Get the multi-dimension padding of the deconvolution.
@@ -2545,7 +2681,10 @@ public:
//!
//! \see setPaddingNd()
//!
- virtual Dims getPaddingNd() const TRTNOEXCEPT = 0;
+ Dims getPaddingNd() const noexcept
+ {
+ return mImpl->getPaddingNd();
+ }
//!
//! \brief Append or replace an input of this layer with a specific tensor
@@ -2564,7 +2703,7 @@ public:
//!
//! If this function is called with a value greater than 0, then the function getNbInputs() changes
//!
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ using ILayer::setInput;
//! \brief Set the multi-dimension dilation of the deconvolution.
//!
@@ -2572,14 +2711,24 @@ public:
//!
//! \see getDilationNd()
//!
- virtual void setDilationNd(Dims dilation) TRTNOEXCEPT = 0;
+ void setDilationNd(Dims dilation) noexcept
+ {
+ mImpl->setDilationNd(dilation);
+ }
//!
//! \brief Get the multi-dimension dilation of the deconvolution.
//!
//! \see setDilationNd()
//!
- virtual Dims getDilationNd() const TRTNOEXCEPT = 0;
+ Dims getDilationNd() const noexcept
+ {
+ return mImpl->getDilationNd();
+ }
+
+protected:
+ virtual ~IDeconvolutionLayer() noexcept = default;
+ apiv::VDeconvolutionLayer* mImpl;
};
//!
@@ -2607,12 +2756,15 @@ enum class ElementWiseOperation : int32_t
kLESS = 13 //!< Check if element in first tensor is less than corresponding element in second tensor.
};
+namespace impl
+{
//! Maximum number of elements in ElementWiseOperation enum. \see ElementWiseOperation
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 14;
-}
+ static constexpr int32_t kVALUE = 14;
+};
+} // namespace impl
//!
//! \class IElementWiseLayer
@@ -2621,7 +2773,12 @@ constexpr inline int32_t EnumMax()
//!
//! This layer applies a per-element binary operation between corresponding elements of two tensors.
//!
-//! The input dimensions of the two input tensors must be equal, and the output tensor is the same size as each input.
+//! The input tensors must have the same number of dimensions. For each dimension, their lengths must
+//! match, or one of them must be one. In the latter case, the tensor is broadcast along that axis.
+//!
+//! The output tensor has the same number of dimensions as the inputs. For each output dimension,
+//! its length is equal to the lengths of the corresponding input dimensions if they match,
+//! otherwise it is equal to the length that is not one.
//！
//! \warning When running this layer on the DLA with Int8 data type, the dynamic ranges of two input tensors shall be
//! equal. If the dynamic ranges are generated using calibrator, the largest value shall be used.
@@ -2640,7 +2797,10 @@ public:
//!
//! \see getBiasWeights()
//!
- virtual void setOperation(ElementWiseOperation op) TRTNOEXCEPT = 0;
+ void setOperation(ElementWiseOperation op) noexcept
+ {
+ return mImpl->setOperation(op);
+ }
//!
//! \brief Get the binary operation for the layer.
@@ -2649,10 +2809,14 @@ public:
//!
//! \see setBiasWeights()
//!
- virtual ElementWiseOperation getOperation() const TRTNOEXCEPT = 0;
+ ElementWiseOperation getOperation() const noexcept
+ {
+ return mImpl->getOperation();
+ }
protected:
- virtual ~IElementWiseLayer() {}
+ apiv::VElementWiseLayer* mImpl;
+ virtual ~IElementWiseLayer() noexcept = default;
};
//!
@@ -2667,14 +2831,20 @@ public:
//!
//! \see getGatherAxis()
//!
- virtual void setGatherAxis(int32_t axis) TRTNOEXCEPT = 0;
+ void setGatherAxis(int32_t axis) noexcept
+ {
+ mImpl->setGatherAxis(axis);
+ }
//!
//! \brief Get the axis to gather on.
//!
//! \see setGatherAxis()
//!
- virtual int32_t getGatherAxis() const TRTNOEXCEPT = 0;
+ int32_t getGatherAxis() const noexcept
+ {
+ return mImpl->getGatherAxis();
+ }
//!
//! \brief Set the number of leading dimensions of indices tensor to be handled elementwise.
@@ -2683,17 +2853,24 @@ public:
//!
//! \see getNbElementWiseDims()
//!
- virtual void setNbElementWiseDims(int32_t k) TRTNOEXCEPT = 0;
+ void setNbElementWiseDims(int32_t k) noexcept
+ {
+ mImpl->setNbElementWiseDims(k);
+ }
//!
//! \brief Get the number of leading dimensions of indices tensor to be handled elementwise.
//!
//! \see setNbElementWiseDims()
//!
- virtual int32_t getNbElementWiseDims() const TRTNOEXCEPT = 0;
+ int32_t getNbElementWiseDims() const noexcept
+ {
+ return mImpl->getNbElementWiseDims();
+ }
protected:
- virtual ~IGatherLayer() {}
+ apiv::VGatherLayer* mImpl;
+ virtual ~IGatherLayer() noexcept = default;
};
//!
@@ -2773,7 +2950,7 @@ protected:
//! H[t] := (1 - z[t])*h[t] + z[t]*H[t-1]
//! ~~~
//!
-//! \see IRNNLayer, IRNNv2Layer
+//! \see IRNNv2Layer
//!
enum class RNNOperation : int32_t
{
@@ -2785,7 +2962,7 @@ enum class RNNOperation : int32_t
//! Maximum number of elements in RNNOperation enum. \see RNNOperation
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 4;
}
@@ -2795,7 +2972,7 @@ constexpr inline int32_t EnumMax()
//!
//! \brief Enumerates the RNN direction that may be performed by an RNN layer.
//!
-//! \see IRNNLayer, IRNNv2Layer
+//! \see IRNNv2Layer
//!
enum class RNNDirection : int32_t
{
@@ -2805,7 +2982,7 @@ enum class RNNDirection : int32_t
//! Maximum number of elements in RNNDirection enum. \see RNNDirection
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 2;
}
@@ -2823,7 +3000,7 @@ constexpr inline int32_t EnumMax()
//! and `W[g]` is conceptually an identity matrix. In this case, the input vector `X[t]` must have length `H`
//! (the size of the hidden state).
//!
-//! \see IRNNLayer, IRNNv2Layer
+//! \see IRNNv2Layer
//!
enum class RNNInputMode : int32_t
{
@@ -2833,178 +3010,163 @@ enum class RNNInputMode : int32_t
//! Maximum number of elements in RNNInputMode enum. \see RNNInputMode
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 2;
}
//!
-//! \class IRNNLayer
+//! \enum RNNGateType
//!
-//! \brief A RNN layer in a network definition.
+//! \brief Identifies an individual gate within an RNN cell.
//!
-//! This layer applies an RNN operation on the inputs. This layer only works with networks that
-//! that have an implicit batch dimension. For dynamic shapes and explicit batch dimension networks,
-//! use IRNNv2Layer.
+//! \see RNNOperation
//!
-//! \deprecated This interface is superseded by IRNNv2Layer and will be removed in TensorRT 8.0.
+enum class RNNGateType : int32_t
+{
+ kINPUT = 0, //!< Input gate (i).
+ kOUTPUT = 1, //!< Output gate (o).
+ kFORGET = 2, //!< Forget gate (f).
+ kUPDATE = 3, //!< Update gate (z).
+ kRESET = 4, //!< Reset gate (r).
+ kCELL = 5, //!< Cell gate (c).
+ kHIDDEN = 6 //!< Hidden gate (h).
+};
+
+template <>
+constexpr inline int32_t EnumMax() noexcept
+{
+ return 7;
+} //!< Maximum number of elements in RNNGateType enum. \see RNNGateType
+
+//!
+//! \class IRNNv2Layer
+//!
+//! \brief An RNN layer in a network definition, version 2.
+//!
+//! This layer supersedes IRNNLayer.
+//!
+//! \deprecated IRNNv2Layer will be removed in TensorRT 9.0, use INetworkDefinition::addLoop instead.
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class TRT_DEPRECATED IRNNLayer : public ILayer
+class TRT_DEPRECATED IRNNv2Layer : public ILayer
{
public:
- //!
- //! \brief Get the number of layers in the RNN.
- //!
- //! \return The number of layers in the RNN.
- //!
- virtual uint32_t getLayerCount() const TRTNOEXCEPT = 0;
+ int32_t getLayerCount() const noexcept
+ {
+ return mImpl->getLayerCount();
+ } //!< Get the layer count of the RNN.
+ int32_t getHiddenSize() const noexcept
+ {
+ return mImpl->getHiddenSize();
+ } //!< Get the hidden size of the RNN.
+ int32_t getMaxSeqLength() const noexcept
+ {
+ return mImpl->getMaxSeqLength();
+ } //!< Get the maximum sequence length of the RNN.
+ int32_t getDataLength() const noexcept
+ {
+ return mImpl->getDataLength();
+ } //!< Get the maximum data length of the RNN.
//!
- //! \brief Get the size of the hidden layers.
+ //! \brief Specify individual sequence lengths in the batch with the ITensor pointed to by
+ //! \p seqLengths.
//!
- //! The hidden size is the value of hiddenSize parameter passed into addRNN().
+ //! The \p seqLengths ITensor should be a {N1, ..., Np} tensor, where N1..Np are the index dimensions
+ //! of the input tensor to the RNN.
//!
- //! \return The internal hidden layer size for the RNN.
- //! \see getDirection(), addRNN()
+ //! If this is not specified, then the RNN layer assumes all sequences are size getMaxSeqLength().
//!
- virtual std::size_t getHiddenSize() const TRTNOEXCEPT = 0;
+ //! All sequence lengths in \p seqLengths should be in the range [1, getMaxSeqLength()]. Zero-length
+ //! sequences are not supported.
+ //!
+ //! This tensor must be of type DataType::kINT32.
+ //!
+ void setSequenceLengths(ITensor& seqLengths) noexcept
+ {
+ return mImpl->setSequenceLengths(seqLengths);
+ }
//!
- //! \brief Get the sequence length.
+ //! \brief Get the sequence lengths specified for the RNN.
//!
- //! The sequence length is the maximum number of time steps passed into the addRNN() function.
- //! This is also the maximum number of input tensors that the RNN can process at once.
+ //! \return nullptr if no sequence lengths were specified, the sequence length data otherwise.
//!
- //! \return the maximum number of time steps that can be executed by a single call RNN layer.
+ //! \see setSequenceLengths()
//!
- virtual int32_t getSeqLength() const TRTNOEXCEPT = 0;
+ ITensor* getSequenceLengths() const noexcept
+ {
+ return mImpl->getSequenceLengths();
+ }
//!
//! \brief Set the operation of the RNN layer.
- //!
//! \see getOperation(), RNNOperation
//!
- virtual void setOperation(RNNOperation op) TRTNOEXCEPT = 0;
+ void setOperation(RNNOperation op) noexcept
+ {
+ mImpl->setOperation(op);
+ }
//!
//! \brief Get the operation of the RNN layer.
- //!
//! \see setOperation(), RNNOperation
//!
- virtual RNNOperation getOperation() const TRTNOEXCEPT = 0;
+ RNNOperation getOperation() const noexcept
+ {
+ return mImpl->getOperation();
+ }
//!
- //! \brief Set the operation of the RNN layer.
- //!
+ //! \brief Set the input mode of the RNN layer.
//! \see getInputMode(), RNNInputMode
//!
- virtual void setInputMode(RNNInputMode op) TRTNOEXCEPT = 0;
+ void setInputMode(RNNInputMode op) noexcept
+ {
+ mImpl->setInputMode(op);
+ }
//!
- //! \brief Get the operation of the RNN layer.
- //!
+ //! \brief Get the input mode of the RNN layer.
//! \see setInputMode(), RNNInputMode
//!
- virtual RNNInputMode getInputMode() const TRTNOEXCEPT = 0;
+ RNNInputMode getInputMode() const noexcept
+ {
+ return mImpl->getInputMode();
+ }
//!
//! \brief Set the direction of the RNN layer.
//!
- //! The direction determines if the RNN is run
- //! as a unidirectional(left to right) or
+ //! The direction determines if the RNN is run as a unidirectional(left to right) or
//! bidirectional(left to right and right to left).
- //! In the ::kBIDIRECTION case the
- //! output is concatenated together, resulting
+ //! In the ::kBIDIRECTION case the output is concatenated together, resulting
//! in output size of 2x getHiddenSize().
+ //!
//! \see getDirection(), RNNDirection
//!
- virtual void setDirection(RNNDirection op) TRTNOEXCEPT = 0;
+ void setDirection(RNNDirection op) noexcept
+ {
+ mImpl->setDirection(op);
+ }
//!
//! \brief Get the direction of the RNN layer.
- //!
//! \see setDirection(), RNNDirection
//!
- virtual RNNDirection getDirection() const TRTNOEXCEPT = 0;
+ RNNDirection getDirection() const noexcept
+ {
+ return mImpl->getDirection();
+ }
//!
- //! \param weights The weight structure holding the weight parameters.
+ //! \brief Set the weight parameters for an individual gate in the RNN.
//!
- //! \brief Set the weight parameters for the RNN.
- //!
- //! The trained weights for the weight parameter matrices of the RNN.
//! The #DataType for this structure must be ::kFLOAT or ::kHALF, and must be the same
//! datatype as the input tensor.
//!
- //! The layout of the weight structure depends on the #RNNOperation, #RNNInputMode, and
- //! #RNNDirection of the layer. The array specified by `weights.values` contains a sequence of
- //! parameter matrices, where each parameter matrix is linearly appended after the previous
- //! without padding; e.g., if parameter matrix 0 and 1 have M and N elements respectively, then
- //! the layout of `weights.values` in memory looks like:
- //!
- //! ~~~
- //! index | 0 1 2 3 4 ... M-2 M-1 | M M+1 ... M+N-2 M+N-1 | M+N M+N+1 M+N+2 ... | ...
- //! data |-- parameter matrix 0 --|-- parameter matrix 1 --|-- parameter matrix 2 --| ...
- //! ~~~
- //!
- //! The following sections describe \ref setRNNWeightsOrder "the order of weight matrices" and
- //! \ref setRNNWeightsLayout "the layout of elements within a weight matrix".
- //!
- //! \section setRNNWeightsOrder Order of weight matrices
- //!
- //! The parameter matrices are ordered as described below:
- //!
- //! ~~~
- //! Let G(op, l) be defined to be a function that produces lists of parameter names, as follows:
- //!
- //! G(::kRELU, l) := [ Wl[i], Rl[i] ]
- //! G(::kTANH, l) := [ Wl[i], Rl[i] ]
- //! G(::kLSTM, l) := [ Wl[f], Wl[i], Wl[c], Wl[o], Rl[f], Rl[i], Rl[c], Rl[o] ]
- //! G(::kGRU, l) := [ Wl[z], Wl[r], Wl[h], Rl[z], Rl[r], Rl[h] ]
- //!
- //! where Wl[g] and Rl[g] are the names of the input and recurrent
- //! input weight matrices for gate g, layer index l.
- //!
- //! See RNNOperation for an overview of the naming convention used for gates.
- //!
- //! If getDirection() == ::kUNIDIRECTION, then l identifies the stacked layer of the
- //! RNN, with l=0 being the first recurrent layer and l=L-1 being the last recurrent layer.
- //!
- //! If getDirection() == ::kBIDIRECTION, then (l % 2) identifies the direction of the
- //! recurrent layer (forward if 0, or backward if 1), and (l / 2) identifies the position
- //! of the recurrent layer within the (forward or backward) stack.
- //!
- //! Let op := getOperation(),
- //! L := { ::kUNIDIRECTION => getLayerCount()
- //! { ::kBIDIRECTION => (2 * getLayerCount())
- //!
- //! Then the ordering of parameter matrices is the list produced by concatenating
- //! G(op, 0), G(op, 1), G(op, 2), ..., G(op, L-1).
- //! ~~~
- //!
- //! For example:
- //!
- //! - an RNN with `getLayerCount() == 3`, `getDirection() == ::kUNIDIRECTION`,
- //! and `getOperation() == ::kRELU` has the following order:
- //!
- //! `[ W0[i], R0[i], W1[i], R1[i], W2[i], R2[i] ]`
- //!
- //! - an RNN with `getLayerCount() == 2`, `getDirection() == ::kUNIDIRECTION`,
- //! and `getOperation() == ::kGRU` has the following order:
- //!
- //! `[ W0[z], W0[r], W0[h], R0[z], R0[r], R0[h], W1[z], W1[r], W1[h], R1[z], R1[r], R1[h] ]`
- //!
- //! - an RNN with `getLayerCount() == 2`, `getDirection() == ::kBIDIRECTION`,
- //! and `getOperation() == ::kRELU` has the following order:
- //!
- //! `[ W0_fw[i], R0_fw[i], W0_bw[i], R0_bw[i], W1_fw[i], R1_fw[i], W1_bw[i], R1_bw[i] ]`
- //!
- //! (fw = "forward", bw = "backward")
- //!
- //! \section setRNNWeightsLayout Layout of elements within a weight matrix
- //!
//! Each parameter matrix is row-major in memory, and has the following dimensions:
//!
//! ~~~
@@ -3041,254 +3203,7 @@ public:
//! backward) RNN cell operates on the previous (forward or
//! backward) RNN cell's hidden state, which is size `H`).
//!
- //! \see getWeights(), #RNNOperation
- //!
- virtual void setWeights(Weights weights) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the W weights for the RNN.
- //!
- //! \see setWeights()
- //!
- virtual Weights getWeights() const TRTNOEXCEPT = 0;
-
- //!
- //! \param bias The weight structure holding the bias parameters.
- //!
- //! \brief Set the bias parameters for the RNN.
- //!
- //! The trained weights for the bias parameter vectors of the RNN.
- //! The #DataType for this structure must be ::kFLOAT or ::kHALF, and must be the same
- //! datatype as the input tensor.
- //!
- //! The layout of the weight structure depends on the #RNNOperation, #RNNInputMode, and
- //! #RNNDirection of the layer. The array specified by `weights.values` contains a sequence of
- //! bias vectors, where each bias vector is linearly appended after the previous
- //! without padding; e.g., if bias vector 0 and 1 have M and N elements respectively, then
- //! the layout of `weights.values` in memory looks like:
- //!
- //! ~~~
- //! index | 0 1 2 3 4 ... M-2 M-1 | M M+1 ... M+N-2 M+N-1 | M+N M+N+1 M+N+2 ... | ...
- //! data |-- bias vector 0 --|-- bias vector 1 --|-- bias vector 2 --| ...
- //! ~~~
- //!
- //! The ordering of bias vectors is similar to the \ref setRNNWeightsOrder "ordering of weight matrices"
- //! as described in setWeights(). To determine the order of bias vectors for a given RNN configuration,
- //! determine the ordered list of weight matrices `[ W0, W1, ..., Wn ]`. Then replace each weight matrix
- //! with its corresponding bias vector, i.e. apply the following transform (for layer `l`, gate `g`):
- //!
- //! - `Wl[g]` becomes `Wbl[g]`
- //! - `Rl[g]` becomes `Rbl[g]`
- //!
- //! For example:
- //!
- //! - an RNN with `getLayerCount() == 3`, `getDirection() == ::kUNIDIRECTION`,
- //! and `getOperation() == ::kRELU` has the following order:
- //!
- //! `[ Wb0[i], Rb0[i], Wb1[i], Rb1[i], Wb2[i], Rb2[i] ]`
- //!
- //! - an RNN with `getLayerCount() == 2`, `getDirection() == ::kUNIDIRECTION`,
- //! and `getOperation() == ::kGRU` has the following order:
- //!
- //! `[ Wb0[z], Wb0[r], Wb0[h], Rb0[z], Rb0[r], Rb0[h], Wb1[z], Wb1[r], Wb1[h], Rb1[z], Rb1[r], Rb1[h] ]`
- //!
- //! - an RNN with `getLayerCount() == 2`, `getDirection() == ::kBIDIRECTION`,
- //! and `getOperation() == ::kRELU` has the following order:
- //!
- //! `[ Wb0_fw[i], Rb0_fw[i], Wb0_bw[i], Rb0_bw[i], Wb1_fw[i], Rb1_fw[i], Wb1_bw[i], Rb1_bw[i] ]`
- //!
- //! (fw = "forward", bw = "backward")
- //!
- //! Each bias vector has a fixed size, getHiddenSize().
- //!
- //! \see getBias(), #RNNOperation
- //!
- virtual void setBias(Weights bias) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the bias parameter vector for the RNN.
- //!
- //! \see setBias()
- //!
- virtual Weights getBias() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the length of the data being processed by the RNN for use in computing
- //! other values.
- //!
- //! \see setHiddenState(), setCellState()
- //!
- virtual int32_t getDataLength() const TRTNOEXCEPT = 0;
-
- //!
- //! \param hidden The initial hidden state of the RNN.
- //!
- //! \brief Set the initial hidden state of the RNN with the provided \p hidden ITensor.
- //!
- //! The layout for \p hidden is a linear layout of a 3D matrix:
- //! - C - The number of layers in the RNN, it must match getLayerCount().
- //! - H - The number of mini-batches for each time sequence.
- //! - W - The size of the per layer hidden states, it must match getHiddenSize().
- //!
- //! If getDirection() is ::kBIDIRECTION, the amount of space required is doubled and C is equal to
- //! getLayerCount() * 2.
- //!
- //! If hidden is not specified, then the initial hidden state is set to zero.
- //!
- //! \see getHiddenState()
- //!
- virtual void setHiddenState(ITensor& hidden) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the initial hidden state of the RNN.
- //!
- //! \return nullptr if no initial hidden tensor was specified, the initial hidden data otherwise.
- //!
- virtual ITensor* getHiddenState() const TRTNOEXCEPT = 0;
-
- //!
- //! \param cell The initial cell state of the RNN.
- //!
- //! \brief Set the initial cell state of the RNN with the provided \p cell ITensor.
- //!
- //! The layout for \p cell is a linear layout of a 3D matrix:
- //! - C - The number of layers in the RNN, it must match getLayerCount().
- //! - H - The number of mini-batches for each time sequence.
- //! - W - The size of the per layer hidden states, it must match getHiddenSize().
- //!
- //! If \p cell is not specified, then the initial cell state is set to zero.
- //!
- //! If getDirection() is ::kBIDIRECTION, the amount of space required is doubled and C is equal to
- //! getLayerCount() * 2.
- //!
- //! The cell state only affects LSTM RNN's.
- //!
- //! \see getCellState()
- //!
- virtual void setCellState(ITensor& cell) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the initial cell state of the RNN.
- //!
- //! \return nullptr if no initial cell tensor was specified, the initial cell data otherwise.
- //!
- virtual ITensor* getCellState() const TRTNOEXCEPT = 0;
-
-protected:
- virtual ~IRNNLayer() {}
-};
-
-//!
-//! \enum RNNGateType
-//!
-//! \brief Identifies an individual gate within an RNN cell.
-//!
-//! \see RNNOperation
-//!
-enum class RNNGateType : int32_t
-{
- kINPUT = 0, //!< Input gate (i).
- kOUTPUT = 1, //!< Output gate (o).
- kFORGET = 2, //!< Forget gate (f).
- kUPDATE = 3, //!< Update gate (z).
- kRESET = 4, //!< Reset gate (r).
- kCELL = 5, //!< Cell gate (c).
- kHIDDEN = 6 //!< Hidden gate (h).
-};
-
-//! Maximum number of elements in RNNGateType enum. \see RNNGateType
-template <>
-constexpr inline int32_t EnumMax()
-{
- return 7;
-}
-
-//!
-//! \class IRNNv2Layer
-//!
-//! \brief An RNN layer in a network definition, version 2.
-//!
-//! This layer supersedes IRNNLayer.
-//!
-//! \deprecated IRNNv2Layer will be removed in TensorRT 9.0, use ILoop::addLoop instead.
-//!
-//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
-//!
-class TRT_DEPRECATED IRNNv2Layer : public ILayer
-{
-public:
- virtual int32_t getLayerCount() const TRTNOEXCEPT = 0; //!< Get the layer count of the RNN
- virtual int32_t getHiddenSize() const TRTNOEXCEPT = 0; //!< Get the hidden size of the RNN
- virtual int32_t getMaxSeqLength() const TRTNOEXCEPT = 0; //!< Get the maximum sequence length of the RNN
- virtual int32_t getDataLength() const TRTNOEXCEPT = 0; //!< Get the maximum data length of the RNN
-
- //!
- //! \brief Specify individual sequence lengths in the batch with the ITensor pointed to by
- //! \p seqLengths.
- //!
- //! The \p seqLengths ITensor should be a {N1, ..., Np} tensor, where N1..Np are the index dimensions
- //! of the input tensor to the RNN.
- //!
- //! If this is not specified, then the RNN layer assumes all sequences are size getMaxSeqLength().
- //!
- //! All sequence lengths in \p seqLengths should be in the range [1, getMaxSeqLength()]. Zero-length
- //! sequences are not supported.
- //!
- //! This tensor must be of type DataType::kINT32.
- //!
- virtual void setSequenceLengths(ITensor& seqLengths) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the sequence lengths specified for the RNN.
- //!
- //! \return nullptr if no sequence lengths were specified, the sequence length data otherwise.
- //!
- //! \see setSequenceLengths()
- //!
- virtual ITensor* getSequenceLengths() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the operation of the RNN layer.
- //! \see getOperation(), RNNOperation
- //!
- virtual void setOperation(RNNOperation op) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the operation of the RNN layer.
- //! \see setOperation(), RNNOperation
- //!
- virtual RNNOperation getOperation() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the input mode of the RNN layer.
- //! \see getInputMode(), RNNInputMode
- //!
- virtual void setInputMode(RNNInputMode op) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the input mode of the RNN layer.
- //! \see setInputMode(), RNNInputMode
- //!
- virtual RNNInputMode getInputMode() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the direction of the RNN layer.
- //! \see getDirection(), RNNDirection
- //!
- virtual void setDirection(RNNDirection op) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the direction of the RNN layer.
- //! \see setDirection(), RNNDirection
- //!
- virtual RNNDirection getDirection() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the weight parameters for an individual gate in the RNN.
- //!
//! \param layerIndex The index of the layer that contains this gate. See the section
- //! \ref setRNNWeightsOrder "Order of weight matrices" in IRNNLayer::setWeights()
- //! for a description of the layer index.
//! \param gate The name of the gate within the RNN layer. The gate name must correspond
//! to one of the gates used by this layer's #RNNOperation.
//! \param isW True if the weight parameters are for the input matrix W[g]
@@ -3300,17 +3215,28 @@ public:
//! in IRNNLayer::setWeights() for documentation on the expected
//! dimensions of this matrix.
//!
- virtual void setWeightsForGate(int32_t layerIndex, RNNGateType gate, bool isW, Weights weights) TRTNOEXCEPT = 0;
+ void setWeightsForGate(int32_t layerIndex, RNNGateType gate, bool isW, Weights weights) noexcept
+ {
+ mImpl->setWeightsForGate(layerIndex, gate, isW, weights);
+ }
//!
//! \brief Get the weight parameters for an individual gate in the RNN.
//! \see setWeightsForGate()
//!
- virtual Weights getWeightsForGate(int32_t layerIndex, RNNGateType gate, bool isW) const TRTNOEXCEPT = 0;
+ Weights getWeightsForGate(int32_t layerIndex, RNNGateType gate, bool isW) const noexcept
+ {
+ return mImpl->getWeightsForGate(layerIndex, gate, isW);
+ }
//!
//! \brief Set the bias parameters for an individual gate in the RNN.
//!
+ //! The #DataType for this structure must be ::kFLOAT or ::kHALF, and must be the same
+ //! datatype as the input tensor.
+ //!
+ //! Each bias vector has a fixed size, getHiddenSize().
+ //!
//! \param layerIndex The index of the layer that contains this gate. See the section
//! \ref setRNNWeightsOrder "Order of weight matrices" in IRNNLayer::setWeights()
//! for a description of the layer index.
@@ -3323,13 +3249,19 @@ public:
//! \param bias The weight structure holding the bias parameters, which should be an
//! array of size getHiddenSize().
//!
- virtual void setBiasForGate(int32_t layerIndex, RNNGateType gate, bool isW, Weights bias) TRTNOEXCEPT = 0;
+ void setBiasForGate(int32_t layerIndex, RNNGateType gate, bool isW, Weights bias) noexcept
+ {
+ mImpl->setBiasForGate(layerIndex, gate, isW, bias);
+ }
//!
//! \brief Get the bias parameters for an individual gate in the RNN.
//! \see setBiasForGate()
//!
- virtual Weights getBiasForGate(int32_t layerIndex, RNNGateType gate, bool isW) const TRTNOEXCEPT = 0;
+ Weights getBiasForGate(int32_t layerIndex, RNNGateType gate, bool isW) const noexcept
+ {
+ return mImpl->getBiasForGate(layerIndex, gate, isW);
+ }
//!
//! \brief Set the initial hidden state of the RNN with the provided \p hidden ITensor.
@@ -3343,13 +3275,19 @@ public:
//! final backward hidden state is stored in `L= 2*l + 1`.
//! - `H` is the hidden state for each layer, equal to getHiddenSize().
//!
- virtual void setHiddenState(ITensor& hidden) TRTNOEXCEPT = 0;
+ void setHiddenState(ITensor& hidden) noexcept
+ {
+ mImpl->setHiddenState(hidden);
+ }
//!
//! \brief Get the initial hidden state of the RNN.
//! \see setHiddenState()
//!
- virtual ITensor* getHiddenState() const TRTNOEXCEPT = 0;
+ ITensor* getHiddenState() const noexcept
+ {
+ return mImpl->getHiddenState();
+ }
//!
//! \brief Set the initial cell state of the LSTM with the provided \p cell ITensor.
@@ -3365,71 +3303,23 @@ public:
//!
//! It is an error to call setCellState() on an RNN layer that is not configured with RNNOperation::kLSTM.
//!
- virtual void setCellState(ITensor& cell) TRTNOEXCEPT = 0;
+ void setCellState(ITensor& cell) noexcept
+ {
+ mImpl->setCellState(cell);
+ }
//!
//! \brief Get the initial cell state of the RNN.
//! \see setCellState()
//!
- virtual ITensor* getCellState() const TRTNOEXCEPT = 0;
+ ITensor* getCellState() const noexcept
+ {
+ return mImpl->getCellState();
+ }
protected:
- virtual ~IRNNv2Layer() {}
-};
-
-//!
-//! \class IOutputDimensionsFormula
-//!
-//! \brief Application-implemented interface to compute layer output sizes.
-//!
-//! \deprecated IOutputDimensionsFormula has been superseded by PaddingMode and will be removed in TensorRT 9.0.
-//!
-class TRT_DEPRECATED IOutputDimensionsFormula
-{
-public:
- //!
- //! \brief Application-implemented interface to compute the HW output dimensions of a layer from the layer input
- //! and parameters.
- //!
- //! \param inputDims The input dimensions of the layer.
- //! \param kernelSize The kernel size (or window size, for a pooling layer) parameter of the layer operation.
- //! \param stride The stride parameter for the layer.
- //! \param padding The padding parameter of the layer.
- //! \param dilation The dilation parameter of the layer (only applicable to convolutions).
- //! \param layerName The name of the layer.
- //!
- //! \return The output size of the layer
- //!
- //! Note that for dilated convolutions, the dilation is applied to the kernel size before this routine is called.
- //!
- virtual DimsHW compute(DimsHW inputDims, DimsHW kernelSize, DimsHW stride, DimsHW padding, DimsHW dilation, const char* layerName) const TRTNOEXCEPT = 0;
-
- virtual ~IOutputDimensionsFormula() {}
-};
-
-//!
-//! \class IPluginLayer
-//!
-//! \brief Layer type for plugins.
-//!
-//! \see IPluginExt
-//!
-//! \deprecated This interface is superseded by IPluginV2Layer and will be removed in TensorRT 8.0.
-//!
-//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
-//!
-class TRT_DEPRECATED IPluginLayer : public ILayer
-{
-public:
- //!
- //! \brief Get the plugin for the layer.
- //!
- //! \see IPluginExt
- //!
- virtual IPlugin& getPlugin() TRTNOEXCEPT = 0;
-
-protected:
- virtual ~IPluginLayer() {}
+ apiv::VRNNv2Layer* mImpl;
+ virtual ~IRNNv2Layer() noexcept = default;
};
//!
@@ -3449,10 +3339,14 @@ public:
//!
//! \see IPluginV2
//!
- virtual IPluginV2& getPlugin() TRTNOEXCEPT = 0;
+ IPluginV2& getPlugin() noexcept
+ {
+ return mImpl->getPlugin();
+ }
protected:
- virtual ~IPluginV2Layer() {}
+ apiv::VPluginV2Layer* mImpl;
+ virtual ~IPluginV2Layer() noexcept = default;
};
//!
@@ -3489,7 +3383,7 @@ enum class UnaryOperation : int32_t
//! Maximum number of elements in UnaryOperation enum. \see UnaryOperation
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 21;
}
@@ -3509,17 +3403,24 @@ public:
//!
//! \see getOperation(), UnaryOperation
//!
- virtual void setOperation(UnaryOperation op) TRTNOEXCEPT = 0;
+ void setOperation(UnaryOperation op) noexcept
+ {
+ mImpl->setOperation(op);
+ }
//!
//! \brief Get the unary operation for the layer.
//!
//! \see setOperation(), UnaryOperation
//!
- virtual UnaryOperation getOperation() const TRTNOEXCEPT = 0;
+ UnaryOperation getOperation() const noexcept
+ {
+ return mImpl->getOperation();
+ }
protected:
- virtual ~IUnaryLayer() {}
+ apiv::VUnaryLayer* mImpl;
+ virtual ~IUnaryLayer() noexcept = default;
};
//!
@@ -3551,7 +3452,7 @@ enum class ReduceOperation : int32_t
//! Maximum number of elements in ReduceOperation enum. \see ReduceOperation
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 5;
}
@@ -3571,45 +3472,64 @@ public:
//!
//! \see getOperation(), ReduceOperation
//!
- virtual void setOperation(ReduceOperation op) TRTNOEXCEPT = 0;
+ void setOperation(ReduceOperation op) noexcept
+ {
+ mImpl->setOperation(op);
+ }
//!
//! \brief Get the reduce operation for the layer.
//!
//! \see setOperation(), ReduceOperation
//!
- virtual ReduceOperation getOperation() const TRTNOEXCEPT = 0;
+ ReduceOperation getOperation() const noexcept
+ {
+ return mImpl->getOperation();
+ }
//!
//! \brief Set the axes over which to reduce.
//!
//! \see getReduceAxes
//!
- virtual void setReduceAxes(uint32_t reduceAxes) TRTNOEXCEPT = 0;
+ void setReduceAxes(uint32_t reduceAxes) noexcept
+ {
+ mImpl->setReduceAxes(reduceAxes);
+ }
//!
//! \brief Get the axes over which to reduce for the layer.
//!
//! \see setReduceAxes
//!
- virtual uint32_t getReduceAxes() const TRTNOEXCEPT = 0;
+ uint32_t getReduceAxes() const noexcept
+ {
+ return mImpl->getReduceAxes();
+ }
//!
//! \brief Set the boolean that specifies whether or not to keep the reduced dimensions for the layer.
//!
//! \see getKeepDimensions
//!
- virtual void setKeepDimensions(bool keepDimensions) TRTNOEXCEPT = 0;
+ void setKeepDimensions(bool keepDimensions) noexcept
+ {
+ mImpl->setKeepDimensions(keepDimensions);
+ }
//!
//! \brief Get the boolean that specifies whether or not to keep the reduced dimensions for the layer.
//!
//! \see setKeepDimensions
//!
- virtual bool getKeepDimensions() const TRTNOEXCEPT = 0;
+ bool getKeepDimensions() const noexcept
+ {
+ return mImpl->getKeepDimensions();
+ }
protected:
- virtual ~IReduceLayer() {}
+ apiv::VReduceLayer* mImpl;
+ virtual ~IReduceLayer() noexcept = default;
};
//!
@@ -3634,7 +3554,10 @@ public:
//!
//! \deprecated Superseded by setPrePaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setPrePadding(DimsHW padding) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setPrePadding(DimsHW padding) noexcept
+ {
+ mImpl->setPrePadding(padding);
+ }
//!
//! \brief Get the padding that is applied at the start of the tensor.
@@ -3643,7 +3566,10 @@ public:
//!
//! \deprecated Superseded by getPrePaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getPrePadding() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getPrePadding() const noexcept
+ {
+ return mImpl->getPrePadding();
+ }
//!
//! \brief Set the padding that is applied at the end of the tensor.
@@ -3654,7 +3580,10 @@ public:
//!
//! \deprecated Superseded by setPostPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual void setPostPadding(DimsHW padding) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void setPostPadding(DimsHW padding) noexcept
+ {
+ mImpl->setPostPadding(padding);
+ }
//!
//! \brief Get the padding that is applied at the end of the tensor.
@@ -3663,51 +3592,66 @@ public:
//!
//! \deprecated Superseded by getPostPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual DimsHW getPostPadding() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED DimsHW getPostPadding() const noexcept
+ {
+ return mImpl->getPostPadding();
+ }
-protected:
- virtual ~IPaddingLayer() {}
-
-public:
//!
//! \brief Set the padding that is applied at the start of the tensor.
//!
//! Negative padding results in trimming the edge by the specified amount.
//!
- //! \warning Only 2 dimensionsional padding is currently supported.
+ //! \warning Only 2 dimensional padding is currently supported.
//!
//! \see getPrePaddingNd
//!
- virtual void setPrePaddingNd(Dims padding) TRTNOEXCEPT = 0;
+ void setPrePaddingNd(Dims padding) noexcept
+ {
+ mImpl->setPrePaddingNd(padding);
+ }
//!
//! \brief Get the padding that is applied at the start of the tensor.
//!
- //! \warning Only 2 dimensionsional padding is currently supported.
+ //! \warning Only 2 dimensional padding is currently supported.
//!
//! \see setPrePaddingNd
//!
- virtual Dims getPrePaddingNd() const TRTNOEXCEPT = 0;
+ Dims getPrePaddingNd() const noexcept
+ {
+ return mImpl->getPrePaddingNd();
+ }
//!
//! \brief Set the padding that is applied at the end of the tensor.
//!
//! Negative padding results in trimming the edge by the specified amount
//!
- //! \warning Only 2 dimensionsional padding is currently supported.
+ //! \warning Only 2 dimensional padding is currently supported.
//!
//! \see getPostPaddingNd
//!
- virtual void setPostPaddingNd(Dims padding) TRTNOEXCEPT = 0;
+ void setPostPaddingNd(Dims padding) noexcept
+ {
+ mImpl->setPostPaddingNd(padding);
+ }
//!
//! \brief Get the padding that is applied at the end of the tensor.
//!
- //! \warning Only 2 dimensionsional padding is currently supported.
+ //! \warning Only 2 dimensional padding is currently supported.
//!
//! \see setPostPaddingNd
//!
- virtual Dims getPostPaddingNd() const TRTNOEXCEPT = 0;
+ Dims getPostPaddingNd() const noexcept
+ {
+ return mImpl->getPostPaddingNd();
+ }
+
+protected:
+ apiv::VPaddingLayer* mImpl;
+ virtual ~IPaddingLayer() noexcept = default;
};
struct Permutation
@@ -3725,7 +3669,7 @@ struct Permutation
//!
//! \brief Layer type for shuffling data.
//!
-//! This class shuffles data by applying in sequence: a transpose operation, a reshape operation
+//! This layer shuffles data by applying in sequence: a transpose operation, a reshape operation
//! and a second transpose operation. The dimension types of the output are those of the reshape dimension.
//!
//! The layer has an optional second input. If present, it must be a 1D Int32 shape tensor,
@@ -3745,7 +3689,10 @@ public:
//!
//! \see getFirstTranspose
//!
- virtual void setFirstTranspose(Permutation permutation) TRTNOEXCEPT = 0;
+ void setFirstTranspose(Permutation permutation) noexcept
+ {
+ mImpl->setFirstTranspose(permutation);
+ }
//!
//! \brief Get the permutation applied by the first transpose operation.
@@ -3754,7 +3701,10 @@ public:
//!
//! \see setFirstTranspose
//!
- virtual Permutation getFirstTranspose() const TRTNOEXCEPT = 0;
+ Permutation getFirstTranspose() const noexcept
+ {
+ return mImpl->getFirstTranspose();
+ }
//!
//! \brief Set the reshaped dimensions.
@@ -3774,9 +3724,12 @@ public:
//!
//! The product of the new dimensions must be equal to the product of the old.
//!
- //! If the second input is set, it is reset to null.
+ //! If a second input had been used to create this layer, that input is reset to null by this method.
//!
- virtual void setReshapeDimensions(Dims dimensions) TRTNOEXCEPT = 0;
+ void setReshapeDimensions(Dims dimensions) noexcept
+ {
+ mImpl->setReshapeDimensions(dimensions);
+ }
//!
//! \brief Get the reshaped dimensions.
@@ -3786,7 +3739,10 @@ public:
//! If a second input is present and non-null, or setReshapeDimensions has
//! not yet been called, this function returns Dims with nbDims == -1.
//!
- virtual Dims getReshapeDimensions() const TRTNOEXCEPT = 0;
+ Dims getReshapeDimensions() const noexcept
+ {
+ return mImpl->getReshapeDimensions();
+ }
//!
//! \brief Append or replace an input of this layer with a specific tensor
@@ -3807,7 +3763,16 @@ public:
//! If this function is called with a value 1, then the function getNbInputs() changes
//! from returning 1 to 2.
//!
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ //! The reshape dimensions are treated identically to how they are treated if set statically
+ //! via setReshapeDimensions. In particular, a -1 is treated as a wildcard even if dynamically
+ //! supplied at runtime, and a 0 is treated as a placeholder if getZeroIsPlaceholder() = true,
+ //! which is the default. If the placeholder interpretation of 0 is unwanted because the
+ //! runtime dimension should be 0 when the reshape dimension is 0, be sure to call
+ //! setZeroIsPlacholder(false) on the IShuffleLayer.
+ //!
+ //! \see setReshapeDimensions.
+ //!
+ using ILayer::setInput;
//!
//! \brief Set the permutation applied by the second transpose operation.
@@ -3821,7 +3786,10 @@ public:
//!
//! \see getSecondTranspose
//!
- virtual void setSecondTranspose(Permutation permutation) TRTNOEXCEPT = 0;
+ void setSecondTranspose(Permutation permutation) noexcept
+ {
+ mImpl->setSecondTranspose(permutation);
+ }
//!
//! \brief Get the permutation applied by the second transpose operation.
@@ -3830,12 +3798,11 @@ public:
//!
//! \see setSecondTranspose
//!
- virtual Permutation getSecondTranspose() const TRTNOEXCEPT = 0;
+ Permutation getSecondTranspose() const noexcept
+ {
+ return mImpl->getSecondTranspose();
+ }
-protected:
- virtual ~IShuffleLayer() {}
-
-public:
//!
//! \brief Set meaning of 0 in reshape dimensions.
//!
@@ -3847,7 +3814,10 @@ public:
//!
//! \see getZeroIsPlaceholder();
//!
- virtual void setZeroIsPlaceholder(bool zeroIsPlaceholder) = 0;
+ void setZeroIsPlaceholder(bool zeroIsPlaceholder) noexcept
+ {
+ return mImpl->setZeroIsPlaceholder(zeroIsPlaceholder);
+ }
//!
//! \brief Get meaning of 0 in reshape dimensions.
@@ -3857,7 +3827,14 @@ public:
//!
//! \see setZeroIsPlaceholder
//!
- virtual bool getZeroIsPlaceholder() const = 0;
+ bool getZeroIsPlaceholder() const noexcept
+ {
+ return mImpl->getZeroIsPlaceholder();
+ }
+
+protected:
+ apiv::VShuffleLayer* mImpl;
+ virtual ~IShuffleLayer() noexcept = default;
};
//!
@@ -3873,7 +3850,7 @@ enum class SliceMode : int32_t
//! Maximum number of elements in SliceMode enum. \see SliceMode
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 2;
}
@@ -3894,8 +3871,10 @@ constexpr inline int32_t EnumMax()
//! copies elements to the output tensor using the specified stride across the input tensor.
//! Start, size, and stride tensors must be 1D Int32 shape tensors if not specified via Dims.
//!
-//! Furthermore, if the slice layer must produce a shape tensor, then start, size, and stride must be
-//! build time constants, i.e. as static Dims, or be computable by constant folding.
+//! A slice layer can produce a shape tensor if the following conditions are met:
+//!
+//! * start, size, and stride are build time constants, either as static Dims, or computable by constant folding.
+//! * The number of elements in the output tensor does not exceed 2*Dims::MAX_DIMS.
//!
//! For example using slice on a tensor:
//! input = {{0, 2, 4}, {1, 3, 5}}
@@ -3914,11 +3893,14 @@ public:
//!
//! \param start The start offset to read data from the input tensor.
//!
- //! If the second input is set, it is reset to null.
+ //! If a second input had been used to create this layer, that input is reset to null by this method.
//!
//! \see getStart
//!
- virtual void setStart(Dims start) TRTNOEXCEPT = 0;
+ void setStart(Dims start) noexcept
+ {
+ mImpl->setStart(start);
+ }
//!
//! \brief Get the start offset for the slice layer.
@@ -3930,18 +3912,24 @@ public:
//!
//! \see setStart
//!
- virtual Dims getStart() const TRTNOEXCEPT = 0;
+ Dims getStart() const noexcept
+ {
+ return mImpl->getStart();
+ }
//!
//! \brief Set the dimensions of the output slice.
//!
//! \param size The dimensions of the output slice.
//!
- //! If the third input is set, it is reset to null.
+ //! If a third input had been used to create this layer, that input is reset to null by this method.
//!
//! \see getSize
//!
- virtual void setSize(Dims size) TRTNOEXCEPT = 0;
+ void setSize(Dims size) noexcept
+ {
+ return mImpl->setSize(size);
+ }
//!
//! \brief Get dimensions of the output slice.
@@ -3953,18 +3941,24 @@ public:
//!
//! \see setSize
//!
- virtual Dims getSize() const TRTNOEXCEPT = 0;
+ Dims getSize() const noexcept
+ {
+ return mImpl->getSize();
+ }
//!
//! \brief Set the stride for computing the output slice data.
//!
//! \param stride The dimensions of the stride to compute the values to store in the output slice.
//!
- //! If the fourth input is set, it is reset to null.
+ //! If a fourth input had been used to create this layer, that input is reset to null by this method.
//!
//! \see getStride
//!
- virtual void setStride(Dims stride) TRTNOEXCEPT = 0;
+ void setStride(Dims stride) noexcept
+ {
+ mImpl->setStride(stride);
+ }
//!
//! \brief Get the stride for the output slice.
@@ -3976,21 +3970,30 @@ public:
//!
//! \see setStride
//!
- virtual Dims getStride() const TRTNOEXCEPT = 0;
+ Dims getStride() const noexcept
+ {
+ return mImpl->getStride();
+ }
//!
//! \brief Set the slice mode.
//!
//! \see getMode()
//!
- virtual void setMode(SliceMode mode) TRTNOEXCEPT = 0;
+ void setMode(SliceMode mode) noexcept
+ {
+ mImpl->setMode(mode);
+ }
//!
//! \brief Get the slice mode.
//!
//! \see setMode()
//!
- virtual SliceMode getMode() const TRTNOEXCEPT = 0;
+ SliceMode getMode() const noexcept
+ {
+ return mImpl->getMode();
+ }
//!
//! \brief Append or replace an input of this layer with a specific tensor
@@ -4010,17 +4013,18 @@ public:
//! If this function is called with a value greater than 0, then the function getNbInputs() changes
//! from returning 1 to index + 1.
//!
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ using ILayer::setInput;
protected:
- virtual ~ISliceLayer() {}
+ apiv::VSliceLayer* mImpl;
+ virtual ~ISliceLayer() noexcept = default;
};
//! \class IShapeLayer
//!
//! \brief Layer type for getting shape of a tensor.
//!
-//! This class sets the output to a one-dimensional tensor with the dimensions of the input tensor.
+//! This layer sets the output to a one-dimensional tensor with the dimensions of the input tensor.
//!
//! For example, if the input is a four-dimensional tensor (of any type) with
//! dimensions [2,3,5,7], the output tensor is a one-dimensional Int32 tensor
@@ -4031,7 +4035,8 @@ protected:
class IShapeLayer : public ILayer
{
protected:
- virtual ~IShapeLayer() {}
+ apiv::VShapeLayer* mImpl;
+ virtual ~IShapeLayer() noexcept = default;
};
//!
@@ -4047,7 +4052,7 @@ enum class TopKOperation : int32_t
//! Maximum number of elements in TopKOperation enum. \see TopKOperation
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 2;
}
@@ -4067,14 +4072,20 @@ public:
//!
//! \see getOperation(), TopKOperation
//!
- virtual void setOperation(TopKOperation op) TRTNOEXCEPT = 0;
+ void setOperation(TopKOperation op) noexcept
+ {
+ mImpl->setOperation(op);
+ }
//!
//! \brief Get the operation for the layer.
//!
//! \see setOperation(), TopKOperation
//!
- virtual TopKOperation getOperation() const TRTNOEXCEPT = 0;
+ TopKOperation getOperation() const noexcept
+ {
+ return mImpl->getOperation();
+ }
//!
//! \brief Set the k value for the layer.
@@ -4083,31 +4094,44 @@ public:
//!
//! \see getK()
//!
- virtual void setK(int32_t k) TRTNOEXCEPT = 0;
+ void setK(int32_t k) noexcept
+ {
+ mImpl->setK(k);
+ }
//!
//! \brief Get the k value for the layer.
//!
//! \see setK()
//!
- virtual int32_t getK() const TRTNOEXCEPT = 0;
+ int32_t getK() const noexcept
+ {
+ return mImpl->getK();
+ }
//!
//! \brief Set which axes to reduce for the layer.
//!
//! \see getReduceAxes()
//!
- virtual void setReduceAxes(uint32_t reduceAxes) TRTNOEXCEPT = 0;
+ void setReduceAxes(uint32_t reduceAxes) noexcept
+ {
+ mImpl->setReduceAxes(reduceAxes);
+ }
//!
//! \brief Get the axes to reduce for the layer.
//!
//! \see setReduceAxes()
//!
- virtual uint32_t getReduceAxes() const TRTNOEXCEPT = 0;
+ uint32_t getReduceAxes() const noexcept
+ {
+ return mImpl->getReduceAxes();
+ }
protected:
- virtual ~ITopKLayer() {}
+ apiv::VTopKLayer* mImpl;
+ virtual ~ITopKLayer() noexcept = default;
};
//!
@@ -4140,7 +4164,7 @@ enum class MatrixOperation : int32_t
//! Maximum number of elements in MatrixOperation enum. \see DataType
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 3;
}
@@ -4177,38 +4201,26 @@ public:
//! \brief Set the operation for an input tensor.
//! \param index Input tensor number (0 or 1).
//! \param op New operation.
- //! \see getTranspose()
+ //! \see getOperation()
//!
- virtual void setOperation(int32_t index, MatrixOperation op) TRTNOEXCEPT = 0;
+ void setOperation(int32_t index, MatrixOperation op) noexcept
+ {
+ mImpl->setOperation(index, op);
+ }
//!
//! \brief Get the operation for an input tensor.
//! \param index Input tensor number (0 or 1).
- //! \see setTranspose()
+ //! \see setOperation()
//!
- virtual MatrixOperation getOperation(int32_t index) const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the transpose flag for an input tensor.
- //! \param index Input tensor number (0 or 1).
- //! \param val New transpose flag.
- //! \see getTranspose()
- //!
- //! \deprecated setTranspose is superseded by setOperation and will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED virtual void setTranspose(int32_t index, bool val) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the transpose flag for an input tensor.
- //! \param index Input tensor number (0 or 1).
- //! \see setTranspose()
- //!
- //! \deprecated getTranspose is superseded by getOperation and will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED virtual bool getTranspose(int32_t index) const TRTNOEXCEPT = 0;
+ MatrixOperation getOperation(int32_t index) const noexcept
+ {
+ return mImpl->getOperation(index);
+ }
protected:
- virtual ~IMatrixMultiplyLayer() {}
+ apiv::VMatrixMultiplyLayer* mImpl;
+ virtual ~IMatrixMultiplyLayer() noexcept = default;
};
//!
@@ -4228,23 +4240,26 @@ protected:
class IRaggedSoftMaxLayer : public ILayer
{
protected:
- virtual ~IRaggedSoftMaxLayer() {}
+ apiv::VRaggedSoftMaxLayer* mImpl;
+ virtual ~IRaggedSoftMaxLayer() noexcept = default;
};
//! \class IIdentityLayer
//!
//! \brief A layer that represents the identity function.
//!
-//! If tensor precision is being explicitly specified, it can be used to transform from one precision to another.
-//! Other than transforming between the same precision (kFLOAT -> kFLOAT for example), the only valid
-//! tranformations supported are: (kFLOAT -> kHALF), (kFLOAT -> kINT8), (kHALF -> kFLOAT) and (kINT8 -> kFLOAT).
+//! If tensor precision is being explicitly specified, it can be used to convert from one precision to another.
+//! Other than conversion between the same precision (kFLOAT -> kFLOAT for example), the only valid
+//! tranformations supported are: (kHALF -> kINT32), (kHALF -> kFLOAT), (kFLOAT -> kINT32), (kINT32 -> kHALF),
+//! (kINT32 -> kFLOAT), (kBOOL -> kBOOL), (kBOOL -> kHALF), (kBOOL -> kFLOAT).
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
class IIdentityLayer : public ILayer
{
protected:
- virtual ~IIdentityLayer() {}
+ apiv::VIdentityLayer* mImpl;
+ virtual ~IIdentityLayer() noexcept = default;
};
//! \class IConstantLayer
@@ -4266,14 +4281,20 @@ public:
//!
//! \see getWeights()
//!
- virtual void setWeights(Weights weights) TRTNOEXCEPT = 0;
+ void setWeights(Weights weights) noexcept
+ {
+ mImpl->setWeights(weights);
+ }
//!
//! \brief Get the weights for the layer.
//!
//! \see setWeights
//!
- virtual Weights getWeights() const TRTNOEXCEPT = 0;
+ Weights getWeights() const noexcept
+ {
+ return mImpl->getWeights();
+ }
//!
//! \brief Set the dimensions for the layer.
@@ -4282,7 +4303,10 @@ public:
//!
//! \see setDimensions
//!
- virtual void setDimensions(Dims dimensions) TRTNOEXCEPT = 0;
+ void setDimensions(Dims dimensions) noexcept
+ {
+ mImpl->setDimensions(dimensions);
+ }
//!
//! \brief Get the dimensions for the layer.
@@ -4291,10 +4315,14 @@ public:
//!
//! \see getDimensions
//!
- virtual Dims getDimensions() const TRTNOEXCEPT = 0;
+ Dims getDimensions() const noexcept
+ {
+ return mImpl->getDimensions();
+ }
protected:
- virtual ~IConstantLayer() {}
+ apiv::VConstantLayer* mImpl;
+ virtual ~IConstantLayer() noexcept = default;
};
//!
@@ -4307,7 +4335,8 @@ protected:
class IParametricReLULayer : public ILayer
{
protected:
- virtual ~IParametricReLULayer() noexcept {}
+ apiv::VParametricReLULayer* mImpl;
+ virtual ~IParametricReLULayer() noexcept = default;
};
//! \enum ResizeMode
@@ -4321,24 +4350,137 @@ enum class ResizeMode : int32_t
kLINEAR = 1 //!< Can handle linear (1D), bilinear (2D), and trilinear (3D) resizing.
};
+namespace impl
+{
//! Maximum number of elements in ResizeMode enum. \see ResizeMode
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 2;
-}
+ static constexpr int32_t kVALUE = 2;
+};
+} // namespace impl
+
+//!
+//! \enum ResizeCoordinateTransformation
+//!
+//! \brief The resize coordinate transformation function.
+//!
+//! \see IResizeLayer::setCoordinateTransformation()
+//!
+enum class ResizeCoordinateTransformation : int32_t
+{
+ //! We can think each value in tensor has a volume, and the coordinate is a point inside this volume.
+ //! The coordinate point is drawn as star(*) in below diagram, and multiple values range has a length.
+ //! Let's use x_origin as the coordinate of axis x in the input tensor, x_resized as the coordinate of axis x in the
+ //! output tensor, length_origin as length of the input tensor in axis x, and length_resize as length of the output
+ //! tensor in axis x.
+ //!
+ //! |<--------------length---------->|
+ //! | 0 | 1 | 2 | 3 |
+ //! * * * *
+ //!
+ //! x_origin = x_resized * (length_origin - 1) / (length_resize - 1)
+ //!
+ kALIGN_CORNERS = 0,
+
+ //! |<--------------length--------------------->|
+ //! | 0 | 1 | 2 | 3 |
+ //! * * * *
+ //!
+ //! x_origin = x_resized * (length_origin / length_resize)
+ //!
+ kASYMMETRIC = 1,
+
+ //! |<--------------length--------------------->|
+ //! | 0 | 1 | 2 | 3 |
+ //! * * * *
+ //!
+ //! x_origin = (x_resized + 0.5) * (length_origin / length_resize) - 0.5
+ //!
+ kHALF_PIXEL = 2,
+};
+
+namespace impl
+{
+//! Maximum number of elements in ResizeCoordinateTransformation enum. \see ResizeCoordinateTransformation
+template <>
+struct EnumMaxImpl
+{
+ static constexpr int32_t kVALUE = 3;
+};
+} // namespace impl
+
+//!
+//! \enum ResizeSelector
+//!
+//! \brief The coordinate selector when resize to single pixel output.
+//!
+//! \see IResizeLayer::setSelectorForSinglePixel()
+//!
+enum class ResizeSelector : int32_t
+{
+ //! Use formula to map the original index.
+ kFORMULA = 0,
+
+ //! Select the upper left pixel.
+ kUPPER = 1,
+};
+
+namespace impl
+{
+//! Maximum number of elements in ResizeSelector enum. \see ResizeSelector
+template <>
+struct EnumMaxImpl
+{
+ static constexpr int32_t kVALUE = 2;
+};
+} // namespace impl
+
+//!
+//! \enum ResizeRoundMode
+//!
+//! \brief The rounding mode for nearest neighbor resize.
+//!
+//!
+//! \see IResizeLayer::setNearestRounding()
+//!
+enum class ResizeRoundMode : int32_t
+{
+ //! Round half up.
+ kHALF_UP = 0,
+
+ //! Round half down.
+ kHALF_DOWN = 1,
+
+ //! Round to floor.
+ kFLOOR = 2,
+
+ //! Round to ceil.
+ kCEIL = 3,
+};
+
+namespace impl
+{
+//! Maximum number of elements in ResizeRoundMode enum. \see ResizeRoundMode
+template <>
+struct EnumMaxImpl
+{
+ static constexpr int32_t kVALUE = 4;
+};
+} // namespace impl
//! \class IResizeLayer
//!
//! \brief A resize layer in a network definition.
//!
-//! Resize layer can be used for resizing a ND tensor.
+//! Resize layer can be used for resizing a N-D tensor.
//!
//! Resize layer currently supports the following configurations:
-//! - ResizeMode::kNEAREST - resizes innermost `m` dimensions of ND, where 0 < m <= min(8, N) and N > 0
-//! - ResizeMode::kLINEAR - resizes innermost `m` dimensions of ND, where 0 < m <= min(3, N) and N > 0
+//! - ResizeMode::kNEAREST - resizes innermost `m` dimensions of N-D, where 0 < m <= min(8, N) and N > 0
+//! - ResizeMode::kLINEAR - resizes innermost `m` dimensions of N-D, where 0 < m <= min(3, N) and N > 0
//!
//! Default resize mode is ResizeMode::kNEAREST.
+//!
//! Resize layer provides two ways to resize tensor dimensions.
//! - Set output dimensions directly. It can be done for static as well as dynamic resize layer.
//! Static resize layer requires output dimensions to be known at build-time.
@@ -4354,7 +4496,8 @@ public:
//!
//! \brief Set the output dimensions.
//!
- //! \param dimensions The output dimensions. Number of output dimensions must be the same as the number of input dimensions.
+ //! \param dimensions The output dimensions. Number of output dimensions must be the same as the number of input
+ //! dimensions.
//!
//! If there is a second input, i.e. resize layer is dynamic,
//! calling setOutputDimensions() is an error and does not update the
@@ -4366,14 +4509,20 @@ public:
//! \see setScales
//! \see getOutputDimensions
//!
- virtual void setOutputDimensions(Dims dimensions) TRTNOEXCEPT = 0;
+ void setOutputDimensions(Dims dimensions) noexcept
+ {
+ return mImpl->setOutputDimensions(dimensions);
+ }
//!
//! \brief Get the output dimensions.
//!
//! \return The output dimensions.
//!
- virtual Dims getOutputDimensions() const TRTNOEXCEPT = 0;
+ Dims getOutputDimensions() const noexcept
+ {
+ return mImpl->getOutputDimensions();
+ }
//!
//! \brief Set the resize scales.
@@ -4393,7 +4542,10 @@ public:
//! \see setOutputDimensions
//! \see getScales
//!
- virtual void setScales(const float* scales, int32_t nbScales) TRTNOEXCEPT = 0;
+ void setScales(const float* scales, int32_t nbScales) noexcept
+ {
+ mImpl->setScales(scales, nbScales);
+ }
//!
//! \brief Copies resize scales to scales[0, ..., nbScales-1], where nbScales is the number of scales that were set.
@@ -4409,7 +4561,10 @@ public:
//! \return The number of resize scales i.e. nbScales if scales were set.
//! Return -1 in case no scales were set or resize layer is used in dynamic mode.
//!
- virtual int32_t getScales(int32_t size, float* scales) const TRTNOEXCEPT = 0;
+ int32_t getScales(int32_t size, float* scales) const noexcept
+ {
+ return mImpl->getScales(size, scales);
+ }
//!
//! \brief Set resize mode for an input tensor.
@@ -4418,14 +4573,20 @@ public:
//!
//! \see ResizeMode
//!
- virtual void setResizeMode(ResizeMode resizeMode) TRTNOEXCEPT = 0;
+ void setResizeMode(ResizeMode resizeMode) noexcept
+ {
+ mImpl->setResizeMode(resizeMode);
+ }
//!
//! \brief Get resize mode for an input tensor.
//!
//! \return The resize mode.
//!
- virtual ResizeMode getResizeMode() const TRTNOEXCEPT = 0;
+ ResizeMode getResizeMode() const noexcept
+ {
+ return mImpl->getResizeMode();
+ }
//!
//! \brief Set whether to align corners while resizing.
@@ -4436,20 +4597,32 @@ public:
//!
//! Default: false.
//!
- virtual void setAlignCorners(bool alignCorners) TRTNOEXCEPT = 0;
+ //! \deprecated Superseded by IResizeLayer::setCoordinateTransformation() and
+ //! will be removed in TensorRT 10.0.
+ //!
+ TRT_DEPRECATED void setAlignCorners(bool alignCorners) noexcept
+ {
+ mImpl->setAlignCorners(alignCorners);
+ }
//!
//! \brief True if align corners has been set.
//!
//! \return True if align corners has been set, false otherwise.
//!
- virtual bool getAlignCorners() const TRTNOEXCEPT = 0;
+ //! \deprecated Superseded by IResizeLayer::getCoordinateTransformation() and
+ //! will be removed in TensorRT 10.0.
+ //!
+ TRT_DEPRECATED bool getAlignCorners() const noexcept
+ {
+ return mImpl->getAlignCorners();
+ }
//!
//! \brief Append or replace an input of this layer with a specific tensor
//!
//! \param index the index of the input to modify.
- //! \param tensor the new input tensor
+ //! \param tensor the new input tensor.
//!
//! Sets the input tensor for the given index. The index must be 0 for a static resize layer.
//! A static resize layer is converted to a dynamic resize layer by calling setInput with an index 1.
@@ -4464,10 +4637,84 @@ public:
//! If this function is called with a value 1, then the function getNbInputs() changes
//! from returning 1 to 2.
//!
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ using ILayer::setInput;
+
+ //!
+ //! \brief Set coordinate transformation function.
+ //!
+ //! We have different functions mapping the coordinate in output tensor to the coordinate in input tensor.
+ //!
+ //! Default is ResizeCoordinateTransformation::kASYMMETRIC.
+ //!
+ //! \see ResizeCoordinateTransformation
+ //!
+ void setCoordinateTransformation(ResizeCoordinateTransformation coordTransform) noexcept
+ {
+ mImpl->setCoordinateTransformation(coordTransform);
+ }
+
+ //!
+ //! \brief Get coordinate transformation function.
+ //!
+ //! \return The coordinate transformation function.
+ //!
+ ResizeCoordinateTransformation getCoordinateTransformation() const noexcept
+ {
+ return mImpl->getCoordinateTransformation();
+ }
+
+ //!
+ //! \brief Set coordinate selector function when resized to single pixel.
+ //!
+ //! When resize to single pixel image, use this function to decide how to map the coordinate in the original
+ //! image.
+ //!
+ //! Default is ResizeSelector::kFORMULA.
+ //!
+ //! \see ResizeSelector
+ //!
+ void setSelectorForSinglePixel(ResizeSelector selector) noexcept
+ {
+ mImpl->setSelectorForSinglePixel(selector);
+ }
+
+ //!
+ //! \brief Get the coordinate selector function when resized to single pixel.
+ //!
+ //! \return The selector function.
+ //!
+ ResizeSelector getSelectorForSinglePixel() const noexcept
+ {
+ return mImpl->getSelectorForSinglePixel();
+ }
+
+ //!
+ //! \brief Set rounding mode for nearest neighbor resize.
+ //!
+ //! This value is used for nearest neighbor interpolation rounding. It is applied after coordinate transformation.
+ //!
+ //! Default is kFLOOR.
+ //!
+ //! \see ResizeRoundMode
+ //!
+ void setNearestRounding(ResizeRoundMode value) noexcept
+ {
+ mImpl->setNearestRounding(value);
+ }
+
+ //!
+ //! \brief Get rounding mode for nearest neighbor resize.
+ //!
+ //! \return The rounding mode.
+ //!
+ ResizeRoundMode getNearestRounding() const noexcept
+ {
+ return mImpl->getNearestRounding();
+ }
protected:
- virtual ~IResizeLayer() {}
+ virtual ~IResizeLayer() noexcept = default;
+ apiv::VResizeLayer* mImpl;
};
//! Enum that describes kinds of loop outputs.
@@ -4485,7 +4732,7 @@ enum class LoopOutput : int32_t
//! Maximum number of elements in LoopOutput enum. \see DataType
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 3;
}
@@ -4500,7 +4747,7 @@ enum class TripLimit : int32_t
//! Maximum number of elements in TripLimit enum. \see DataType
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 2;
}
@@ -4511,7 +4758,14 @@ class ILoopBoundaryLayer : public ILayer
{
public:
//! Return pointer to ILoop associated with this boundary layer.
- virtual ILoop* getLoop() const noexcept = 0;
+ ILoop* getLoop() const noexcept
+ {
+ return mBoundary->getLoop();
+ }
+
+protected:
+ virtual ~ILoopBoundaryLayer() noexcept = default;
+ apiv::VLoopBoundaryLayer* mBoundary;
};
class IRecurrenceLayer : public ILoopBoundaryLayer
@@ -4535,7 +4789,11 @@ public:
//! If this function is called with a value 1, then the function getNbInputs() changes
//! from returning 1 to 2.
//!
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ using ILayer::setInput;
+
+protected:
+ virtual ~IRecurrenceLayer() noexcept = default;
+ apiv::VRecurrenceLayer* mImpl;
};
//!
@@ -4558,7 +4816,10 @@ public:
class ILoopOutputLayer : public ILoopBoundaryLayer
{
public:
- virtual LoopOutput getLoopOutput() const noexcept = 0;
+ LoopOutput getLoopOutput() const noexcept
+ {
+ return mImpl->getLoopOutput();
+ }
//!
//! \brief Set where to insert the contenation axis. Ignored if getLoopOutput() is kLAST_VALUE.
@@ -4572,10 +4833,16 @@ public:
//! setAxis(3) causes the output to have dimensions [b,c,d,a].
//! Default is axis is 0.
//!
- virtual void setAxis(int32_t axis) noexcept = 0;
+ void setAxis(int32_t axis) noexcept
+ {
+ mImpl->setAxis(axis);
+ }
//! Get axis being concatenated over.
- virtual int32_t getAxis() const noexcept = 0;
+ int32_t getAxis() const noexcept
+ {
+ return mImpl->getAxis();
+ }
//!
//! \brief Append or replace an input of this layer with a specific tensor
@@ -4597,49 +4864,80 @@ public:
//! If this function is called with a value 1, then the function getNbInputs() changes
//! from returning 1 to 2.
//!
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ using ILayer::setInput;
+
+protected:
+ virtual ~ILoopOutputLayer() noexcept = default;
+ apiv::VLoopOutputLayer* mImpl;
};
class ITripLimitLayer : public ILoopBoundaryLayer
{
public:
- virtual TripLimit getTripLimit() const noexcept = 0;
+ TripLimit getTripLimit() const noexcept
+ {
+ return mImpl->getTripLimit();
+ }
+
+protected:
+ virtual ~ITripLimitLayer() noexcept = default;
+ apiv::VTripLimitLayer* mImpl;
};
class IIteratorLayer : public ILoopBoundaryLayer
{
public:
//! Set axis to iterate over.
- virtual void setAxis(int32_t axis) noexcept = 0;
+ void setAxis(int32_t axis) noexcept
+ {
+ mImpl->setAxis(axis);
+ }
//! Get axis being iterated over.
- virtual int32_t getAxis() const noexcept = 0;
+ int32_t getAxis() const noexcept
+ {
+ return mImpl->getAxis();
+ }
//! For reverse=false, the layer is equivalent to addGather(tensor, I, 0) where I is a
//! scalar tensor containing the loop iteration number.
//! For reverse=true, the layer is equivalent to addGather(tensor, M-1-I, 0) where M is the trip count
//! computed from TripLimits of kind kCOUNT.
//! The default is reverse=false.
- virtual void setReverse(bool reverse) noexcept = 0;
+ void setReverse(bool reverse) noexcept
+ {
+ mImpl->setReverse(reverse);
+ }
//! True if and only if reversing input.
- virtual bool getReverse() const noexcept = 0;
+ bool getReverse() const noexcept
+ {
+ return mImpl->getReverse();
+ }
+
+protected:
+ virtual ~IIteratorLayer() noexcept = default;
+ apiv::VIteratorLayer* mImpl;
};
//!
//! Helper for creating a recurrent subgraph.
//!
-class ILoop
+//! An ILoop cannot be added to an INetworkDefinition where hasImplicitBatchDimensions() returns true.
+//!
+class ILoop : public INoCopy
{
public:
//!
//! \brief Create a recurrence layer for this loop with initialValue as its first input.
//!
- //! IRecurrenceLayer requires exactly two inputs. The 2nd input must be added, via method IRecurrenceLayer::setInput(1,...)
- //! before an Engine can be built.
- //
+ //! IRecurrenceLayer requires exactly two inputs. The 2nd input must be added, via method
+ //! IRecurrenceLayer::setInput(1,...) before an Engine can be built.
//!
- virtual IRecurrenceLayer* addRecurrence(ITensor& initialValue) noexcept = 0;
+ IRecurrenceLayer* addRecurrence(ITensor& initialValue) noexcept
+ {
+ return mImpl->addRecurrence(initialValue);
+ }
//!
//! \brief Add a trip-count limiter, based on the given tensor.
@@ -4649,7 +4947,7 @@ public:
//! count is reached or condition is falsified.
//! It is an error to not add at least one trip limiter.
//!
- //! For kTRIP_LIMIT, the input tensor must be available before the loop starts.
+ //! For kCOUNT, the input tensor must be available before the loop starts.
//!
//! For kWHILE, the input tensor must be the output of a subgraph that contains
//! only layers that are not ITripLimitLayer, IIteratorLayer or ILoopOutputLayer.
@@ -4657,7 +4955,10 @@ public:
//! ITripLimitLayer. A trivial example of this rule is that the input to the kWHILE
//! is the output of an IRecurrenceLayer for the same loop.
//!
- virtual ITripLimitLayer* addTripLimit(ITensor& tensor, TripLimit limit) noexcept = 0;
+ ITripLimitLayer* addTripLimit(ITensor& tensor, TripLimit limit) noexcept
+ {
+ return mImpl->addTripLimit(tensor, limit);
+ }
//!
//! \brief Return layer that subscripts tensor by loop iteration.
@@ -4667,7 +4968,10 @@ public:
//! For reverse=true, this is equivalent to addGather(tensor, M-1-I, 0) where M is the trip count
//! computed from TripLimits of kind kCOUNT.
//!
- virtual IIteratorLayer* addIterator(ITensor& tensor, int32_t axis = 0, bool reverse = false) noexcept = 0;
+ IIteratorLayer* addIterator(ITensor& tensor, int32_t axis = 0, bool reverse = false) noexcept
+ {
+ return mImpl->addIterator(tensor, axis, reverse);
+ }
//! \brief Make an output for this loop, based on the given tensor.
//!
@@ -4676,7 +4980,10 @@ public:
//! If outputKind is kCONCATENATE or kREVERSE, a second input specifying the
//! concatenation dimension must be added via method ILoopOutputLayer::setInput.
//!
- virtual ILoopOutputLayer* addLoopOutput(ITensor& tensor, LoopOutput outputKind, int32_t axis = 0) noexcept = 0;
+ ILoopOutputLayer* addLoopOutput(ITensor& tensor, LoopOutput outputKind, int32_t axis = 0) noexcept
+ {
+ return mImpl->addLoopOutput(tensor, outputKind, axis);
+ }
//!
//! \brief Set the name of the loop.
@@ -4686,17 +4993,24 @@ public:
//!
//! \see getName()
//!
- virtual void setName(const char* name) noexcept = 0;
+ void setName(const char* name) noexcept
+ {
+ mImpl->setName(name);
+ }
//!
//! \brief Return the name of the loop.
//!
//! \see setName()
//!
- virtual const char* getName() const noexcept = 0;
+ const char* getName() const noexcept
+ {
+ return mImpl->getName();
+ }
protected:
- virtual ~ILoop() {}
+ virtual ~ILoop() noexcept = default;
+ apiv::VLoop* mImpl;
};
//!
@@ -4705,7 +5019,8 @@ protected:
class ISelectLayer : public ILayer
{
protected:
- virtual ~ISelectLayer() {}
+ virtual ~ISelectLayer() noexcept = default;
+ apiv::VSelectLayer* mImpl;
};
//!
@@ -4723,7 +5038,7 @@ enum class FillOperation : int32_t
//! Maximum number of elements in FillOperation enum. \see FillOperation
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 2;
}
@@ -4743,6 +5058,13 @@ constexpr inline int32_t EnumMax()
//! Alpha and Beta are treated differently based on the Fill Operation specified. See details in
//! IFillLayer::setAlpha(), IFillLayer::setBeta(), and IFillLayer::setInput().
//!
+//! A fill layer can produce a shape tensor if the following restrictions are met:
+//!
+//! * The FillOperation is kLINSPACE.
+//! * The output is a 1D Int32 tensor with length not exceeding 2*Dims::MAX_DIMS.
+//! * There is at most one input, and if so, that input is input 0.
+//! * If input 0 exists, the length of the output tensor must be computable by constant folding.
+//!
//! \see FillOperation
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
@@ -4754,11 +5076,14 @@ public:
//!
//! \param dimensions The output tensor's dimensions.
//!
- //! If the first input is set, it is reset to null.
+ //! If the first input had been used to create this layer, that input is reset to null by this method.
//!
//! \see getDimensions
//
- virtual void setDimensions(Dims dimensions) noexcept = 0;
+ void setDimensions(Dims dimensions) noexcept
+ {
+ mImpl->setDimensions(dimensions);
+ }
//!
//! \brief Get the output tensor's dimensions.
@@ -4770,21 +5095,30 @@ public:
//!
//! \see setDimensions
//!
- virtual Dims getDimensions() const noexcept = 0;
+ Dims getDimensions() const noexcept
+ {
+ return mImpl->getDimensions();
+ }
//!
//! \brief Set the fill operation for the layer.
//!
//! \see getOperation(), FillOperation
//!
- virtual void setOperation(FillOperation op) noexcept = 0;
+ void setOperation(FillOperation op) noexcept
+ {
+ mImpl->setOperation(op);
+ }
//!
//! \brief Get the fill operation for the layer.
//!
//! \see setOperation(), FillOperation
//!
- virtual FillOperation getOperation() const noexcept = 0;
+ FillOperation getOperation() const noexcept
+ {
+ return mImpl->getOperation();
+ }
//!
//! \brief Set the alpha parameter.
@@ -4795,11 +5129,14 @@ public:
//! kLINSPACE | the start value;
//! kRANDOMUNIFORM | the minimum value;
//!
- //! If the second input is set, it is reset to null.
+ //! If a second input had been used to create this layer, that input is reset to null by this method.
//!
//! \see getAlpha
//
- virtual void setAlpha(double alpha) noexcept = 0;
+ void setAlpha(double alpha) noexcept
+ {
+ mImpl->setAlpha(alpha);
+ }
//!
//! \brief Get the value of alpha parameter.
@@ -4811,7 +5148,10 @@ public:
//!
//! \see setAlpha
//!
- virtual double getAlpha() const noexcept = 0;
+ double getAlpha() const noexcept
+ {
+ return mImpl->getAlpha();
+ }
//!
//! \brief Set the beta parameter.
@@ -4822,11 +5162,14 @@ public:
//! kLINSPACE | the delta value;
//! kRANDOMUNIFORM | the maximal value;
//!
- //! If the third input is set, it is reset to null.
+ //! If a third input had been used to create this layer, that input is reset to null by this method.
//!
//! \see getBeta
//!
- virtual void setBeta(double beta) noexcept = 0;
+ void setBeta(double beta) noexcept
+ {
+ mImpl->setBeta(beta);
+ }
//!
//! \brief Get the value of beta parameter.
@@ -4838,7 +5181,10 @@ public:
//!
//! \see setBeta
//!
- virtual double getBeta() const noexcept = 0;
+ double getBeta() const noexcept
+ {
+ return mImpl->getBeta();
+ }
//!
//! \brief replace an input of this layer with a specific tensor.
@@ -4866,10 +5212,187 @@ public:
//! then afterwards getNbInputs() returns index + 1, and any missing intervening
//! inputs are set to null.
//!
- void setInput(int32_t index, ITensor& tensor) _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ using ILayer::setInput;
protected:
- virtual ~IFillLayer() {}
+ virtual ~IFillLayer() noexcept = default;
+ apiv::VFillLayer* mImpl;
+};
+
+//!
+//! \class IQuantizeLayer
+//!
+//! \brief A Quantize layer in a network definition.
+//!
+//! This layer accepts a floating-point data input tensor, and uses the scale and zeroPt inputs to
+//! quantize the data to an 8-bit signed integer according to:
+//! \p output = clamp(round(\p input / \p scale) + \p zeroPt)
+//!
+//! Rounding type is rounding-to-nearest ties-to-even (https://en.wikipedia.org/wiki/Rounding#Round_half_to_even).
+//! Clamping is in the range [-128, 127].
+//!
+//! The first input (index 0) is the tensor to be quantized.
+//! The second (index 1) and third (index 2) are the scale and zero point respectively.
+//! Each of \p scale and \p zeroPt must be either a scalar, or a 1D tensor.
+//!
+//! The \p zeroPt tensor is optional, and if not set, will be assumed to be zero. Its data type must be
+//! DataType::kINT8. \p zeroPt must only contain zero-valued coefficients, because only symmetric quantization is
+//! supported.
+//! The \p scale value must be either a scalar for per-tensor quantization, or a 1D tensor for per-channel
+//! quantization. All \p scale coefficients must have positive values. The size of the 1-D \p scale tensor must match
+//! the size of the quantization axis. The size of the \p scale must match the size of the \p zeroPt.
+//!
+//! The subgraph which terminates with the \p scale tensor must be a build-time constant. The same restrictions apply
+//! to the \p zeroPt.
+//! The output type, if constrained, must be constrained to DataType::kINT8. The input type, if constrained, must be
+//! constrained to DataType::kFLOAT (FP16 input is not supported).
+//! The output size is the same as the input size. The quantization axis is in reference to the input tensor's
+//! dimensions.
+//!
+//! IQuantizeLayer only supports DataType::kFLOAT precision and will default to this precision during instantiation.
+//! IQuantizeLayer only supports DataType::kINT8 output.
+//!
+//! As an example of the operation of this layer, imagine a 4D NCHW activation input which can be quantized using a
+//! single scale coefficient (referred to as per-tensor quantization):
+//! For each n in N:
+//! For each c in C:
+//! For each h in H:
+//! For each w in W:
+//! output[n,c,h,w] = clamp(round(\p input[n,c,h,w] / \p scale) + \p zeroPt)
+//!
+//! Per-channel quantization is supported only for weight inputs. Thus, Activations cannot be quantized per-channel.
+//! As an example of per-channel operation, imagine a 4D KCRS weights input and K (dimension 0) as the quantization
+//! axis. The scale is an array of coefficients, and must have the same size as the quantization axis.
+//! For each k in K:
+//! For each c in C:
+//! For each r in R:
+//! For each s in S:
+//! output[k,c,r,s] = clamp(round(\p input[k,c,r,s] / \p scale[k]) + \p zeroPt[k])
+//!
+//! \note Only symmetric quantization is supported.
+//! \note Currently the only allowed build-time constant \p scale and \zeroPt subgraphs are:
+//! 1. Constant -> Quantize
+//! 2. Constant -> Cast -> Quantize
+//!
+//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
+//!
+class IQuantizeLayer : public ILayer
+{
+public:
+ //!
+ //! \brief Get the quantization axis.
+ //!
+ //! \return axis parameter set by setAxis().
+ //! The return value is the index of the quantization axis in the input tensor's dimensions.
+ //! A value of -1 indicates per-tensor quantization.
+ //! The default value is -1.
+ //!
+ int32_t getAxis() const noexcept
+ {
+ return mImpl->getAxis();
+ }
+ //!
+ //! \brief Set the quantization axis.
+ //!
+ //! Set the index of the quantization axis (with reference to the input tensor's dimensions).
+ //! The axis must be a valid axis if the scale tensor has more than one coefficient.
+ //! The axis value will be ignored if the scale tensor has exactly one coefficient (per-tensor quantization).
+ //!
+ void setAxis(int32_t axis) noexcept
+ {
+ mImpl->setAxis(axis);
+ }
+
+protected:
+ virtual ~IQuantizeLayer() noexcept = default;
+ apiv::VQuantizeLayer* mImpl;
+};
+
+//!
+//! \class IDequantizeLayer
+//!
+//! \brief A Dequantize layer in a network definition.
+//!
+//! This layer accepts a signed 8-bit integer input tensor, and uses the configured scale and zeroPt inputs to
+//! dequantize the input according to:
+//! \p output = (\p input - \p zeroPt) * \p scale
+//!
+//! The first input (index 0) is the tensor to be quantized.
+//! The second (index 1) and third (index 2) are the scale and zero point respectively.
+//! Each of \p scale and \p zeroPt must be either a scalar, or a 1D tensor.
+//!
+//! The \p zeroPt tensor is optional, and if not set, will be assumed to be zero. Its data type must be
+//! DataType::kINT8. \p zeroPt must only contain zero-valued coefficients, because only symmetric quantization is
+//! supported.
+//! The \p scale value must be either a scalar for per-tensor quantization, or a 1D tensor for per-channel
+//! quantization. All \p scale coefficients must have positive values. The size of the 1-D \p scale tensor must match
+//! the size of the quantization axis. The size of the \p scale must match the size of the \p zeroPt.
+//!
+//! The subgraph which terminates with the \p scale tensor must be a build-time constant. The same restrictions apply
+//! to the \p zeroPt.
+//! The output type, if constrained, must be constrained to DataType::kINT8. The input type, if constrained, must be
+//! constrained to DataType::kFLOAT (FP16 input is not supported).
+//! The output size is the same as the input size. The quantization axis is in reference to the input tensor's
+//! dimensions.
+//!
+//! IDequantizeLayer only supports DataType::kINT8 precision and will default to this precision during instantiation.
+//! IDequantizeLayer only supports DataType::kFLOAT output.
+//!
+//! As an example of the operation of this layer, imagine a 4D NCHW activation input which can be quantized using a
+//! single scale coefficient (referred to as per-tensor quantization):
+//! For each n in N:
+//! For each c in C:
+//! For each h in H:
+//! For each w in W:
+//! output[n,c,h,w] = (\p input[n,c,h,w] - \p zeroPt) * \p scale
+//!
+//! Per-channel dequantization is supported only for input that is rooted at an IConstantLayer (i.e. weights).
+//! Activations cannot be quantized per-channel. As an example of per-channel operation, imagine a 4D KCRS weights input
+//! and K (dimension 0) as the quantization axis. The scale is an array of coefficients, which is the same size as the
+//! quantization axis.
+//! For each k in K:
+//! For each c in C:
+//! For each r in R:
+//! For each s in S:
+//! output[k,c,r,s] = (\p input[k,c,r,s] - \p zeroPt[k]) * \p scale[k]
+//!
+//! \note Only symmetric quantization is supported.
+//! \note Currently the only allowed build-time constant \p scale and \zeroPt subgraphs are:
+//! 1. Constant -> Quantize
+//! 2. Constant -> Cast -> Quantize
+//!
+//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
+//!
+class IDequantizeLayer : public ILayer
+{
+public:
+ //!
+ //! \brief Get the quantization axis.
+ //!
+ //! \return axis parameter set by setAxis().
+ //! The return value is the index of the quantization axis in the input tensor's dimensions.
+ //! A value of -1 indicates per-tensor quantization.
+ //! The default value is -1.
+ //!
+ int32_t getAxis() const noexcept
+ {
+ return mImpl->getAxis();
+ }
+ //!
+ //! \brief Set the quantization axis.
+ //!
+ //! Set the index of the quantization axis (with reference to the input tensor's dimensions).
+ //! The axis must be a valid axis if the scale tensor has more than one coefficient.
+ //! The axis value will be ignored if the scale tensor has exactly one coefficient (per-tensor quantization).
+ //!
+ void setAxis(int32_t axis) noexcept
+ {
+ mImpl->setAxis(axis);
+ }
+
+protected:
+ virtual ~IDequantizeLayer() noexcept = default;
+ apiv::VDequantizeLayer* mImpl;
};
//!
@@ -4880,7 +5403,7 @@ protected:
//! A network definition defines the structure of the network, and combined with a IBuilderConfig, is built
//! into an engine using an IBuilder. An INetworkDefinition can either have an implicit batch dimensions, specified
//! at runtime, or all dimensions explicit, full dims mode, in the network definition. When a network has been
-//! created using createNetwork(), only implicit batch size mode is supported. The function hasImplicitBatchSize()
+//! created using createNetwork(), only implicit batch size mode is supported. The function hasImplicitBatchDimension()
//! is used to query the mode of the network.
//!
//! A network with implicit batch dimensions returns the dimensions of a layer without the implicit dimension,
@@ -4891,15 +5414,17 @@ protected:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class INetworkDefinition
+class INetworkDefinition : public INoCopy
{
public:
+ virtual ~INetworkDefinition() noexcept = default;
+
//!
//! \brief Add an input tensor to the network.
//!
//! The name of the input tensor is used to find the index into the buffer array for an engine built from
//! the network. The volume of the dimensions must be less than 2^30 elements.
-
+ //!
//! For networks with an implicit batch dimension, this volume includes the batch dimension with its length set
//! to the maximum batch size. For networks with all explicit dimensions and with wildcard dimensions, the volume
//! is based on the maxima specified by an IOptimizationProfile.Dimensions are normally non-negative integers. The
@@ -4907,7 +5432,7 @@ public:
//! be specified at runtime. Input tensors with such a wildcard must have a corresponding entry in the
//! IOptimizationProfiles indicating the permitted extrema, and the input dimensions must be set by
//! IExecutionContext::setBindingDimensions. Different IExecutionContext instances can have different dimensions.
- //! Wildcard dimensions are only supported for EngineCapability::kDEFAULT. They are not
+ //! Wildcard dimensions are only supported for EngineCapability::kSTANDARD. They are not
//! supported in safety contexts. DLA does not support Wildcard dimensions.
//!
//! Tensor dimensions are specified independent of format. For example, if a
@@ -4929,7 +5454,10 @@ public:
//!
//! \return The new tensor or nullptr if there is an error.
//!
- virtual ITensor* addInput(const char* name, DataType type, Dims dimensions) TRTNOEXCEPT = 0;
+ ITensor* addInput(const char* name, DataType type, Dims dimensions) noexcept
+ {
+ return mImpl->addInput(name, type, dimensions);
+ }
//!
//! \brief Mark a tensor as a network output.
@@ -4938,7 +5466,10 @@ public:
//!
//! \warning It is an error to mark a network input as an output.
//!
- virtual void markOutput(ITensor& tensor) TRTNOEXCEPT = 0;
+ void markOutput(ITensor& tensor) noexcept
+ {
+ mImpl->markOutput(tensor);
+ }
//!
//! \brief Add a convolution layer to the network.
@@ -4958,8 +5489,11 @@ public:
//!
//! \deprecated Superseded by addConvolutionNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual IConvolutionLayer* addConvolution(ITensor& input, int32_t nbOutputMaps, DimsHW kernelSize,
- Weights kernelWeights, Weights biasWeights) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED IConvolutionLayer* addConvolution(
+ ITensor& input, int32_t nbOutputMaps, DimsHW kernelSize, Weights kernelWeights, Weights biasWeights) noexcept
+ {
+ return mImpl->addConvolution(input, nbOutputMaps, kernelSize, kernelWeights, biasWeights);
+ }
//!
//! \brief Add a fully connected layer to the network.
@@ -4976,8 +5510,11 @@ public:
//!
//! \return The new fully connected layer, or nullptr if it could not be created.
//!
- virtual IFullyConnectedLayer* addFullyConnected(
- ITensor& input, int32_t nbOutputs, Weights kernelWeights, Weights biasWeights) TRTNOEXCEPT = 0;
+ IFullyConnectedLayer* addFullyConnected(
+ ITensor& input, int32_t nbOutputs, Weights kernelWeights, Weights biasWeights) noexcept
+ {
+ return mImpl->addFullyConnected(input, nbOutputs, kernelWeights, biasWeights);
+ }
//!
//! \brief Add an activation layer to the network.
@@ -4993,7 +5530,10 @@ public:
//!
//! \return The new activation layer, or nullptr if it could not be created.
//!
- virtual IActivationLayer* addActivation(ITensor& input, ActivationType type) TRTNOEXCEPT = 0;
+ IActivationLayer* addActivation(ITensor& input, ActivationType type) noexcept
+ {
+ return mImpl->addActivation(input, type);
+ }
//!
//! \brief Add a pooling layer to the network.
@@ -5009,8 +5549,10 @@ public:
//!
//! \deprecated Superseded by addPoolingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual IPoolingLayer* addPooling(
- ITensor& input, PoolingType type, DimsHW windowSize) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED IPoolingLayer* addPooling(ITensor& input, PoolingType type, DimsHW windowSize) noexcept
+ {
+ return mImpl->addPooling(input, type, windowSize);
+ }
//!
//! \brief Add a LRN layer to the network.
@@ -5026,12 +5568,17 @@ public:
//!
//! \return The new LRN layer, or nullptr if it could not be created.
//!
- virtual ILRNLayer* addLRN(ITensor& input, int32_t window, float alpha, float beta, float k) TRTNOEXCEPT = 0;
+ ILRNLayer* addLRN(ITensor& input, int32_t window, float alpha, float beta, float k) noexcept
+ {
+ return mImpl->addLRN(input, window, alpha, beta, k);
+ }
//!
//! \brief Add a Scale layer to the network.
//!
- //! \param input The input tensor to the layer. This tensor is required to have a minimum of 3 dimensions.
+ //! \param input The input tensor to the layer.
+ //! This tensor is required to have a minimum of 3 dimensions in implicit batch mode
+ //! and a minimum of 4 dimensions in explicit batch mode.
//! \param mode The scaling mode.
//! \param shift The shift value.
//! \param scale The scale value.
@@ -5048,7 +5595,10 @@ public:
//!
//! \return The new Scale layer, or nullptr if it could not be created.
//!
- virtual IScaleLayer* addScale(ITensor& input, ScaleMode mode, Weights shift, Weights scale, Weights power) TRTNOEXCEPT = 0;
+ IScaleLayer* addScale(ITensor& input, ScaleMode mode, Weights shift, Weights scale, Weights power) noexcept
+ {
+ return mImpl->addScale(input, mode, shift, scale, power);
+ }
//!
//! \brief Add a SoftMax layer to the network.
@@ -5058,7 +5608,10 @@ public:
//!
//! \return The new SoftMax layer, or nullptr if it could not be created.
//!
- virtual ISoftMaxLayer* addSoftMax(ITensor& input) TRTNOEXCEPT = 0;
+ ISoftMaxLayer* addSoftMax(ITensor& input) noexcept
+ {
+ return mImpl->addSoftMax(input);
+ }
//!
//! \brief Add a concatenation layer to the network.
@@ -5070,9 +5623,12 @@ public:
//!
//! \return The new concatenation layer, or nullptr if it could not be created.
//!
- //! \warning All tensors must have the same dimensions for all dimensions except for channel.
+ //! \warning All tensors must have the same dimensions except along the concatenation axis.
//!
- virtual IConcatenationLayer* addConcatenation(ITensor* const* inputs, int32_t nbInputs) TRTNOEXCEPT = 0;
+ IConcatenationLayer* addConcatenation(ITensor* const* inputs, int32_t nbInputs) noexcept
+ {
+ return mImpl->addConcatenation(inputs, nbInputs);
+ }
//!
//! \brief Add a deconvolution layer to the network.
@@ -5092,8 +5648,11 @@ public:
//!
//! \deprecated Superseded by addDeconvolutionNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual IDeconvolutionLayer* addDeconvolution(ITensor& input, int32_t nbOutputMaps,
- DimsHW kernelSize, Weights kernelWeights, Weights biasWeights) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED IDeconvolutionLayer* addDeconvolution(
+ ITensor& input, int32_t nbOutputMaps, DimsHW kernelSize, Weights kernelWeights, Weights biasWeights) noexcept
+ {
+ return mImpl->addDeconvolution(input, nbOutputMaps, kernelSize, kernelWeights, biasWeights);
+ }
//!
//! \brief Add an elementwise layer to the network.
@@ -5115,89 +5674,10 @@ public:
//!
//! \return The new elementwise layer, or nullptr if it could not be created.
//!
- virtual IElementWiseLayer* addElementWise(ITensor& input1, ITensor& input2, ElementWiseOperation op) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Add an \p layerCount deep RNN layer to the network with a
- //! sequence length of \p maxSeqLen and \p hiddenSize internal state per
- //! layer.
- //!
- //! \param inputs The input tensor to the layer.
- //! \param layerCount The number of layers in the RNN.
- //! \param hiddenSize The size of the internal hidden state for each layer.
- //! \param maxSeqLen The maximum length of the time sequence.
- //! \param op The type of RNN to execute.
- //! \param mode The input mode for the RNN.
- //! \param dir The direction to run the RNN.
- //! \param weights The weights for the weight matrix parameters of the RNN.
- //! \param bias The weights for the bias vectors parameters of the RNN.
- //!
- //! The inputs tensor must be of the type DataType::kFLOAT or DataType::kHALF,
- //! and have non-zero volume.
- //!
- //! See IRNNLayer::setWeights() and IRNNLayer::setBias() for details on the required input
- //! format for \p weights and \p bias.
- //!
- //! The layout for the \p input tensor should be `{1, S_max, N, E}`, where:
- //! - `S_max` is the maximum allowed sequence length (number of RNN iterations)
- //! - `N` is the batch size
- //! - `E` specifies the embedding length (unless ::kSKIP is set, in which case it should match
- //! getHiddenSize()).
- //!
- //! The first output tensor is the output of the final RNN layer across all timesteps, with dimensions
- //! `{S_max, N, H}`:
- //!
- //! - `S_max` is the maximum allowed sequence length (number of RNN iterations)
- //! - `N` is the batch size
- //! - `H` is an output hidden state (equal to getHiddenSize() or 2x getHiddenSize())
- //!
- //! The second tensor is the final hidden state of the RNN across all layers, and if the RNN
- //! is an LSTM (i.e. getOperation() is ::kLSTM), then the third tensor is the final cell
- //! state of the RNN across all layers. Both the second and third output tensors have dimensions
- //! `{L, N, H}`:
- //!
- //! - `L` is equal to getLayerCount() if getDirection is ::kUNIDIRECTION,
- //! and 2*getLayerCount() if getDirection is ::kBIDIRECTION. In the bi-directional
- //! case, layer `l`'s final forward hidden state is stored in `L = 2*l`, and
- //! final backward hidden state is stored in `L = 2*l + 1`.
- //! - `N` is the batch size
- //! - `H` is getHiddenSize().
- //!
- //! Note that in bidirectional RNNs, the full "hidden state" for a layer `l`
- //! is the concatenation of its forward hidden state and its backward hidden
- //! state, and its size is 2*H.
- //!
- //! \deprecated Superseded by addRNNv2 and will be removed in TensorRT 8.0.
- //!
- //! \see IRNNLayer
- //!
- //! \warning This layer does not support wildcard dimensions or explicit batch size networks.
- //! \warning Int32 tensors are not valid input tensors.
- //!
- //! \return The new RNN layer, or nullptr if it could not be created.
- //!
- TRT_DEPRECATED virtual IRNNLayer* addRNN(ITensor& inputs, int32_t layerCount, std::size_t hiddenSize,
- int32_t maxSeqLen, RNNOperation op, RNNInputMode mode, RNNDirection dir, Weights weights,
- Weights bias) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Add a plugin layer to the network.
- //!
- //! \param inputs The input tensors to the layer.
- //! \param nbInputs The number of input tensors.
- //! \param plugin The layer plugin.
- //!
- //! \see IPluginLayer
- //!
- //! \deprecated Superseded by addPluginV2 and will be removed in TensorRT 8.0.
- //!
- //! \warning Plugin inputs do not support wildcard dimensions or explicit batch size networks.
- //! \warning Int32 tensors are not valid input tensors.
- //!
- //! \return the new plugin layer, or nullptr if it could not be created.
- //!
- TRT_DEPRECATED virtual IPluginLayer* addPlugin(
- ITensor* const* inputs, int32_t nbInputs, IPlugin& plugin) TRTNOEXCEPT = 0;
+ IElementWiseLayer* addElementWise(ITensor& input1, ITensor& input2, ElementWiseOperation op) noexcept
+ {
+ return mImpl->addElementWise(input1, input2, op);
+ }
//!
//! \brief Add a unary layer to the network.
@@ -5213,7 +5693,10 @@ public:
//!
//! \return The new unary layer, or nullptr if it could not be created
//!
- virtual IUnaryLayer* addUnary(ITensor& input, UnaryOperation operation) TRTNOEXCEPT = 0;
+ IUnaryLayer* addUnary(ITensor& input, UnaryOperation operation) noexcept
+ {
+ return mImpl->addUnary(input, operation);
+ }
//! \brief Add a padding layer to the network.
//!
@@ -5227,8 +5710,10 @@ public:
//!
//! \deprecated Superseded by addPaddingNd and will be removed in TensorRT 9.0.
//!
- TRT_DEPRECATED virtual IPaddingLayer* addPadding(
- ITensor& input, DimsHW prePadding, DimsHW postPadding) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED IPaddingLayer* addPadding(ITensor& input, DimsHW prePadding, DimsHW postPadding) noexcept
+ {
+ return mImpl->addPadding(input, prePadding, postPadding);
+ }
//!
//! \brief Add a shuffle layer to the network.
@@ -5239,96 +5724,10 @@ public:
//!
//! \return The new shuffle layer, or nullptr if it could not be created.
//!
- virtual IShuffleLayer* addShuffle(ITensor& input) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the pooling output dimensions formula.
- //!
- //! \deprecated This method does not currently work reliably and will be removed in TensorRT 8.0.
- //!
- //! \param formula The formula from computing the pooling output dimensions. If null is passed, the default
- //! formula is used.
- //!
- //! The default formula in each dimension is (inputDim + padding * 2 - kernelSize) / stride + 1.
- //!
- //! \warning Custom output dimensions formulas are not supported with wildcard dimensions.
- //!
- //! \see IOutputDimensionsFormula getPoolingOutputDimensionsFormula()
- //!
- TRT_DEPRECATED virtual void setPoolingOutputDimensionsFormula(IOutputDimensionsFormula* formula) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the pooling output dimensions formula.
- //!
- //! \deprecated This method does not currently work reliably and will be removed in TensorRT 8.0.
- //!
- //! \return The formula from computing the pooling output dimensions.
- //!
- //! \warning Custom output dimensions formulas are not supported with wildcard dimensions.
- //!
- //! \see IOutputDimensionsFormula setPoolingOutputDimensionsFormula()
- //!
- TRT_DEPRECATED virtual IOutputDimensionsFormula& getPoolingOutputDimensionsFormula() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the convolution output dimensions formula.
- //!
- //! \deprecated This method does not currently work reliably and will be removed in TensorRT 8.0.
- //!
- //! \param formula The formula from computing the convolution output dimensions. If null is passed, the default
- //! formula is used.
- //!
- //! The default formula in each dimension is (inputDim + padding * 2 - kernelSize) / stride + 1.
- //!
- //! \warning Custom output dimensions formulas are not supported with wildcard dimensions.
- //!
- //! \see IOutputDimensionsFormula getConvolutionOutputDimensionsFormula()
- //!
- TRT_DEPRECATED virtual void setConvolutionOutputDimensionsFormula(
- IOutputDimensionsFormula* formula) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the convolution output dimensions formula.
- //!
- //! \deprecated This method does not currently work reliably and will be removed in TensorRT 8.0.
- //!
- //! \return The formula from computing the convolution output dimensions.
- //!
- //! \warning Custom output dimensions formulas are not supported with wildcard dimensions.
- //!
- //! \see IOutputDimensionsFormula setConvolutionOutputDimensionsFormula()
- //!
- TRT_DEPRECATED virtual IOutputDimensionsFormula& getConvolutionOutputDimensionsFormula() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the deconvolution output dimensions formula.
- //!
- //! \deprecated This method does not currently work reliably and will be removed in TensorRT 8.0.
- //!
- //! \param formula The formula from computing the deconvolution output dimensions. If null is passed, the default!
- //! formula is used.
- //!
- //! The default formula in each dimension is (inputDim - 1) * stride + kernelSize - 2 * padding.
- //!
- //! \warning Custom output dimensions formulas are not supported with wildcard dimensions.
- //!
- //! \see IOutputDimensionsFormula getDevonvolutionOutputDimensionsFormula()
- //!
- TRT_DEPRECATED virtual void setDeconvolutionOutputDimensionsFormula(
- IOutputDimensionsFormula* formula) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the deconvolution output dimensions formula.
- //!
- //! \return The formula from computing the deconvolution output dimensions.
- //!
- //! \deprecated This method does not currently work reliably and will be removed in TensorRT 8.0.
- //!
- //! \warning Custom output dimensions formulas are not supported with wildcard dimensions.
- //!
- //! \see IOutputDimensionsFormula setDeconvolutionOutputDimensionsFormula()
- //!
- TRT_DEPRECATED virtual IOutputDimensionsFormula& getDeconvolutionOutputDimensionsFormula() const TRTNOEXCEPT = 0;
+ IShuffleLayer* addShuffle(ITensor& input) noexcept
+ {
+ return mImpl->addShuffle(input);
+ }
//!
//! \brief Get the number of layers in the network.
@@ -5337,7 +5736,10 @@ public:
//!
//! \see getLayer()
//!
- virtual int32_t getNbLayers() const TRTNOEXCEPT = 0;
+ int32_t getNbLayers() const noexcept
+ {
+ return mImpl->getNbLayers();
+ }
//!
//! \brief Get the layer specified by the given index.
@@ -5348,7 +5750,10 @@ public:
//!
//! \see getNbLayers()
//!
- virtual ILayer* getLayer(int32_t index) const TRTNOEXCEPT = 0;
+ ILayer* getLayer(int32_t index) const noexcept
+ {
+ return mImpl->getLayer(index);
+ }
//!
//! \brief Get the number of inputs in the network.
@@ -5357,7 +5762,10 @@ public:
//!
//! \see getInput()
//!
- virtual int32_t getNbInputs() const TRTNOEXCEPT = 0;
+ int32_t getNbInputs() const noexcept
+ {
+ return mImpl->getNbInputs();
+ }
//!
//! \brief Get the input tensor specified by the given index.
@@ -5366,9 +5774,14 @@ public:
//!
//! \return The input tensor, or nullptr if the index is out of range.
//!
+ //! \note adding inputs invalidates indexing here
+ //!
//! \see getNbInputs()
//!
- virtual ITensor* getInput(int32_t index) const TRTNOEXCEPT = 0; // adding inputs invalidates indexing here
+ ITensor* getInput(int32_t index) const noexcept
+ {
+ return mImpl->getInput(index);
+ }
//!
//! \brief Get the number of outputs in the network.
@@ -5379,7 +5792,10 @@ public:
//!
//! \see getOutput()
//!
- virtual int32_t getNbOutputs() const TRTNOEXCEPT = 0;
+ int32_t getNbOutputs() const noexcept
+ {
+ return mImpl->getNbOutputs();
+ }
//!
//! \brief Get the output tensor specified by the given index.
@@ -5388,19 +5804,27 @@ public:
//!
//! \return The output tensor, or nullptr if the index is out of range.
//!
+ //! \note adding inputs invalidates indexing here
+ //!
//! \see getNbOutputs()
//!
- virtual ITensor* getOutput(int32_t index) const TRTNOEXCEPT = 0; // adding outputs invalidates indexing here
+ ITensor* getOutput(int32_t index) const noexcept
+ {
+ return mImpl->getOutput(index);
+ }
//!
//! \brief Destroy this INetworkDefinition object.
//!
- virtual void destroy() TRTNOEXCEPT = 0;
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
+ //!
+ TRT_DEPRECATED void destroy() noexcept
+ {
+ delete this;
+ }
-protected:
- virtual ~INetworkDefinition() {}
-
-public:
//!
//! \brief Add a reduce layer to the network.
//!
@@ -5424,7 +5848,11 @@ public:
//!
//! \return The new reduce layer, or nullptr if it could not be created.
//!
- virtual IReduceLayer* addReduce(ITensor& input, ReduceOperation operation, uint32_t reduceAxes, bool keepDimensions) TRTNOEXCEPT = 0;
+ IReduceLayer* addReduce(
+ ITensor& input, ReduceOperation operation, uint32_t reduceAxes, bool keepDimensions) noexcept
+ {
+ return mImpl->addReduce(input, operation, reduceAxes, keepDimensions);
+ }
//!
//! \brief Add a TopK layer to the network.
@@ -5454,7 +5882,10 @@ public:
//!
//! \return The new TopK layer, or nullptr if it could not be created.
//!
- virtual ITopKLayer* addTopK(ITensor& input, TopKOperation op, int32_t k, uint32_t reduceAxes) TRTNOEXCEPT = 0;
+ ITopKLayer* addTopK(ITensor& input, TopKOperation op, int32_t k, uint32_t reduceAxes) noexcept
+ {
+ return mImpl->addTopK(input, op, k, reduceAxes);
+ }
//!
//! \brief Add a gather layer to the network.
@@ -5467,7 +5898,10 @@ public:
//!
//! \return The new gather layer, or nullptr if it could not be created.
//!
- virtual IGatherLayer* addGather(ITensor& data, ITensor& indices, int32_t axis) TRTNOEXCEPT = 0;
+ IGatherLayer* addGather(ITensor& data, ITensor& indices, int32_t axis) noexcept
+ {
+ return mImpl->addGather(data, indices, axis);
+ }
//!
//! \brief Add a RaggedSoftMax layer to the network.
@@ -5482,7 +5916,10 @@ public:
//!
//! \return The new RaggedSoftMax layer, or nullptr if it could not be created.
//!
- virtual IRaggedSoftMaxLayer* addRaggedSoftMax(ITensor& input, ITensor& bounds) TRTNOEXCEPT = 0;
+ IRaggedSoftMaxLayer* addRaggedSoftMax(ITensor& input, ITensor& bounds) noexcept
+ {
+ return mImpl->addRaggedSoftMax(input, bounds);
+ }
//!
//! \brief Add a MatrixMultiply layer to the network.
@@ -5498,28 +5935,11 @@ public:
//!
//! \return The new matrix multiply layer, or nullptr if it could not be created.
//!
- virtual IMatrixMultiplyLayer* addMatrixMultiply(
- ITensor& input0, MatrixOperation op0, ITensor& input1, MatrixOperation op1) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Add a MatrixMultiply layer to the network.
- //!
- //! \param input0 The first input tensor (commonly A).
- //! \param transpose0 If true, op(input0)=transpose(input0), else op(input0)=input0.
- //! \param input1 The second input tensor (commonly B).
- //! \param transpose1 If true, op(input1)=transpose(input1), else op(input1)=input1.
- //!
- //! \see IMatrixMultiplyLayer
- //!
- //! \return The new matrix multiply layer, or nullptr if it could not be created.
- //!
- //! \warning Int32 tensors are not valid input tensors.
- //!
- //! \deprecated This interface is superseded by the overload that replaces bool with MatrixOperation and will be
- //! removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED virtual IMatrixMultiplyLayer* addMatrixMultiply(
- ITensor& input0, bool transpose0, ITensor& input1, bool transpose1) TRTNOEXCEPT = 0;
+ IMatrixMultiplyLayer* addMatrixMultiply(
+ ITensor& input0, MatrixOperation op0, ITensor& input1, MatrixOperation op1) noexcept
+ {
+ return mImpl->addMatrixMultiply(input0, op0, input1, op1);
+ }
//!
//! \brief Add a constant layer to the network.
@@ -5541,7 +5961,10 @@ public:
//! If a wildcard dimension is used, the volume of the runtime dimensions must equal
//! the number of weights specified.
//!
- virtual IConstantLayer* addConstant(Dims dimensions, Weights weights) TRTNOEXCEPT = 0;
+ IConstantLayer* addConstant(Dims dimensions, Weights weights) noexcept
+ {
+ return mImpl->addConstant(dimensions, weights);
+ }
//!
//! \brief Add an \p layerCount deep RNN layer to the network with \p hiddenSize internal states that can
@@ -5599,34 +6022,18 @@ public:
//!
//! \see IRNNv2Layer
//!
- //! \deprecated Superseded by ILoop::addLoop and will be removed in TensorRT 9.0.
+ //! \deprecated Superseded by INetworkDefinition::addLoop and will be removed in TensorRT 9.0.
//!
//! \warning RNN inputs do not support wildcard dimensions or explicit batch size networks.
//! \warning Int32 tensors are not valid input tensors, only for sequence lengths.
//!
//! \return The new RNN layer, or nullptr if it could not be created.
//!
- TRT_DEPRECATED virtual IRNNv2Layer* addRNNv2(
- ITensor& input, int32_t layerCount, int32_t hiddenSize, int32_t maxSeqLen, RNNOperation op) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Add a plugin layer to the network using an IPluginExt interface.
- //!
- //! \param inputs The input tensors to the layer.
- //! \param nbInputs The number of input tensors.
- //! \param plugin The layer plugin.
- //!
- //! \see IPluginLayer
- //!
- //! \deprecated Superseded by addPluginV2 and will be removed in TensorRT 8.0.
- //!
- //! \warning Plugin inputs do not support wildcard dimensions or explicit batch size networks.
- //! \warning Int32 tensors are not valid input tensors.
- //!
- //! \return The new plugin layer, or nullptr if it could not be created.
- //!
- TRT_DEPRECATED virtual IPluginLayer* addPluginExt(
- ITensor* const* inputs, int32_t nbInputs, IPluginExt& plugin) TRTNOEXCEPT = 0;
+ TRT_DEPRECATED IRNNv2Layer* addRNNv2(
+ ITensor& input, int32_t layerCount, int32_t hiddenSize, int32_t maxSeqLen, RNNOperation op) noexcept
+ {
+ return mImpl->addRNNv2(input, layerCount, hiddenSize, maxSeqLen, op);
+ }
//!
//! \brief Add an identity layer.
@@ -5639,7 +6046,10 @@ public:
//!
//! \return The new identity layer, or nullptr if it could not be created.
//!
- virtual IIdentityLayer* addIdentity(ITensor& input) TRTNOEXCEPT = 0;
+ IIdentityLayer* addIdentity(ITensor& input) noexcept
+ {
+ return mImpl->addIdentity(input);
+ }
//!
//! \brief remove a tensor from the network definition.
@@ -5651,7 +6061,10 @@ public:
//! and the call will be ignored. Its intended use is to remove detached tensors after
//! e.g. concatenating two networks with Layer::setInput().
//!
- virtual void removeTensor(ITensor& tensor) TRTNOEXCEPT = 0;
+ void removeTensor(ITensor& tensor) noexcept
+ {
+ mImpl->removeTensor(tensor);
+ }
//!
//! \brief unmark a tensor as a network output.
@@ -5660,7 +6073,10 @@ public:
//!
//! see markOutput()
//!
- virtual void unmarkOutput(ITensor& tensor) TRTNOEXCEPT = 0;
+ void unmarkOutput(ITensor& tensor) noexcept
+ {
+ mImpl->unmarkOutput(tensor);
+ }
//!
//! \brief Add a plugin layer to the network using the IPluginV2 interface.
@@ -5676,7 +6092,10 @@ public:
//!
//! \return The new plugin layer, or nullptr if it could not be created.
//!
- virtual IPluginV2Layer* addPluginV2(ITensor* const* inputs, int32_t nbInputs, IPluginV2& plugin) TRTNOEXCEPT = 0;
+ IPluginV2Layer* addPluginV2(ITensor* const* inputs, int32_t nbInputs, IPluginV2& plugin) noexcept
+ {
+ return mImpl->addPluginV2(inputs, nbInputs, plugin);
+ }
//!
//! \brief Add a slice layer to the network.
@@ -5692,7 +6111,10 @@ public:
//!
//! \return The new slice layer, or nullptr if it could not be created.
//!
- virtual ISliceLayer* addSlice(ITensor& input, Dims start, Dims size, Dims stride) TRTNOEXCEPT = 0;
+ ISliceLayer* addSlice(ITensor& input, Dims start, Dims size, Dims stride) noexcept
+ {
+ return mImpl->addSlice(input, start, size, stride);
+ }
//!
//! \brief Sets the name of the network.
@@ -5711,7 +6133,10 @@ public:
//!
//! \return none
//!
- virtual void setName(const char* name) TRTNOEXCEPT = 0;
+ void setName(const char* name) noexcept
+ {
+ mImpl->setName(name);
+ }
//!
//! \brief Returns the name associated with the network.
@@ -5722,7 +6147,10 @@ public:
//!
//! \return A zero delimited C-style string representing the name of the network.
//!
- virtual const char* getName() const TRTNOEXCEPT = 0;
+ const char* getName() const noexcept
+ {
+ return mImpl->getName();
+ }
//!
//! \brief Add a shape layer to the network.
@@ -5737,7 +6165,10 @@ public:
//!
//! \return The new shape layer, or nullptr if it could not be created.
//!
- virtual IShapeLayer* addShape(ITensor& input) TRTNOEXCEPT = 0;
+ IShapeLayer* addShape(ITensor& input) noexcept
+ {
+ return mImpl->addShape(input);
+ }
//!
//! \brief Query whether the network was created with an implicit batch dimension.
@@ -5753,7 +6184,10 @@ public:
//!
//! \see createNetworkV2
//!
- virtual bool hasImplicitBatchDimension() const TRTNOEXCEPT = 0;
+ bool hasImplicitBatchDimension() const noexcept
+ {
+ return mImpl->hasImplicitBatchDimension();
+ }
//!
//! \brief Enable tensor's value to be computed by IExecutionContext::getShapeBinding.
@@ -5768,7 +6202,10 @@ public:
//!
//! \see isShapeBinding(), getShapeBinding()
//!
- virtual bool markOutputForShapes(ITensor& tensor) TRTNOEXCEPT = 0;
+ bool markOutputForShapes(ITensor& tensor) noexcept
+ {
+ return mImpl->markOutputForShapes(tensor);
+ }
//!
//! \brief Undo markOutputForShapes.
@@ -5777,7 +6214,10 @@ public:
//!
//! \return True if successful, false if tensor is not marked as an output.
//!
- virtual bool unmarkOutputForShapes(ITensor& tensor) TRTNOEXCEPT = 0;
+ bool unmarkOutputForShapes(ITensor& tensor) noexcept
+ {
+ return mImpl->unmarkOutputForShapes(tensor);
+ }
//!
//! \brief Add a parametric ReLU layer to the network.
@@ -5792,7 +6232,10 @@ public:
//!
//! \return The new parametric ReLU layer, or nullptr if it could not be created.
//!
- virtual IParametricReLULayer* addParametricReLU(ITensor& input, ITensor& slope) noexcept = 0;
+ IParametricReLULayer* addParametricReLU(ITensor& input, ITensor& slope) noexcept
+ {
+ return mImpl->addParametricReLU(input, slope);
+ }
//!
//! \brief Add a multi-dimension convolution layer to the network.
@@ -5811,8 +6254,11 @@ public:
//!
//! \return The new convolution layer, or nullptr if it could not be created.
//!
- virtual IConvolutionLayer* addConvolutionNd(ITensor& input, int32_t nbOutputMaps, Dims kernelSize,
- Weights kernelWeights, Weights biasWeights) TRTNOEXCEPT = 0;
+ IConvolutionLayer* addConvolutionNd(
+ ITensor& input, int32_t nbOutputMaps, Dims kernelSize, Weights kernelWeights, Weights biasWeights) noexcept
+ {
+ return mImpl->addConvolutionNd(input, nbOutputMaps, kernelSize, kernelWeights, biasWeights);
+ }
//!
//! \brief Add a multi-dimension pooling layer to the network.
@@ -5828,7 +6274,10 @@ public:
//!
//! \return The new pooling layer, or nullptr if it could not be created.
//!
- virtual IPoolingLayer* addPoolingNd(ITensor& input, PoolingType type, Dims windowSize) TRTNOEXCEPT = 0;
+ IPoolingLayer* addPoolingNd(ITensor& input, PoolingType type, Dims windowSize) noexcept
+ {
+ return mImpl->addPoolingNd(input, type, windowSize);
+ }
//!
//! \brief Add a multi-dimension deconvolution layer to the network.
@@ -5847,8 +6296,11 @@ public:
//
//! \return The new deconvolution layer, or nullptr if it could not be created.
//!
- virtual IDeconvolutionLayer* addDeconvolutionNd(ITensor& input, int32_t nbOutputMaps, Dims kernelSize,
- Weights kernelWeights, Weights biasWeights) TRTNOEXCEPT = 0;
+ IDeconvolutionLayer* addDeconvolutionNd(
+ ITensor& input, int32_t nbOutputMaps, Dims kernelSize, Weights kernelWeights, Weights biasWeights) noexcept
+ {
+ return mImpl->addDeconvolutionNd(input, nbOutputMaps, kernelSize, kernelWeights, biasWeights);
+ }
//!
//! \brief Add a multi-dimension scale layer to the network.
@@ -5870,14 +6322,21 @@ public:
//! For ::kCHANNEL, the number of weights is C.
//! For ::kELEMENTWISE, the number of weights is C*D*E*F.
//!
+ //! channelAxis can also be set explicitly using setChannelAxis().
+ //!
//! \see IScaleLayer
+ //! \see setChannelAxis()
+ //!
//! \warning Int32 tensors are not valid input tensors.
//! \warning Only 2D or 3D scale is supported.
//!
//! \return The new Scale layer, or nullptr if it could not be created.
//!
- virtual IScaleLayer* addScaleNd(ITensor& input, ScaleMode mode, Weights shift, Weights scale, Weights power,
- int32_t channelAxis) TRTNOEXCEPT = 0;
+ IScaleLayer* addScaleNd(
+ ITensor& input, ScaleMode mode, Weights shift, Weights scale, Weights power, int32_t channelAxis) noexcept
+ {
+ return mImpl->addScaleNd(input, mode, shift, scale, power, channelAxis);
+ }
//! \brief Add a resize layer to the network.
//!
@@ -5889,11 +6348,16 @@ public:
//!
//! \return The new resize layer, or nullptr if it could not be created.
//!
- virtual IResizeLayer* addResize(ITensor& input) TRTNOEXCEPT = 0;
+ IResizeLayer* addResize(ITensor& input) noexcept
+ {
+ return mImpl->addResize(input);
+ }
//!
//! \brief True if network is an explicit precision network
//!
+ //! \deprecated Will be removed in TensorRT 10.0.
+ //!
//! hasExplicitPrecision() is true if and only if this INetworkDefinition
//! was created with createNetworkV2() with NetworkDefinitionCreationFlag::kEXPLICIT_PRECISION set.
//!
@@ -5901,7 +6365,10 @@ public:
//!
//! \return True if network has explicit precision, false otherwise.
//!
- virtual bool hasExplicitPrecision() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED bool hasExplicitPrecision() const noexcept
+ {
+ return mImpl->hasExplicitPrecision();
+ }
//!
//! \brief Add a loop to the network.
@@ -5912,7 +6379,12 @@ public:
//! or nullptr if network has an implicit batch dimension or this version
//! of TensorRT does not support loops.
//!
- virtual ILoop* addLoop() noexcept = 0;
+ //! The network must not have an implicit batch dimension.
+ //!
+ ILoop* addLoop() noexcept
+ {
+ return mImpl->addLoop();
+ }
//! \brief Add a select layer to the network.
//!
@@ -5942,22 +6414,33 @@ public:
//!
//! then the output dimensions are [1,3,0,9].
//!
+ //! The network must not have an implicit batch dimension.
+ //!
//! \see ISelectLayer
//!
//! \return The new select layer, or nullptr if it could not be created.
- virtual ISelectLayer* addSelect(ITensor& condition, ITensor& thenInput, ITensor& elseInput) TRTNOEXCEPT = 0;
+ ISelectLayer* addSelect(ITensor& condition, ITensor& thenInput, ITensor& elseInput) noexcept
+ {
+ return mImpl->addSelect(condition, thenInput, elseInput);
+ }
//! \brief Add a fill layer to the network.
//!
//! \param dimensions The output tensor dimensions.
//! \param op The fill operation that the layer applies.
//!
- //! \warning The dimensions's nbDims must be 1.
+ //! \warning For FillOperation::kLINSPACE, dimensions.nbDims must be 1.
+ //!
+ //! The network must not have an implicit batch dimension.
//!
//! \see IFillLayer
//!
//! \return The new fill layer, or nullptr if it could not be created.
- virtual IFillLayer* addFill(Dims dimensions, FillOperation op) noexcept = 0;
+ //!
+ IFillLayer* addFill(Dims dimensions, FillOperation op) noexcept
+ {
+ return mImpl->addFill(dimensions, op);
+ }
//! \brief Add a padding layer to the network. Only 2D padding is currently supported.
//!
@@ -5969,8 +6452,105 @@ public:
//!
//! \return The new padding layer, or nullptr if it could not be created.
//!
- virtual IPaddingLayer* addPaddingNd(
- ITensor& input, Dims prePadding, Dims postPadding) TRTNOEXCEPT = 0;
+ IPaddingLayer* addPaddingNd(ITensor& input, Dims prePadding, Dims postPadding) noexcept
+ {
+ return mImpl->addPaddingNd(input, prePadding, postPadding);
+ }
+
+ //! \brief Associate a name with all current uses of the given weights.
+ //!
+ //! The name must be set after the Weights are used in the network.
+ //! Lookup is associative. The name applies to all Weights with matching
+ //! type, value pointer, and count. If Weights with a matching value
+ //! pointer, but different type or count exists in the network, an
+ //! error message is issued, the name is rejected, and return false.
+ //! If the name has already been used for other weights,
+ //! return false. A nullptr causes the weights to become unnamed,
+ //! i.e. clears any previous name.
+ //!
+ //! \param weights The weights to be named.
+ //! \param name The name to associate with the weights.
+ //!
+ //! \return true on success.
+ bool setWeightsName(Weights weights, const char* name) noexcept
+ {
+ return mImpl->setWeightsName(weights, name);
+ }
+
+ //!
+ //! \brief Set the ErrorRecorder for this interface
+ //!
+ //! Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution.
+ //! This function will call incRefCount of the registered ErrorRecorder at least once. Setting
+ //! recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if
+ //! a recorder has been registered.
+ //!
+ //! If an error recorder is not set, messages will be sent to the global log stream.
+ //!
+ //! \param recorder The error recorder to register with this interface.
+ //
+ //! \see getErrorRecorder()
+ //!
+ void setErrorRecorder(IErrorRecorder* recorder) noexcept
+ {
+ mImpl->setErrorRecorder(recorder);
+ }
+
+ //!
+ //! \brief get the ErrorRecorder assigned to this interface.
+ //!
+ //! Retrieves the assigned error recorder object for the given class.
+ //! A nullptr will be returned if setErrorRecorder has not been called.
+ //!
+ //! \return A pointer to the IErrorRecorder object that has been registered.
+ //!
+ //! \see setErrorRecorder()
+ //!
+ IErrorRecorder* getErrorRecorder() const noexcept
+ {
+ return mImpl->getErrorRecorder();
+ }
+
+ //!
+ //! \brief Add a dequantization layer to the network.
+ //!
+ //! \param input The input tensor to be quantized.
+ //! \param scale A tensor with the scale value.
+ //!
+ //! \see IDequantizeLayer
+ //!
+ //! \p input tensor data type must be DataType::kFLOAT.
+ //! \p scale tensor data type must be DataType::kFLOAT. The subgraph which terminates with the \p scale tensor must
+ //! be a build-time constant.
+ //!
+ //! \return The new quantization layer, or nullptr if it could not be created.
+ //!
+ IDequantizeLayer* addDequantize(ITensor& input, ITensor& scale) noexcept
+ {
+ return mImpl->addDequantize(input, scale);
+ }
+
+ //!
+ //! \brief Add a quantization layer to the network.
+ //!
+ //! \param input The input tensor to be quantized.
+ //! \param scale A tensor with the scale value.
+ //!
+ //! \see IQuantizeLayer
+ //!
+ //! \p input tensor data type must be DataType::kFLOAT.
+ //! \p scale tensor data type must be DataType::kFLOAT. The subgraph which terminates with the \p scale tensor must
+ //! be a build-time constant.
+ //!
+ //! \return The new quantization layer, or nullptr if it could not be created.
+ //!
+ IQuantizeLayer* addQuantize(ITensor& input, ITensor& scale) noexcept
+ {
+ return mImpl->addQuantize(input, scale);
+ }
+
+protected:
+ apiv::VNetworkDefinition* mImpl;
};
//!
@@ -5988,7 +6568,7 @@ enum class CalibrationAlgoType : int32_t
//! Maximum number of elements in CalibrationAlgoType enum. \see DataType
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 4;
}
@@ -6012,7 +6592,7 @@ public:
//!
//! \return The batch size.
//!
- virtual int32_t getBatchSize() const TRTNOEXCEPT = 0;
+ virtual int32_t getBatchSize() const noexcept = 0;
//!
//! \brief Get a batch of input for calibration.
@@ -6027,7 +6607,7 @@ public:
//!
//! \see getBatchSize()
//!
- virtual bool getBatch(void* bindings[], const char* names[], int32_t nbBindings) TRTNOEXCEPT = 0;
+ virtual bool getBatch(void* bindings[], const char* names[], int32_t nbBindings) noexcept = 0;
//!
//! \brief Load a calibration cache.
@@ -6043,7 +6623,7 @@ public:
//!
//! \return A pointer to the cache, or nullptr if there is no data.
//!
- virtual const void* readCalibrationCache(std::size_t& length) TRTNOEXCEPT = 0;
+ virtual const void* readCalibrationCache(std::size_t& length) noexcept = 0;
//!
//! \brief Save a calibration cache.
@@ -6053,16 +6633,16 @@ public:
//!
//! \see readCalibrationCache()
//!
- virtual void writeCalibrationCache(const void* ptr, std::size_t length) TRTNOEXCEPT = 0;
+ virtual void writeCalibrationCache(const void* ptr, std::size_t length) noexcept = 0;
//!
//! \brief Get the algorithm used by this calibrator.
//!
//! \return The algorithm used by the calibrator.
//!
- virtual CalibrationAlgoType getAlgorithm() TRTNOEXCEPT = 0;
+ virtual CalibrationAlgoType getAlgorithm() noexcept = 0;
- virtual ~IInt8Calibrator() {}
+ virtual ~IInt8Calibrator() noexcept = default;
};
//!
@@ -6075,9 +6655,12 @@ public:
//!
//! Signal that this is the entropy calibrator.
//!
- CalibrationAlgoType getAlgorithm() TRTNOEXCEPT override { return CalibrationAlgoType::kENTROPY_CALIBRATION; }
+ CalibrationAlgoType getAlgorithm() noexcept override
+ {
+ return CalibrationAlgoType::kENTROPY_CALIBRATION;
+ }
- virtual ~IInt8EntropyCalibrator() {}
+ virtual ~IInt8EntropyCalibrator() noexcept = default;
};
//!
@@ -6090,14 +6673,16 @@ public:
//!
//! Signal that this is the entropy calibrator 2.
//!
- CalibrationAlgoType getAlgorithm() TRTNOEXCEPT override { return CalibrationAlgoType::kENTROPY_CALIBRATION_2; }
+ CalibrationAlgoType getAlgorithm() noexcept override
+ {
+ return CalibrationAlgoType::kENTROPY_CALIBRATION_2;
+ }
- virtual ~IInt8EntropyCalibrator2() {}
+ virtual ~IInt8EntropyCalibrator2() noexcept = default;
};
//!
-//! MinMax Calibrator. This is the preferred calibrator for NLP tasks. It supports per
-//! activation tensor scaling.
+//! MinMax Calibrator. It supports per activation tensor scaling.
//!
class IInt8MinMaxCalibrator : public IInt8Calibrator
{
@@ -6105,9 +6690,12 @@ public:
//!
//! Signal that this is the MinMax Calibrator.
//!
- CalibrationAlgoType getAlgorithm() TRTNOEXCEPT override { return CalibrationAlgoType::kMINMAX_CALIBRATION; }
+ CalibrationAlgoType getAlgorithm() noexcept override
+ {
+ return CalibrationAlgoType::kMINMAX_CALIBRATION;
+ }
- virtual ~IInt8MinMaxCalibrator() {}
+ virtual ~IInt8MinMaxCalibrator() noexcept = default;
};
//!
@@ -6120,7 +6708,10 @@ public:
//!
//! Signal that this is the legacy calibrator.
//!
- CalibrationAlgoType getAlgorithm() TRTNOEXCEPT override { return CalibrationAlgoType::kLEGACY_CALIBRATION; }
+ CalibrationAlgoType getAlgorithm() noexcept override
+ {
+ return CalibrationAlgoType::kLEGACY_CALIBRATION;
+ }
//!
//! \brief The quantile (between 0 and 1) that will be used to select the region maximum when the quantile method
@@ -6128,7 +6719,7 @@ public:
//!
//! See the user guide for more details on how the quantile is used.
//!
- virtual double getQuantile() const TRTNOEXCEPT = 0;
+ virtual double getQuantile() const noexcept = 0;
//!
//! \brief The fraction (between 0 and 1) of the maximum used to define the regression cutoff when using regression
@@ -6136,7 +6727,7 @@ public:
//!
//! See the user guide for more details on how the regression cutoff is used
//!
- virtual double getRegressionCutoff() const TRTNOEXCEPT = 0;
+ virtual double getRegressionCutoff() const noexcept = 0;
//!
//! \brief Load a histogram.
@@ -6150,7 +6741,7 @@ public:
//!
//! \return A pointer to the cache, or nullptr if there is no data.
//!
- virtual const void* readHistogramCache(std::size_t& length) TRTNOEXCEPT = 0;
+ virtual const void* readHistogramCache(std::size_t& length) noexcept = 0;
//!
//! \brief Save a histogram cache.
@@ -6160,9 +6751,9 @@ public:
//!
//! \see readHistogramCache()
//!
- virtual void writeHistogramCache(const void* ptr, std::size_t length) TRTNOEXCEPT = 0;
+ virtual void writeHistogramCache(const void* ptr, std::size_t length) noexcept = 0;
- virtual ~IInt8LegacyCalibrator() {}
+ virtual ~IInt8LegacyCalibrator() noexcept = default;
};
//!
@@ -6175,26 +6766,36 @@ public:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class IAlgorithmIOInfo
+class IAlgorithmIOInfo : public INoCopy
{
public:
//!
//! \brief Return TensorFormat of the input/output of algorithm.
//!
- virtual TensorFormat getTensorFormat() const TRTNOEXCEPT = 0;
+ TensorFormat getTensorFormat() const noexcept
+ {
+ return mImpl->getTensorFormat();
+ }
//!
//! \brief Return DataType of the input/output of algorithm.
//!
- virtual DataType getDataType() const TRTNOEXCEPT = 0;
+ DataType getDataType() const noexcept
+ {
+ return mImpl->getDataType();
+ }
//!
//! \brief Return strides of the input/output tensor of algorithm.
//!
- virtual Dims getStrides() const TRTNOEXCEPT = 0;
+ Dims getStrides() const noexcept
+ {
+ return mImpl->getStrides();
+ }
protected:
- virtual ~IAlgorithmIOInfo() {}
+ virtual ~IAlgorithmIOInfo() noexcept = default;
+ apiv::VAlgorithmIOInfo* mImpl;
};
//!
@@ -6208,21 +6809,28 @@ protected:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class IAlgorithmVariant
+class IAlgorithmVariant : public INoCopy
{
public:
//!
//! \brief Return implementation of the algorithm.
//!
- virtual int64_t getImplementation() const TRTNOEXCEPT = 0;
+ int64_t getImplementation() const noexcept
+ {
+ return mImpl->getImplementation();
+ }
//!
//! \brief Return tactic of the algorithm.
//!
- virtual int64_t getTactic() const TRTNOEXCEPT = 0;
+ int64_t getTactic() const noexcept
+ {
+ return mImpl->getTactic();
+ }
protected:
- virtual ~IAlgorithmVariant() {}
+ virtual ~IAlgorithmVariant() noexcept = default;
+ apiv::VAlgorithmVariant* mImpl;
};
//!
@@ -6233,14 +6841,17 @@ protected:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class IAlgorithmContext
+class IAlgorithmContext : public INoCopy
{
public:
//!
//! \brief Return name of the algorithm node.
//! This is a unique identifier for the IAlgorithmContext.
//!
- virtual const char* getName() const TRTNOEXCEPT = 0;
+ const char* getName() const noexcept
+ {
+ return mImpl->getName();
+ }
//!
//! \brief Get the minimum / optimum / maximum dimensions for input or output tensor.
@@ -6248,20 +6859,30 @@ public:
//! and the outputs.
//! \param select Which of the minimum, optimum, or maximum dimensions to be queried.
//!
- virtual Dims getDimensions(int32_t index, OptProfileSelector select) const TRTNOEXCEPT = 0;
+ Dims getDimensions(int32_t index, OptProfileSelector select) const noexcept
+ {
+ return mImpl->getDimensions(index, select);
+ }
//!
//! \brief Return number of inputs of the algorithm.
//!
- virtual int32_t getNbInputs() const TRTNOEXCEPT = 0;
+ int32_t getNbInputs() const noexcept
+ {
+ return mImpl->getNbInputs();
+ }
//!
//! \brief Return number of outputs of the algorithm.
//!
- virtual int32_t getNbOutputs() const TRTNOEXCEPT = 0;
+ int32_t getNbOutputs() const noexcept
+ {
+ return mImpl->getNbOutputs();
+ }
protected:
- virtual ~IAlgorithmContext() {}
+ virtual ~IAlgorithmContext() noexcept = default;
+ apiv::VAlgorithmContext* mImpl;
};
//!
@@ -6273,7 +6894,7 @@ protected:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class IAlgorithm
+class IAlgorithm : public INoCopy
{
public:
//!
@@ -6282,26 +6903,56 @@ public:
//! \param index Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs
//! and the outputs.
//!
- virtual const IAlgorithmIOInfo& getAlgorithmIOInfo(int32_t index) const TRTNOEXCEPT = 0;
+ //! \return a reference to IAlgorithmIOInfo specified by index or the first algorithm if index is out of range.
+ //!
+ //! \deprecated API will be removed in TensorRT 10.0, use IAlgorithm::getAlgorithmIOInfoByIndex instead.
+ //!
+ TRT_DEPRECATED const IAlgorithmIOInfo& getAlgorithmIOInfo(int32_t index) const noexcept
+ {
+ return mImpl->getAlgorithmIOInfo(index);
+ }
//!
//! \brief Returns the algorithm variant.
//!
- virtual const IAlgorithmVariant& getAlgorithmVariant() const TRTNOEXCEPT = 0;
+ const IAlgorithmVariant& getAlgorithmVariant() const noexcept
+ {
+ return mImpl->getAlgorithmVariant();
+ }
//!
//! \brief The time in milliseconds to execute the algorithm.
//!
- virtual float getTimingMSec() const TRTNOEXCEPT = 0;
+ float getTimingMSec() const noexcept
+ {
+ return mImpl->getTimingMSec();
+ }
//!
//! \brief The size of the GPU temporary memory in bytes which the algorithm uses at execution time.
//!
- virtual std::size_t getWorkspaceSize() const TRTNOEXCEPT = 0;
+ std::size_t getWorkspaceSize() const noexcept
+ {
+ return mImpl->getWorkspaceSize();
+ }
+
+ //!
+ //! \brief Returns the format of an Algorithm input or output. Algorithm inputs are incrementally numbered first,
+ //! followed by algorithm outputs.
+ //! \param index Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs
+ //! and the outputs.
+ //!
+ //! \return a pointer to a IAlgorithmIOInfo interface or nullptr if index is out of range.
+ //!
+ const IAlgorithmIOInfo* getAlgorithmIOInfoByIndex(int32_t index) const noexcept
+ {
+ return mImpl->getAlgorithmIOInfoByIndex(index);
+ }
protected:
- virtual ~IAlgorithm() {}
-};
+ virtual ~IAlgorithm() noexcept = default;
+ apiv::VAlgorithm* mImpl;
+}; // IAlgorithm
//!
//! \class IAlgorithmSelector
@@ -6311,7 +6962,7 @@ protected:
//! \note A layer in context of algorithm selection may be different from ILayer in INetworkDefiniton.
//! For example, an algorithm might be implementing a conglomeration of multiple ILayers in INetworkDefinition.
//!
-class IAlgorithmSelector
+class IAlgorithmSelector
{
public:
//!
@@ -6323,12 +6974,13 @@ public:
//! \param nbChoices Number of algorithm choices.
//! \param selection The user writes indices of selected choices in to selection buffer which is of size nbChoices.
//!
- //! \note TRT uses its default algorithm selection to choose from the list provided.
- //! If return value is 0, TRT’s default algorithm selection is used unless strict type constraints are set.
- //! The list of choices is valid only for this specific algorithm context.
+ //! \note TensorRT uses its default algorithm selection to choose from the list provided.
+ //! If return value is 0, TensorRT’s default algorithm selection is used unless strict type constraints are
+ //! set. The list of choices is valid only for this specific algorithm context.
//!
virtual int32_t selectAlgorithms(const IAlgorithmContext& context, const IAlgorithm* const* choices,
- int32_t nbChoices, int32_t* selection) TRTNOEXCEPT = 0;
+ int32_t nbChoices, int32_t* selection) noexcept
+ = 0;
//!
//! \brief Called by TensorRT to report choices it made.
//!
@@ -6340,18 +6992,19 @@ public:
//! \param nbAlgorithms The size of algoContexts as well as algoChoices.
//!
virtual void reportAlgorithms(const IAlgorithmContext* const* algoContexts, const IAlgorithm* const* algoChoices,
- int32_t nbAlgorithms) TRTNOEXCEPT = 0;
+ int32_t nbAlgorithms) noexcept
+ = 0;
- virtual ~IAlgorithmSelector() {}
+ virtual ~IAlgorithmSelector() noexcept = default;
};
//!
-//! \brief Represents a collection of one or more QuantizationFlag values using binary OR
+//! \brief Represents one or more QuantizationFlag values using binary OR
//! operations.
//!
//! \see IBuilderConfig::getQuantizationFlags(), IBuilderConfig::setQuantizationFlags()
//!
-typedef uint32_t QuantizationFlags;
+using QuantizationFlags = uint32_t;
//!
//! \enum QuantizationFlag
@@ -6370,18 +7023,18 @@ enum class QuantizationFlag : int32_t
//! Maximum number of quantization flags in QuantizationFlag enum. \see QuantizationFlag
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 1;
}
//!
-//! \brief Represents a collection of one or more QuantizationFlag values using binary OR
+//! \brief Represents one or more QuantizationFlag values using binary OR
//! operations, e.g., 1U << BuilderFlag::kFP16 | 1U << BuilderFlag::kDEBUG.
//!
//! \see IBuilderConfig::getFlags(), ITensor::setFlags(),
//!
-typedef uint32_t BuilderFlags;
+using BuilderFlags = uint32_t;
//!
//! \enum BuilderFlag
@@ -6403,14 +7056,24 @@ enum class BuilderFlag : int32_t
//! Allow (but not require) computations on tensors of type DataType::kFLOAT to use TF32.
//! TF32 computes inner products by rounding the inputs to 10-bit mantissas before
//! multiplying, but accumulates the sum using 23-bit mantissas. Enabled by default.
- kTF32 = 7
+ kTF32 = 7,
+
+ //! Allow the builder to examine weights and use optimized functions when weights have suitable sparsity.
+ kSPARSE_WEIGHTS = 8,
+
+ //! Change the allowed parameters in the EngineCapability::kSTANDARD flow to
+ //! match the restrictions that EngineCapability::kSAFETY check against for DeviceType::kGPU
+ //! and EngineCapability::kDLA_STANDALONE check against the DeviceType::kDLA case. This flag
+ //! is forced to true if EngineCapability::kSAFETY at build time if it is unset.
+ //!
+ kSAFETY_SCOPE = 9
};
//! Maximum number of builder flags in BuilderFlag enum. \see BuilderFlag
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
- return 8;
+ return 10;
}
//!
@@ -6430,49 +7093,108 @@ enum class ProfilingVerbosity : int32_t
//! Maximum number of profile verbosity levels in ProfilingVerbosity enum. \see ProfilingVerbosity
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 3;
}
-
//!
-//! \enum TacticSource
+//! \class ITimingCache
//!
-//! \brief List of tactic sources for TensorRT.
+//! \brief Class to handle tactic timing info collected from builder.
//!
-//! \see TacticSources, IBuilderConfig::setTacticSources(), IBuilderConfig::getTacticSources()
+//! The timing cache is created or initialized by IBuilderConfig. It can be shared across builder instances
+//! to accelerate the builder wallclock time.
//!
-enum class TacticSource : int32_t
+//! \see IBuilderConfig
+//!
+//!
+//! \class ITimingCache
+//!
+//! \brief Class to handle tactic timing info collected from builder.
+//!
+//! The timing cache is created or initialized by IBuilderConfig. It can be shared across builder instances
+//! to accelerate the builder wallclock time.
+//!
+//! \see IBuilderConfig
+//!
+//!
+//! \class ITimingCache
+//!
+//! \brief Class to handle tactic timing info collected from builder.
+//!
+//! The timing cache is created or initialized by IBuilderConfig. It can be shared across builder instances
+//! to accelerate the builder wallclock time.
+//!
+//! \see IBuilderConfig
+//!
+class ITimingCache : public INoCopy
{
- //! \note Disabling kCUBLAS will cause the cublas handle passed to plugins in attachToContext to be null.
- kCUBLAS = 0, //!< cuBLAS tactics.
- kCUBLAS_LT = 1 //!< cuBLAS LT tactics
+public:
+ virtual ~ITimingCache() noexcept = default;
+
+ //!
+ //! \brief Serialize a timing cache to IHostMemory object.
+ //!
+ //! This function allows serialization of current timing cache.
+ //!
+ //! \return A pointer to a IHostMemory object that contains a serialized timing cache.
+ //!
+ //! \see IHostMemory
+ //!
+ nvinfer1::IHostMemory* serialize() const noexcept
+ {
+ return mImpl->serialize();
+ }
+
+ //!
+ //! \brief Combine input timing cache into local instance.
+ //!
+ //! This function allows combining entries in the input timing cache to local cache object.
+ //!
+ //! \param inputCache The input timing cache.
+ //! \param ignoreMismatch Whether or not to allow cache verification header mismatch.
+ //!
+ //! \return True if combined successfully, false otherwise.
+ //!
+ //! Append entries in input cache to local cache. Conflicting entries will be skipped
+ //! The input cache must be generated by a TensorRT build of exact same version, otherwise
+ //! combine will be skipped and return false.
+ //! ignoreMismatch must be set to true if combining a timing cache created from a
+ //! different device.
+ //!
+ //! \warning Combining caches generated from devices with different device properties may
+ //! lead to functional/performance bugs!
+ //!
+ bool combine(const ITimingCache& inputCache, bool ignoreMismatch) noexcept
+ {
+ return mImpl->combine(inputCache, ignoreMismatch);
+ }
+
+ //!
+ //! \brief Empty the timing cache
+ //!
+ //! \return True if reset successfully, false otherwise.
+ //!
+ bool reset() noexcept
+ {
+ return mImpl->reset();
+ }
+
+protected:
+ apiv::VTimingCache* mImpl;
};
-//! Maximum number of tactic sources in TacticSource enum. \see TacticSource
-template <>
-constexpr inline int32_t EnumMax()
-{
- return 2;
-}
-
-//!
-//! \brief Represents a collection of one or more TacticSource values
-//! combine using bitwise-OR operations.
-//!
-//! \see IBuilderConfig::setTacticSources(), IBuilderConfig::getTacticSources()
-//!
-using TacticSources = uint32_t;
-
//!
//! \class IBuilderConfig
//!
//! \brief Holds properties for configuring a builder to produce an engine. \see BuilderFlags
//!
-class IBuilderConfig
+class IBuilderConfig : public INoCopy
{
public:
+ virtual ~IBuilderConfig() noexcept = default;
+
//!
//! \brief Set the number of minimization iterations used when timing layers.
//!
@@ -6483,7 +7205,10 @@ public:
//!
//! \see getMinTimingIterations()
//!
- virtual void setMinTimingIterations(int32_t minTiming) TRTNOEXCEPT = 0;
+ virtual void setMinTimingIterations(int32_t minTiming) noexcept
+ {
+ mImpl->setMinTimingIterations(minTiming);
+ }
//!
//! \brief Query the number of minimization iterations.
@@ -6492,7 +7217,10 @@ public:
//!
//! \see setMinTimingIterations()
//!
- virtual int32_t getMinTimingIterations() const TRTNOEXCEPT = 0;
+ virtual int32_t getMinTimingIterations() const noexcept
+ {
+ return mImpl->getMinTimingIterations();
+ }
//!
//! \brief Set the number of averaging iterations used when timing layers.
@@ -6502,7 +7230,10 @@ public:
//!
//! \see getAvgTimingIterations()
//!
- virtual void setAvgTimingIterations(int32_t avgTiming) TRTNOEXCEPT = 0;
+ virtual void setAvgTimingIterations(int32_t avgTiming) noexcept
+ {
+ mImpl->setAvgTimingIterations(avgTiming);
+ }
//!
//! \brief Query the number of averaging iterations.
@@ -6511,7 +7242,10 @@ public:
//!
//! \see setAvgTimingIterations()
//!
- virtual int32_t getAvgTimingIterations() const TRTNOEXCEPT = 0;
+ int32_t getAvgTimingIterations() const noexcept
+ {
+ return mImpl->getAvgTimingIterations();
+ }
//!
//! \brief Configure the builder to target specified EngineCapability flow.
@@ -6521,28 +7255,40 @@ public:
//!
//! The supported flows are specified in the EngineCapability enum.
//!
- virtual void setEngineCapability(EngineCapability capability) TRTNOEXCEPT = 0;
+ void setEngineCapability(EngineCapability capability) noexcept
+ {
+ mImpl->setEngineCapability(capability);
+ }
//!
//! \brief Query EngineCapability flow configured for the builder.
//!
- //! By default it returns EngineCapability::kDEFAULT.
+ //! By default it returns EngineCapability::kSTANDARD.
//!
//! \see setEngineCapability()
//!
- virtual EngineCapability getEngineCapability() const TRTNOEXCEPT = 0;
+ EngineCapability getEngineCapability() const noexcept
+ {
+ return mImpl->getEngineCapability();
+ }
//!
//! \brief Set Int8 Calibration interface.
//!
//! The calibrator is to minimize the information loss during the INT8 quantization process.
//!
- virtual void setInt8Calibrator(IInt8Calibrator* calibrator) TRTNOEXCEPT = 0;
+ void setInt8Calibrator(IInt8Calibrator* calibrator) noexcept
+ {
+ mImpl->setInt8Calibrator(calibrator);
+ }
//!
//! \brief Get Int8 Calibration interface.
//!
- virtual IInt8Calibrator* getInt8Calibrator() const TRTNOEXCEPT = 0;
+ IInt8Calibrator* getInt8Calibrator() const noexcept
+ {
+ return mImpl->getInt8Calibrator();
+ }
//!
//! \brief Set the maximum workspace size.
@@ -6551,7 +7297,10 @@ public:
//!
//! \see getMaxWorkspaceSize()
//!
- virtual void setMaxWorkspaceSize(std::size_t workspaceSize) TRTNOEXCEPT = 0;
+ void setMaxWorkspaceSize(std::size_t workspaceSize) noexcept
+ {
+ mImpl->setMaxWorkspaceSize(workspaceSize);
+ }
//!
//! \brief Get the maximum workspace size.
@@ -6562,7 +7311,10 @@ public:
//!
//! \see setMaxWorkspaceSize()
//!
- virtual std::size_t getMaxWorkspaceSize() const TRTNOEXCEPT = 0;
+ std::size_t getMaxWorkspaceSize() const noexcept
+ {
+ return mImpl->getMaxWorkspaceSize();
+ }
//!
//! \brief Set the build mode flags to turn on builder options for this network.
@@ -6576,7 +7328,10 @@ public:
//!
//! \see getFlags()
//!
- virtual void setFlags(BuilderFlags builderFlags) TRTNOEXCEPT = 0;
+ void setFlags(BuilderFlags builderFlags) noexcept
+ {
+ mImpl->setFlags(builderFlags);
+ }
//!
//! \brief Get the build mode flags for this builder config. Defaults to 0.
@@ -6585,7 +7340,10 @@ public:
//!
//! \see setFlags()
//!
- virtual BuilderFlags getFlags() const TRTNOEXCEPT = 0;
+ BuilderFlags getFlags() const noexcept
+ {
+ return mImpl->getFlags();
+ }
//!
//! \brief clear a single build mode flag.
@@ -6594,7 +7352,10 @@ public:
//!
//! \see setFlags()
//!
- virtual void clearFlag(BuilderFlag builderFlag) TRTNOEXCEPT = 0;
+ void clearFlag(BuilderFlag builderFlag) noexcept
+ {
+ mImpl->clearFlag(builderFlag);
+ }
//!
//! \brief Set a single build mode flag.
@@ -6603,7 +7364,10 @@ public:
//!
//! \see setFlags()
//!
- virtual void setFlag(BuilderFlag builderFlag) TRTNOEXCEPT = 0;
+ void setFlag(BuilderFlag builderFlag) noexcept
+ {
+ mImpl->setFlag(builderFlag);
+ }
//!
//! \brief Returns true if the build mode flag is set
@@ -6612,7 +7376,10 @@ public:
//!
//! \return True if flag is set, false if unset.
//!
- virtual bool getFlag(BuilderFlag builderFlag) const TRTNOEXCEPT = 0;
+ bool getFlag(BuilderFlag builderFlag) const noexcept
+ {
+ return mImpl->getFlag(builderFlag);
+ }
//!
//! \brief Set the device that this layer must execute on.
@@ -6620,37 +7387,52 @@ public:
//! If DeviceType is not set or is reset, TensorRT will use the default DeviceType set in the builder.
//!
//! \note The device type for a layer must be compatible with the safety flow (if specified).
- //! For example a layer cannot be marked for DLA execution while the builder is configured for kSAFE_GPU.
+ //! For example a layer cannot be marked for DLA execution while the builder is configured for kSAFETY.
//!
//! \see getDeviceType()
//!
- virtual void setDeviceType(const ILayer* layer, DeviceType deviceType) TRTNOEXCEPT = 0;
+ void setDeviceType(const ILayer* layer, DeviceType deviceType) noexcept
+ {
+ mImpl->setDeviceType(layer, deviceType);
+ }
//!
//! \brief Get the device that this layer executes on.
//! \return Returns DeviceType of the layer.
//!
- virtual DeviceType getDeviceType(const ILayer* layer) const TRTNOEXCEPT = 0;
+ DeviceType getDeviceType(const ILayer* layer) const noexcept
+ {
+ return mImpl->getDeviceType(layer);
+ }
//!
//! \brief whether the DeviceType has been explicitly set for this layer
//! \return true if device type is not default
//! \see setDeviceType() getDeviceType() resetDeviceType()
//!
- virtual bool isDeviceTypeSet(const ILayer* layer) const TRTNOEXCEPT = 0;
+ bool isDeviceTypeSet(const ILayer* layer) const noexcept
+ {
+ return mImpl->isDeviceTypeSet(layer);
+ }
//!
//! \brief reset the DeviceType for this layer
//!
//! \see setDeviceType() getDeviceType() isDeviceTypeSet()
//!
- virtual void resetDeviceType(const ILayer* layer) TRTNOEXCEPT = 0;
+ void resetDeviceType(const ILayer* layer) noexcept
+ {
+ mImpl->resetDeviceType(layer);
+ }
//!
//! \brief Checks if a layer can run on DLA.
//! \return status true if the layer can on DLA else returns false.
//!
- virtual bool canRunOnDLA(const ILayer* layer) const TRTNOEXCEPT = 0;
+ bool canRunOnDLA(const ILayer* layer) const noexcept
+ {
+ return mImpl->canRunOnDLA(layer);
+ }
//!
//! \brief Sets the DLA core used by the network.
@@ -6662,7 +7444,10 @@ public:
//!
//! \warning Starting with TensorRT 8, the default value will be -1 if the DLA is not specified or unused.
//!
- virtual void setDLACore(int32_t dlaCore) TRTNOEXCEPT = 0;
+ void setDLACore(int32_t dlaCore) noexcept
+ {
+ mImpl->setDLACore(dlaCore);
+ }
//!
//! \brief Get the DLA core that the engine executes on.
@@ -6670,53 +7455,78 @@ public:
//!
//! \warning Starting with TensorRT 8, the default value will be -1 if the DLA is not specified or unused.
//!
- virtual int32_t getDLACore() const TRTNOEXCEPT = 0;
+ int32_t getDLACore() const noexcept
+ {
+ return mImpl->getDLACore();
+ }
//!
//! \brief Sets the default DeviceType to be used by the builder. It ensures that all the layers that can run on
//! this device will run on it, unless setDeviceType is used to override the default DeviceType for a layer.
//! \see getDefaultDeviceType()
//!
- virtual void setDefaultDeviceType(DeviceType deviceType) TRTNOEXCEPT = 0;
+ void setDefaultDeviceType(DeviceType deviceType) noexcept
+ {
+ mImpl->setDefaultDeviceType(deviceType);
+ }
//!
//! \brief Get the default DeviceType which was set by setDefaultDeviceType.
//!
//! By default it returns DeviceType::kGPU.
//!
- virtual DeviceType getDefaultDeviceType() const TRTNOEXCEPT = 0;
+ DeviceType getDefaultDeviceType() const noexcept
+ {
+ return mImpl->getDefaultDeviceType();
+ }
//!
//! \brief Resets the builder configuration to defaults.
//!
//! When initializing a builder config object, we can call this function.
//!
- virtual void reset() TRTNOEXCEPT = 0;
+ void reset() noexcept
+ {
+ mImpl->reset();
+ }
//!
//! \brief De-allocates any internally allocated memory.
//!
//! When destroying a builder config object, we can call this function.
//!
- virtual void destroy() TRTNOEXCEPT = 0;
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
+ //!
+ TRT_DEPRECATED void destroy() noexcept
+ {
+ delete this;
+ }
//!
- //! \brief Set the cudaStream that is used to profile this network.
+ //! \brief Set the cuda stream that is used to profile this network.
//!
//! \param stream The cuda stream used for profiling by the builder.
//!
//! \see getProfileStream()
//!
- virtual void setProfileStream(const cudaStream_t stream) TRTNOEXCEPT = 0;
+ void setProfileStream(const cudaStream_t stream) noexcept
+ {
+ return mImpl->setProfileStream(stream);
+ }
//!
- //! \brief Get the cudaStream that is used to profile this network.
+ //! \brief Get the cuda stream that is used to profile this network.
//!
- //! \return The cuda stream used for profiling by the builder.
+ //! \return The cuda stream set by setProfileStream, nullptr if setProfileStream has not been called.
//!
//! \see setProfileStream()
//!
- virtual cudaStream_t getProfileStream() const TRTNOEXCEPT = 0;
+ cudaStream_t getProfileStream() const noexcept
+ {
+ return mImpl->getProfileStream();
+ }
//!
//! \brief Add an optimization profile.
@@ -6729,7 +7539,10 @@ public:
//! \return The index of the optimization profile (starting from 0) if the input is valid, or -1 if the input is
//! not valid.
//!
- virtual int32_t addOptimizationProfile(const IOptimizationProfile* profile) noexcept = 0;
+ int32_t addOptimizationProfile(const IOptimizationProfile* profile) noexcept
+ {
+ return mImpl->addOptimizationProfile(profile);
+ }
//!
//! \brief Get number of optimization profiles.
@@ -6739,12 +7552,11 @@ public:
//!
//! \return The number of the optimization profiles.
//!
- virtual int32_t getNbOptimizationProfiles() const noexcept = 0;
+ int32_t getNbOptimizationProfiles() const noexcept
+ {
+ return mImpl->getNbOptimizationProfiles();
+ }
-protected:
- virtual ~IBuilderConfig() {}
-
-public:
//!
//! \brief Set verbosity level of layer information exposed in NVTX annotations.
//!
@@ -6752,7 +7564,10 @@ public:
//!
//! \see ProfilingVerbosity, getProfilingVerbosity()
//!
- virtual void setProfilingVerbosity(ProfilingVerbosity verbosity) TRTNOEXCEPT = 0;
+ void setProfilingVerbosity(ProfilingVerbosity verbosity) noexcept
+ {
+ mImpl->setProfilingVerbosity(verbosity);
+ }
//!
//! \brief Get verbosity level of layer information exposed in NVTX annotations.
@@ -6762,36 +7577,52 @@ public:
//!
//! \see ProfilingVerbosity, setProfilingVerbosity()
//!
- virtual ProfilingVerbosity getProfilingVerbosity() const TRTNOEXCEPT = 0;
+ ProfilingVerbosity getProfilingVerbosity() const noexcept
+ {
+ return mImpl->getProfilingVerbosity();
+ }
//!
//! \brief Set Algorithm Selector.
//!
//! \param selector The algorithm selector to be set in the build config.
- virtual void setAlgorithmSelector(IAlgorithmSelector* selector) TRTNOEXCEPT = 0;
+ void setAlgorithmSelector(IAlgorithmSelector* selector) noexcept
+ {
+ mImpl->setAlgorithmSelector(selector);
+ }
//!
//! \brief Get Algorithm Selector.
//!
- virtual IAlgorithmSelector* getAlgorithmSelector() const TRTNOEXCEPT = 0;
+ IAlgorithmSelector* getAlgorithmSelector() const noexcept
+ {
+ return mImpl->getAlgorithmSelector();
+ }
//!
//! \brief Add a calibration profile.
//!
- //! Calibration optimization profile must be set if int8 calibration is used to set scales for a network with runtime dimensions.
+ //! Calibration optimization profile must be set if int8 calibration is used to set scales for a network with
+ //! runtime dimensions.
//!
//! \param profile The new calibration profile, which must satisfy profile->isValid() == true or be nullptr.
//! MIN and MAX values will be overwritten by kOPT.
//! \return True if the calibration profile was set correctly.
//!
- virtual bool setCalibrationProfile(const IOptimizationProfile* profile) noexcept = 0;
+ bool setCalibrationProfile(const IOptimizationProfile* profile) noexcept
+ {
+ return mImpl->setCalibrationProfile(profile);
+ }
//!
//! \brief Get the current calibration profile.
//!
//! \return A pointer to the current calibration profile or nullptr if calibration profile is unset.
//!
- virtual const IOptimizationProfile* getCalibrationProfile() noexcept = 0;
+ const IOptimizationProfile* getCalibrationProfile() noexcept
+ {
+ return mImpl->getCalibrationProfile();
+ }
//!
//! \brief Set the quantization flags.
@@ -6805,7 +7636,10 @@ public:
//!
//! \see getQuantizationFlags()
//!
- virtual void setQuantizationFlags(QuantizationFlags flags) TRTNOEXCEPT = 0;
+ void setQuantizationFlags(QuantizationFlags flags) noexcept
+ {
+ mImpl->setQuantizationFlags(flags);
+ }
//!
//! \brief Get the quantization flags.
@@ -6814,7 +7648,10 @@ public:
//!
//! \see setQuantizationFlag()
//!
- virtual QuantizationFlags getQuantizationFlags() const TRTNOEXCEPT = 0;
+ QuantizationFlags getQuantizationFlags() const noexcept
+ {
+ return mImpl->getQuantizationFlags();
+ }
//!
//! \brief clear a quantization flag.
@@ -6823,7 +7660,10 @@ public:
//!
//! \see setQuantizationFlags()
//!
- virtual void clearQuantizationFlag(QuantizationFlag flag) TRTNOEXCEPT = 0;
+ void clearQuantizationFlag(QuantizationFlag flag) noexcept
+ {
+ mImpl->clearQuantizationFlag(flag);
+ }
//!
//! \brief Set a single quantization flag.
@@ -6832,7 +7672,10 @@ public:
//!
//! \see setQuantizationFlags()
//!
- virtual void setQuantizationFlag(QuantizationFlag flag) TRTNOEXCEPT = 0;
+ void setQuantizationFlag(QuantizationFlag flag) noexcept
+ {
+ mImpl->setQuantizationFlag(flag);
+ }
//!
//! \brief Returns true if the quantization flag is set.
@@ -6841,7 +7684,10 @@ public:
//!
//! \return True if quantization flag is set, false if unset.
//!
- virtual bool getQuantizationFlag(QuantizationFlag flag) const TRTNOEXCEPT = 0;
+ bool getQuantizationFlag(QuantizationFlag flag) const noexcept
+ {
+ return mImpl->getQuantizationFlag(flag);
+ }
//!
//! \brief Set tactic sources.
@@ -6849,8 +7695,8 @@ public:
//! This bitset controls which tactic sources TensorRT is allowed to use for tactic
//! selection.
//!
- //! By default, kCUBLAS is always enabled. kCUBLAS_LT is enabled for x86
- //! platforms, as well as non-x86 platforms if CUDA >= 11.0
+ //! By default, kCUBLAS and kCUDNN are always enabled. kCUBLAS_LT is enabled for x86
+ //! platforms as well as non-x86 platforms when CUDA >= 11.0.
//!
//! Multiple tactic sources may be combined with a bitwise OR operation. For example,
//! to enable cublas and cublasLt as tactic sources, use a value of:
@@ -6863,7 +7709,10 @@ public:
//! \return true if the tactic sources in the build configuration were updated.
//! The tactic sources in the build configuration will not be updated if the provided value is invalid.
//!
- virtual bool setTacticSources(TacticSources tacticSources) TRTNOEXCEPT = 0;
+ bool setTacticSources(TacticSources tacticSources) noexcept
+ {
+ return mImpl->setTacticSources(tacticSources);
+ }
//!
//! \brief Get tactic sources.
@@ -6875,18 +7724,74 @@ public:
//!
//! \return tactic sources
//!
- virtual TacticSources getTacticSources() const TRTNOEXCEPT = 0;
+ TacticSources getTacticSources() const noexcept
+ {
+ return mImpl->getTacticSources();
+ }
+
+ //!
+ //! \brief Create timing cache
+ //!
+ //! Create ITimingCache instance from serialized raw data. The created timing cache doesn’t belong to
+ //! a specific IBuilderConfig. It can be shared by multiple builder instances. Call setTimingCache()
+ //! before launching a builder to attach cache to builder instance.
+ //!
+ //! \param blob A pointer to the raw data that contains serialized timing cache
+ //! \param size The size in bytes of the serialized timing cache. Size 0 means create a new cache from scratch
+ //!
+ //! \see setTimingCache
+ //!
+ //! \return the pointer to ITimingCache created
+ //!
+ nvinfer1::ITimingCache* createTimingCache(const void* blob, std::size_t size) const noexcept
+ {
+ return mImpl->createTimingCache(blob, size);
+ }
+
+ //!
+ //! \brief Attach a timing cache to IBuilderConfig
+ //!
+ //! The timing cache has verification header to make sure the provided cache can be used in current environment.
+ //! A failure will be reported if the CUDA device property in the provided cache is different from current
+ //! environment. ignoreMismatch = true skips strict verification and allows loading cache created from a different
+ //! device.
+ //!
+ //! The cache must not be destroyed until after the engine is built.
+ //!
+ //! \param cache the timing cache to be used
+ //! \param ignoreMismatch whether or not allow using a cache that contains different CUDA device property
+ //!
+ //! \return true if set successfully, false otherwise
+ //!
+ //! \warning Using cache generated from devices with different CUDA device properties may lead to
+ //! functional/performance bugs.
+ //!
+ bool setTimingCache(const ITimingCache& cache, bool ignoreMismatch) noexcept
+ {
+ return mImpl->setTimingCache(cache, ignoreMismatch);
+ }
+
+ //!
+ //! \brief Get the pointer to the timing cache from current IBuilderConfig
+ //!
+ //! \return pointer to the timing cache used in current IBuilderConfig
+ //!
+ const nvinfer1::ITimingCache* getTimingCache() const noexcept
+ {
+ return mImpl->getTimingCache();
+ }
+
+protected:
+ apiv::VBuilderConfig* mImpl;
};
-//! \typedef NetworkDefinitionCreationFlags
-//!
-//! \brief This bitset is capable of representing one or more NetworkDefinitionCreationFlag flags
-//! constructed with binary OR operations.
+//! \brief Represents one or more NetworkDefinitionCreationFlag flags
+//! using binary OR operations.
//! e.g., 1U << NetworkDefinitionCreationFlag::kEXPLICIT_BATCH
//!
//! \see IBuilder::createNetworkV2
//!
-typedef uint32_t NetworkDefinitionCreationFlags;
+using NetworkDefinitionCreationFlags = uint32_t;
//! \enum NetworkDefinitionCreationFlag
//!
@@ -6917,12 +7822,12 @@ enum class NetworkDefinitionCreationFlag : int32_t
//! precision network is [-127,127].
//! 5) Quantizing and dequantizing activation values between higher (FP32) and lower (INT8) precision
//! will be performed using explicit Scale layers with input/output precision set appropriately.
- kEXPLICIT_PRECISION = 1, //!< Mark the network to be an explicit precision network
+ kEXPLICIT_PRECISION TRT_DEPRECATED_ENUM = 1, //! <-- Deprecated, used for backward compatibility
};
//! Maximum number of elements in NetworkDefinitionCreationFlag enum. \see NetworkDefinitionCreationFlag
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 2;
}
@@ -6934,21 +7839,10 @@ constexpr inline int32_t EnumMax()
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class IBuilder
+class IBuilder : public INoCopy
{
public:
- //!
- //! \brief Create a network definition object where all tensors have an implicit batch dimension.
- //!
- //! This method is equivalent to createNetworkV2(0U), and retained for
- //! compatibility
- //! with earlier version of TensorRT. The network does not support dynamic shapes or explicit batch sizes.
- //!
- //! \see INetworkDefinition, createNetworkV2
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilder::createNetworkV2 instead.
- //!
- TRT_DEPRECATED virtual nvinfer1::INetworkDefinition* createNetwork() TRTNOEXCEPT = 0;
+ virtual ~IBuilder() noexcept = default;
//!
//! \brief Set the maximum batch size.
@@ -6958,7 +7852,10 @@ public:
//!
//! \see getMaxBatchSize()
//!
- virtual void setMaxBatchSize(int32_t batchSize) TRTNOEXCEPT = 0;
+ void setMaxBatchSize(int32_t batchSize) noexcept
+ {
+ mImpl->setMaxBatchSize(batchSize);
+ }
//!
//! \brief Get the maximum batch size.
@@ -6968,231 +7865,38 @@ public:
//! \see setMaxBatchSize()
//! \see getMaxDLABatchSize()
//!
- virtual int32_t getMaxBatchSize() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the maximum workspace size.
- //!
- //! \param workspaceSize The maximum GPU temporary memory which the engine can use at execution time.
- //!
- //! \see getMaxWorkspaceSize()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setMaxWorkspaceSize instead.
- //!
- TRT_DEPRECATED virtual void setMaxWorkspaceSize(std::size_t workspaceSize) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the maximum workspace size.
- //!
- //! \return The maximum workspace size.
- //!
- //! \see setMaxWorkspaceSize()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getMaxWorkspaceSize instead.
- //!
- TRT_DEPRECATED virtual std::size_t getMaxWorkspaceSize() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set whether half2 mode is used.
- //!
- //! half2 mode is a paired-image mode that is significantly faster for batch sizes greater than one on platforms
- //! with fp16 support.
- //!
- //! \param mode Whether half2 mode is used.
- //!
- //! \see getHalf2Mode()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setFlag instead.
- //!
- TRT_DEPRECATED virtual void setHalf2Mode(bool mode) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query whether half2 mode is used.
- //!
- //! \see setHalf2Mode()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getFlag instead.
- //!
- TRT_DEPRECATED virtual bool getHalf2Mode() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set whether the builder should use debug synchronization.
- //!
- //! If this flag is true, the builder will synchronize after timing each layer, and report the layer name. It can
- //! be useful when diagnosing issues at build time.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setFlag instead.
- //!
- TRT_DEPRECATED virtual void setDebugSync(bool sync) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query whether the builder will use debug synchronization.
- //!
- //! \see setDebugSync()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getFlag instead.
- //!
- TRT_DEPRECATED virtual bool getDebugSync() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the number of minimization iterations used when timing layers.
- //!
- //! When timing layers, the builder minimizes over a set of average times for layer execution. This parameter
- //! controls the number of iterations used in minimization.
- //!
- //! \see getMinFindIterations()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setMinTimingIterations instead.
- //!
- TRT_DEPRECATED virtual void setMinFindIterations(int32_t minFind) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query the number of minimization iterations.
- //!
- //! \see setMinFindIterations()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getMinTimingIterations instead.
- //!
- TRT_DEPRECATED virtual int32_t getMinFindIterations() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the number of averaging iterations used when timing layers.
- //!
- //! When timing layers, the builder minimizes over a set of average times for layer execution. This parameter
- //! controls the number of iterations used in averaging.
- //!
- //! \see getAverageFindIterations()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setAvgTimingIterations instead.
- //!
- TRT_DEPRECATED virtual void setAverageFindIterations(int32_t avgFind) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query the number of averaging iterations.
- //!
- //! \see setAverageFindIterations()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getAvgTimingIterations instead.
- //!
- TRT_DEPRECATED virtual int32_t getAverageFindIterations() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Build a CUDA engine from a network definition.
- //!
- //! \see INetworkDefinition ICudaEngine
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilder::buildEngineWithConfig instead.
- //!
- TRT_DEPRECATED virtual nvinfer1::ICudaEngine* buildCudaEngine(
- nvinfer1::INetworkDefinition& network) TRTNOEXCEPT = 0;
+ int32_t getMaxBatchSize() const noexcept
+ {
+ return mImpl->getMaxBatchSize();
+ }
//!
//! \brief Determine whether the platform has fast native fp16.
//!
- virtual bool platformHasFastFp16() const TRTNOEXCEPT = 0;
+ bool platformHasFastFp16() const noexcept
+ {
+ return mImpl->platformHasFastFp16();
+ }
//!
//! \brief Determine whether the platform has fast native int8.
//!
- virtual bool platformHasFastInt8() const TRTNOEXCEPT = 0;
+ bool platformHasFastInt8() const noexcept
+ {
+ return mImpl->platformHasFastInt8();
+ }
//!
//! \brief Destroy this object.
//!
- virtual void destroy() TRTNOEXCEPT = 0;
-
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
//!
- //! \brief Set whether or not quantized 8-bit kernels are permitted.
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
//!
- //! During engine build int8 kernels will also be tried when this mode is enabled.
- //!
- //! \param mode Whether quantized 8-bit kernels are permitted.
- //!
- //! \see getInt8Mode()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setFlag instead.
- //!
- TRT_DEPRECATED virtual void setInt8Mode(bool mode) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query whether Int8 mode is used.
- //!
- //! \see setInt8Mode()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getFlag instead.
- //!
- TRT_DEPRECATED virtual bool getInt8Mode() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set Int8 Calibration interface.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setInt8Calibrator instead.
- //!
- TRT_DEPRECATED virtual void setInt8Calibrator(IInt8Calibrator* calibrator) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the device that this layer must execute on.
- //! \param DeviceType that this layer must execute on.
- //! If DeviceType is not set or is reset, TensorRT will use the default DeviceType set in the builder.
- //!
- //! \note The device type for a layer must be compatible with the safety flow (if specified).
- //! For example a layer cannot be marked for DLA execution while the builder is configured for kSAFE_GPU.
- //!
- //! \see getDeviceType()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setDeviceType instead.
- //!
- TRT_DEPRECATED virtual void setDeviceType(ILayer* layer, DeviceType deviceType) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the device that this layer executes on.
- //! \return Returns DeviceType of the layer.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getDeviceType instead.
- //!
- TRT_DEPRECATED virtual DeviceType getDeviceType(const ILayer* layer) const TRTNOEXCEPT = 0;
-
- //!
- //! \brief whether the DeviceType has been explicitly set for this layer
- //! \return whether the DeviceType has been explicitly set
- //! \see setDeviceType() getDeviceType() resetDeviceType()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::isDeviceTypeSet instead.
- //!
- TRT_DEPRECATED virtual bool isDeviceTypeSet(const ILayer* layer) const TRTNOEXCEPT = 0;
-
- //!
- //! \brief reset the DeviceType for this layer
- //!
- //! \see setDeviceType() getDeviceType() isDeviceTypeSet()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::resetDeviceType instead.
- //!
- TRT_DEPRECATED virtual void resetDeviceType(ILayer* layer) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Checks if a layer can run on DLA.
- //! \return status true if the layer can on DLA else returns false.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::canRunOnDLA instead.
- //!
- TRT_DEPRECATED virtual bool canRunOnDLA(const ILayer* layer) const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Sets the default DeviceType to be used by the builder. It ensures that all the layers that can run on
- //! this device will run on it, unless setDeviceType is used to override the default DeviceType for a layer.
- //! \see getDefaultDeviceType()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setDefaultDeviceType instead.
- //!
- TRT_DEPRECATED virtual void setDefaultDeviceType(DeviceType deviceType) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the default DeviceType which was set by setDefaultDeviceType.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getDefaultDeviceType instead.
- //!
- TRT_DEPRECATED virtual DeviceType getDefaultDeviceType() const TRTNOEXCEPT = 0;
+ TRT_DEPRECATED void destroy() noexcept
+ {
+ delete this;
+ }
//!
//! \brief Get the maximum batch size DLA can support.
@@ -7201,57 +7905,19 @@ public:
//!
//! \warning getMaxDLABatchSize does not work with dynamic shapes.
//!
- virtual int32_t getMaxDLABatchSize() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Sets the builder to use GPU if a layer that was supposed to run on DLA can not run on DLA.
- //! \param Allows fallback if setFallBackMode is true else disables fallback option.
- //!
- //! \note GPU fallback may only be specified for non-safety modes. \see EngineCapability
- //! Simultaneously enabling GPU fallback and safety-restricted modes is disallowed.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setFlag instead.
- //!
- TRT_DEPRECATED virtual void allowGPUFallback(bool setFallBackMode) TRTNOEXCEPT = 0;
+ int32_t getMaxDLABatchSize() const noexcept
+ {
+ return mImpl->getMaxDLABatchSize();
+ }
//!
//! \brief Return the number of DLA engines available to this builder.
//!
- virtual int32_t getNbDLACores() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the DLA core that the engine must execute on.
- //! \param dlaCore The DLA core to execute the engine on (0 to N-1, where N is the maximum number of DLA cores
- //! present on the device). Default value is 0.
- //! DLA Core is not a property of the engine that is preserved by serialization: when the engine is deserialized
- //! it will be associated with the DLA core which is configured for the runtime.
- //! \see IRuntime::setDLACore() getDLACore()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setDLACore instead.
- //!
- TRT_DEPRECATED virtual void setDLACore(int32_t dlaCore) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the DLA core that the engine executes on.
- //! \return If setDLACore is called, returns DLA core from 0 to N-1, else returns 0.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getDLACore instead.
- //!
- TRT_DEPRECATED virtual int32_t getDLACore() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Resets the builder state
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilder::reset instead.
- //!
- TRT_DEPRECATED virtual void reset(nvinfer1::INetworkDefinition& network) TRTNOEXCEPT = 0;
-
-protected:
- virtual ~IBuilder()
+ int32_t getNbDLACores() const noexcept
{
+ return mImpl->getNbDLACores();
}
-public:
//!
//! \brief Set the GPU allocator.
//! \param allocator Set the GPU allocator to be used by the builder. All GPU memory acquired will use this
@@ -7263,97 +7929,20 @@ public:
//! must span the lifetime of those engines as
//! well as that of the builder. If nullptr is passed, the default allocator will be used.
//!
- virtual void setGpuAllocator(IGpuAllocator* allocator) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set whether or not 16-bit kernels are permitted.
- //!
- //! During engine build fp16 kernels will also be tried when this mode is enabled.
- //!
- //! \param mode Whether 16-bit kernels are permitted.
- //!
- //! \see getFp16Mode()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setFlag instead.
- //!
- TRT_DEPRECATED virtual void setFp16Mode(bool mode) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query whether 16-bit kernels are permitted.
- //!
- //! \see setFp16Mode()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getFlag instead.
- //!
- TRT_DEPRECATED virtual bool getFp16Mode() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set whether or not type constraints are strict.
- //!
- //! When strict type constraints are in use, TensorRT will always choose a layer implementation that conforms to the
- //! type constraints specified, if one exists. If this flag is not set, a higher-precision implementation may be
- //! chosen if it results in higher performance.
- //!
- //! If no conformant layer exists, TensorRT will choose a non-conformant layer if available regardless of the
- //! setting of this flag.
- //!
- //! See the developer guide for the definition of strictness.
- //!
- //! \param mode Whether type constraints are strict
- //!
- //! \see getStrictTypeConstraints()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setFlag instead.
- //!
- TRT_DEPRECATED virtual void setStrictTypeConstraints(bool mode) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query whether or not type constraints are strict.
- //!
- //! \see setStrictTypeConstraints()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getFlag instead.
- //!
- TRT_DEPRECATED virtual bool getStrictTypeConstraints() const TRTNOEXCEPT = 0;
-
- //!
- //! Set whether engines will be refittable.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setFlag instead.
- //!
- TRT_DEPRECATED virtual void setRefittable(bool canRefit) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query whether or not engines will be refittable.
- //!
- //! \see getRefittable()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getFlag instead.
- //!
- TRT_DEPRECATED virtual bool getRefittable() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Configure the builder to target specified EngineCapability flow.
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::setEngineCapability instead.
- //!
- TRT_DEPRECATED virtual void setEngineCapability(EngineCapability capability) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Query EngineCapability flow configured for the builder.
- //!
- //! \see setEngineCapability()
- //!
- //! \deprecated API will be removed in TensorRT 8.0, use IBuilderConfig::getEngineCapability instead.
- //!
- TRT_DEPRECATED virtual EngineCapability getEngineCapability() const TRTNOEXCEPT = 0;
+ void setGpuAllocator(IGpuAllocator* allocator) noexcept
+ {
+ mImpl->setGpuAllocator(allocator);
+ }
//!
//! \brief Create a builder configuration object.
//!
//! \see IBuilderConfig
//!
- virtual nvinfer1::IBuilderConfig* createBuilderConfig() TRTNOEXCEPT = 0;
+ nvinfer1::IBuilderConfig* createBuilderConfig() noexcept
+ {
+ return mImpl->createBuilderConfig();
+ }
//!
//! \brief Builds an engine for the given INetworkDefinition and given IBuilderConfig.
@@ -7361,9 +7950,15 @@ public:
//! It enables the builder to build multiple engines based on the same network definition, but with different
//! builder configurations.
//!
- virtual nvinfer1::ICudaEngine* buildEngineWithConfig(
- INetworkDefinition& network, IBuilderConfig& config) TRTNOEXCEPT = 0;
-
+ //! \note This function will synchronize the cuda stream returned by \p config.getProfileStream() before returning.
+ //!
+ //! \deprecated API will be removed in TensorRT 10.0, use IBuilder::buildSerializedNetwork instead.
+ //!
+ TRT_DEPRECATED nvinfer1::ICudaEngine* buildEngineWithConfig(
+ INetworkDefinition& network, IBuilderConfig& config) noexcept
+ {
+ return mImpl->buildEngineWithConfig(network, config);
+ }
//! \brief Create a network definition object
//!
@@ -7376,8 +7971,10 @@ public:
//!
//! \see INetworkDefinition, NetworkDefinitionCreationFlags
//!
- virtual nvinfer1::INetworkDefinition* createNetworkV2(NetworkDefinitionCreationFlags flags) TRTNOEXCEPT = 0;
-
+ nvinfer1::INetworkDefinition* createNetworkV2(NetworkDefinitionCreationFlags flags) noexcept
+ {
+ return mImpl->createNetworkV2(flags);
+ }
//! \brief Create a new optimization profile.
//!
@@ -7388,7 +7985,10 @@ public:
//!
//! \see IOptimizationProfile
//!
- virtual nvinfer1::IOptimizationProfile* createOptimizationProfile() noexcept = 0;
+ nvinfer1::IOptimizationProfile* createOptimizationProfile() noexcept
+ {
+ return mImpl->createOptimizationProfile();
+ }
//!
//! \brief Set the ErrorRecorder for this interface
@@ -7398,33 +7998,91 @@ public:
//! recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if
//! a recorder has been registered.
//!
+ //! If an error recorder is not set, messages will be sent to the global log stream.
+ //!
//! \param recorder The error recorder to register with this interface.
//
- //! \see getErrorRecorder
+ //! \see getErrorRecorder()
//!
- virtual void setErrorRecorder(IErrorRecorder* recorder) TRTNOEXCEPT = 0;
+ void setErrorRecorder(IErrorRecorder* recorder) noexcept
+ {
+ mImpl->setErrorRecorder(recorder);
+ }
//!
//! \brief get the ErrorRecorder assigned to this interface.
//!
- //! Retrieves the assigned error recorder object for the given class. A default error recorder does not exist,
- //! so a nullptr will be returned if setErrorRecorder has not been called.
+ //! Retrieves the assigned error recorder object for the given class.
+ //! A nullptr will be returned if setErrorRecorder has not been called.
//!
//! \return A pointer to the IErrorRecorder object that has been registered.
//!
- //! \see setErrorRecorder
+ //! \see setErrorRecorder()
//!
- virtual IErrorRecorder* getErrorRecorder() const TRTNOEXCEPT = 0;
+ IErrorRecorder* getErrorRecorder() const noexcept
+ {
+ return mImpl->getErrorRecorder();
+ }
//!
//! \brief Resets the builder state to default values.
//!
- virtual void reset() TRTNOEXCEPT = 0;
+ void reset() noexcept
+ {
+ mImpl->reset();
+ }
//!
//! \brief Determine whether the platform has TF32 support.
//!
- virtual bool platformHasTf32() const TRTNOEXCEPT = 0;
+ bool platformHasTf32() const noexcept
+ {
+ return mImpl->platformHasTf32();
+ }
+
+ //!
+ //! \brief Builds and serializes a network for the given INetworkDefinition and IBuilderConfig.
+ //!
+ //! This function allows building and serialization of a network without creating an engine.
+ //!
+ //! \param network Network definition.
+ //! \param config Builder configuration.
+ //!
+ //! \return A pointer to a IHostMemory object that contains a serialized network.
+ //!
+ //! \note This function will synchronize the cuda stream returned by \p config.getProfileStream() before returning.
+ //!
+ //! \see INetworkDefinition, IBuilderConfig, IHostMemory
+ //!
+ nvinfer1::IHostMemory* buildSerializedNetwork(INetworkDefinition& network, IBuilderConfig& config) noexcept
+ {
+ return mImpl->buildSerializedNetwork(network, config);
+ }
+
+ //!
+ //! \brief Checks that a network is within the scope of the IBuilderConfig settings.
+ //!
+ //! \param network The network definition to check for configuration compliance.
+ //! \param config The configuration of the builder to use when checking \p network.
+ //!
+ //! Given an INetworkDefinition, \p network, and an IBuilderConfig, \p config, check if
+ //! the network falls within the constraints of the builder configuration based on the
+ //! EngineCapability, BuilderFlag, and DeviceType. If the network is within the constraints,
+ //! then the function returns true, and false if a violation occurs. This function reports
+ //! the conditions that are violated to the registered ErrorRecorder.
+ //!
+ //! \return True if network is within the scope of the restrictions specified by the builder config,
+ //! false otherwise.
+ //!
+ //! \note This function will synchronize the cuda stream returned by \p config.getProfileStream() before returning.
+ //!
+ bool isNetworkSupported(INetworkDefinition const& network, IBuilderConfig const& config) const noexcept
+ {
+ return mImpl->isNetworkSupported(network, config);
+ }
+
+protected:
+ apiv::VBuilder* mImpl;
};
} // namespace nvinfer1
@@ -7433,24 +8091,26 @@ public:
//! Internal C entry point for creating IBuilder.
//! @private
//!
-extern "C" TENSORRTAPI void* createInferBuilder_INTERNAL(void* logger, int32_t version);
+extern "C" TENSORRTAPI void* createInferBuilder_INTERNAL(void* logger, int32_t version) noexcept;
namespace nvinfer1
{
namespace
{
+
//!
//! \brief Create an instance of an IBuilder class.
//!
-//! This class is the logging class for the builder.
+//! This is the logging class for the builder.
//!
//! unnamed namespace avoids linkage surprises when linking objects built with different versions of this header.
//!
-inline IBuilder* createInferBuilder(ILogger& logger)
+inline IBuilder* createInferBuilder(ILogger& logger) noexcept
{
return static_cast(createInferBuilder_INTERNAL(&logger, NV_TENSORRT_VERSION));
}
-}
-}
-#endif
+} // namespace
+} // namespace nvinfer1
+
+#endif // NV_INFER_H
diff --git a/include/NvInferImpl.h b/include/NvInferImpl.h
new file mode 100644
index 00000000..2b283e45
--- /dev/null
+++ b/include/NvInferImpl.h
@@ -0,0 +1,939 @@
+/*
+ * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef NV_INFER_IMPL_H
+#define NV_INFER_IMPL_H
+
+#include "NvInferLegacyDims.h"
+#include "NvInferRuntimeCommon.h"
+
+namespace nvinfer1
+{
+
+class IActivationLayer;
+class IAlgorithm;
+class IAlgorithmContext;
+class IAlgorithmIOInfo;
+class IAlgorithmSelector;
+class IAlgorithmVariant;
+class IBuilderConfig;
+class IConcatenationLayer;
+class IConstantLayer;
+class IConvolutionLayer;
+class ICudaEngine;
+class IDeconvolutionLayer;
+class IDequantizeLayer;
+class IDimensionExpr;
+class IElementWiseLayer;
+class IExecutionContext;
+class IFillLayer;
+class IFullyConnectedLayer;
+class IGatherLayer;
+class IHostMemory;
+class IIdentityLayer;
+class IInt8Calibrator;
+class IIteratorLayer;
+class ILayer;
+class ILoop;
+class ILoopOutputLayer;
+class ILRNLayer;
+class IMatrixMultiplyLayer;
+class INetworkDefinition;
+class IOptimizationProfile;
+class IPaddingLayer;
+class IParametricReLULayer;
+class IPlugin;
+class IPluginExt;
+class IPluginFactory;
+class IPluginLayer;
+class IPluginV2Layer;
+class IPoolingLayer;
+class IProfiler;
+class IQuantizeLayer;
+class IRaggedSoftMaxLayer;
+class IRecurrenceLayer;
+class IReduceLayer;
+class IResizeLayer;
+class IRNNv2Layer;
+class IScaleLayer;
+class ISelectLayer;
+class IShapeLayer;
+class IShuffleLayer;
+class ISliceLayer;
+class ISoftMaxLayer;
+class ITensor;
+class ITimingCache;
+class ITopKLayer;
+class ITripLimitLayer;
+class IUnaryLayer;
+struct Permutation;
+class Weights;
+
+enum class ActivationType : int32_t;
+enum class BuilderFlag : int32_t;
+enum class CalibrationAlgoType : int32_t;
+enum class DeviceType : int32_t;
+enum class DimensionOperation : int32_t;
+enum class ElementWiseOperation : int32_t;
+enum class EngineCapability : int32_t;
+enum class FillOperation : int32_t;
+enum class LayerType : int32_t;
+enum class LoopOutput : int32_t;
+enum class MatrixOperation : int32_t;
+enum class NetworkDefinitionCreationFlag : int32_t;
+enum class OptProfileSelector : int32_t;
+enum class PaddingMode : int32_t;
+enum class PoolingType : int32_t;
+enum class ProfilingVerbosity : int32_t;
+enum class QuantizationFlag : int32_t;
+enum class ReduceOperation : int32_t;
+enum class ResizeCoordinateTransformation : int32_t;
+enum class ResizeMode : int32_t;
+enum class ResizeRoundMode : int32_t;
+enum class ResizeSelector : int32_t;
+enum class RNNDirection : int32_t;
+enum class RNNGateType : int32_t;
+enum class RNNInputMode : int32_t;
+enum class RNNOperation : int32_t;
+enum class ScaleMode : int32_t;
+enum class SliceMode : int32_t;
+enum class TensorLocation : int32_t;
+enum class TopKOperation : int32_t;
+enum class TripLimit : int32_t;
+enum class UnaryOperation : int32_t;
+enum class WeightsRole : int32_t;
+
+using TacticSources = uint32_t;
+using TensorFormats = uint32_t;
+using BuilderFlags = uint32_t;
+using NetworkDefinitionCreationFlags = uint32_t;
+using QuantizationFlags = uint32_t;
+
+//!
+//! \file NvInferImpl.h
+//!
+//! This file contains definitions for API methods that cross the shared library boundary. These
+//! methods must not be called directly by applications; they should only be called through the
+//! API classes.
+//!
+
+namespace apiv
+{
+
+class VRoot
+{
+public:
+ virtual ~VRoot() noexcept = default;
+};
+
+class VHostMemory : public VRoot
+{
+public:
+ virtual void* data() const noexcept = 0;
+ virtual std::size_t size() const noexcept = 0;
+ virtual DataType type() const noexcept = 0;
+};
+
+class VDimensionExpr : public VRoot
+{
+public:
+ virtual bool isConstant() const = 0;
+ virtual int32_t getConstantValue() const = 0;
+};
+
+class VExprBuilder : public VRoot
+{
+public:
+ virtual const IDimensionExpr* constant(int32_t value) = 0;
+ virtual const IDimensionExpr* operation(
+ DimensionOperation op, const IDimensionExpr& first, const IDimensionExpr& second)
+ = 0;
+};
+
+class VRuntime : public VRoot
+{
+public:
+ virtual nvinfer1::ICudaEngine* deserializeCudaEngine(
+ const void* blob, std::size_t size, IPluginFactory* pluginFactory) noexcept
+ = 0;
+ virtual void setDLACore(int32_t dlaCore) noexcept = 0;
+ virtual int32_t getDLACore() const noexcept = 0;
+ virtual int32_t getNbDLACores() const noexcept = 0;
+ virtual void setGpuAllocator(IGpuAllocator* allocator) noexcept = 0;
+ virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+};
+
+class VRefitter : public VRoot
+{
+public:
+ virtual bool setWeights(const char* layerName, WeightsRole role, const Weights weights) noexcept = 0;
+ virtual bool refitCudaEngine() noexcept = 0;
+ virtual int32_t getMissing(int32_t size, const char** layerNames, WeightsRole* roles) noexcept = 0;
+ virtual int32_t getAll(int32_t size, const char** layerNames, WeightsRole* roles) noexcept = 0;
+ virtual bool setDynamicRange(const char* tensorName, float min, float max) noexcept = 0;
+ virtual float getDynamicRangeMin(const char* tensorName) const noexcept = 0;
+ virtual float getDynamicRangeMax(const char* tensorName) const noexcept = 0;
+ virtual int32_t getTensorsWithDynamicRange(int32_t size, const char** tensorNames) const noexcept = 0;
+ virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+ virtual bool setNamedWeights(const char* name, Weights weights) noexcept = 0;
+ virtual int32_t getMissingWeights(int32_t size, const char** weightsNames) noexcept = 0;
+ virtual int32_t getAllWeights(int32_t size, const char** weightsNames) noexcept = 0;
+};
+
+class VOptimizationProfile : public VRoot
+{
+public:
+ virtual bool setDimensions(const char* inputName, OptProfileSelector select, Dims dims) noexcept = 0;
+ virtual Dims getDimensions(const char* inputName, OptProfileSelector select) const noexcept = 0;
+ virtual bool setShapeValues(
+ const char* inputName, OptProfileSelector select, const int32_t* values, int32_t nbValues) noexcept
+ = 0;
+ virtual int32_t getNbShapeValues(const char* inputName) const noexcept = 0;
+ virtual int32_t const* getShapeValues(const char* inputName, OptProfileSelector select) const noexcept = 0;
+ virtual bool setExtraMemoryTarget(float target) noexcept = 0;
+ virtual float getExtraMemoryTarget() const noexcept = 0;
+ virtual bool isValid() const noexcept = 0;
+};
+
+class VCudaEngine : public VRoot
+{
+public:
+ virtual int32_t getNbBindings() const noexcept = 0;
+ virtual int32_t getBindingIndex(const char* name) const noexcept = 0;
+ virtual const char* getBindingName(int32_t bindingIndex) const noexcept = 0;
+ virtual bool bindingIsInput(int32_t bindingIndex) const noexcept = 0;
+ virtual Dims getBindingDimensions(int32_t bindingIndex) const noexcept = 0;
+ virtual DataType getBindingDataType(int32_t bindingIndex) const noexcept = 0;
+ virtual int32_t getMaxBatchSize() const noexcept = 0;
+ virtual int32_t getNbLayers() const noexcept = 0;
+ virtual IHostMemory* serialize() const noexcept = 0;
+ virtual IExecutionContext* createExecutionContext() noexcept = 0;
+ virtual TensorLocation getLocation(int32_t bindingIndex) const noexcept = 0;
+ virtual IExecutionContext* createExecutionContextWithoutDeviceMemory() noexcept = 0;
+ virtual size_t getDeviceMemorySize() const noexcept = 0;
+ virtual bool isRefittable() const noexcept = 0;
+ virtual int32_t getBindingBytesPerComponent(int32_t bindingIndex) const noexcept = 0;
+ virtual int32_t getBindingComponentsPerElement(int32_t bindingIndex) const noexcept = 0;
+ virtual TensorFormat getBindingFormat(int32_t bindingIndex) const noexcept = 0;
+ virtual const char* getBindingFormatDesc(int32_t bindingIndex) const noexcept = 0;
+ virtual int32_t getBindingVectorizedDim(int32_t bindingIndex) const noexcept = 0;
+ virtual const char* getName() const noexcept = 0;
+ virtual int32_t getNbOptimizationProfiles() const noexcept = 0;
+ virtual Dims getProfileDimensions(int32_t bindingIndex, int32_t profileIndex, OptProfileSelector select) const
+ noexcept
+ = 0;
+ virtual const int32_t* getProfileShapeValues(
+ int32_t profileIndex, int32_t inputIndex, OptProfileSelector select) const noexcept
+ = 0;
+ virtual bool isShapeBinding(int32_t bindingIndex) const noexcept = 0;
+ virtual bool isExecutionBinding(int32_t bindingIndex) const noexcept = 0;
+ virtual EngineCapability getEngineCapability() const noexcept = 0;
+ virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+ virtual bool hasImplicitBatchDimension() const noexcept = 0;
+ virtual TacticSources getTacticSources() const noexcept = 0;
+};
+
+class VExecutionContext : public VRoot
+{
+public:
+ virtual bool execute(int32_t batchSize, void* const* bindings) noexcept = 0;
+ virtual bool enqueue(
+ int32_t batchSize, void* const* bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept
+ = 0;
+ virtual void setDebugSync(bool sync) noexcept = 0;
+ virtual bool getDebugSync() const noexcept = 0;
+ virtual void setProfiler(IProfiler*) noexcept = 0;
+ virtual IProfiler* getProfiler() const noexcept = 0;
+ virtual const ICudaEngine& getEngine() const noexcept = 0;
+ virtual void setName(const char* name) noexcept = 0;
+ virtual const char* getName() const noexcept = 0;
+ virtual void setDeviceMemory(void* memory) noexcept = 0;
+ virtual Dims getStrides(int32_t bindingIndex) const noexcept = 0;
+ virtual bool setOptimizationProfile(int32_t profileIndex) noexcept = 0;
+ virtual int32_t getOptimizationProfile() const noexcept = 0;
+ virtual bool setBindingDimensions(int32_t bindingIndex, Dims dimensions) noexcept = 0;
+ virtual Dims getBindingDimensions(int32_t bindingIndex) const noexcept = 0;
+ virtual bool setInputShapeBinding(int32_t bindingIndex, int32_t const* data) noexcept = 0;
+ virtual bool getShapeBinding(int32_t bindingIndex, int32_t* data) const noexcept = 0;
+ virtual bool allInputDimensionsSpecified() const noexcept = 0;
+ virtual bool allInputShapesSpecified() const noexcept = 0;
+ virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+ virtual bool executeV2(void* const* bindings) noexcept = 0;
+ virtual bool enqueueV2(void* const* bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept = 0;
+ virtual bool setOptimizationProfileAsync(int32_t profileIndex, cudaStream_t stream) noexcept = 0;
+};
+
+class VTensor : public VRoot
+{
+public:
+ virtual void setName(const char* name) noexcept = 0;
+ virtual const char* getName() const noexcept = 0;
+ virtual void setDimensions(Dims dimensions) noexcept = 0;
+ virtual Dims getDimensions() const noexcept = 0;
+ virtual void setType(DataType type) noexcept = 0;
+ virtual DataType getType() const noexcept = 0;
+ virtual bool setDynamicRange(float min, float max) noexcept = 0;
+ virtual bool isNetworkInput() const noexcept = 0;
+ virtual bool isNetworkOutput() const noexcept = 0;
+ virtual void setBroadcastAcrossBatch(bool broadcastAcrossBatch) noexcept = 0;
+ virtual bool getBroadcastAcrossBatch() const noexcept = 0;
+ virtual TensorLocation getLocation() const noexcept = 0;
+ virtual void setLocation(TensorLocation location) noexcept = 0;
+ virtual bool dynamicRangeIsSet() const noexcept = 0;
+ virtual void resetDynamicRange() noexcept = 0;
+ virtual float getDynamicRangeMin() const noexcept = 0;
+ virtual float getDynamicRangeMax() const noexcept = 0;
+ virtual void setAllowedFormats(TensorFormats formats) noexcept = 0;
+ virtual TensorFormats getAllowedFormats() const noexcept = 0;
+ virtual bool isShapeTensor() const noexcept = 0;
+ virtual bool isExecutionTensor() const noexcept = 0;
+};
+class VLayer : public VRoot
+{
+public:
+ virtual LayerType getType() const noexcept = 0;
+ virtual void setName(const char* name) noexcept = 0;
+ virtual const char* getName() const noexcept = 0;
+ virtual int32_t getNbInputs() const noexcept = 0;
+ virtual ITensor* getInput(int32_t index) const noexcept = 0;
+ virtual int32_t getNbOutputs() const noexcept = 0;
+ virtual ITensor* getOutput(int32_t index) const noexcept = 0;
+ virtual void setInput(int32_t index, ITensor& tensor) noexcept = 0;
+ virtual void setPrecision(DataType dataType) noexcept = 0;
+ virtual DataType getPrecision() const noexcept = 0;
+ virtual bool precisionIsSet() const noexcept = 0;
+ virtual void resetPrecision() noexcept = 0;
+ virtual void setOutputType(int32_t index, DataType dataType) noexcept = 0;
+ virtual DataType getOutputType(int32_t index) const noexcept = 0;
+ virtual bool outputTypeIsSet(int32_t index) const noexcept = 0;
+ virtual void resetOutputType(int32_t index) noexcept = 0;
+};
+
+class VConvolutionLayer : public VRoot
+{
+public:
+ virtual void setKernelSize(DimsHW kernelSize) noexcept = 0;
+ virtual DimsHW getKernelSize() const noexcept = 0;
+ virtual void setNbOutputMaps(int32_t nbOutputMaps) noexcept = 0;
+ virtual int32_t getNbOutputMaps() const noexcept = 0;
+ virtual void setStride(DimsHW stride) noexcept = 0;
+ virtual DimsHW getStride() const noexcept = 0;
+ virtual void setPadding(DimsHW padding) noexcept = 0;
+ virtual DimsHW getPadding() const noexcept = 0;
+ virtual void setNbGroups(int32_t nbGroups) noexcept = 0;
+ virtual int32_t getNbGroups() const noexcept = 0;
+ virtual void setKernelWeights(Weights weights) noexcept = 0;
+ virtual Weights getKernelWeights() const noexcept = 0;
+ virtual void setBiasWeights(Weights weights) noexcept = 0;
+ virtual Weights getBiasWeights() const noexcept = 0;
+ virtual void setDilation(DimsHW dilation) noexcept = 0;
+ virtual DimsHW getDilation() const noexcept = 0;
+ virtual void setPrePadding(Dims padding) noexcept = 0;
+ virtual Dims getPrePadding() const noexcept = 0;
+ virtual void setPostPadding(Dims padding) noexcept = 0;
+ virtual Dims getPostPadding() const noexcept = 0;
+ virtual void setPaddingMode(PaddingMode paddingMode) noexcept = 0;
+ virtual PaddingMode getPaddingMode() const noexcept = 0;
+ virtual void setKernelSizeNd(Dims kernelSize) noexcept = 0;
+ virtual Dims getKernelSizeNd() const noexcept = 0;
+ virtual void setStrideNd(Dims stride) noexcept = 0;
+ virtual Dims getStrideNd() const noexcept = 0;
+ virtual void setPaddingNd(Dims padding) noexcept = 0;
+ virtual Dims getPaddingNd() const noexcept = 0;
+ virtual void setDilationNd(Dims dilation) noexcept = 0;
+ virtual Dims getDilationNd() const noexcept = 0;
+};
+
+class VFullyConnectedLayer : public VRoot
+{
+public:
+ virtual void setNbOutputChannels(int32_t nbOutputs) noexcept = 0;
+ virtual int32_t getNbOutputChannels() const noexcept = 0;
+ virtual void setKernelWeights(Weights weights) noexcept = 0;
+ virtual Weights getKernelWeights() const noexcept = 0;
+ virtual void setBiasWeights(Weights weights) noexcept = 0;
+ virtual Weights getBiasWeights() const noexcept = 0;
+};
+
+class VActivationLayer : public VRoot
+{
+public:
+ virtual void setActivationType(ActivationType type) noexcept = 0;
+ virtual ActivationType getActivationType() const noexcept = 0;
+ virtual void setAlpha(float alpha) noexcept = 0;
+ virtual void setBeta(float beta) noexcept = 0;
+ virtual float getAlpha() const noexcept = 0;
+ virtual float getBeta() const noexcept = 0;
+};
+
+class VPoolingLayer : public VRoot
+{
+public:
+ virtual void setPoolingType(PoolingType type) noexcept = 0;
+ virtual PoolingType getPoolingType() const noexcept = 0;
+ virtual void setWindowSize(DimsHW windowSize) noexcept = 0;
+ virtual DimsHW getWindowSize() const noexcept = 0;
+ virtual void setStride(DimsHW stride) noexcept = 0;
+ virtual DimsHW getStride() const noexcept = 0;
+ virtual void setPadding(DimsHW padding) noexcept = 0;
+ virtual DimsHW getPadding() const noexcept = 0;
+ virtual void setBlendFactor(float blendFactor) noexcept = 0;
+ virtual float getBlendFactor() const noexcept = 0;
+ virtual void setAverageCountExcludesPadding(bool exclusive) noexcept = 0;
+ virtual bool getAverageCountExcludesPadding() const noexcept = 0;
+ virtual void setPrePadding(Dims padding) noexcept = 0;
+ virtual Dims getPrePadding() const noexcept = 0;
+ virtual void setPostPadding(Dims padding) noexcept = 0;
+ virtual Dims getPostPadding() const noexcept = 0;
+ virtual void setPaddingMode(PaddingMode paddingMode) noexcept = 0;
+ virtual PaddingMode getPaddingMode() const noexcept = 0;
+ virtual void setWindowSizeNd(Dims windowSize) noexcept = 0;
+ virtual Dims getWindowSizeNd() const noexcept = 0;
+ virtual void setStrideNd(Dims stride) noexcept = 0;
+ virtual Dims getStrideNd() const noexcept = 0;
+ virtual void setPaddingNd(Dims padding) noexcept = 0;
+ virtual Dims getPaddingNd() const noexcept = 0;
+};
+
+class VLRNLayer : public VRoot
+{
+public:
+ virtual void setWindowSize(int32_t windowSize) noexcept = 0;
+ virtual int32_t getWindowSize() const noexcept = 0;
+ virtual void setAlpha(float alpha) noexcept = 0;
+ virtual float getAlpha() const noexcept = 0;
+ virtual void setBeta(float beta) noexcept = 0;
+ virtual float getBeta() const noexcept = 0;
+ virtual void setK(float k) noexcept = 0;
+ virtual float getK() const noexcept = 0;
+};
+
+class VScaleLayer : public VRoot
+{
+public:
+ virtual void setMode(ScaleMode mode) noexcept = 0;
+ virtual ScaleMode getMode() const noexcept = 0;
+ virtual void setShift(Weights shift) noexcept = 0;
+ virtual Weights getShift() const noexcept = 0;
+ virtual void setScale(Weights scale) noexcept = 0;
+ virtual Weights getScale() const noexcept = 0;
+ virtual void setPower(Weights power) noexcept = 0;
+ virtual Weights getPower() const noexcept = 0;
+ virtual int32_t getChannelAxis() const noexcept = 0;
+ virtual void setChannelAxis(int32_t channelAxis) noexcept = 0;
+};
+
+class VSoftMaxLayer : public VRoot
+{
+public:
+ virtual void setAxes(uint32_t axes) noexcept = 0;
+ virtual uint32_t getAxes() const noexcept = 0;
+};
+
+class VConcatenationLayer : public VRoot
+{
+public:
+ virtual void setAxis(int32_t axis) noexcept = 0;
+ virtual int32_t getAxis() const noexcept = 0;
+};
+
+class VDeconvolutionLayer : public VRoot
+{
+public:
+ virtual void setKernelSize(DimsHW kernelSize) noexcept = 0;
+ virtual DimsHW getKernelSize() const noexcept = 0;
+ virtual void setNbOutputMaps(int32_t nbOutputMaps) noexcept = 0;
+ virtual int32_t getNbOutputMaps() const noexcept = 0;
+ virtual void setStride(DimsHW stride) noexcept = 0;
+ virtual DimsHW getStride() const noexcept = 0;
+ virtual void setPadding(DimsHW padding) noexcept = 0;
+ virtual DimsHW getPadding() const noexcept = 0;
+ virtual void setNbGroups(int32_t nbGroups) noexcept = 0;
+ virtual int32_t getNbGroups() const noexcept = 0;
+ virtual void setKernelWeights(Weights weights) noexcept = 0;
+ virtual Weights getKernelWeights() const noexcept = 0;
+ virtual void setBiasWeights(Weights weights) noexcept = 0;
+ virtual Weights getBiasWeights() const noexcept = 0;
+ virtual void setPrePadding(Dims padding) noexcept = 0;
+ virtual Dims getPrePadding() const noexcept = 0;
+ virtual void setPostPadding(Dims padding) noexcept = 0;
+ virtual Dims getPostPadding() const noexcept = 0;
+ virtual void setPaddingMode(PaddingMode paddingMode) noexcept = 0;
+ virtual PaddingMode getPaddingMode() const noexcept = 0;
+ virtual void setKernelSizeNd(Dims kernelSize) noexcept = 0;
+ virtual Dims getKernelSizeNd() const noexcept = 0;
+ virtual void setStrideNd(Dims stride) noexcept = 0;
+ virtual Dims getStrideNd() const noexcept = 0;
+ virtual void setPaddingNd(Dims padding) noexcept = 0;
+ virtual Dims getPaddingNd() const noexcept = 0;
+ virtual void setDilationNd(Dims dilation) noexcept = 0;
+ virtual Dims getDilationNd() const noexcept = 0;
+};
+
+class VElementWiseLayer : public VRoot
+{
+public:
+ virtual void setOperation(ElementWiseOperation op) noexcept = 0;
+ virtual ElementWiseOperation getOperation() const noexcept = 0;
+};
+
+class VGatherLayer : public VRoot
+{
+public:
+ virtual void setGatherAxis(int32_t axis) noexcept = 0;
+ virtual int32_t getGatherAxis() const noexcept = 0;
+ virtual void setNbElementWiseDims(int32_t k) noexcept = 0;
+ virtual int32_t getNbElementWiseDims() const noexcept = 0;
+};
+
+class VRNNv2Layer : public VRoot
+{
+public:
+ virtual int32_t getLayerCount() const noexcept = 0;
+ virtual int32_t getHiddenSize() const noexcept = 0;
+ virtual int32_t getMaxSeqLength() const noexcept = 0;
+ virtual int32_t getDataLength() const noexcept = 0;
+ virtual void setSequenceLengths(ITensor& seqLengths) noexcept = 0;
+ virtual ITensor* getSequenceLengths() const noexcept = 0;
+ virtual void setOperation(RNNOperation op) noexcept = 0;
+ virtual RNNOperation getOperation() const noexcept = 0;
+ virtual void setInputMode(RNNInputMode op) noexcept = 0;
+ virtual RNNInputMode getInputMode() const noexcept = 0;
+ virtual void setDirection(RNNDirection op) noexcept = 0;
+ virtual RNNDirection getDirection() const noexcept = 0;
+ virtual void setWeightsForGate(int32_t layerIndex, RNNGateType gate, bool isW, Weights weights) noexcept = 0;
+ virtual Weights getWeightsForGate(int32_t layerIndex, RNNGateType gate, bool isW) const noexcept = 0;
+ virtual void setBiasForGate(int32_t layerIndex, RNNGateType gate, bool isW, Weights bias) noexcept = 0;
+ virtual Weights getBiasForGate(int32_t layerIndex, RNNGateType gate, bool isW) const noexcept = 0;
+ virtual void setHiddenState(ITensor& hidden) noexcept = 0;
+ virtual ITensor* getHiddenState() const noexcept = 0;
+ virtual void setCellState(ITensor& cell) noexcept = 0;
+ virtual ITensor* getCellState() const noexcept = 0;
+};
+
+class VPluginLayer : public VRoot
+{
+public:
+ virtual IPlugin& getPlugin() noexcept = 0;
+};
+
+class VPluginV2Layer : public VRoot
+{
+public:
+ virtual IPluginV2& getPlugin() noexcept = 0;
+};
+
+class VUnaryLayer : public VRoot
+{
+public:
+ virtual void setOperation(UnaryOperation op) noexcept = 0;
+ virtual UnaryOperation getOperation() const noexcept = 0;
+};
+
+class VReduceLayer : public VRoot
+{
+public:
+ virtual void setOperation(ReduceOperation op) noexcept = 0;
+ virtual ReduceOperation getOperation() const noexcept = 0;
+ virtual void setReduceAxes(uint32_t reduceAxes) noexcept = 0;
+ virtual uint32_t getReduceAxes() const noexcept = 0;
+ virtual void setKeepDimensions(bool keepDimensions) noexcept = 0;
+ virtual bool getKeepDimensions() const noexcept = 0;
+};
+
+class VPaddingLayer : public VRoot
+{
+public:
+ virtual void setPrePadding(DimsHW padding) noexcept = 0;
+ virtual DimsHW getPrePadding() const noexcept = 0;
+ virtual void setPostPadding(DimsHW padding) noexcept = 0;
+ virtual DimsHW getPostPadding() const noexcept = 0;
+ virtual void setPrePaddingNd(Dims padding) noexcept = 0;
+ virtual Dims getPrePaddingNd() const noexcept = 0;
+ virtual void setPostPaddingNd(Dims padding) noexcept = 0;
+ virtual Dims getPostPaddingNd() const noexcept = 0;
+};
+
+class VShuffleLayer : public VRoot
+{
+public:
+ virtual void setFirstTranspose(const Permutation& permutation) noexcept = 0;
+ virtual const Permutation& getFirstTranspose() const noexcept = 0;
+ virtual void setReshapeDimensions(Dims dimensions) noexcept = 0;
+ virtual Dims getReshapeDimensions() const noexcept = 0;
+ virtual void setSecondTranspose(const Permutation& permutation) noexcept = 0;
+ virtual const Permutation& getSecondTranspose() const noexcept = 0;
+ virtual void setZeroIsPlaceholder(bool zeroIsPlaceholder) = 0;
+ virtual bool getZeroIsPlaceholder() const = 0;
+};
+
+class VSliceLayer : public VRoot
+{
+public:
+ virtual void setStart(Dims start) noexcept = 0;
+ virtual Dims getStart() const noexcept = 0;
+ virtual void setSize(Dims size) noexcept = 0;
+ virtual Dims getSize() const noexcept = 0;
+ virtual void setStride(Dims stride) noexcept = 0;
+ virtual Dims getStride() const noexcept = 0;
+ virtual void setMode(SliceMode mode) noexcept = 0;
+ virtual SliceMode getMode() const noexcept = 0;
+};
+
+class VShapeLayer : public VRoot
+{
+public:
+};
+
+class VTopKLayer : public VRoot
+{
+public:
+ virtual void setOperation(TopKOperation op) noexcept = 0;
+ virtual TopKOperation getOperation() const noexcept = 0;
+ virtual void setK(int32_t k) noexcept = 0;
+ virtual int32_t getK() const noexcept = 0;
+ virtual void setReduceAxes(uint32_t reduceAxes) noexcept = 0;
+ virtual uint32_t getReduceAxes() const noexcept = 0;
+};
+
+class VMatrixMultiplyLayer : public VRoot
+{
+public:
+ virtual void setOperation(int32_t index, MatrixOperation op) noexcept = 0;
+ virtual MatrixOperation getOperation(int32_t index) const noexcept = 0;
+};
+
+class VRaggedSoftMaxLayer : public VRoot
+{
+public:
+};
+
+class VIdentityLayer : public VRoot
+{
+public:
+};
+
+class VConstantLayer : public VRoot
+{
+public:
+ virtual void setWeights(Weights weights) noexcept = 0;
+ virtual Weights getWeights() const noexcept = 0;
+ virtual void setDimensions(Dims dimensions) noexcept = 0;
+ virtual Dims getDimensions() const noexcept = 0;
+};
+
+class VParametricReLULayer : public VRoot
+{
+public:
+};
+
+class VResizeLayer : public VRoot
+{
+public:
+ virtual void setOutputDimensions(Dims dimensions) noexcept = 0;
+ virtual Dims getOutputDimensions() const noexcept = 0;
+ virtual void setScales(const float* scales, int32_t nbScales) noexcept = 0;
+ virtual int32_t getScales(int32_t size, float* scales) const noexcept = 0;
+ virtual void setResizeMode(ResizeMode resizeMode) noexcept = 0;
+ virtual ResizeMode getResizeMode() const noexcept = 0;
+ virtual void setAlignCorners(bool alignCorners) noexcept = 0;
+ virtual bool getAlignCorners() const noexcept = 0;
+ virtual void setCoordinateTransformation(ResizeCoordinateTransformation coordTransform) noexcept = 0;
+ virtual ResizeCoordinateTransformation getCoordinateTransformation() const noexcept = 0;
+ virtual void setSelectorForSinglePixel(ResizeSelector selector) noexcept = 0;
+ virtual ResizeSelector getSelectorForSinglePixel() const noexcept = 0;
+ virtual void setNearestRounding(ResizeRoundMode value) noexcept = 0;
+ virtual ResizeRoundMode getNearestRounding() const noexcept = 0;
+};
+
+class VLoopBoundaryLayer : public VRoot
+{
+public:
+ virtual ILoop* getLoop() const noexcept = 0;
+};
+
+class VRecurrenceLayer : public VRoot
+{
+public:
+};
+
+class VLoopOutputLayer : public VRoot
+{
+public:
+ virtual LoopOutput getLoopOutput() const noexcept = 0;
+ virtual void setAxis(int32_t axis) noexcept = 0;
+ virtual int32_t getAxis() const noexcept = 0;
+};
+
+class VTripLimitLayer : public VRoot
+{
+public:
+ virtual TripLimit getTripLimit() const noexcept = 0;
+};
+
+class VIteratorLayer : public VRoot
+{
+public:
+ virtual void setAxis(int32_t axis) noexcept = 0;
+ virtual int32_t getAxis() const noexcept = 0;
+ virtual void setReverse(bool reverse) noexcept = 0;
+ virtual bool getReverse() const noexcept = 0;
+};
+class VLoop : public VRoot
+{
+public:
+ virtual IRecurrenceLayer* addRecurrence(ITensor& initialValue) noexcept = 0;
+ virtual ITripLimitLayer* addTripLimit(ITensor& tensor, TripLimit limit) noexcept = 0;
+ virtual IIteratorLayer* addIterator(ITensor& tensor, int32_t axis = 0, bool reverse = false) noexcept = 0;
+ virtual ILoopOutputLayer* addLoopOutput(ITensor& tensor, LoopOutput outputKind, int32_t axis = 0) noexcept = 0;
+ virtual void setName(const char* name) noexcept = 0;
+ virtual const char* getName() const noexcept = 0;
+};
+class VSelectLayer : public VRoot
+{
+};
+
+class VFillLayer : public VRoot
+{
+public:
+ virtual void setDimensions(Dims dimensions) noexcept = 0;
+ virtual Dims getDimensions() const noexcept = 0;
+ virtual void setOperation(FillOperation op) noexcept = 0;
+ virtual FillOperation getOperation() const noexcept = 0;
+ virtual void setAlpha(double alpha) noexcept = 0;
+ virtual double getAlpha() const noexcept = 0;
+ virtual void setBeta(double beta) noexcept = 0;
+ virtual double getBeta() const noexcept = 0;
+};
+
+class VQuantizeLayer : public VRoot
+{
+public:
+ virtual int32_t getAxis() const noexcept = 0;
+ virtual void setAxis(int32_t axis) noexcept = 0;
+};
+
+class VDequantizeLayer : public VRoot
+{
+public:
+ virtual int32_t getAxis() const noexcept = 0;
+ virtual void setAxis(int32_t axis) noexcept = 0;
+};
+
+class VNetworkDefinition : public VRoot
+{
+public:
+ virtual ITensor* addInput(const char* name, DataType type, Dims dimensions) noexcept = 0;
+ virtual void markOutput(ITensor& tensor) noexcept = 0;
+ virtual IConvolutionLayer* addConvolution(
+ ITensor& input, int32_t nbOutputMaps, DimsHW kernelSize, Weights kernelWeights, Weights biasWeights) noexcept
+ = 0;
+ virtual IFullyConnectedLayer* addFullyConnected(
+ ITensor& input, int32_t nbOutputs, Weights kernelWeights, Weights biasWeights) noexcept
+ = 0;
+ virtual IActivationLayer* addActivation(ITensor& input, ActivationType type) noexcept = 0;
+ virtual IPoolingLayer* addPooling(ITensor& input, PoolingType type, DimsHW windowSize) noexcept = 0;
+ virtual ILRNLayer* addLRN(ITensor& input, int32_t window, float alpha, float beta, float k) noexcept = 0;
+ virtual IScaleLayer* addScale(ITensor& input, ScaleMode mode, Weights shift, Weights scale, Weights power) noexcept
+ = 0;
+ virtual ISoftMaxLayer* addSoftMax(ITensor& input) noexcept = 0;
+ virtual IConcatenationLayer* addConcatenation(ITensor* const* inputs, int32_t nbInputs) noexcept = 0;
+ virtual IDeconvolutionLayer* addDeconvolution(
+ ITensor& input, int32_t nbOutputMaps, DimsHW kernelSize, Weights kernelWeights, Weights biasWeights) noexcept
+ = 0;
+ virtual IElementWiseLayer* addElementWise(ITensor& input1, ITensor& input2, ElementWiseOperation op) noexcept = 0;
+ virtual IUnaryLayer* addUnary(ITensor& input, UnaryOperation operation) noexcept = 0;
+ virtual IPaddingLayer* addPadding(ITensor& input, DimsHW prePadding, DimsHW postPadding) noexcept = 0;
+ virtual IShuffleLayer* addShuffle(ITensor& input) noexcept = 0;
+ virtual int32_t getNbLayers() const noexcept = 0;
+ virtual ILayer* getLayer(int32_t index) const noexcept = 0;
+ virtual int32_t getNbInputs() const noexcept = 0;
+ virtual ITensor* getInput(int32_t index) const noexcept = 0;
+ virtual int32_t getNbOutputs() const noexcept = 0;
+ virtual ITensor* getOutput(int32_t index) const noexcept = 0;
+ virtual IReduceLayer* addReduce(
+ ITensor& input, ReduceOperation operation, uint32_t reduceAxes, bool keepDimensions) noexcept
+ = 0;
+ virtual ITopKLayer* addTopK(ITensor& input, TopKOperation op, int32_t k, uint32_t reduceAxes) noexcept = 0;
+ virtual IGatherLayer* addGather(ITensor& data, ITensor& indices, int32_t axis) noexcept = 0;
+ virtual IRaggedSoftMaxLayer* addRaggedSoftMax(ITensor& input, ITensor& bounds) noexcept = 0;
+ virtual IMatrixMultiplyLayer* addMatrixMultiply(
+ ITensor& input0, MatrixOperation op0, ITensor& input1, MatrixOperation op1) noexcept
+ = 0;
+ virtual IConstantLayer* addConstant(Dims dimensions, Weights weights) noexcept = 0;
+ virtual IRNNv2Layer* addRNNv2(
+ ITensor& input, int32_t layerCount, int32_t hiddenSize, int32_t maxSeqLen, RNNOperation op) noexcept
+ = 0;
+ virtual IIdentityLayer* addIdentity(ITensor& input) noexcept = 0;
+ virtual void removeTensor(ITensor& tensor) noexcept = 0;
+ virtual void unmarkOutput(ITensor& tensor) noexcept = 0;
+ virtual IPluginV2Layer* addPluginV2(ITensor* const* inputs, int32_t nbInputs, IPluginV2& plugin) noexcept = 0;
+ virtual ISliceLayer* addSlice(ITensor& input, Dims start, Dims size, Dims stride) noexcept = 0;
+ virtual void setName(const char* name) noexcept = 0;
+ virtual const char* getName() const noexcept = 0;
+ virtual IShapeLayer* addShape(ITensor& input) noexcept = 0;
+ virtual bool hasImplicitBatchDimension() const noexcept = 0;
+ virtual bool markOutputForShapes(ITensor& tensor) noexcept = 0;
+ virtual bool unmarkOutputForShapes(ITensor& tensor) noexcept = 0;
+ virtual IParametricReLULayer* addParametricReLU(ITensor& input, ITensor& slope) noexcept = 0;
+ virtual IConvolutionLayer* addConvolutionNd(
+ ITensor& input, int32_t nbOutputMaps, Dims kernelSize, Weights kernelWeights, Weights biasWeights) noexcept
+ = 0;
+ virtual IPoolingLayer* addPoolingNd(ITensor& input, PoolingType type, Dims windowSize) noexcept = 0;
+ virtual IDeconvolutionLayer* addDeconvolutionNd(
+ ITensor& input, int32_t nbOutputMaps, Dims kernelSize, Weights kernelWeights, Weights biasWeights) noexcept
+ = 0;
+ virtual IScaleLayer* addScaleNd(
+ ITensor& input, ScaleMode mode, Weights shift, Weights scale, Weights power, int32_t channelAxis) noexcept
+ = 0;
+ virtual IResizeLayer* addResize(ITensor& input) noexcept = 0;
+ virtual bool hasExplicitPrecision() const noexcept = 0;
+ virtual ILoop* addLoop() noexcept = 0;
+ virtual ISelectLayer* addSelect(ITensor& condition, ITensor& thenInput, ITensor& elseInput) noexcept = 0;
+ virtual IFillLayer* addFill(Dims dimensions, FillOperation op) noexcept = 0;
+ virtual IPaddingLayer* addPaddingNd(ITensor& input, Dims prePadding, Dims postPadding) noexcept = 0;
+ virtual bool setWeightsName(Weights weights, const char* name) noexcept = 0;
+ virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+ virtual IDequantizeLayer* addDequantize(ITensor& input, ITensor& scale) noexcept = 0;
+ virtual IQuantizeLayer* addQuantize(ITensor& input, ITensor& scale) noexcept = 0;
+};
+
+class VAlgorithmIOInfo : public VRoot
+{
+public:
+ virtual TensorFormat getTensorFormat() const noexcept = 0;
+ virtual DataType getDataType() const noexcept = 0;
+ virtual Dims getStrides() const noexcept = 0;
+};
+
+class VAlgorithmVariant : public VRoot
+{
+public:
+ virtual int64_t getImplementation() const noexcept = 0;
+ virtual int64_t getTactic() const noexcept = 0;
+};
+
+class VAlgorithmContext : public VRoot
+{
+public:
+ virtual const char* getName() const noexcept = 0;
+ virtual Dims getDimensions(int32_t index, OptProfileSelector select) const noexcept = 0;
+ virtual int32_t getNbInputs() const noexcept = 0;
+ virtual int32_t getNbOutputs() const noexcept = 0;
+};
+
+class VAlgorithm : public VRoot
+{
+public:
+ virtual const IAlgorithmIOInfo& getAlgorithmIOInfo(int32_t index) const noexcept = 0;
+ virtual const IAlgorithmVariant& getAlgorithmVariant() const noexcept = 0;
+ virtual float getTimingMSec() const noexcept = 0;
+ virtual std::size_t getWorkspaceSize() const noexcept = 0;
+ virtual const IAlgorithmIOInfo* getAlgorithmIOInfoByIndex(int32_t index) const noexcept = 0;
+};
+
+class VTimingCache : public VRoot
+{
+public:
+ virtual nvinfer1::IHostMemory* serialize() const noexcept = 0;
+ virtual bool combine(const ITimingCache& inputCache, bool ignoreMismatch) noexcept = 0;
+ virtual bool reset() noexcept = 0;
+};
+
+class VBuilderConfig : public VRoot
+{
+public:
+ virtual void setMinTimingIterations(int32_t minTiming) noexcept = 0;
+ virtual int32_t getMinTimingIterations() const noexcept = 0;
+ virtual void setAvgTimingIterations(int32_t avgTiming) noexcept = 0;
+ virtual int32_t getAvgTimingIterations() const noexcept = 0;
+ virtual void setEngineCapability(EngineCapability capability) noexcept = 0;
+ virtual EngineCapability getEngineCapability() const noexcept = 0;
+ virtual void setInt8Calibrator(IInt8Calibrator* calibrator) noexcept = 0;
+ virtual IInt8Calibrator* getInt8Calibrator() const noexcept = 0;
+ virtual void setMaxWorkspaceSize(std::size_t workspaceSize) noexcept = 0;
+ virtual std::size_t getMaxWorkspaceSize() const noexcept = 0;
+ virtual void setFlags(BuilderFlags builderFlags) noexcept = 0;
+ virtual BuilderFlags getFlags() const noexcept = 0;
+ virtual void clearFlag(BuilderFlag builderFlag) noexcept = 0;
+ virtual void setFlag(BuilderFlag builderFlag) noexcept = 0;
+ virtual bool getFlag(BuilderFlag builderFlag) const noexcept = 0;
+ virtual void setDeviceType(const ILayer* layer, DeviceType deviceType) noexcept = 0;
+ virtual DeviceType getDeviceType(const ILayer* layer) const noexcept = 0;
+ virtual bool isDeviceTypeSet(const ILayer* layer) const noexcept = 0;
+ virtual void resetDeviceType(const ILayer* layer) noexcept = 0;
+ virtual bool canRunOnDLA(const ILayer* layer) const noexcept = 0;
+ virtual void setDLACore(int32_t dlaCore) noexcept = 0;
+ virtual int32_t getDLACore() const noexcept = 0;
+ virtual void setDefaultDeviceType(DeviceType deviceType) noexcept = 0;
+ virtual DeviceType getDefaultDeviceType() const noexcept = 0;
+ virtual void reset() noexcept = 0;
+ virtual void setProfileStream(const cudaStream_t stream) noexcept = 0;
+ virtual cudaStream_t getProfileStream() const noexcept = 0;
+ virtual int32_t addOptimizationProfile(const IOptimizationProfile* profile) noexcept = 0;
+ virtual int32_t getNbOptimizationProfiles() const noexcept = 0;
+ virtual void setProfilingVerbosity(ProfilingVerbosity verbosity) noexcept = 0;
+ virtual ProfilingVerbosity getProfilingVerbosity() const noexcept = 0;
+ virtual void setAlgorithmSelector(IAlgorithmSelector* selector) noexcept = 0;
+ virtual IAlgorithmSelector* getAlgorithmSelector() const noexcept = 0;
+ virtual bool setCalibrationProfile(const IOptimizationProfile* profile) noexcept = 0;
+ virtual const IOptimizationProfile* getCalibrationProfile() noexcept = 0;
+ virtual void setQuantizationFlags(QuantizationFlags flags) noexcept = 0;
+ virtual QuantizationFlags getQuantizationFlags() const noexcept = 0;
+ virtual void clearQuantizationFlag(QuantizationFlag flag) noexcept = 0;
+ virtual void setQuantizationFlag(QuantizationFlag flag) noexcept = 0;
+ virtual bool getQuantizationFlag(QuantizationFlag flag) const noexcept = 0;
+ virtual bool setTacticSources(TacticSources tacticSources) noexcept = 0;
+ virtual TacticSources getTacticSources() const noexcept = 0;
+ virtual nvinfer1::ITimingCache* createTimingCache(const void* blob, std::size_t size) const noexcept = 0;
+ virtual bool setTimingCache(const ITimingCache& cache, bool ignoreMismatch) noexcept = 0;
+ virtual const nvinfer1::ITimingCache* getTimingCache() const noexcept = 0;
+};
+
+class VBuilder : public VRoot
+{
+public:
+ virtual void setMaxBatchSize(int32_t batchSize) noexcept = 0;
+ virtual int32_t getMaxBatchSize() const noexcept = 0;
+ virtual bool platformHasFastFp16() const noexcept = 0;
+ virtual bool platformHasFastInt8() const noexcept = 0;
+ virtual int32_t getMaxDLABatchSize() const noexcept = 0;
+ virtual int32_t getNbDLACores() const noexcept = 0;
+ virtual void setGpuAllocator(IGpuAllocator* allocator) noexcept = 0;
+ virtual nvinfer1::IBuilderConfig* createBuilderConfig() noexcept = 0;
+ virtual nvinfer1::ICudaEngine* buildEngineWithConfig(INetworkDefinition& network, IBuilderConfig& config) noexcept
+ = 0;
+ virtual nvinfer1::INetworkDefinition* createNetworkV2(NetworkDefinitionCreationFlags flags) noexcept = 0;
+ virtual nvinfer1::IOptimizationProfile* createOptimizationProfile() noexcept = 0;
+ virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+ virtual void reset() noexcept = 0;
+ virtual bool platformHasTf32() const noexcept = 0;
+ virtual nvinfer1::IHostMemory* buildSerializedNetwork(INetworkDefinition& network, IBuilderConfig& config) noexcept
+ = 0;
+ virtual bool isNetworkSupported(INetworkDefinition const& network, IBuilderConfig const& config) const noexcept
+ = 0;
+};
+
+} // namespace apiv
+} // namespace nvinfer1
+
+#endif // NV_INFER_RUNTIME_IMPL_H
diff --git a/include/NvInferLegacyDims.h b/include/NvInferLegacyDims.h
new file mode 100644
index 00000000..ee4c588c
--- /dev/null
+++ b/include/NvInferLegacyDims.h
@@ -0,0 +1,222 @@
+/*
+ * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef NV_INFER_LEGACY_DIMS_H
+#define NV_INFER_LEGACY_DIMS_H
+
+#include "NvInferRuntimeCommon.h"
+
+//!
+//! \file NvInferLegacyDims.h
+//!
+//! This file contains declarations of legacy dimensions types which use channel
+//! semantics in their names, and declarations on which those types rely.
+//!
+
+//!
+//! \namespace nvinfer1
+//!
+//! \brief The TensorRT API version 1 namespace.
+//!
+namespace nvinfer1
+{
+//!
+//! \class Dims2
+//! \brief Descriptor for two-dimensional data.
+//!
+class Dims2 : public Dims
+{
+public:
+ //!
+ //! \brief Construct an empty Dims2 object.
+ //!
+ Dims2()
+ {
+ nbDims = 2;
+ for (int32_t i = 0; i < MAX_DIMS; ++i)
+ {
+ d[i] = 0;
+ }
+ }
+
+ //!
+ //! \brief Construct a Dims2 from 2 elements.
+ //!
+ //! \param d0 The first element.
+ //! \param d1 The second element.
+ //!
+ Dims2(int32_t d0, int32_t d1)
+ {
+ nbDims = 2;
+ d[0] = d0;
+ d[1] = d1;
+ for (int32_t i = nbDims; i < MAX_DIMS; ++i)
+ {
+ d[i] = 0;
+ }
+ }
+};
+
+//!
+//! \class DimsHW
+//! \brief Descriptor for two-dimensional spatial data.
+//!
+class DimsHW : public Dims2
+{
+public:
+ //!
+ //! \brief Construct an empty DimsHW object.
+ //!
+ DimsHW()
+ : Dims2()
+ {
+ }
+
+ //!
+ //! \brief Construct a DimsHW given height and width.
+ //!
+ //! \param height the height of the data
+ //! \param width the width of the data
+ //!
+ DimsHW(int32_t height, int32_t width)
+ : Dims2(height, width)
+ {
+ }
+
+ //!
+ //! \brief Get the height.
+ //!
+ //! \return The height.
+ //!
+ int32_t& h()
+ {
+ return d[0];
+ }
+
+ //!
+ //! \brief Get the height.
+ //!
+ //! \return The height.
+ //!
+ int32_t h() const
+ {
+ return d[0];
+ }
+
+ //!
+ //! \brief Get the width.
+ //!
+ //! \return The width.
+ //!
+ int32_t& w()
+ {
+ return d[1];
+ }
+
+ //!
+ //! \brief Get the width.
+ //!
+ //! \return The width.
+ //!
+ int32_t w() const
+ {
+ return d[1];
+ }
+};
+
+//!
+//! \class Dims3
+//! \brief Descriptor for three-dimensional data.
+//!
+class Dims3 : public Dims
+{
+public:
+ //!
+ //! \brief Construct an empty Dims3 object.
+ //!
+ Dims3()
+ {
+ nbDims = 3;
+ for (int32_t i = 0; i < MAX_DIMS; ++i)
+ {
+ d[i] = 0;
+ }
+ }
+
+ //!
+ //! \brief Construct a Dims3 from 3 elements.
+ //!
+ //! \param d0 The first element.
+ //! \param d1 The second element.
+ //! \param d2 The third element.
+ //!
+ Dims3(int32_t d0, int32_t d1, int32_t d2)
+ {
+ nbDims = 3;
+ d[0] = d0;
+ d[1] = d1;
+ d[2] = d2;
+ for (int32_t i = nbDims; i < MAX_DIMS; ++i)
+ {
+ d[i] = 0;
+ }
+ }
+};
+
+//!
+//! \class Dims4
+//! \brief Descriptor for four-dimensional data.
+//!
+class Dims4 : public Dims
+{
+public:
+ //!
+ //! \brief Construct an empty Dims4 object.
+ //!
+ Dims4()
+ {
+ nbDims = 4;
+ for (int32_t i = 0; i < MAX_DIMS; ++i)
+ {
+ d[i] = 0;
+ }
+ }
+
+ //!
+ //! \brief Construct a Dims4 from 4 elements.
+ //!
+ //! \param d0 The first element.
+ //! \param d1 The second element.
+ //! \param d2 The third element.
+ //! \param d3 The fourth element.
+ //!
+ Dims4(int32_t d0, int32_t d1, int32_t d2, int32_t d3)
+ {
+ nbDims = 4;
+ d[0] = d0;
+ d[1] = d1;
+ d[2] = d2;
+ d[3] = d3;
+ for (int32_t i = nbDims; i < MAX_DIMS; ++i)
+ {
+ d[i] = 0;
+ }
+ }
+};
+
+} // namespace nvinfer1
+
+#endif // NV_INFER_LEGCY_DIMS_H
diff --git a/include/NvInferPlugin.h b/include/NvInferPlugin.h
index f06d3caa..639a552d 100644
--- a/include/NvInferPlugin.h
+++ b/include/NvInferPlugin.h
@@ -30,8 +30,9 @@ extern "C"
//!
//! \brief Initialize and register all the existing TensorRT plugins to the Plugin Registry with an optional
//! namespace. The plugin library author should ensure that this function name is unique to the library. This
- //! function should be called once before accessing the Plugin Registry. \param logger Logger object to print plugin
- //! registration information \param libNamespace Namespace used to register all the plugins in this library
+ //! function should be called once before accessing the Plugin Registry.
+ //! \param logger Logger object to print plugin registration information
+ //! \param libNamespace Namespace used to register all the plugins in this library
//!
TENSORRTAPI bool initLibNvInferPlugins(void* logger, const char* libNamespace);
diff --git a/include/NvInferPluginUtils.h b/include/NvInferPluginUtils.h
index 1f22522b..f9c94555 100644
--- a/include/NvInferPluginUtils.h
+++ b/include/NvInferPluginUtils.h
@@ -16,6 +16,7 @@
#ifndef NV_INFER_PLUGIN_UTILS_H
#define NV_INFER_PLUGIN_UTILS_H
+
#include "NvInferRuntimeCommon.h"
//!
@@ -27,34 +28,6 @@
namespace nvinfer1
{
-//!
-//! \enum PluginType
-//!
-//! \brief The type values for the various plugins.
-//!
-//! \see INvPlugin::getPluginType()
-//!
-enum class PluginType : int32_t
-{
- kFASTERRCNN = 0, //!< FasterRCNN fused plugin (RPN + ROI pooling).
- kNORMALIZE = 1, //!< Normalize plugin.
- kPERMUTE = 2, //!< Permute plugin.
- kPRIORBOX = 3, //!< PriorBox plugin.
- kSSDDETECTIONOUTPUT = 4, //!< SSD DetectionOutput plugin.
- kCONCAT = 5, //!< Concat plugin.
- kPRELU = 6, //!< YOLO PReLU Plugin.
- kYOLOREORG = 7, //!< YOLO Reorg Plugin.
- kYOLOREGION = 8, //!< YOLO Region Plugin.
- kANCHORGENERATOR = 9, //!< SSD Grid Anchor Generator.
-};
-
-//! Maximum number of elements in PluginType enum. \see PluginType
-template <>
-constexpr inline int32_t EnumMax()
-{
- return 10;
-}
-
namespace plugin
{
@@ -75,7 +48,7 @@ typedef struct
//! \param minSize Minimum box size in pixels. Can not be nullptr.
//! \param maxSize Maximum box size in pixels. Can be nullptr.
//! \param aspectRatios Aspect ratios of the boxes. Can be nullptr.
-//! \param numMinSize Number of element in minSize. Must be larger than 0.
+//! \param numMinSize Number of elements in minSize. Must be larger than 0.
//! \param numMaxSize Number of elements in maxSize. Can be 0 or same as numMinSize.
//! \param numAspectRatios Number of elements in aspectRatios. Can be 0.
//! \param flip If true, will flip each aspect ratio. For example, if there is an aspect ratio "r", the aspect ratio
@@ -106,8 +79,10 @@ struct PriorBoxParameters
//! \param poolingH Height of the output in pixels after ROI pooling on feature map.
//! \param poolingW Width of the output in pixels after ROI pooling on feature map.
//! \param featureStride Feature stride; ratio of input image size to feature map size. Assuming that max pooling layers
-//! in neural network use square filters. \param preNmsTop Number of proposals to keep before applying NMS. \param
-//! nmsMaxOut Number of remaining proposals after applying NMS. \param anchorsRatioCount Number of anchor box ratios.
+//! in the neural network use square filters.
+//! \param preNmsTop Number of proposals to keep before applying NMS.
+//! \param nmsMaxOut Number of remaining proposals after applying NMS.
+//! \param anchorsRatioCount Number of anchor box ratios.
//! \param anchorsScaleCount Number of anchor box scales.
//! \param iouThreshold IoU (Intersection over Union) threshold used for the NMS step.
//! \param minBoxSize Minimum allowed bounding box size before scaling, used for anchor box calculation.
@@ -162,7 +137,7 @@ enum class CodeTypeSSD : int32_t
//!
//! \brief The DetectionOutput plugin layer generates the detection output based on location and confidence predictions by doing non maximum suppression.
-//! This plugin first decodes the bounding boxes based on the anchors generated. It then performs non_max_suppression on the decoded bouding boxes.
+//! This plugin first decodes the bounding boxes based on the anchors generated. It then performs non_max_suppression on the decoded bounding boxes.
//! DetectionOutputParameters defines a set of parameters for creating the DetectionOutput plugin layer.
//! It contains:
//! \param shareLocation If true, bounding box are shared among different classes.
@@ -192,22 +167,9 @@ struct DetectionOutputParameters
};
//!
-//! \brief The Region plugin layer performs region proposal calculation: generate 5 bounding boxes per cell (for
-//! yolo9000, generate 3 bounding boxes per cell).
+//! \brief When performing yolo9000, softmaxTree is helping to do softmax on confidence scores, for element to get the precise classification through word-tree structured classification definition.
//!
-//! For each box, calculating its probablities of objects detections from 80 pre-defined classifications (yolo9000
-//! has 9418 pre-defined classifications, and these 9418 items are organized as work-tree structure).
-//! RegionParameters defines a set of parameters for creating the Region plugin layer.
-//!
-//! \param num Number of predicted bounding box for each grid cell.
-//! \param coords Number of coordinates for a bounding box.
-//! \param classes Number of classfications to be predicted.
-//! \param softmaxTree When performing yolo9000, softmaxTree is helping to do softmax on confidence scores, for element
-//! to get the precise classfication through word-tree structured classfication definition.
-//!
-//! \deprecated. This plugin is superseded by createRegionPlugin and will be removed in TensorRT 8.0.
-//!
-TRT_DEPRECATED typedef struct
+struct softmaxTree
{
int32_t* leaf;
int32_t n;
@@ -219,9 +181,19 @@ TRT_DEPRECATED typedef struct
int32_t groups;
int32_t* groupSize;
int32_t* groupOffset;
-} softmaxTree; // softmax tree
+};
-struct TRT_DEPRECATED RegionParameters
+//!
+//! \brief The Region plugin layer performs region proposal calculation: generate 5 bounding boxes per cell (for yolo9000, generate 3 bounding boxes per cell).
+//! For each box, calculating its probablities of objects detections from 80 pre-defined classifications (yolo9000 has 9418 pre-defined classifications,
+//! and these 9418 items are organized as work-tree structure).
+//! RegionParameters defines a set of parameters for creating the Region plugin layer.
+//! \param num Number of predicted bounding box for each grid cell.
+//! \param coords Number of coordinates for a bounding box.
+//! \param classes Number of classifications to be predicted.
+//! \param smTree Helping structure to do softmax on confidence scores.
+//!
+struct RegionParameters
{
int32_t num;
int32_t coords;
@@ -254,6 +226,7 @@ struct NMSParameters
bool isNormalized;
};
-} // end plugin namespace
-} // end nvinfer1 namespace
-#endif
+} // namespace plugin
+} // namespace nvinfer1
+
+#endif // NV_INFER_PLUGIN_UTILS_H
diff --git a/include/NvInferRuntime.h b/include/NvInferRuntime.h
index 644f6388..6709a109 100644
--- a/include/NvInferRuntime.h
+++ b/include/NvInferRuntime.h
@@ -23,44 +23,86 @@
//! This is the top-level API file for TensorRT extended runtime library.
//!
+#include "NvInferImpl.h"
#include "NvInferRuntimeCommon.h"
namespace nvinfer1
{
class IExecutionContext; //!< Forward declaration of IExecutionContext for use by other interfaces.
-class ICudaEngine; //!< Forward declaration of ICudaENgine for use by other interfaces.
-class IPluginFactory; //!< Forward declaration of IPluginFactory for use by other interfaces.
+class ICudaEngine; //!< Forward declaration of ICudaEngine for use by other interfaces.
+class IPluginFactory; //!< Forward declaration of IPluginFactory for use by other interfaces.
+
+//!
+//! \class INoCopy
+//!
+//! \brief Base class for all TensorRT interfaces that are implemented by the TensorRT libraries
+//!
+//! Objects of such classes are not movable or copyable, and should only be manipulated
+//! via pointers.
+//!
+
+class INoCopy
+{
+protected:
+ INoCopy() = default;
+ virtual ~INoCopy() = default;
+ INoCopy(const INoCopy& other) = delete;
+ INoCopy& operator=(const INoCopy& other) = delete;
+ INoCopy(INoCopy&& other) = delete;
+ INoCopy& operator=(INoCopy&& other) = delete;
+};
//!
//! \enum EngineCapability
//!
//! \brief List of supported engine capability flows.
//!
-//! The EngineCapability determines the restrictions of a network during build time for what can be executed
-//! at runtime. EngineCapability::kDEFAULT does not provide any restrictions on functionality and the
-//! resulting serialized engine can be executed with TensorRT's standard runtime APIs in the nvinfer1 namespace.
-//! EngineCapabiltiy::kSAFE_GPU provides a restricted subset of network operations that are safety certified and
-//! the resulting serialized engine can be executed with TensorRT's safe runtime APIs in the nvinfer1::safe namespace.
-//! EngineCapability::kSAFE_DLA provides a restricted subset of network operations that are DLA compatible and
-//! the resulting serialized engine can be executed using NvMediaDLA's runtime APIs. See sampleNvmedia for an
-//! example of integrating NvMediaDLA APIs with TensorRT APIs.
+//! \details The EngineCapability determines the restrictions of a network during build time and what runtime
+//! it targets. When BuilderFlag::kSAFETY_SCOPE is not set (by default), EngineCapability::kSTANDARD does not provide
+//! any restrictions on functionality and the resulting serialized engine can be executed with TensorRT's standard
+//! runtime APIs in the nvinfer1 namespace. EngineCapability::kSAFETY provides a restricted subset of network
+//! operations that are safety certified and the resulting serialized engine can be executed with TensorRT's safe
+//! runtime APIs in the nvinfer1::safe namespace. EngineCapability::kDLA_STANDALONE provides a restricted subset of
+//! network operations that are DLA compatible and the resulting serialized engine can be executed using standalone
+//! DLA runtime APIs. See sampleNvmedia for an example of integrating NvMediaDLA APIs with TensorRT APIs.
//!
+
enum class EngineCapability : int32_t
{
- kDEFAULT = 0, //!< Full capability, TensorRT mode without any restrictions using TensorRT nvinfer1 APIs.
- kSAFE_GPU = 1, //!< Safety restricted capability, TensorRT flow that can only run on GPU devices via TensorRT
- //!< nvinfer1::safe APIs.
- kSAFE_DLA = 2, //!< Safety restricted capability, TensorRT flow that can only run on DLA devices via
- //!< NvMediaDLA APIs.
+ //!
+ //! Standard: TensorRT flow without targeting the safety runtime.
+ //! This flow supports both DeviceType::kGPU and DeviceType::kDLA.
+ //!
+ kSTANDARD = 0,
+ kDEFAULT TRT_DEPRECATED_ENUM = kSTANDARD,
+
+ //!
+ //! Safety: TensorRT flow with restrictions targeting the safety runtime.
+ //! See safety documentation for list of supported layers and formats.
+ //! This flow supports only DeviceType::kGPU.
+ //!
+ kSAFETY = 1,
+ kSAFE_GPU TRT_DEPRECATED_ENUM = kSAFETY,
+
+ //!
+ //! DLA Standalone: TensorRT flow with restrictions targeting external, to TensorRT, DLA runtimes.
+ //! See DLA documentation for list of supported layers and formats.
+ //! This flow supports only DeviceType::kDLA.
+ //!
+ kDLA_STANDALONE = 2,
+ kSAFE_DLA TRT_DEPRECATED_ENUM = kDLA_STANDALONE,
};
+namespace impl
+{
//! Maximum number of elements in EngineCapability enum. \see EngineCapability
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 3;
-}
+ static constexpr int32_t kVALUE = 3;
+};
+} // namespace impl
//!
//! \class Weights
@@ -92,194 +134,42 @@ public:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class IHostMemory
+class IHostMemory : public INoCopy
{
public:
- virtual void* data() const noexcept = 0; //!< A pointer to the raw data that is owned by the library.
- virtual std::size_t size() const noexcept = 0; //!< The size in bytes of the data that was allocated.
- virtual DataType type() const noexcept = 0; //!< The type of the memory that was allocated.
- virtual void destroy() noexcept = 0; //!< Destroy the allocated memory.
-protected:
- virtual ~IHostMemory() {}
-};
+ virtual ~IHostMemory() noexcept = default;
-//! \class IPlugin
-//!
-//! \brief Plugin class for user-implemented layers.
-//!
-//! Plugins are a mechanism for applications to implement custom layers. Each plugin is owned by the application, and its lifetime
-//! must span any use of it by TensorRT
-//!
-class IPlugin
-{
-public:
- //!
- //! \brief Get the number of outputs from the layer.
- //!
- //! \return The number of outputs.
- //!
- //! This function is called by the implementations of INetworkDefinition and IBuilder. In particular, it is called
- //! prior to any call to initialize().
- //!
- virtual int32_t getNbOutputs() const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Get the dimension of an output tensor.
- //!
- //! \param index The index of the output tensor.
- //! \param inputs The input tensors.
- //! \param nbInputDims The number of input tensors.
- //!
- //! This function is called by the implementations of INetworkDefinition and IBuilder. In particular, it is called
- //! prior to any call to initialize().
- //!
- virtual Dims getOutputDimensions(int32_t index, const Dims* inputs, int32_t nbInputDims) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Configure the layer.
- //!
- //! This function is called by the builder prior to initialize(). It provides an opportunity for the layer to make
- //! algorithm choices on the basis of its weights, dimensions, and maximum batch size. The type is assumed to be
- //! FP32 and format NCHW.
- //!
- //! \param inputDims The input tensor dimensions.
- //! \param nbInputs The number of inputs.
- //! \param outputDims The output tensor dimensions.
- //! \param nbOutputs The number of outputs.
- //! \param maxBatchSize The maximum batch size.
- //!
- //! The dimensions passed here do not include the outermost batch size (i.e. for 2-D image networks, they will be
- //! 3-dimensional CHW dimensions).
- //!
- //! This method is not called for PluginExt classes, configureWithFormat is called instead.
- //!
- virtual void configure(const Dims* inputDims, int32_t nbInputs, const Dims* outputDims, int32_t nbOutputs,
- int32_t maxBatchSize) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Initialize the layer for execution. This is called when the engine is created.
- //!
- //! \return 0 for success, else non-zero (which will cause engine termination).
- //!
- virtual int32_t initialize() TRTNOEXCEPT = 0;
-
- //!
- //! \brief Release resources acquired during plugin layer initialization. This is called when the engine is
- //! destroyed. \see initialize()
- //!
- virtual void terminate() TRTNOEXCEPT = 0;
-
- //!
- //! \brief Find the workspace size required by the layer.
- //!
- //! This function is called during engine startup, after initialize(). The workspace size returned should be
- //! sufficient for any batch size up to the maximum.
- //!
- //! \return The workspace size.
- //!
- virtual size_t getWorkspaceSize(int32_t maxBatchSize) const TRTNOEXCEPT = 0;
-
- //!
- //! \brief Execute the layer.
- //!
- //! \param batchSize The number of inputs in the batch.
- //! \param inputs The memory for the input tensors.
- //! \param outputs The memory for the output tensors.
- //! \param workspace Workspace for execution.
- //! \param stream The stream in which to execute the kernels.
- //!
- //! \return 0 for success, else non-zero (which will cause engine termination).
- //!
- virtual int32_t enqueue(int32_t batchSize, const void* const* inputs, void** outputs, void* workspace,
- cudaStream_t stream) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Find the size of the serialization buffer required.
- //!
- //! \return The size of the serialization buffer.
- //!
- virtual size_t getSerializationSize() TRTNOEXCEPT = 0;
-
- //!
- //! \brief Serialize the layer.
- //!
- //! \param buffer A pointer to a buffer of size at least that returned by getSerializationSize().
- //!
- //! \see getSerializationSize()
- //!
- virtual void serialize(void* buffer) TRTNOEXCEPT = 0;
-
- virtual ~IPlugin() {}
-};
-
-//!
-//! \class IPluginExt
-//!
-//! \brief Plugin class for user-implemented layers.
-//!
-//! Plugins are a mechanism for applications to implement custom layers. Each plugin is owned by the application, and its lifetime
-//! must span any use of it by TensorRT.
-//!
-class IPluginExt : public IPlugin
-{
-public:
- //!
- //! \brief Return the API version with which this plugin was built.
- //!
- //! Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with
- //! plugins.
- //!
- virtual int32_t getTensorRTVersion() const TRTNOEXCEPT
+ //! A pointer to the raw data that is owned by the library.
+ void* data() const noexcept
{
- return NV_TENSORRT_VERSION;
+ return mImpl->data();
}
- //!
- //! \brief Check format support.
- //!
- //! \param type DataType requested.
- //! \param format PluginFormat requested.
- //! \return true if the plugin supports the type-format combination.
- //!
- //! This function is called by the implementations of INetworkDefinition, IBuilder, and ICudaEngine.
- //! In particular, it is called when creating an engine and when deserializing an engine.
- //!
- //! \warning DataType:kBOOL not supported.
- //!
- virtual bool supportsFormat(DataType type, PluginFormat format) const TRTNOEXCEPT = 0;
+ //! The size in bytes of the data that was allocated.
+ std::size_t size() const noexcept
+ {
+ return mImpl->size();
+ }
+ //! The type of the memory that was allocated.
+ DataType type() const noexcept
+ {
+ return mImpl->type();
+ }
//!
- //! \brief Configure the layer.
+ //! Destroy the allocated memory.
//!
- //! This function is called by the builder prior to initialize(). It provides an opportunity for the layer to make
- //! algorithm choices on the basis of its weights, dimensions, and maximum batch size.
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
//!
- //! \param inputDims The input tensor dimensions.
- //! \param nbInputs The number of inputs.
- //! \param outputDims The output tensor dimensions.
- //! \param nbOutputs The number of outputs.
- //! \param type The data type selected for the engine.
- //! \param format The format selected for the engine.
- //! \param maxBatchSize The maximum batch size.
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
//!
- //! The dimensions passed here do not include the outermost batch size (i.e. for 2-D image networks, they will be
- //! 3-dimensional CHW dimensions).
- //!
- //! \warning DataType:kBOOL not supported.
- //!
- virtual void configureWithFormat(const Dims* inputDims, int32_t nbInputs, const Dims* outputDims, int32_t nbOutputs,
- DataType type, PluginFormat format, int32_t maxBatchSize) TRTNOEXCEPT = 0;
-
- virtual ~IPluginExt() {}
+ TRT_DEPRECATED void destroy() noexcept
+ {
+ delete this;
+ }
protected:
- //!
- //! \brief Derived classes should not implement this. In a C++11 API it would be override final.
- //!
- void configure(const Dims* /*inputDims*/, int32_t /*nbInputs*/, const Dims* /*outputDims*/, int32_t /*nbOutputs*/,
- int32_t /*maxBatchSize*/) _TENSORRT_FINAL TRTNOEXCEPT
- {
- }
+ apiv::VHostMemory* mImpl;
};
//!
@@ -307,11 +197,31 @@ enum class DimensionOperation : int32_t
//! Maximum number of elements in DimensionOperation enum. \see DimensionOperation
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 9;
}
+//!
+//! \enum TensorLocation
+//! \brief The location for tensor data storage, device or host.
+//!
+enum class TensorLocation : int32_t
+{
+ kDEVICE = 0, //!< Data stored on device.
+ kHOST = 1, //!< Data stored on host.
+};
+
+namespace impl
+{
+//! Maximum number of elements in TensorLocation enum. \see TensorLocation
+template <>
+struct EnumMaxImpl
+{
+ static constexpr int32_t kVALUE = 2;
+};
+} // namespace impl
+
//!
//! \class IDimensionExpr
//!
@@ -320,20 +230,29 @@ constexpr inline int32_t EnumMax()
//! in overrides of IPluginV2DynamicExt::getOutputDimensions to define output
//! dimensions in terms of input dimensions.
//!
+//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
+//!
//! \see DimensionOperation, IPluginV2DynamicExt::getOutputDimensions
//!
-class IDimensionExpr
+class IDimensionExpr : public INoCopy
{
public:
//! Return true if expression is a build-time constant.
- virtual bool isConstant() const = 0;
+ bool isConstant() const noexcept
+ {
+ return mImpl->isConstant();
+ }
//! If isConstant(), returns value of the constant.
//! If !isConstant(), return std::numeric_limits::min().
- virtual int32_t getConstantValue() const = 0;
+ int32_t getConstantValue() const noexcept
+ {
+ return mImpl->getConstantValue();
+ }
protected:
- virtual ~IDimensionExpr() {}
+ apiv::VDimensionExpr* mImpl;
+ virtual ~IDimensionExpr() noexcept = default;
};
//!
@@ -353,18 +272,26 @@ protected:
//!
//! \see IDimensionExpr
//!
-class IExprBuilder
+class IExprBuilder : public INoCopy
{
public:
//! Return pointer to IDimensionExp for given value.
- virtual const IDimensionExpr* constant(int32_t value) = 0;
+ const IDimensionExpr* constant(int32_t value) noexcept
+ {
+ return mImpl->constant(value);
+ }
//! Return pointer to IDimensionExp that represents the given operation applied to first and second.
//! Returns nullptr if op is not a valid DimensionOperation.
- virtual const IDimensionExpr* operation(DimensionOperation op, const IDimensionExpr& first, const IDimensionExpr& second) = 0;
+ const IDimensionExpr* operation(
+ DimensionOperation op, const IDimensionExpr& first, const IDimensionExpr& second) noexcept
+ {
+ return mImpl->operation(op, first, second);
+ }
protected:
- virtual ~IExprBuilder() {}
+ apiv::VExprBuilder* mImpl;
+ virtual ~IExprBuilder() noexcept = default;
};
//!
@@ -403,12 +330,12 @@ struct DynamicPluginTensorDesc
//!
//! Clients should override the public methods, including the following inherited methods:
//!
-//! virtual int32_t getNbOutputs() const TRTNOEXCEPT = 0;
+//! virtual int32_t getNbOutputs() const noexcept = 0;
//! virtual nvinfer1::DataType getOutputDataType(int32_t index, const nvinfer1::DataType* inputTypes, int32_t
-//! nbInputs) const TRTNOEXCEPT = 0; virtual size_t getSerializationSize() const TRTNOEXCEPT = 0; virtual void
-//! serialize(void* buffer) const TRTNOEXCEPT = 0; virtual void destroy() TRTNOEXCEPT = 0; virtual void
-//! setPluginNamespace(const char* pluginNamespace) TRTNOEXCEPT = 0; virtual const char* getPluginNamespace() const
-//! TRTNOEXCEPT = 0;
+//! nbInputs) const noexcept = 0; virtual size_t getSerializationSize() const noexcept = 0; virtual void
+//! serialize(void* buffer) const noexcept = 0; virtual void destroy() noexcept = 0; virtual void
+//! setPluginNamespace(const char* pluginNamespace) noexcept = 0; virtual const char* getPluginNamespace() const
+//! noexcept = 0;
//!
//! For getOutputDataType, the inputTypes will always be DataType::kFLOAT or DataType::kINT32,
//! and the returned type is canonicalized to DataType::kFLOAT if it is DataType::kHALF or DataType:kINT8.
@@ -417,14 +344,14 @@ struct DynamicPluginTensorDesc
class IPluginV2DynamicExt : public nvinfer1::IPluginV2Ext
{
public:
- IPluginV2DynamicExt* clone() const _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ IPluginV2DynamicExt* clone() const noexcept override = 0;
//!
//! \brief Get expressions for computing dimensions of an output tensor from dimensions of the input tensors.
//!
//! \param outputIndex The index of the output tensor
//! \param inputs Expressions for dimensions of the input tensors
- //! \param nbInputDims The number of input tensors
+ //! \param nbInputs The number of input tensors
//! \param exprBuilder Object for generating new expressions
//!
//! This function is called by the implementations of IBuilder during analysis of the network.
@@ -444,7 +371,7 @@ public:
//! return output;
//!
virtual DimsExprs getOutputDimensions(
- int32_t outputIndex, const DimsExprs* inputs, int32_t nbInputs, IExprBuilder& exprBuilder)
+ int32_t outputIndex, const DimsExprs* inputs, int32_t nbInputs, IExprBuilder& exprBuilder) noexcept
= 0;
//!
@@ -485,15 +412,41 @@ public:
//! Warning: TensorRT will stop asking for formats once it finds kFORMAT_COMBINATION_LIMIT on combinations.
//!
virtual bool supportsFormatCombination(
- int32_t pos, const PluginTensorDesc* inOut, int32_t nbInputs, int32_t nbOutputs) TRTNOEXCEPT = 0;
+ int32_t pos, const PluginTensorDesc* inOut, int32_t nbInputs, int32_t nbOutputs) noexcept
+ = 0;
//!
- //! \brief Configure the layer.
+ //! \brief Configure the plugin.
//!
- //! This function is called by the builder prior to initialize(). It provides an opportunity for the layer to make
- //! algorithm choices on the basis of bounds on the input and output tensors, and the target value.
+ //! configurePlugin() can be called multiple times in both the build and execution phases. The build phase happens
+ //! before initialize() is called and only occurs during creation of an engine by IBuilder. The execution phase
+ //! happens after initialize() is called and occurs during both creation of an engine by IBuilder and execution
+ //! of an engine by IExecutionContext.
//!
- //! This function is also called once when the resource requirements are changed based on the optimization profiles.
+ //! Build phase:
+ //! IPluginV2DynamicExt->configurePlugin is called when a plugin is being prepared for profiling but not for any
+ //! specific input size. This provides an opportunity for the plugin to make algorithmic choices on the basis of
+ //! input and output formats, along with the bound of possible dimensions. The min and max value of the
+ //! DynamicPluginTensorDesc correspond to the kMIN and kMAX value of the current profile that the plugin is being
+ //! profiled for, with the desc.dims field corresponding to the dimensions of plugin specified at network creation.
+ //! Wildcard dimensions will exist during this phase in the desc.dims field.
+ //!
+ //! Execution phase:
+ //! IPluginV2DynamicExt->configurePlugin is called when a plugin is being prepared for executing the plugin for a
+ //! specific dimensions. This provides an opportunity for the plugin to change algorithmic choices based on the
+ //! explicit input dimensions stored in desc.dims field.
+ //! * IBuilder will call this function once per profile, with desc.dims resolved to the values specified by the
+ //! kOPT
+ //! field of the current profile. Wildcard dimensions will not exist during this phase.
+ //! * IExecutionContext will call this during the next subsequent instance enqueue[V2]() or execute[V2]() if:
+ //! - The batch size is changed from previous call of execute()/enqueue() if hasImplicitBatchDimension() returns
+ //! true.
+ //! - The optimization profile is changed via setOptimizationProfile() or setOptimizationProfileAsync().
+ //! - An input shape binding is changed via setInputShapeBinding().
+ //! - An input execution binding is changed via setBindingDimensions().
+ //! \warning The execution phase is timing critical during IExecutionContext but is not part of the timing loop when
+ //! called from IBuilder. Performance bottlenecks of configurePlugin won't show up during engine building but will
+ //! be visible during execution after calling functions that trigger layer resource updates.
//!
//! \param in The input tensors attributes that are used for configuration.
//! \param nbInputs Number of input tensors.
@@ -501,7 +454,8 @@ public:
//! \param nbOutputs Number of output tensors.
//!
virtual void configurePlugin(const DynamicPluginTensorDesc* in, int32_t nbInputs,
- const DynamicPluginTensorDesc* out, int32_t nbOutputs) TRTNOEXCEPT = 0;
+ const DynamicPluginTensorDesc* out, int32_t nbOutputs) noexcept
+ = 0;
//!
//! \brief Find the workspace size required by the layer.
@@ -513,7 +467,8 @@ public:
//! \return The workspace size.
//!
virtual size_t getWorkspaceSize(const PluginTensorDesc* inputs, int32_t nbInputs, const PluginTensorDesc* outputs,
- int32_t nbOutputs) const TRTNOEXCEPT = 0;
+ int32_t nbOutputs) const noexcept
+ = 0;
//!
//! \brief Execute the layer.
@@ -528,125 +483,58 @@ public:
//! \return 0 for success, else non-zero (which will cause engine termination).
//!
virtual int32_t enqueue(const PluginTensorDesc* inputDesc, const PluginTensorDesc* outputDesc,
- const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) TRTNOEXCEPT = 0;
+ const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept
+ = 0;
protected:
- int32_t getTensorRTVersion() const _TENSORRT_OVERRIDE TRTNOEXCEPT
+ //!
+ //! \brief Return the API version with which this plugin was built. The
+ //! upper byte reserved by TensorRT and is used to differentiate this from IPluginV2.
+ //!
+ //! Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with
+ //! plugins.
+ //!
+ int32_t getTensorRTVersion() const noexcept override
{
return (static_cast(PluginVersion::kV2_DYNAMICEXT) << 24 | (NV_TENSORRT_VERSION & 0xFFFFFF));
}
- virtual ~IPluginV2DynamicExt() {}
+ virtual ~IPluginV2DynamicExt() noexcept {}
- // Rest of the methods below are obsolete inherited methods, and marked final when using a C++11 compiler.
- // Derived classes should not override them.
+private:
+ // Following are obsolete base class methods, and must not be implemented or used.
- //!
- //! \brief Derived classes should not implement this. In a C++11 API it would be override final.
- //!
- //! Instead, derived classes should override the overload of getOutputDimensions that returns DimsExprs.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- Dims getOutputDimensions(
- int32_t /*index*/, const Dims* /*inputs*/, int32_t /*nbInputDims*/) _TENSORRT_FINAL TRTNOEXCEPT
+ void configurePlugin(Dims const*, int32_t, Dims const*, int32_t, DataType const*, DataType const*, bool const*,
+ bool const*, PluginFormat, int32_t) noexcept override final
{
- return Dims{-1, {}, {}};
}
- //!
- //! \brief Derived classes should not implement this. In a C++11 API it would be override final.
- //!
- //! This method is not used because with dynamic shapes there is no implicit batch dimension to broadcast across.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- bool isOutputBroadcastAcrossBatch(int32_t /*outputIndex*/, const bool* /*inputIsBroadcasted*/,
- int32_t /*nbInputs*/) const _TENSORRT_FINAL TRTNOEXCEPT
+ bool supportsFormat(DataType, PluginFormat) const noexcept override final
{
return false;
}
- //!
- //! \brief Derived classes should not implement this. In a C++11 API it would be override final.
- //!
- //! This method is not used because with dynamic shapes there is no implicit batch dimension to broadcast across.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- bool canBroadcastInputAcrossBatch(int32_t /*inputIndex*/) const _TENSORRT_FINAL TRTNOEXCEPT
+ Dims getOutputDimensions(int32_t, Dims const*, int32_t) noexcept override final
+ {
+ return Dims{-1, {}};
+ }
+
+ bool isOutputBroadcastAcrossBatch(int32_t, bool const*, int32_t) const noexcept override final
+ {
+ return false;
+ }
+
+ bool canBroadcastInputAcrossBatch(int32_t) const noexcept override final
{
return true;
}
- //!
- //! \brief Derived classes should not implement this. In a C++11 API it would be override final.
- //!
- //! This method is not used because it does not allow a plugin to specify mixed formats.
- //!
- //! Instead, derived classes should override supportsFormatCombination, which allows plugins
- //! to express mixed formats.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- bool supportsFormat(DataType /*type*/, PluginFormat /*format*/) const _TENSORRT_FINAL TRTNOEXCEPT
- {
- return false;
- }
-
- //!
- //! \brief Derived classes should not implement this. In a C++11 API it would be override final.
- //!
- //! This method is not used because tensors with dynamic shapes do not have an implicit batch dimension,
- //! input dimensions might be variable, and outputs might have different floating-point formats.
- //!
- //! Instead, derived classes should override the overload of configurePlugin that takes poiners to
- //! DynamicPluginTensorDesc.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- void configurePlugin(const Dims* /*inputDims*/, int32_t /*nbInputs*/, const Dims* /*outputDims*/,
- int32_t /*nbOutputs*/, const DataType* /*inputTypes*/, const DataType* /*outputTypes*/,
- const bool* /*inputIsBroadcast*/, const bool* /*outputIsBroadcast*/, PluginFormat /*floatFormat*/,
- int32_t /*maxBatchSize*/) _TENSORRT_FINAL TRTNOEXCEPT
- {
- }
-
- //!
- //! \brief Derived classes should not implement this. In a C++11 API it would be override final.
- //!
- //! This method is not used because tensors with dynamic shapes do not have an implicit batch dimension,
- //! and the other dimensions might not be build-time constants.
- //!
- //! Instead, derived classes should override the overload of getWorkspaceSize that takes pointers to
- //! PluginTensorDesc. The arguments to that overload provide maximum bounds on all dimensions.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- size_t getWorkspaceSize(int32_t /*maxBatchSize*/) const _TENSORRT_FINAL TRTNOEXCEPT
+ size_t getWorkspaceSize(int32_t) const noexcept override final
{
return 0;
}
- //!
- //! \brief Derived classes should not implement this. In a C++11 API it would be override final.
- //!
- //! This method is not used because tensors with dynamic shapes can have different sizes in different execution
- //! contexts.
- //!
- //! Instead, derived classes should override the overload of enqueue that takes pointers to PluginTensorDesc.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- int32_t enqueue(int32_t /*batchSize*/, const void* const* /*inputs*/, void** /*outputs*/, void* /*workspace*/,
- cudaStream_t /*stream*/) _TENSORRT_FINAL TRTNOEXCEPT
+ int32_t enqueue(int32_t, const void* const*, void* const*, void*, cudaStream_t) noexcept override final
{
return 1;
}
@@ -671,9 +559,9 @@ public:
//! \param layerName The name of the layer, set when constructing the network definition.
//! \param ms The time in milliseconds to execute the layer.
//!
- virtual void reportLayerTime(const char* layerName, float ms) TRTNOEXCEPT = 0;
+ virtual void reportLayerTime(const char* layerName, float ms) noexcept = 0;
- virtual ~IProfiler() {}
+ virtual ~IProfiler() noexcept {}
};
//!
@@ -689,13 +577,14 @@ enum class WeightsRole : int32_t
kSHIFT = 2, //!< shift part of IScaleLayer
kSCALE = 3, //!< scale part of IScaleLayer
kCONSTANT = 4, //!< weights for IConstantLayer
+ kANY = 5, //!< Any other weights role
};
//! Maximum number of elements in WeightsRole enum. \see WeightsRole
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
- return 5;
+ return 6;
}
//!
@@ -711,7 +600,7 @@ enum class DeviceType : int32_t
//! Maximum number of elements in DeviceType enum. \see DeviceType
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 2;
}
@@ -723,19 +612,31 @@ constexpr inline int32_t EnumMax()
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class IRuntime
+class IRuntime : public INoCopy
{
public:
+ virtual ~IRuntime() noexcept = default;
+
//!
//! \brief Deserialize an engine from a stream.
//!
+ //! If an error recorder has been set for the runtime, it will also be passed to the engine.
+ //!
//! \param blob The memory that holds the serialized engine.
- //! \param size The size of the memory.
+ //! \param size The size of the memory in bytes.
//! \param pluginFactory The plugin factory, if any plugins are used by the network, otherwise nullptr.
//!
//! \return The engine, or nullptr if it could not be deserialized.
//!
- virtual nvinfer1::ICudaEngine* deserializeCudaEngine(const void* blob, std::size_t size, IPluginFactory* pluginFactory) noexcept = 0;
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning IPluginFactory is no longer supported, therefore pluginFactory must be a nullptr.
+ //!
+ TRT_DEPRECATED nvinfer1::ICudaEngine* deserializeCudaEngine(
+ const void* blob, std::size_t size, IPluginFactory* pluginFactory) noexcept
+ {
+ return mImpl->deserializeCudaEngine(blob, size, nullptr);
+ }
//!
//! \brief Set the DLA core that the deserialized engine must execute on.
@@ -744,7 +645,10 @@ public:
//!
//! \warning Starting with TensorRT 8, the default value will be -1 if the DLA is not specified or unused.
//!
- virtual void setDLACore(int32_t dlaCore) noexcept = 0;
+ void setDLACore(int32_t dlaCore) noexcept
+ {
+ mImpl->setDLACore(dlaCore);
+ }
//!
//! \brief Get the DLA core that the engine executes on.
@@ -752,31 +656,44 @@ public:
//!
//! \warning Starting with TensorRT 8, the default value will be -1 if the DLA is not specified or unused.
//!
- virtual int32_t getDLACore() const noexcept = 0;
+ int32_t getDLACore() const noexcept
+ {
+ return mImpl->getDLACore();
+ }
//!
//! \brief Returns number of DLA hardware cores accessible.
//!
- virtual int32_t getNbDLACores() const noexcept = 0;
+ int32_t getNbDLACores() const noexcept
+ {
+ return mImpl->getNbDLACores();
+ }
//!
//! \brief Destroy this object.
//!
- virtual void destroy() noexcept = 0;
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
+ //!
+ TRT_DEPRECATED void destroy() noexcept
+ {
+ delete this;
+ }
-protected:
- virtual ~IRuntime() {}
-
-public:
//!
//! \brief Set the GPU allocator.
- //! \param allocator Set the GPU allocator to be used by the runtime. All GPU memory acquired will use this allocator. If NULL is passed, the default allocator will be used.
+ //! \param allocator Set the GPU allocator to be used by the runtime. All GPU memory acquired will use this
+ //! allocator. If NULL is passed, the default allocator will be used.
//!
//! Default: uses cudaMalloc/cudaFree.
//!
//! If nullptr is passed, the default allocator will be used.
//!
- virtual void setGpuAllocator(IGpuAllocator* allocator) noexcept = 0;
+ void setGpuAllocator(IGpuAllocator* allocator) noexcept
+ {
+ mImpl->setGpuAllocator(allocator);
+ }
//!
//! \brief Set the ErrorRecorder for this interface
@@ -786,26 +703,34 @@ public:
//! recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if
//! a recorder has been registered.
//!
+ //! If an error recorder is not set, messages will be sent to the global log stream.
+ //!
//! \param recorder The error recorder to register with this interface.
//
- //! \see getErrorRecorder
+ //! \see getErrorRecorder()
//!
- virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ void setErrorRecorder(IErrorRecorder* recorder) noexcept
+ {
+ mImpl->setErrorRecorder(recorder);
+ }
//!
//! \brief get the ErrorRecorder assigned to this interface.
//!
- //! Retrieves the assigned error recorder object for the given class. A default error recorder does not exist,
- //! so a nullptr will be returned if setErrorRecorder has not been called.
+ //! Retrieves the assigned error recorder object for the given class. A nullptr will be returned if
+ //! an error handler has not been set.
//!
//! \return A pointer to the IErrorRecorder object that has been registered.
//!
- //! \see setErrorRecorder
+ //! \see setErrorRecorder()
//!
- virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+ IErrorRecorder* getErrorRecorder() const noexcept
+ {
+ return mImpl->getErrorRecorder();
+ }
//!
- //! \brief Deserialize an engine from a stream when plugin factory is not used.
+ //! \brief Deserialize an engine from a stream.
//!
//! \param blob The memory that holds the serialized engine.
//! \param size The size of the memory.
@@ -814,8 +739,11 @@ public:
//!
nvinfer1::ICudaEngine* deserializeCudaEngine(const void* blob, std::size_t size) noexcept
{
- return deserializeCudaEngine(blob, size, nullptr);
+ return mImpl->deserializeCudaEngine(blob, size, nullptr);
}
+
+protected:
+ apiv::VRuntime* mImpl;
};
//!
@@ -825,9 +753,11 @@ public:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class IRefitter
+class IRefitter : public INoCopy
{
public:
+ virtual ~IRefitter() noexcept = default;
+
//!
//! \brief Specify new weights for a layer of given name.
//! Returns true on success, or false if new weights are rejected.
@@ -838,14 +768,25 @@ public:
//! * The number of weights is inconsistent with the layer’s original specification.
//!
//! Modifying the weights before method refit() completes will result in undefined behavior.
- virtual bool setWeights(const char* layerName, WeightsRole role, Weights weights) TRTNOEXCEPT = 0;
+ bool setWeights(const char* layerName, WeightsRole role, Weights weights) noexcept
+ {
+ return mImpl->setWeights(layerName, role, weights);
+ }
//!
//! \brief Updates associated engine. Return true if successful.
//!
//! Failure occurs if getMissing() != 0 before the call.
//!
- virtual bool refitCudaEngine() TRTNOEXCEPT = 0;
+ //! The behavior is undefined if the engine has pending enqueued work.
+ //!
+ //! Extant IExecutionContexts associated with the engine should not be used afterwards.
+ //! Instead, create new IExecutionContexts after refitting.
+ //!
+ bool refitCudaEngine() noexcept
+ {
+ return mImpl->refitCudaEngine();
+ }
//!
//! \brief Get description of missing weights.
@@ -861,9 +802,12 @@ public:
//! \return The number of missing Weights.
//!
//! If layerNames!=nullptr, each written pointer points to a string owned by
- //! the engine being refitted, and becomes invalid when the engine is destroyed.
+ //! the engine being refit, and becomes invalid when the engine is destroyed.
//!
- virtual int32_t getMissing(int32_t size, const char** layerNames, WeightsRole* roles) TRTNOEXCEPT = 0;
+ int32_t getMissing(int32_t size, const char** layerNames, WeightsRole* roles) noexcept
+ {
+ return mImpl->getMissing(size, layerNames, roles);
+ }
//!
//! \brief Get description of all weights that could be refit.
@@ -875,16 +819,23 @@ public:
//! \return The number of Weights that could be refit.
//!
//! If layerNames!=nullptr, each written pointer points to a string owned by
- //! the engine being refitted, and becomes invalid when the engine is destroyed.
+ //! the engine being refit, and becomes invalid when the engine is destroyed.
//!
- virtual int32_t getAll(int32_t size, const char** layerNames, WeightsRole* roles) TRTNOEXCEPT = 0;
+ int32_t getAll(int32_t size, const char** layerNames, WeightsRole* roles) noexcept
+ {
+ return mImpl->getAll(size, layerNames, roles);
+ }
- virtual void destroy() TRTNOEXCEPT = 0;
+ //!
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
+ //!
+ TRT_DEPRECATED void destroy() noexcept
+ {
+ delete this;
+ }
-protected:
- virtual ~IRefitter() {}
-
-public:
//!
//! Update dynamic range for a tensor.
//!
@@ -897,7 +848,10 @@ public:
//! Returns false if there is no Int8 engine tensor derived from
//! a network tensor of that name. If successful, then getMissing
//! may report that some weights need to be supplied.
- virtual bool setDynamicRange(const char* tensorName, float min, float max) TRTNOEXCEPT = 0;
+ bool setDynamicRange(const char* tensorName, float min, float max) noexcept
+ {
+ return mImpl->setDynamicRange(tensorName, min, max);
+ }
//!
//! \brief Get minimum of dynamic range.
@@ -906,7 +860,10 @@ public:
//!
//! If the dynamic range was never set, returns the minimum computed during calibration.
//!
- virtual float getDynamicRangeMin(const char* tensorName) const TRTNOEXCEPT = 0;
+ float getDynamicRangeMin(const char* tensorName) const noexcept
+ {
+ return mImpl->getDynamicRangeMin(tensorName);
+ }
//!
//! \brief Get maximum of dynamic range.
@@ -915,7 +872,10 @@ public:
//!
//! If the dynamic range was never set, returns the maximum computed during calibration.
//!
- virtual float getDynamicRangeMax(const char* tensorName) const TRTNOEXCEPT = 0;
+ float getDynamicRangeMax(const char* tensorName) const noexcept
+ {
+ return mImpl->getDynamicRangeMax(tensorName);
+ }
//!
//! \brief Get names of all tensors that have refittable dynamic ranges.
@@ -926,9 +886,12 @@ public:
//! \return The number of Weights that could be refit.
//!
//! If tensorNames!=nullptr, each written pointer points to a string owned by
- //! the engine being refitted, and becomes invalid when the engine is destroyed.
+ //! the engine being refit, and becomes invalid when the engine is destroyed.
//!
- virtual int32_t getTensorsWithDynamicRange(int32_t size, const char** tensorNames) const TRTNOEXCEPT = 0;
+ int32_t getTensorsWithDynamicRange(int32_t size, const char** tensorNames) const noexcept
+ {
+ return mImpl->getTensorsWithDynamicRange(size, tensorNames);
+ }
//!
//! \brief Set the ErrorRecorder for this interface
@@ -938,51 +901,88 @@ public:
//! recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if
//! a recorder has been registered.
//!
+ //! If an error recorder is not set, messages will be sent to the global log stream.
+ //!
//! \param recorder The error recorder to register with this interface.
//
- //! \see getErrorRecorder
+ //! \see getErrorRecorder()
//!
- virtual void setErrorRecorder(IErrorRecorder* recorder) TRTNOEXCEPT = 0;
+ void setErrorRecorder(IErrorRecorder* recorder) noexcept
+ {
+ mImpl->setErrorRecorder(recorder);
+ }
//!
- //! \brief get the ErrorRecorder assigned to this interface.
+ //! \brief Get the ErrorRecorder assigned to this interface.
//!
- //! Retrieves the assigned error recorder object for the given class. A default error recorder does not exist,
- //! so a nullptr will be returned if setErrorRecorder has not been called.
+ //! Retrieves the assigned error recorder object for the given class. A nullptr will be returned if
+ //! an error handler has not been set.
//!
//! \return A pointer to the IErrorRecorder object that has been registered.
//!
- //! \see setErrorRecorder
+ //! \see setErrorRecorder()
//!
- virtual IErrorRecorder* getErrorRecorder() const TRTNOEXCEPT = 0;
-};
+ IErrorRecorder* getErrorRecorder() const noexcept
+ {
+ return mImpl->getErrorRecorder();
+ }
-//!
-//! \class IPluginFactory
-//!
-//! \brief Plugin factory for deserialization.
-//!
-//! This Interface is guaranteed not to change for the same major version of TensorRT.
-class IPluginFactory
-{
-public:
//!
- //! \brief Create a plugin from serialized data.
+ //! \brief Specify new weights of given name.
//!
- //! Responsibility of destroying this plugin lies with the application.
- //! It can be done anytime after consumers of this plugin are destroyed.
+ //! \param name The name of the weights to be refit.
+ //! \param weights The new weights to associate with the name.
//!
- //! \param layerName The name of the layer.
- //! \param serialData The serialized data.
- //! \param serialLength The length of the serialized data.
+ //! Returns true on success, or false if new weights are rejected.
+ //! Possible reasons for rejection are:
//!
- //! \return The plugin.
+ //! * The name of weights is nullptr or does not correspond to any refittable weights.
+ //! * The number of weights is inconsistent with the original specification.
//!
- //! \see IPlugin::serialize()
- //!
- virtual IPlugin* createPlugin(const char* layerName, const void* serialData, size_t serialLength) TRTNOEXCEPT = 0;
+ //! Modifying the weights before method refitCudaEngine() completes will result in undefined behavior.
+ bool setNamedWeights(const char* name, Weights weights) noexcept
+ {
+ return mImpl->setNamedWeights(name, weights);
+ }
- virtual ~IPluginFactory() {}
+ //!
+ //! \brief Get names of missing weights.
+ //!
+ //! For example, if some Weights have been set, but the engine was optimized
+ //! in a way that combines weights, any unsupplied Weights in the combination
+ //! are considered missing.
+ //!
+ //! \param size The number of weights names that can be safely written to.
+ //! \param weightsNames The names of the weights to be updated, or nullptr for unnamed weights.
+ //!
+ //! \return The number of missing Weights.
+ //!
+ //! If layerNames!=nullptr, each written pointer points to a string owned by
+ //! the engine being refit, and becomes invalid when the engine is destroyed.
+ //!
+ int32_t getMissingWeights(int32_t size, const char** weightsNames) noexcept
+ {
+ return mImpl->getMissingWeights(size, weightsNames);
+ }
+
+ //!
+ //! \brief Get names of all weights that could be refit.
+ //!
+ //! \param size The number of weights names that can be safely written to.
+ //! \param weightsNames The names of the weights to be updated, or nullptr for unnamed weights.
+ //!
+ //! \return The number of Weights that could be refit.
+ //!
+ //! If layerNames!=nullptr, each written pointer points to a string owned by
+ //! the engine being refit, and becomes invalid when the engine is destroyed.
+ //!
+ int32_t getAllWeights(int32_t size, const char** weightsNames) noexcept
+ {
+ return mImpl->getAllWeights(size, weightsNames);
+ }
+
+protected:
+ apiv::VRefitter* mImpl;
};
//!
@@ -1004,7 +1004,7 @@ enum class OptProfileSelector : int32_t
//!< Number of different values of OptProfileSelector enum. \see OptProfileSelector
template <>
-constexpr inline int32_t EnumMax()
+constexpr inline int32_t EnumMax() noexcept
{
return 3;
}
@@ -1031,7 +1031,7 @@ constexpr inline int32_t EnumMax()
//!
//! \see IBuilderConfig::addOptimizationProfile()
//!
-class IOptimizationProfile
+class IOptimizationProfile : public INoCopy
{
public:
//!
@@ -1059,14 +1059,20 @@ public:
//!
//! \warning If run on DLA, minimum, optimum, and maximum dimensions must to be the same.
//!
- virtual bool setDimensions(const char* inputName, OptProfileSelector select, Dims dims) noexcept = 0;
+ bool setDimensions(const char* inputName, OptProfileSelector select, Dims dims) noexcept
+ {
+ return mImpl->setDimensions(inputName, select, dims);
+ }
//!
//! \brief Get the minimum / optimum / maximum dimensions for a dynamic input tensor.
//!
//! If the dimensions have not been previously set via setDimensions(), return an invalid Dims with nbDims == -1.
//!
- virtual Dims getDimensions(const char* inputName, OptProfileSelector select) const noexcept = 0;
+ Dims getDimensions(const char* inputName, OptProfileSelector select) const noexcept
+ {
+ return mImpl->getDimensions(inputName, select);
+ }
//!
//! \brief Set the minimum / optimum / maximum values for an input shape tensor.
@@ -1075,10 +1081,25 @@ public:
//! This implies that the datatype of t is DataType::kINT32, the rank is either 0 or 1, and the dimensions of t
//! are fixed at network definition time. This function must not be called for any input tensor that is not a
//! shape tensor.
+ //!
//! Each time this function is called for the same input tensor, the same nbValues must be supplied (either 1
//! if the tensor rank is 0, or dims.d[0] if the rank is 1). Furthermore, if minVals, optVals, maxVals are the
//! minimum, optimum, and maximum values, it must be true that minVals[i] <= optVals[i] <= maxVals[i] for
- //! i = 0, ..., nbValues - 1.
+ //! i = 0, ..., nbValues - 1. Execution of the network must be valid for the optVals.
+ //!
+ //! Shape tensors are tensors that contribute to shape calculations in some way, and can contain
+ //! any int32_t values appropriate for the network. Examples:
+ //!
+ //! * A shape tensor used as the second input to IShuffleLayer can contain a -1 wildcard.
+ //! The corresponding minVal[i] should be -1.
+ //!
+ //! * A shape tensor used as the stride input to ISliceLayer can contain any valid strides.
+ //! The values could be positive, negative, or zero.
+ //!
+ //! * A shape tensor subtracted from zero to compute the size input of an ISliceLayer can
+ //! contain any non-positive values that yield a valid slice operation.
+ //!
+ //! Tightening the minVals and maxVals bounds to cover only values that are necessary may help optimization.
//!
//! \param inputName The input tensor name
//! \param select Whether to set the minimum, optimum, or maximum input values.
@@ -1091,9 +1112,11 @@ public:
//!
//! \warning If run on DLA, minimum, optimum, and maximum shape values must to be the same.
//!
- virtual bool setShapeValues(
+ bool setShapeValues(
const char* inputName, OptProfileSelector select, const int32_t* values, int32_t nbValues) noexcept
- = 0;
+ {
+ return mImpl->setShapeValues(inputName, select, values, nbValues);
+ }
//!
//! \brief Get the number of values for an input shape tensor.
@@ -1101,14 +1124,20 @@ public:
//! This will return the number of shape values if setShapeValues() has been called before for this input tensor.
//! Otherwise, return -1.
//!
- virtual int32_t getNbShapeValues(const char* inputName) const noexcept = 0;
+ int32_t getNbShapeValues(const char* inputName) const noexcept
+ {
+ return mImpl->getNbShapeValues(inputName);
+ }
//!
//! \brief Get the minimum / optimum / maximum values for an input shape tensor.
//!
//! If the shape values have not been set previously with setShapeValues(), this returns nullptr.
//!
- virtual const int32_t* getShapeValues(const char* inputName, OptProfileSelector select) const noexcept = 0;
+ int32_t const* getShapeValues(const char* inputName, OptProfileSelector select) const noexcept
+ {
+ return mImpl->getShapeValues(inputName, select);
+ }
//!
//! \brief Set a target for extra GPU memory that may be used by this profile.
@@ -1123,12 +1152,18 @@ public:
//!
//! \return true if the input is in the valid range (between 0 and 1 inclusive), else false
//!
- virtual bool setExtraMemoryTarget(float target) noexcept = 0;
+ bool setExtraMemoryTarget(float target) noexcept
+ {
+ return mImpl->setExtraMemoryTarget(target);
+ }
//!
//! \brief Get the extra memory target that has been defined for this profile.
//!
- virtual float getExtraMemoryTarget() const noexcept = 0;
+ float getExtraMemoryTarget() const noexcept
+ {
+ return mImpl->getExtraMemoryTarget();
+ }
//!
//! \brief Check whether the optimization profile can be passed to an IBuilderConfig object.
@@ -1141,12 +1176,45 @@ public:
//!
//! \return true if the optimization profile is valid and may be passed to an IBuilderConfig, else false
//!
- virtual bool isValid() const noexcept = 0;
+ bool isValid() const noexcept
+ {
+ return mImpl->isValid();
+ }
protected:
- ~IOptimizationProfile() noexcept = default;
+ apiv::VOptimizationProfile* mImpl;
+ virtual ~IOptimizationProfile() noexcept = default;
};
+//!
+//! \enum TacticSource
+//!
+//! \brief List of tactic sources for TensorRT.
+//!
+//! \see TacticSources, IBuilderConfig::setTacticSources(), IBuilderConfig::getTacticSources()
+//!
+enum class TacticSource : int32_t
+{
+ //! \note Disabling kCUBLAS will cause the cublas handle passed to plugins in attachToContext to be null.
+ kCUBLAS = 0, //!< cuBLAS tactics.
+ kCUBLAS_LT = 1, //!< cuBLAS LT tactics
+ kCUDNN = 2 //!< cuDNN tactics
+};
+
+template <>
+constexpr inline int32_t EnumMax() noexcept
+{
+ return 3;
+} //!< Maximum number of tactic sources in TacticSource enum. \see TacticSource
+
+//!
+//! \brief Represents a collection of one or more TacticSource values
+//! combine using bitwise-OR operations.
+//!
+//! \see IBuilderConfig::setTacticSources(), IBuilderConfig::getTacticSources()
+//!
+using TacticSources = uint32_t;
+
//!
//! \class ICudaEngine
//!
@@ -1154,9 +1222,11 @@ protected:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
-class ICudaEngine
+class ICudaEngine : public INoCopy
{
public:
+ virtual ~ICudaEngine() noexcept = default;
+
//!
//! \brief Get the number of binding indices.
//!
@@ -1167,7 +1237,10 @@ public:
//!
//! \see getBindingIndex();
//!
- virtual int32_t getNbBindings() const noexcept = 0;
+ int32_t getNbBindings() const noexcept
+ {
+ return mImpl->getNbBindings();
+ }
//!
//! \brief Retrieve the binding index for a named tensor.
@@ -1186,7 +1259,10 @@ public:
//!
//! \see getNbBindings() getBindingName()
//!
- virtual int32_t getBindingIndex(const char* name) const noexcept = 0;
+ int32_t getBindingIndex(const char* name) const noexcept
+ {
+ return mImpl->getBindingIndex(name);
+ }
//!
//! \brief Retrieve the name corresponding to a binding index.
@@ -1203,7 +1279,10 @@ public:
//!
//! \see getBindingIndex()
//!
- virtual const char* getBindingName(int32_t bindingIndex) const noexcept = 0;
+ const char* getBindingName(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingName(bindingIndex);
+ }
//!
//! \brief Determine whether a binding is an input binding.
@@ -1213,7 +1292,10 @@ public:
//!
//! \see getBindingIndex()
//!
- virtual bool bindingIsInput(int32_t bindingIndex) const noexcept = 0;
+ bool bindingIsInput(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->bindingIsInput(bindingIndex);
+ }
//!
//! \brief Get the dimensions of a binding.
@@ -1235,7 +1317,10 @@ public:
//!
//! \see getBindingIndex()
//!
- virtual Dims getBindingDimensions(int32_t bindingIndex) const noexcept = 0;
+ Dims getBindingDimensions(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingDimensions(bindingIndex);
+ }
//!
//! \brief Determine the required data type for a buffer from its binding index.
@@ -1245,7 +1330,10 @@ public:
//!
//! \see getBindingIndex()
//!
- virtual DataType getBindingDataType(int32_t bindingIndex) const noexcept = 0;
+ DataType getBindingDataType(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingDataType(bindingIndex);
+ }
//!
//! \brief Get the maximum batch size which can be used for inference.
@@ -1254,7 +1342,10 @@ public:
//!
//! \return The maximum batch size for this engine.
//!
- virtual int32_t getMaxBatchSize() const noexcept = 0;
+ int32_t getMaxBatchSize() const noexcept
+ {
+ return mImpl->getMaxBatchSize();
+ }
//!
//! \brief Get the number of layers in the network.
@@ -1265,32 +1356,24 @@ public:
//!
//! \return The number of layers in the network.
//!
- virtual int32_t getNbLayers() const noexcept = 0;
-
- //!
- //! \brief Get the amount of workspace the engine uses.
- //!
- //! The workspace size will be no greater than the value provided to the builder when the engine was built, and will
- //! typically be smaller. Workspace will be allocated for each execution context.
- //!
- //! This method is not used because getDeviceMemorySize returns the total amount of device memory required by an
- //! execution context.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- virtual std::size_t getWorkspaceSize() const noexcept = 0;
+ int32_t getNbLayers() const noexcept
+ {
+ return mImpl->getNbLayers();
+ }
//!
//! \brief Serialize the network to a stream.
//!
//! \return A IHostMemory object that contains the serialized engine.
//!
- //! The network may be deserialized with IRuntime::deserializeCudaEngine() and also safe::IRuntime::deserializeCudaEngine() if only functional-safe features are used in the engine.
+ //! The network may be deserialized with IRuntime::deserializeCudaEngine().
//!
- //! \see IRuntime::deserializeCudaEngine() safe::IRuntime::deserializeCudaEngine()
+ //! \see IRuntime::deserializeCudaEngine()
//!
- virtual IHostMemory* serialize() const noexcept = 0;
+ IHostMemory* serialize() const noexcept
+ {
+ return mImpl->serialize();
+ }
//!
//! \brief Create an execution context.
@@ -1298,16 +1381,27 @@ public:
//! If the engine supports dynamic shapes, each execution context in concurrent use must use a separate optimization
//! profile. The first execution context created will call setOptimizationProfile(0) implicitly. For other execution
//! contexts, setOptimizationProfile() must be called with unique profile index before calling execute or enqueue.
+ //! If an error recorder has been set for the engine, it will also be passed to the execution context.
//!
//! \see IExecutionContext.
//! \see IExecutionContext::setOptimizationProfile()
//!
- virtual IExecutionContext* createExecutionContext() noexcept = 0;
+ IExecutionContext* createExecutionContext() noexcept
+ {
+ return mImpl->createExecutionContext();
+ }
//!
//! \brief Destroy this object;
//!
- virtual void destroy() noexcept = 0;
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
+ //!
+ TRT_DEPRECATED void destroy() noexcept
+ {
+ delete this;
+ }
//!
//! \brief Get location of binding
@@ -1319,33 +1413,39 @@ public:
//! \param bindingIndex The binding index.
//! \return The location of the bound tensor with given index.
//!
- virtual TensorLocation getLocation(int32_t bindingIndex) const noexcept = 0;
+ TensorLocation getLocation(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getLocation(bindingIndex);
+ }
-protected:
- virtual ~ICudaEngine() {}
-
-public:
//! \brief create an execution context without any device memory allocated
//!
//! The memory for execution of this device context must be supplied by the application.
//!
- //! \see getDeviceMemorySize() IExecutionContext::setDeviceMemory()
- //!
- virtual IExecutionContext* createExecutionContextWithoutDeviceMemory() noexcept = 0;
+ IExecutionContext* createExecutionContextWithoutDeviceMemory() noexcept
+ {
+ return mImpl->createExecutionContextWithoutDeviceMemory();
+ }
//!
//! \brief Return the amount of device memory required by an execution context.
//!
//! \see IExecutionContext::setDeviceMemory()
//!
- virtual size_t getDeviceMemorySize() const noexcept = 0;
+ size_t getDeviceMemorySize() const noexcept
+ {
+ return mImpl->getDeviceMemorySize();
+ }
//!
- //! \brief Return true if engine can be refit.
+ //! \brief Return true if an engine can be refit.
//!
//! \see nvinfer1::createInferRefitter()
//!
- virtual bool isRefittable() const noexcept = 0;
+ bool isRefittable() const noexcept
+ {
+ return mImpl->isRefittable();
+ }
//!
//! \brief Return the number of bytes per component of an element.
@@ -1356,7 +1456,10 @@ public:
//!
//! \see ICudaEngine::getBindingVectorizedDim()
//!
- virtual int32_t getBindingBytesPerComponent(int32_t bindingIndex) const noexcept = 0;
+ int32_t getBindingBytesPerComponent(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingBytesPerComponent(bindingIndex);
+ }
//!
//! \brief Return the number of components included in one element.
@@ -1367,14 +1470,20 @@ public:
//!
//! \see ICudaEngine::getBindingVectorizedDim()
//!
- virtual int32_t getBindingComponentsPerElement(int32_t bindingIndex) const noexcept = 0;
+ int32_t getBindingComponentsPerElement(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingComponentsPerElement(bindingIndex);
+ }
//!
//! \brief Return the binding format.
//!
//! \param bindingIndex The binding Index.
//!
- virtual TensorFormat getBindingFormat(int32_t bindingIndex) const noexcept = 0;
+ TensorFormat getBindingFormat(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingFormat(bindingIndex);
+ }
//!
//! \brief Return the human readable description of the tensor format.
@@ -1390,7 +1499,10 @@ public:
//!
//! \param bindingIndex The binding Index.
//!
- virtual const char* getBindingFormatDesc(int32_t bindingIndex) const noexcept = 0;
+ const char* getBindingFormatDesc(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingFormatDesc(bindingIndex);
+ }
//!
//! \brief Return the dimension index that the buffer is vectorized.
@@ -1399,7 +1511,10 @@ public:
//!
//! \param bindingIndex The binding Index.
//!
- virtual int32_t getBindingVectorizedDim(int32_t bindingIndex) const noexcept = 0;
+ int32_t getBindingVectorizedDim(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingVectorizedDim(bindingIndex);
+ }
//!
//! \brief Returns the name of the network associated with the engine.
@@ -1411,7 +1526,10 @@ public:
//!
//! \return A zero delimited C-style string representing the name of the network.
//!
- virtual const char* getName() const noexcept = 0;
+ const char* getName() const noexcept
+ {
+ return mImpl->getName();
+ }
//!
//! \brief Get the number of optimization profiles defined for this engine.
@@ -1419,7 +1537,10 @@ public:
//! \return Number of optimization profiles. It is always at least 1.
//!
//! \see IExecutionContext::setOptimizationProfile()
- virtual int32_t getNbOptimizationProfiles() const noexcept = 0;
+ int32_t getNbOptimizationProfiles() const noexcept
+ {
+ return mImpl->getNbOptimizationProfiles();
+ }
//!
//! \brief Get the minimum / optimum / maximum dimensions for a particular binding under an optimization profile.
@@ -1443,9 +1564,10 @@ public:
//!
//! Otherwise the bindingIndex is considered invalid.
//!
- virtual Dims getProfileDimensions(int32_t bindingIndex, int32_t profileIndex, OptProfileSelector select) const
- noexcept
- = 0;
+ Dims getProfileDimensions(int32_t bindingIndex, int32_t profileIndex, OptProfileSelector select) const noexcept
+ {
+ return mImpl->getProfileDimensions(bindingIndex, profileIndex, select);
+ }
//!
//! \brief Get minimum / optimum / maximum values for an input shape binding under an optimization profile.
@@ -1468,9 +1590,11 @@ public:
//!
//! \see ICudaEngine::getProfileDimensions
//!
- virtual const int32_t* getProfileShapeValues(
- int32_t profileIndex, int32_t inputIndex, OptProfileSelector select) const noexcept
- = 0;
+ const int32_t* getProfileShapeValues(int32_t profileIndex, int32_t inputIndex, OptProfileSelector select) const
+ noexcept
+ {
+ return mImpl->getProfileShapeValues(profileIndex, inputIndex, select);
+ }
//!
//! \brief True if tensor is required as input for shape calculations or output from them.
@@ -1503,7 +1627,10 @@ public:
//!
//! \see isExecutionBinding()
//!
- virtual bool isShapeBinding(int32_t bindingIndex) const noexcept = 0;
+ bool isShapeBinding(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->isShapeBinding(bindingIndex);
+ }
//!
//! \brief True if pointer to tensor data is required for execution phase, false if nullptr can be supplied.
@@ -1514,19 +1641,25 @@ public:
//!
//! \see isShapeBinding()
//!
- virtual bool isExecutionBinding(int32_t bindingIndex) const noexcept = 0;
+ bool isExecutionBinding(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->isExecutionBinding(bindingIndex);
+ }
//!
- //! \brief determine that execution capability this engine has.
+ //! \brief Determine what execution capability this engine has.
//!
- //! If the engine has EngineCapability::kDEFAULT, then all engine functionality is valid..
- //! If the engine has EngineCapability::kSAFE_GPU, then only the functionality in safe::ICudaEngine is valid.
- //! If the engine has EngineCapability::kSAFE_DLA, then only serialize, destroy, and const-accessor functions are
+ //! If the engine has EngineCapability::kSTANDARD, then all engine functionality is valid.
+ //! If the engine has EngineCapability::kSAFETY, then only the functionality in safe engine is valid.
+ //! If the engine has EngineCapability::kDLA_STANDALONE, then only serialize, destroy, and const-accessor functions are
//! valid.
//!
//! \return The EngineCapability flag that the engine was built for.
//!
- virtual EngineCapability getEngineCapability() const noexcept = 0;
+ EngineCapability getEngineCapability() const noexcept
+ {
+ return mImpl->getEngineCapability();
+ }
//! \brief Set the ErrorRecorder for this interface
//!
@@ -1535,23 +1668,31 @@ public:
//! recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if
//! a recorder has been registered.
//!
+ //! If an error recorder is not set, messages will be sent to the global log stream.
+ //!
//! \param recorder The error recorder to register with this interface.
//
- //! \see getErrorRecorder
+ //! \see getErrorRecorder()
//!
- virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ void setErrorRecorder(IErrorRecorder* recorder) noexcept
+ {
+ return mImpl->setErrorRecorder(recorder);
+ }
//!
- //! \brief get the ErrorRecorder assigned to this interface.
+ //! \brief Get the ErrorRecorder assigned to this interface.
//!
- //! Retrieves the assigned error recorder object for the given class. A default error recorder does not exist,
- //! so a nullptr will be returned if setErrorRecorder has not been called.
+ //! Retrieves the assigned error recorder object for the given class. A nullptr will be returned if
+ //! an error handler has not been set.
//!
//! \return A pointer to the IErrorRecorder object that has been registered.
//!
- //! \see setErrorRecorder
+ //! \see setErrorRecorder()
//!
- virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+ IErrorRecorder* getErrorRecorder() const noexcept
+ {
+ return mImpl->getErrorRecorder();
+ }
//!
//! \brief Query whether the engine was built with an implicit batch dimension.
@@ -1567,7 +1708,22 @@ public:
//!
//! \see createNetworkV2
//!
- virtual bool hasImplicitBatchDimension() const TRTNOEXCEPT = 0;
+ bool hasImplicitBatchDimension() const noexcept
+ {
+ return mImpl->hasImplicitBatchDimension();
+ }
+
+ //! \brief return the tactic sources required by this engine
+ //!
+ //! \see IBuilderConfig::setTacticSources()
+ //!
+ TacticSources getTacticSources() const noexcept
+ {
+ return mImpl->getTacticSources();
+ }
+
+protected:
+ apiv::VCudaEngine* mImpl;
};
//!
@@ -1580,38 +1736,61 @@ public:
//! dynamic shapes, each execution context in concurrent use must use a separate optimization profile.
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
-class IExecutionContext
+class IExecutionContext : public INoCopy
{
public:
+ virtual ~IExecutionContext() noexcept = default;
+
//!
//! \brief Synchronously execute inference on a batch.
//!
- //! This method requires an array of input and output buffers. The mapping from tensor names to indices can be
- //! queried using ICudaEngine::getBindingIndex() \param batchSize The batch size. This is at most the value supplied
- //! when the engine was built. \param bindings An array of pointers to input and output buffers for the network.
+ //! This method requires an array of input and output buffers. The mapping from tensor names to indices
+ //! can be queried using ICudaEngine::getBindingIndex()
+ //!
+ //! \param batchSize The batch size. This is at most the value supplied when the engine was built.
+ //! \param bindings An array of pointers to input and output buffers for the network.
//!
//! \return True if execution succeeded.
//!
+ //! \warning This function will trigger layer resource updates if hasImplicitBatchDimension()
+ //! returns true and batchSize changes between subsequent calls, possibly resulting
+ //! in performance bottlenecks.
+ //!
//! \see ICudaEngine::getBindingIndex() ICudaEngine::getMaxBatchSize()
//!
- virtual bool execute(int32_t batchSize, void** bindings) noexcept = 0;
+ bool execute(int32_t batchSize, void* const* bindings) noexcept
+ {
+ return mImpl->execute(batchSize, bindings);
+ }
//!
//! \brief Asynchronously execute inference on a batch.
//!
//! This method requires an array of input and output buffers. The mapping from tensor names to indices can be
//! queried using ICudaEngine::getBindingIndex() \param batchSize The batch size. This is at most the value supplied
- //! when the engine was built. \param bindings An array of pointers to input and output buffers for the network.
- //! \param stream A cuda stream on which the inference kernels will be enqueued
+ //! when the engine was built.
+ //!
+ //! \param bindings An array of pointers to input and output buffers for the network.
+ //! \param stream A cuda stream on which the inference kernels will be enqueued.
//! \param inputConsumed An optional event which will be signaled when the input buffers can be refilled with new
- //! data
+ //! data.
//!
//! \return True if the kernels were enqueued successfully.
//!
//! \see ICudaEngine::getBindingIndex() ICudaEngine::getMaxBatchSize()
//!
- virtual bool enqueue(int32_t batchSize, void** bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept
- = 0;
+ //! \warning Calling enqueue() in from the same IExecutionContext object with different CUDA streams concurrently
+ //! results in undefined behavior. To perform inference concurrently in multiple streams, use one execution
+ //! context per stream.
+ //!
+ //! \warning This function will trigger layer resource updates if hasImplicitBatchDimension()
+ //! returns true and batchSize changes between subsequent calls, possibly resulting in performance
+ //! bottlenecks.
+ //!
+ bool enqueue(int32_t batchSize, void* const* bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept
+ {
+ return mImpl->enqueue(batchSize, bindings, stream, inputConsumed);
+ }
//!
//! \brief Set the debug sync flag.
@@ -1621,45 +1800,63 @@ public:
//!
//! \see getDebugSync()
//!
- virtual void setDebugSync(bool sync) noexcept = 0;
+ void setDebugSync(bool sync) noexcept
+ {
+ mImpl->setDebugSync(sync);
+ }
//!
//! \brief Get the debug sync flag.
//!
//! \see setDebugSync()
//!
- virtual bool getDebugSync() const noexcept = 0;
+ bool getDebugSync() const noexcept
+ {
+ return mImpl->getDebugSync();
+ }
//!
//! \brief Set the profiler.
//!
//! \see IProfiler getProfiler()
//!
- virtual void setProfiler(IProfiler*) noexcept = 0;
+ void setProfiler(IProfiler* profiler) noexcept
+ {
+ mImpl->setProfiler(profiler);
+ }
//!
//! \brief Get the profiler.
//!
//! \see IProfiler setProfiler()
//!
- virtual IProfiler* getProfiler() const noexcept = 0;
+ IProfiler* getProfiler() const noexcept
+ {
+ return mImpl->getProfiler();
+ }
//!
//! \brief Get the associated engine.
//!
//! \see ICudaEngine
//!
- virtual const ICudaEngine& getEngine() const noexcept = 0;
+ const ICudaEngine& getEngine() const noexcept
+ {
+ return mImpl->getEngine();
+ }
//!
//! \brief Destroy this object.
//!
- virtual void destroy() noexcept = 0;
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
+ //!
+ TRT_DEPRECATED void destroy() noexcept
+ {
+ delete this;
+ }
-protected:
- virtual ~IExecutionContext() noexcept {}
-
-public:
//!
//! \brief Set the name of the execution context.
//!
@@ -1667,14 +1864,20 @@ public:
//!
//! \see getName()
//!
- virtual void setName(const char* name) noexcept = 0;
+ void setName(const char* name) noexcept
+ {
+ mImpl->setName(name);
+ }
//!
//! \brief Return the name of the execution context.
//!
//! \see setName()
//!
- virtual const char* getName() const noexcept = 0;
+ const char* getName() const noexcept
+ {
+ return mImpl->getName();
+ }
//!
//! \brief Set the device memory for use by this execution context.
@@ -1687,7 +1890,12 @@ public:
//!
//! \see ICudaEngine::getDeviceMemorySize() ICudaEngine::createExecutionContextWithoutDeviceMemory()
//!
- virtual void setDeviceMemory(void* memory) noexcept = 0;
+
+ TRT_DEPRECATED
+ void setDeviceMemory(void* memory) noexcept
+ {
+ mImpl->setDeviceMemory(memory);
+ }
//!
//! \brief Return the strides of the buffer for the given binding.
@@ -1705,7 +1913,10 @@ public:
//!
//! \see getBindingComponentsPerElement
//!
- virtual Dims getStrides(int32_t bindingIndex) const noexcept = 0;
+ Dims getStrides(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getStrides(bindingIndex);
+ }
public:
//!
@@ -1734,6 +1945,9 @@ public:
//! setInputShapeBinding() for all dynamic input tensors or input shape tensors, which in
//! turn must be called before either execute() or enqueue().
//!
+ //! \warning This function will trigger layer resource updates on the next
+ //! call of enqueue[V2]()/execute[V2](), possibly resulting in performance bottlenecks.
+ //!
//! \return true if the call succeeded, else false (e.g. input out of range)
//!
//! \deprecated This API is superseded by setOptimizationProfileAsync and will be removed in TensorRT 9.0.
@@ -1741,7 +1955,10 @@ public:
//! \see ICudaEngine::getNbOptimizationProfiles() IExecutionContext::setOptimizationProfileAsync()
//!
TRT_DEPRECATED
- virtual bool setOptimizationProfile(int32_t profileIndex) noexcept = 0;
+ bool setOptimizationProfile(int32_t profileIndex) noexcept
+ {
+ return mImpl->setOptimizationProfile(profileIndex);
+ }
//!
//! \brief Get the index of the currently selected optimization profile.
@@ -1750,18 +1967,23 @@ public:
//! to be created, or explicitly for all subsequent contexts), an invalid value of -1 will be returned
//! and all calls to enqueue() or execute() will fail until a valid profile index has been set.
//!
- virtual int32_t getOptimizationProfile() const noexcept = 0;
+ int32_t getOptimizationProfile() const noexcept
+ {
+ return mImpl->getOptimizationProfile();
+ }
//!
//! \brief Set the dynamic dimensions of a binding
//!
- //! Requires the engine to be built without an implicit batch dimension.
- //! The binding must be an input tensor, and all dimensions must be compatible with
- //! the network definition (i.e. only the wildcard dimension -1 can be replaced with a
- //! new dimension > 0). Furthermore, the dimensions must be in the valid range for the
- //! currently selected optimization profile, and the corresponding engine must not be
- //! safety-certified.
+ //! \param bindingIndex index of an input tensor whose dimensions must be compatible with
+ //! the network definition (i.e. only the wildcard dimension -1 can be replaced with a
+ //! new dimension >= 0).
//!
+ //! \param dimensions specifies the dimensions of the input tensor. It must be in the valid
+ //! range for the currently selected optimization profile, and the corresponding engine must
+ //! not be safety-certified.
+ //!
+ //! This method requires the engine to be built without an implicit batch dimension.
//! This method will fail unless a valid optimization profile is defined for the current
//! execution context (getOptimizationProfile() must not be -1).
//!
@@ -1769,11 +1991,23 @@ public:
//! this method needs to be called before either enqueue() or execute() may be called.
//! This can be checked using the method allInputDimensionsSpecified().
//!
- //! \return false if an error occurs (e.g. index out of range), else true
+ //! \warning This function will trigger layer resource updates on the next
+ //! call of enqueue[V2]()/execute[V2](), possibly resulting in performance bottlenecks,
+ //! if the dimensions are different than the previous set dimensions.
+ //!
+ //! \return false if an error occurs (e.g. bindingIndex is out of range for the currently selected
+ //! optimization profile or binding dimension is inconsistent with min-max range of the
+ //! optimization profile), else true. Note that the network can still be invalid for certain
+ //! combinations of input shapes that lead to invalid output shapes. To confirm the correctness
+ //! of the network input shapes, check whether the output binding has valid
+ //! dimensions using getBindingDimensions() on the output bindingIndex.
//!
//! \see ICudaEngine::getBindingIndex
//!
- virtual bool setBindingDimensions(int32_t bindingIndex, Dims dimensions) noexcept = 0;
+ bool setBindingDimensions(int32_t bindingIndex, Dims dimensions) noexcept
+ {
+ return mImpl->setBindingDimensions(bindingIndex, dimensions);
+ }
//!
//! \brief Get the dynamic dimensions of a binding
@@ -1800,7 +2034,10 @@ public:
//!
//! \see ICudaEngine::getProfileDimensions
//!
- virtual Dims getBindingDimensions(int32_t bindingIndex) const noexcept = 0;
+ Dims getBindingDimensions(int32_t bindingIndex) const noexcept
+ {
+ return mImpl->getBindingDimensions(bindingIndex);
+ }
//!
//! \brief Set values of input tensor required by shape calculations.
@@ -1817,7 +2054,20 @@ public:
//! This method will fail unless a valid optimization profile is defined for the current
//! execution context (getOptimizationProfile() must not be -1).
//!
- virtual bool setInputShapeBinding(int32_t bindingIndex, const int32_t* data) noexcept = 0;
+ //! \warning This function will trigger layer resource updates on the next call of
+ //! enqueue[V2]()/execute[V2](), possibly resulting in performance bottlenecks, if the
+ //! shapes are different than the previous set shapes.
+ //!
+ //! \return false if an error occurs (e.g. bindingIndex is out of range for the currently selected
+ //! optimization profile or shape data is inconsistent with min-max range of the
+ //! optimization profile), else true. Note that the network can still be invalid for certain
+ //! combinations of input shapes that lead to invalid output shapes. To confirm the correctness
+ //! of the network input shapes, check whether the output binding has valid
+ //! dimensions using getBindingDimensions() on the output bindingIndex.
+ bool setInputShapeBinding(int32_t bindingIndex, int32_t const* data) noexcept
+ {
+ return mImpl->setInputShapeBinding(bindingIndex, data);
+ }
//!
//! \brief Get values of an input tensor required for shape calculations or an output tensor produced by shape
@@ -1836,7 +2086,10 @@ public:
//!
//! \see isShapeBinding(bindingIndex)
//!
- virtual bool getShapeBinding(int32_t bindingIndex, int32_t* data) const noexcept = 0;
+ bool getShapeBinding(int32_t bindingIndex, int32_t* data) const noexcept
+ {
+ return mImpl->getShapeBinding(bindingIndex, data);
+ }
//!
//! \brief Whether all dynamic dimensions of input tensors have been specified
@@ -1848,7 +2101,10 @@ public:
//!
//! \see setBindingDimensions(bindingIndex,dimensions)
//!
- virtual bool allInputDimensionsSpecified() const noexcept = 0;
+ bool allInputDimensionsSpecified() const noexcept
+ {
+ return mImpl->allInputDimensionsSpecified();
+ }
//!
//! \brief Whether all input shape bindings have been specified
@@ -1859,7 +2115,11 @@ public:
//!
//! \see isShapeBinding(bindingIndex)
//!
- virtual bool allInputShapesSpecified() const noexcept = 0;
+ bool allInputShapesSpecified() const noexcept
+
+ {
+ return mImpl->allInputShapesSpecified();
+ }
//!
//! \brief Set the ErrorRecorder for this interface
@@ -1869,23 +2129,31 @@ public:
//! recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if
//! a recorder has been registered.
//!
+ //! If an error recorder is not set, messages will be sent to the global log stream.
+ //!
//! \param recorder The error recorder to register with this interface.
//
- //! \see getErrorRecorder
+ //! \see getErrorRecorder()
//!
- virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ void setErrorRecorder(IErrorRecorder* recorder) noexcept
+ {
+ mImpl->setErrorRecorder(recorder);
+ }
//!
- //! \brief get the ErrorRecorder assigned to this interface.
+ //! \brief Get the ErrorRecorder assigned to this interface.
//!
- //! Retrieves the assigned error recorder object for the given class. A default error recorder does not exist,
- //! so a nullptr will be returned if setErrorRecorder has not been called.
+ //! Retrieves the assigned error recorder object for the given class. A nullptr will be returned if
+ //! an error handler has not been set.
//!
//! \return A pointer to the IErrorRecorder object that has been registered.
//!
- //! \see setErrorRecorder
+ //! \see setErrorRecorder()
//!
- virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+ IErrorRecorder* getErrorRecorder() const noexcept
+ {
+ return mImpl->getErrorRecorder();
+ }
//!
//! \brief Synchronously execute inference a network.
@@ -1899,7 +2167,10 @@ public:
//!
//! \see ICudaEngine::getBindingIndex() ICudaEngine::getMaxBatchSize()
//!
- virtual bool executeV2(void** bindings) noexcept = 0;
+ bool executeV2(void* const* bindings) noexcept
+ {
+ return mImpl->executeV2(bindings);
+ }
//!
//! \brief Asynchronously execute inference.
@@ -1920,13 +2191,20 @@ public:
//! used, the first enqueueV2() call after a setInputShapeBinding() call will cause failure in stream capture
//! due to resource allocation. Please call enqueueV2() once before capturing the graph.
//!
- virtual bool enqueueV2(void** bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept = 0;
+ //! \warning Calling enqueueV2() in from the same IExecutionContext object with different CUDA streams concurrently
+ //! results in undefined behavior. To perform inference concurrently in multiple streams, use one execution
+ //! context per stream.
+ //!
+ bool enqueueV2(void* const* bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept
+ {
+ return mImpl->enqueueV2(bindings, stream, inputConsumed);
+ }
//!
//! \brief Select an optimization profile for the current context with async
//! semantics.
//!
- //! \param profileIndex Index of the profile. It must lie between 0 and
+ //! \param profileIndex Index of the profile. The value must lie between 0 and
//! getEngine().getNbOptimizationProfiles() - 1
//!
//! \param stream A cuda stream on which the cudaMemcpyAsyncs may be
@@ -1954,6 +2232,9 @@ public:
//! tensors or input shape tensors, which in turn must be called before
//! either execute() or enqueue().
//!
+ //! \warning This function will trigger layer resource updates on the next call of
+ //! enqueue[V2]()/execute[V2](), possibly resulting in performance bottlenecks.
+ //!
//! \warning Not synchronizing the stream used at enqueue with the stream
//! used to set optimization profile asynchronously using this API will
//! result in undefined behavior.
@@ -1961,8 +2242,14 @@ public:
//! \return true if the call succeeded, else false (e.g. input out of range)
//!
//! \see ICudaEngine::getNbOptimizationProfiles()
- //! IExecutionContext::setOptimizationProfile()
- virtual bool setOptimizationProfileAsync(int32_t profileIndex, cudaStream_t stream) noexcept = 0;
+ //! \see IExecutionContext::setOptimizationProfile()
+ bool setOptimizationProfileAsync(int32_t profileIndex, cudaStream_t stream) noexcept
+ {
+ return mImpl->setOptimizationProfileAsync(profileIndex, stream);
+ }
+
+protected:
+ apiv::VExecutionContext* mImpl;
}; // class IExecutionContext
} // namespace nvinfer1
@@ -1970,24 +2257,35 @@ public:
//! Internal C entry point for creating IRuntime.
//! @private
//!
-extern "C" TENSORRTAPI void* createInferRuntime_INTERNAL(void* logger, int32_t version);
+extern "C" TENSORRTAPI void* createInferRuntime_INTERNAL(void* logger, int32_t version) noexcept;
//!
//! Internal C entry point for creating IRefitter.
//! @private
//!
-extern "C" TENSORRTAPI void* createInferRefitter_INTERNAL(void* engine, void* logger, int32_t version);
+extern "C" TENSORRTAPI void* createInferRefitter_INTERNAL(void* engine, void* logger, int32_t version) noexcept;
+
+//!
+//! \brief Return the plugin registry
+//!
+extern "C" TENSORRTAPI nvinfer1::IPluginRegistry* getPluginRegistry() noexcept;
+
+//!
+//! \brief Return the logger object.
+//!
+extern "C" TENSORRTAPI nvinfer1::ILogger* getLogger() noexcept;
namespace nvinfer1
{
-namespace // unnamed namespace avoids linkage surprises when linking objects built with different versions of this header.
+namespace // unnamed namespace avoids linkage surprises when linking objects built with different versions of this
+ // header.
{
//!
//! \brief Create an instance of an IRuntime class.
//!
//! This class is the logging class for the runtime.
//!
-inline IRuntime* createInferRuntime(ILogger& logger)
+inline IRuntime* createInferRuntime(ILogger& logger) noexcept
{
return static_cast(createInferRuntime_INTERNAL(&logger, NV_TENSORRT_VERSION));
}
@@ -1995,13 +2293,42 @@ inline IRuntime* createInferRuntime(ILogger& logger)
//!
//! \brief Create an instance of an IRefitter class.
//!
-//! This class is the logging class for the refitter.
+//! This is the logging class for the refitter.
//!
-inline IRefitter* createInferRefitter(ICudaEngine& engine, ILogger& logger)
+inline IRefitter* createInferRefitter(ICudaEngine& engine, ILogger& logger) noexcept
{
return static_cast(createInferRefitter_INTERNAL(&engine, &logger, NV_TENSORRT_VERSION));
}
-}
-}
+} // namespace
+
+//!
+//! \brief Register the plugin creator to the registry
+//! The static registry object will be instantiated when the plugin library is
+//! loaded. This static object will register all creators available in the
+//! library to the registry.
+//!
+//! \warning Statically registering plugins should be avoided in the automotive
+//! safety context as the application developer should first register an error recorder
+//! with the plugin registry via IPluginRegistry::setErrorRecorder() before using
+//! IPluginRegistry::registerCreator() or other methods.
+//!
+template
+class PluginRegistrar
+{
+public:
+ PluginRegistrar()
+ {
+ getPluginRegistry()->registerCreator(instance, "");
+ }
+
+private:
+ //! Plugin instance.
+ T instance{};
+};
+
+} // namespace nvinfer1
+
+#define REGISTER_TENSORRT_PLUGIN(name) \
+ static nvinfer1::PluginRegistrar pluginRegistrar##name {}
#endif // NV_INFER_RUNTIME_H
diff --git a/include/NvInferRuntimeCommon.h b/include/NvInferRuntimeCommon.h
index dd681cf2..ef60bcbc 100644
--- a/include/NvInferRuntimeCommon.h
+++ b/include/NvInferRuntimeCommon.h
@@ -17,17 +17,9 @@
#ifndef NV_INFER_RUNTIME_COMMON_H
#define NV_INFER_RUNTIME_COMMON_H
+#include "NvInferVersion.h"
#include
#include
-#include "NvInferVersion.h"
-
-#if __cplusplus >= 201103L
-#define _TENSORRT_FINAL final
-#define _TENSORRT_OVERRIDE override
-#else
-#define _TENSORRT_FINAL
-#define _TENSORRT_OVERRIDE
-#endif
//!< Items that are marked as deprecated will be removed in a future release.
#if __cplusplus >= 201402L
@@ -64,9 +56,6 @@
#else
#define TENSORRTAPI
#endif
-
-//! Defined for use with legacy APIs that have not been updated to noexcept yet.
-//! Do not use with new APIs, use noexcept instead.
#define TRTNOEXCEPT
//!
//! \file NvInferRuntimeCommon.h
@@ -76,15 +65,21 @@
// forward declare some CUDA types to avoid an include dependency
-struct cublasContext;
-struct cudnnContext;
+extern "C"
+{
+ //! Forward declaration of cublasContext to use in other interfaces
+ struct cublasContext;
+ //! Forward declaration of cudnnContext to use in other interfaces
+ struct cudnnContext;
-typedef struct CUstream_st* cudaStream_t; //!< Forward declaration of cudaStream_t.
-typedef struct CUevent_st* cudaEvent_t; //!< Forward declaration of cudaEvent_t.
+ //! Forward declaration of cudaStream_t.
+ using cudaStream_t = struct CUstream_st*;
-static const int32_t NV_TENSORRT_VERSION
- = (NV_TENSORRT_MAJOR * 1000) + (NV_TENSORRT_MINOR * 100) + NV_TENSORRT_PATCH; // major, minor, patch
+ //! Forward declaration of cudaEvent_t.
+ using cudaEvent_t = struct CUevent_st*;
+}
+#define NV_TENSORRT_VERSION nvinfer1::kNV_TENSORRT_VERSION_IMPL
//!
//! \namespace nvinfer1
//!
@@ -93,44 +88,35 @@ static const int32_t NV_TENSORRT_VERSION
namespace nvinfer1
{
-class IErrorRecorder; //!< Forward declare IErrorRecorder for use in other interfaces.
-class IGpuAllocator; //!< Forward declare IGpuAllocator for use in other interfaces.
+static constexpr int32_t kNV_TENSORRT_VERSION_IMPL
+ = (NV_TENSORRT_MAJOR * 1000) + (NV_TENSORRT_MINOR * 100) + NV_TENSORRT_PATCH; // major, minor, patch
+
+//! char_t is the type used by TensorRT to represent all valid characters.
+using char_t = char;
+//! AsciiChar is the type used by TensorRT to represent valid ASCII characters.
+using AsciiChar = char_t;
+
+//! Forward declare IErrorRecorder for use in other interfaces.
+class IErrorRecorder;
+//! Forward declare IGpuAllocator for use in other interfaces.
+class IGpuAllocator;
+
+namespace impl
+{
+//! Declaration of EnumMaxImpl struct to store maximum number of elements in an enumeration type.
+template
+struct EnumMaxImpl;
+} // namespace impl
//! Maximum number of elements in an enumeration type.
template
-constexpr inline int32_t EnumMax();
-
-//!
-//! \enum ActivationType
-//!
-//! \brief Enumerates the types of activation to perform in an activation layer.
-//!
-enum class ActivationType : int32_t
+constexpr int32_t EnumMax() noexcept
{
- kRELU = 0, //!< Rectified linear activation.
- kSIGMOID = 1, //!< Sigmoid activation.
- kTANH = 2, //!< TanH activation.
- kLEAKY_RELU = 3, //!< LeakyRelu activation: x>=0 ? x : alpha * x.
- kELU = 4, //!< Elu activation: x>=0 ? x : alpha * (exp(x) - 1).
- kSELU = 5, //!< Selu activation: x>0 ? beta * x : beta * (alpha*exp(x) - alpha)
- kSOFTSIGN = 6, //!< Softsign activation: x / (1+|x|)
- kSOFTPLUS = 7, //!< Parametric softplus activation: alpha*log(exp(beta*x)+1)
- kCLIP = 8, //!< Clip activation: max(alpha, min(beta, x))
- kHARD_SIGMOID = 9, //!< Hard sigmoid activation: max(0, min(1, alpha*x+beta))
- kSCALED_TANH = 10, //!< Scaled tanh activation: alpha*tanh(beta*x)
- kTHRESHOLDED_RELU = 11 //!< Thresholded ReLU activation: x>alpha ? x : 0
-};
-
-//! Maximum number of elements in ActivationType enum. \see ActivationType
-template <>
-constexpr inline int32_t EnumMax()
-{
- return 12;
+ return impl::EnumMaxImpl::kVALUE;
}
//!
//! \enum DataType
-//!
//! \brief The type of weights and tensors.
//!
enum class DataType : int32_t
@@ -151,63 +137,44 @@ enum class DataType : int32_t
kBOOL = 4
};
+namespace impl
+{
//! Maximum number of elements in DataType enum. \see DataType
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 5;
-}
-
-//!
-//! \enum DimensionType
-//! \brief The type of data encoded across this dimension.
-//!
-enum class DimensionType : int32_t
-{
- kSPATIAL = 0, //!< Elements correspond to different spatial data.
- kCHANNEL = 1, //!< Elements correspond to different channels.
- kINDEX = 2, //!< Elements correspond to different batch index.
- kSEQUENCE = 3 //!< Elements correspond to different sequence values.
+ // Declaration of kVALUE that represents maximum number of elements in DataType enum
+ static constexpr int32_t kVALUE = 5;
};
-
-//! Maximum number of elements in DimensionType enum. \see DimensionType
-template <>
-constexpr inline int32_t EnumMax()
-{
- return 4;
-}
+} // namespace impl
//!
//! \class Dims
//! \brief Structure to define the dimensions of a tensor.
//!
-//! \note: Currently the following formats are supported for layer inputs and outputs:
-//! * zero or more index dimensions followed by one channel and two spatial dimensions (e.g. CHW)
-//! * one time series dimension followed by one index dimension followed by one channel dimension (i.e. TNC)
-//!
//! TensorRT can also return an invalid dims structure. This structure is represented by nbDims == -1
//! and d[i] == 0 for all d.
//!
//! TensorRT can also return an "unknown rank" dims structure. This structure is represented by nbDims == -1
//! and d[i] == -1 for all d.
//!
-class Dims
+class Dims32
{
public:
- static const int32_t MAX_DIMS = 8; //!< The maximum number of dimensions supported for a tensor.
- int32_t nbDims; //!< The number of dimensions.
- int32_t d[MAX_DIMS]; //!< The extent of each dimension.
- TRT_DEPRECATED DimensionType type[MAX_DIMS]; //!< The type of each dimension, provided for backwards compatibility
- //!< and will be removed in TensorRT 8.0.
+ //! The maximum number of dimensions supported for a tensor.
+ static constexpr int32_t MAX_DIMS{8};
+ //! The number of dimensions.
+ int32_t nbDims;
+ //! The extent of each dimension.
+ int32_t d[MAX_DIMS];
};
//!
-//! \brief It is capable of representing one or more TensorFormat by binary OR
-//! operations, e.g., 1U << TensorFormats::kCHW4 | 1U << TensorFormats::kCHW32.
+//! Alias for Dims32.
//!
-//! \see ITensor::getAllowedFormats(), ITensor::setAllowedFormats(),
+//! \warning: This alias might change in the future.
//!
-typedef uint32_t TensorFormats;
+using Dims = Dims32;
//!
//! \enum TensorFormat
@@ -217,22 +184,21 @@ typedef uint32_t TensorFormats;
//! This enum is extended to be used by both plugins and reformat-free network
//! I/O tensors.
//!
-//! \see IPluginExt::getPluginFormats(), safe::ICudaEngine::getBindingFormat()
+//! \see IPluginV2::supportsFormat(), safe::ICudaEngine::getBindingFormat()
//!
//! For more information about data formats, see the topic "Data Format Description" located in the
-//! TensorRT Developer Guide (https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html).
+//! TensorRT Developer Guide.
//!
enum class TensorFormat : int32_t
{
//! Row major linear format.
- //! For a tensor with dimensions {N, C, H, W}, the W axis always has
- //! unit stride, and the stride of every other axis is at least the the
- //! product of of the next dimension times the next stride. the strides
- //! are the same as for a C array with dimensions [N][C][H][W].
+ //! For a tensor with dimensions {N, C, H, W} or {numbers, channels,
+ //! columns, rows}, the dimensional index corresponds to {3, 2, 1, 0}
+ //! and thus the order is W minor.
+ //!
+ //! For DLA usage, the tensor sizes are limited to C,H,W in the range [1,8192].
+ //!
kLINEAR = 0,
- //! Deprecated name of kLINEAR, provided for backwards compatibility and will
- //! be removed in TensorRT 8.0.
- kNCHW TRT_DEPRECATED_ENUM = kLINEAR,
//! Two wide channel vectorized row major format. This format is bound to
//! FP16. It is only available for dimensions >= 3.
@@ -241,9 +207,6 @@ enum class TensorFormat : int32_t
//! [N][(C+1)/2][H][W][2], with the tensor coordinates (n, c, h, w)
//! mapping to array subscript [n][c/2][h][w][c%2].
kCHW2 = 1,
- //! Deprecated name of kCHW2, provided for backwards compatibility and will
- //! be removed in TensorRT 8.0.
- kNC2HW2 TRT_DEPRECATED_ENUM = kCHW2,
//! Eight channel format where C is padded to a multiple of 8. This format
//! is bound to FP16. It is only available for dimensions >= 3.
@@ -252,9 +215,6 @@ enum class TensorFormat : int32_t
//! [N][H][W][(C+7)/8*8], with the tensor coordinates (n, c, h, w)
//! mapping to array subscript [n][h][w][c].
kHWC8 = 2,
- //! Deprecated name of kHWC8, provided for backwards compatibility and will
- //! be removed in TensorRT 8.0.
- kNHWC8 TRT_DEPRECATED_ENUM = kHWC8,
//! Four wide channel vectorized row major format. This format is bound to
//! INT8 or FP16. It is only available for dimensions >= 3.
@@ -280,8 +240,9 @@ enum class TensorFormat : int32_t
//! [N][(C+15)/16][H][W][16], with the tensor coordinates (n, c, h, w)
//! mapping to array subscript [n][c/16][h][w][c%16].
//!
- //! For DLA usage, this format maps to the native format for FP16,
+ //! For DLA usage, this format maps to the native image format for FP16,
//! and the tensor sizes are limited to C,H,W in the range [1,8192].
+ //!
kCHW16 = 4,
//! Thirty-two wide channel vectorized row major format. This format is
@@ -291,7 +252,7 @@ enum class TensorFormat : int32_t
//! [N][(C+31)/32][H][W][32], with the tensor coordinates (n, c, h, w)
//! mapping to array subscript [n][c/32][h][w][c%32].
//!
- //! For DLA usage, this format maps to the native format for INT8,
+ //! For DLA usage, this format maps to the native image format for INT8,
//! and the tensor sizes are limited to C,H,W in the range [1,8192].
kCHW32 = 5,
@@ -315,7 +276,7 @@ enum class TensorFormat : int32_t
//! and is only available for dimensions >= 3.
kHWC = 8,
- //! DLA planar format. For a tensor with dimension {N, C, H, W}, the W axis
+ //! DLA planar format. For a tensor with dimension {N, C, H, W}, the W axis
//! always has unit stride. The stride for stepping along the H axis is
//! rounded up to 64 bytes.
//!
@@ -325,7 +286,7 @@ enum class TensorFormat : int32_t
//! mapping to array subscript [n][c][h][w].
kDLA_LINEAR = 9,
- //! DLA image format. For a tensor with dimension {N, C, H, W} the C axis
+ //! DLA image format. For a tensor with dimension {N, C, H, W} the C axis
//! always has unit stride. The stride for stepping along the H axis is rounded up
//! to 32 bytes. C can only be 1, 3 or 4.
//! If C == 1, it will map to grayscale format.
@@ -337,22 +298,34 @@ enum class TensorFormat : int32_t
//! [N][H][roundUp(W, 32/C'/elementSize)][C'] where elementSize is 2 for FP16
//! and 1 for Int8. The tensor coordinates (n, c, h, w) mapping to array
//! subscript [n][h][w][c].
- kDLA_HWC4 = 10
+ kDLA_HWC4 = 10,
+
+ //! Sixteen channel format where C is padded to a multiple of 16. This format
+ //! is bound to FP16. It is only available for dimensions >= 3.
+ //! For a tensor with dimensions {N, C, H, W},
+ //! the memory layout is equivalent to the array with dimensions
+ //! [N][H][W][(C+15)/16*16], with the tensor coordinates (n, c, h, w)
+ //! mapping to array subscript [n][h][w][c].
+ kHWC16 = 11
};
//!
//! \brief PluginFormat is reserved for backward compatibility.
//!
-//! \see IPluginExt::getPluginFormats()
+//! \see IPluginV2::supportsFormat()
//!
using PluginFormat = TensorFormat;
+namespace impl
+{
//! Maximum number of elements in TensorFormat enum. \see TensorFormat
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 11;
-}
+ //! Declaration of kVALUE that represents maximum number of elements in TensorFormat enum
+ static constexpr int32_t kVALUE = 12;
+};
+} // namespace impl
//! \struct PluginTensorDesc
//!
@@ -361,14 +334,18 @@ constexpr inline int32_t EnumMax()
//! Scale is only valid when data type is DataType::kINT8. TensorRT will set
//! the value to -1.0f if it is invalid.
//!
-//! \see IPluginV2IOExt::supportsFormat
+//! \see IPluginV2IOExt::supportsFormatCombination
//! \see IPluginV2IOExt::configurePlugin
//!
struct PluginTensorDesc
{
+ //! Dimensions.
Dims dims;
- DataType type; //!< \warning DataType:kBOOL not supported.
+ //! \warning DataType:kBOOL not supported.
+ DataType type;
+ //! Tensor format.
TensorFormat format;
+ //! Scale for INT8 data type.
float scale;
};
@@ -380,10 +357,14 @@ struct PluginTensorDesc
//!
enum class PluginVersion : uint8_t
{
- kV2 = 0, //! IPluginV2
- kV2_EXT = 1, //! IPluginV2Ext
- kV2_IOEXT = 2, //! IPluginV2IOExt
- kV2_DYNAMICEXT = 3, //! IPluginV2DynamicExt
+ //! IPluginV2
+ kV2 = 0,
+ //! IPluginV2Ext
+ kV2_EXT = 1,
+ //! IPluginV2IOExt
+ kV2_IOEXT = 2,
+ //! IPluginV2DynamicExt
+ kV2_DYNAMICEXT = 3,
};
//! \class IPluginV2
@@ -403,10 +384,9 @@ public:
//!
//! \brief Return the API version with which this plugin was built.
//!
- //! Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with
- //! plugins.
+ //! Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with plugins.
//!
- virtual int32_t getTensorRTVersion() const TRTNOEXCEPT
+ virtual int32_t getTensorRTVersion() const noexcept
{
return NV_TENSORRT_VERSION;
}
@@ -415,13 +395,13 @@ public:
//! \brief Return the plugin type. Should match the plugin name returned by the corresponding plugin creator
//! \see IPluginCreator::getPluginName()
//!
- virtual const char* getPluginType() const TRTNOEXCEPT = 0;
+ virtual AsciiChar const* getPluginType() const noexcept = 0;
//!
//! \brief Return the plugin version. Should match the plugin version returned by the corresponding plugin creator
//! \see IPluginCreator::getPluginVersion()
//!
- virtual const char* getPluginVersion() const TRTNOEXCEPT = 0;
+ virtual AsciiChar const* getPluginVersion() const noexcept = 0;
//!
//! \brief Get the number of outputs from the layer.
@@ -431,7 +411,7 @@ public:
//! This function is called by the implementations of INetworkDefinition and IBuilder. In particular, it is called
//! prior to any call to initialize().
//!
- virtual int32_t getNbOutputs() const TRTNOEXCEPT = 0;
+ virtual int32_t getNbOutputs() const noexcept = 0;
//!
//! \brief Get the dimension of an output tensor.
@@ -443,7 +423,7 @@ public:
//! This function is called by the implementations of INetworkDefinition and IBuilder. In particular, it is called
//! prior to any call to initialize().
//!
- virtual Dims getOutputDimensions(int32_t index, const Dims* inputs, int32_t nbInputDims) TRTNOEXCEPT = 0;
+ virtual Dims getOutputDimensions(int32_t index, Dims const* inputs, int32_t nbInputDims) noexcept = 0;
//!
//! \brief Check format support.
@@ -462,7 +442,7 @@ public:
//!
//! \warning DataType:kBOOL not supported.
//!
- virtual bool supportsFormat(DataType type, PluginFormat format) const TRTNOEXCEPT = 0;
+ virtual bool supportsFormat(DataType type, PluginFormat format) const noexcept = 0;
//!
//! \brief Configure the layer.
@@ -487,21 +467,22 @@ public:
//!
//! \warning DataType:kBOOL not supported.
//!
- virtual void configureWithFormat(const Dims* inputDims, int32_t nbInputs, const Dims* outputDims, int32_t nbOutputs,
- DataType type, PluginFormat format, int32_t maxBatchSize) TRTNOEXCEPT = 0;
+ virtual void configureWithFormat(Dims const* inputDims, int32_t nbInputs, Dims const* outputDims, int32_t nbOutputs,
+ DataType type, PluginFormat format, int32_t maxBatchSize) noexcept = 0;
//!
//! \brief Initialize the layer for execution. This is called when the engine is created.
//!
//! \return 0 for success, else non-zero (which will cause engine termination).
//!
- virtual int32_t initialize() TRTNOEXCEPT = 0;
+ virtual int32_t initialize() noexcept = 0;
//!
//! \brief Release resources acquired during plugin layer initialization. This is called when the engine is
- //! destroyed. \see initialize()
+ //! destroyed.
+ //! \see initialize()
//!
- virtual void terminate() TRTNOEXCEPT = 0;
+ virtual void terminate() noexcept = 0;
//!
//! \brief Find the workspace size required by the layer.
@@ -511,7 +492,7 @@ public:
//!
//! \return The workspace size.
//!
- virtual size_t getWorkspaceSize(int32_t maxBatchSize) const TRTNOEXCEPT = 0;
+ virtual size_t getWorkspaceSize(int32_t maxBatchSize) const noexcept = 0;
//!
//! \brief Execute the layer.
@@ -524,48 +505,61 @@ public:
//!
//! \return 0 for success, else non-zero (which will cause engine termination).
//!
- virtual int32_t enqueue(int32_t batchSize, const void* const* inputs, void** outputs, void* workspace,
- cudaStream_t stream) TRTNOEXCEPT = 0;
+ virtual int32_t enqueue(int32_t batchSize, void const* const* inputs, void* const* outputs, void* workspace,
+ cudaStream_t stream) noexcept
+ = 0;
//!
//! \brief Find the size of the serialization buffer required.
//!
//! \return The size of the serialization buffer.
//!
- virtual size_t getSerializationSize() const TRTNOEXCEPT = 0;
+ virtual size_t getSerializationSize() const noexcept = 0;
//!
//! \brief Serialize the layer.
//!
- //! \param buffer A pointer to a buffer to serialize data. Size of buffer must be equal to value returned by getSerializationSize.
+ //! \param buffer A pointer to a buffer to serialize data. Size of buffer must be equal to value returned by
+ //! getSerializationSize.
//!
//! \see getSerializationSize()
//!
- virtual void serialize(void* buffer) const TRTNOEXCEPT = 0;
+ virtual void serialize(void* buffer) const noexcept = 0;
//!
//! \brief Destroy the plugin object. This will be called when the network, builder or engine is destroyed.
//!
- virtual void destroy() TRTNOEXCEPT = 0;
+ virtual void destroy() noexcept = 0;
//!
- //! \brief Clone the plugin object. This copies over internal plugin parameters and returns a new plugin object with these parameters.
+ //! \brief Clone the plugin object. This copies over internal plugin parameters and returns a new plugin object with
+ //! these parameters.
//!
- virtual IPluginV2* clone() const TRTNOEXCEPT = 0;
+ //! The TensorRT runtime calls clone() to clone the plugin when an execution context is created for an engine,
+ //! after the engine has been created. The runtime does not call initialize() on the cloned plugin,
+ //! so the cloned plugin should be created in an initialized state.
+ //!
+ virtual IPluginV2* clone() const noexcept = 0;
//!
//! \brief Set the namespace that this plugin object belongs to. Ideally, all plugin
//! objects from the same plugin library should have the same namespace.
//!
- virtual void setPluginNamespace(const char* pluginNamespace) TRTNOEXCEPT = 0;
+ virtual void setPluginNamespace(AsciiChar const* pluginNamespace) noexcept = 0;
//!
//! \brief Return the namespace of the plugin object.
//!
- virtual const char* getPluginNamespace() const TRTNOEXCEPT = 0;
+ virtual AsciiChar const* getPluginNamespace() const noexcept = 0;
+
+ IPluginV2() = default;
+ virtual ~IPluginV2() noexcept = default;
protected:
- virtual ~IPluginV2() {}
+ IPluginV2(IPluginV2 const&) = default;
+ IPluginV2(IPluginV2&&) = default;
+ IPluginV2& operator=(IPluginV2 const&) & = default;
+ IPluginV2& operator=(IPluginV2&&) & = default;
};
//! \class IPluginV2Ext
@@ -583,13 +577,14 @@ class IPluginV2Ext : public IPluginV2
public:
//!
//! \brief Return the DataType of the plugin output at the requested index.
- //! The default behavior should be to return the type of the first input, or DataType::kFLOAT if the layer has no
- //! inputs. The returned data type must have a format that is supported by the plugin. \see supportsFormat()
+ //! The default behavior should be to return the type of the first input, or DataType::kFLOAT if the layer has no inputs.
+ //! The returned data type must have a format that is supported by the plugin.
+ //! \see supportsFormat()
//!
//! \warning DataType:kBOOL not supported.
//!
virtual nvinfer1::DataType getOutputDataType(
- int32_t index, const nvinfer1::DataType* inputTypes, int32_t nbInputs) const TRTNOEXCEPT = 0;
+ int32_t index, nvinfer1::DataType const* inputTypes, int32_t nbInputs) const noexcept = 0;
//! \brief Return true if output tensor is broadcast across a batch.
//!
@@ -602,7 +597,7 @@ public:
//! physical replication of the values.
//!
virtual bool isOutputBroadcastAcrossBatch(
- int32_t outputIndex, const bool* inputIsBroadcasted, int32_t nbInputs) const TRTNOEXCEPT = 0;
+ int32_t outputIndex, bool const* inputIsBroadcasted, int32_t nbInputs) const noexcept = 0;
//! \brief Return true if plugin can use input that is broadcast across batch without replication.
//!
@@ -617,7 +612,7 @@ public:
//!
//! This method is called only for inputs that can be broadcast.
//!
- virtual bool canBroadcastInputAcrossBatch(int32_t inputIndex) const TRTNOEXCEPT = 0;
+ virtual bool canBroadcastInputAcrossBatch(int32_t inputIndex) const noexcept = 0;
//!
//! \brief Configure the layer with input and output data types.
@@ -638,68 +633,83 @@ public:
//!
//! The dimensions passed here do not include the outermost batch size (i.e. for 2-D image networks, they will be
//! 3-dimensional CHW dimensions). When inputIsBroadcast or outputIsBroadcast is true, the outermost batch size for
- //! that input or output should be treated as if it is one. \ref inputIsBroadcast[i] is true only if the input is
- //! semantically broadcast across the batch and \ref canBroadcastInputAcrossBatch(i) returned true. \ref
- //! outputIsBroadcast[i] is true only if \ref isOutputBroadcastAcrossBatch(i) returned true.
+ //! that input or output should be treated as if it is one.
+ //! \ref inputIsBroadcast[i] is true only if the input is semantically broadcast across the batch and
+ //! \ref canBroadcastInputAcrossBatch(i) returned true.
+ //! \ref outputIsBroadcast[i] is true only if \ref isOutputBroadcastAcrossBatch(i) returns true.
//!
//! \warning for the floatFormat field, the values PluginFormat::kCHW4, PluginFormat::kCHW16, and
//! PluginFormat::kCHW32 will not be passed in, this is to keep backward compatibility with TensorRT 5.x series. Use
//! PluginV2IOExt or PluginV2DynamicExt for other PluginFormats.
//!
- virtual void configurePlugin(const Dims* inputDims, int32_t nbInputs, const Dims* outputDims, int32_t nbOutputs,
- const DataType* inputTypes, const DataType* outputTypes, const bool* inputIsBroadcast,
- const bool* outputIsBroadcast, PluginFormat floatFormat, int32_t maxBatchSize) TRTNOEXCEPT = 0;
+ virtual void configurePlugin(Dims const* inputDims, int32_t nbInputs, Dims const* outputDims, int32_t nbOutputs,
+ DataType const* inputTypes, DataType const* outputTypes, bool const* inputIsBroadcast,
+ bool const* outputIsBroadcast, PluginFormat floatFormat, int32_t maxBatchSize) noexcept = 0;
- virtual ~IPluginV2Ext() {}
+ IPluginV2Ext() = default;
+ ~IPluginV2Ext() override = default;
//!
//! \brief Attach the plugin object to an execution context and grant the plugin the access to some context resource.
//!
- //! \param cudnn The cudnn context handle of the execution context
+ //! \param cudnn The CUDNN context handle of the execution context
//! \param cublas The cublas context handle of the execution context
//! \param allocator The allocator used by the execution context
//!
- //! This function is called automatically for each plugin when a new execution context is created.
+ //! This function is called automatically for each plugin when a new execution context is created. If the context
+ //! was created without resources, this method is not called until the resources are assigned. It is also called if
+ //! new resources are assigned to the context.
+ //!
//! If the plugin needs per-context resource, it can be allocated here.
//! The plugin can also get context-owned CUDNN and CUBLAS context here.
//!
- virtual void attachToContext(cudnnContext* /*cudnn*/, cublasContext* /*cublas*/, IGpuAllocator* /*allocator*/) TRTNOEXCEPT {}
+ //! \note In the automotive safety context, the CUDNN and CUBLAS parameters will be nullptr because CUDNN and CUBLAS
+ //! is not used by the safe runtime.
+ //!
+ virtual void attachToContext(cudnnContext* /*cudnn*/, cublasContext* /*cublas*/, IGpuAllocator* /*allocator*/) noexcept {}
//!
//! \brief Detach the plugin object from its execution context.
//!
- //! This function is called automatically for each plugin when a execution context is destroyed.
+ //! This function is called automatically for each plugin when a execution context is destroyed or the context
+ //! resources are unassigned from the context.
+ //!
//! If the plugin owns per-context resource, it can be released here.
//!
- virtual void detachFromContext() TRTNOEXCEPT {}
+ virtual void detachFromContext() noexcept {}
//!
//! \brief Clone the plugin object. This copies over internal plugin parameters as well and returns a new plugin object with these parameters.
//! If the source plugin is pre-configured with configurePlugin(), the returned object should also be pre-configured. The returned object should allow attachToContext() with a new execution context.
//! Cloned plugin objects can share the same per-engine immutable resource (e.g. weights) with the source object (e.g. via ref-counting) to avoid duplication.
//!
- virtual IPluginV2Ext* clone() const _TENSORRT_OVERRIDE TRTNOEXCEPT = 0;
+ IPluginV2Ext* clone() const noexcept override = 0;
protected:
+ IPluginV2Ext(IPluginV2Ext const&) = default;
+ IPluginV2Ext(IPluginV2Ext&&) = default;
+ IPluginV2Ext& operator=(IPluginV2Ext const&) & = default;
+ IPluginV2Ext& operator=(IPluginV2Ext&&) & = default;
+
//!
//! \brief Return the API version with which this plugin was built. The
- //! upper byte reserved by TensorRT and is used to differentiate this from IPlguinV2.
+ //! upper byte reserved by TensorRT and is used to differentiate this from IPluginV2.
//!
//! Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with
//! plugins.
//!
- int32_t getTensorRTVersion() const _TENSORRT_OVERRIDE TRTNOEXCEPT
+ int32_t getTensorRTVersion() const noexcept override
{
- return (static_cast(PluginVersion::kV2_EXT) << 24 | (NV_TENSORRT_VERSION & 0xFFFFFF));
+ return static_cast((static_cast(PluginVersion::kV2_EXT) << 24U)
+ | (static_cast(NV_TENSORRT_VERSION) & 0xFFFFFFU));
}
//!
//! \brief Derived classes should not implement this. In a C++11 API it would be override final.
//!
- void configureWithFormat(const Dims* /*inputDims*/, int32_t /*nbInputs*/, const Dims* /*outputDims*/,
- int32_t /*nbOutputs*/, DataType /*type*/, PluginFormat /*format*/,
- int32_t /*maxBatchSize*/) _TENSORRT_OVERRIDE TRTNOEXCEPT
+ void configureWithFormat(Dims const* /*inputDims*/, int32_t /*nbInputs*/, Dims const* /*outputDims*/,
+ int32_t /*nbOutputs*/, DataType /*type*/, PluginFormat /*format*/, int32_t /*maxBatchSize*/) noexcept override
{
}
};
@@ -728,7 +738,7 @@ public:
//! \param nbOutput Number of output tensors.
//!
virtual void configurePlugin(
- const PluginTensorDesc* in, int32_t nbInput, const PluginTensorDesc* out, int32_t nbOutput) TRTNOEXCEPT = 0;
+ PluginTensorDesc const* in, int32_t nbInput, PluginTensorDesc const* out, int32_t nbOutput) noexcept = 0;
//!
//! \brief Return true if plugin supports the format and datatype for the input/output indexed by pos.
@@ -752,8 +762,8 @@ public:
//! * A definition for a plugin that supports only FP16 NCHW for its two inputs,
//! and FP32 NCHW for its single output:
//!
- //! return inOut.format[pos] == TensorFormat::kLINEAR && (inOut.type[pos] == pos < 2 ? DataType::kHALF :
- //! DataType::kFLOAT);
+ //! return inOut.format[pos] == TensorFormat::kLINEAR &&
+ //! (inOut.type[pos] == pos < 2 ? DataType::kHALF : DataType::kFLOAT);
//!
//! * A definition for a "polymorphic" plugin with two inputs and one output that supports
//! any format or type, but the inputs and output must have the same format and type:
@@ -763,56 +773,39 @@ public:
//! Warning: TensorRT will stop asking for formats once it finds kFORMAT_COMBINATION_LIMIT on combinations.
//!
virtual bool supportsFormatCombination(
- int32_t pos, const PluginTensorDesc* inOut, int32_t nbInputs, int32_t nbOutputs) const TRTNOEXCEPT = 0;
+ int32_t pos, PluginTensorDesc const* inOut, int32_t nbInputs, int32_t nbOutputs) const noexcept = 0;
+
+ IPluginV2IOExt() = default;
+ ~IPluginV2IOExt() override = default;
protected:
+ IPluginV2IOExt(IPluginV2IOExt const&) = default;
+ IPluginV2IOExt(IPluginV2IOExt&&) = default;
+ IPluginV2IOExt& operator=(IPluginV2IOExt const&) & = default;
+ IPluginV2IOExt& operator=(IPluginV2IOExt&&) & = default;
+
//!
//! \brief Return the API version with which this plugin was built. The upper byte is reserved by TensorRT and is
- //! used to differentiate this from IPlguinV2 and IPluginV2Ext.
+ //! used to differentiate this from IPluginV2 and IPluginV2Ext.
//!
//! Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with
//! plugins.
//!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- int32_t getTensorRTVersion() const _TENSORRT_OVERRIDE
+ int32_t getTensorRTVersion() const noexcept override
{
- return (static_cast(PluginVersion::kV2_IOEXT) << 24 | (NV_TENSORRT_VERSION & 0xFFFFFF));
+ return static_cast((static_cast(PluginVersion::kV2_IOEXT) << 24U)
+ | (static_cast(NV_TENSORRT_VERSION) & 0xFFFFFFU));
}
- //!
- //! \brief Deprecated interface inheriting from base class. Derived classes should not implement this. In a C++11
- //! API it would be override final.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- void configureWithFormat(
- const Dims*, int32_t, const Dims*, int32_t, DataType, PluginFormat, int32_t) _TENSORRT_OVERRIDE _TENSORRT_FINAL
+private:
+ // Following are obsolete base class methods, and must not be implemented or used.
+
+ void configurePlugin(Dims const*, int32_t, Dims const*, int32_t, DataType const*, DataType const*, bool const*,
+ bool const*, PluginFormat, int32_t) noexcept override final
{
}
- //!
- //! \brief Deprecated interface inheriting from base class. Derived classes should not implement this. In a C++11
- //! API it would be override final.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- void configurePlugin(const Dims*, int32_t, const Dims*, int32_t, const DataType*, const DataType*, const bool*,
- const bool*, PluginFormat, int32_t) _TENSORRT_OVERRIDE _TENSORRT_FINAL
- {
- }
-
- //!
- //! \brief Deprecated interface inheriting from base class. Derived classes should not implement this. In a C++11
- //! API it would be override final.
- //!
- //! \deprecated Deprecated interface will be removed in TensorRT 8.0.
- //!
- TRT_DEPRECATED
- bool supportsFormat(DataType, PluginFormat) const _TENSORRT_OVERRIDE _TENSORRT_FINAL
+ bool supportsFormat(DataType, PluginFormat) const noexcept override final
{
return false;
}
@@ -825,14 +818,23 @@ protected:
enum class PluginFieldType : int32_t
{
- kFLOAT16 = 0, //!< FP16 field type.
- kFLOAT32 = 1, //!< FP32 field type.
- kFLOAT64 = 2, //!< FP64 field type.
- kINT8 = 3, //!< INT8 field type.
- kINT16 = 4, //!< INT16 field type.
- kINT32 = 5, //!< INT32 field type.
- kCHAR = 6, //!< char field type.
- kDIMS = 7, //!< nvinfer1::Dims field type.
+ //! FP16 field type.
+ kFLOAT16 = 0,
+ //! FP32 field type.
+ kFLOAT32 = 1,
+ //! FP64 field type.
+ kFLOAT64 = 2,
+ //! INT8 field type.
+ kINT8 = 3,
+ //! INT16 field type.
+ kINT16 = 4,
+ //! INT32 field type.
+ kINT32 = 5,
+ //! char field type.
+ kCHAR = 6,
+ //! nvinfer1::Dims field type.
+ kDIMS = 7,
+ //! Unknown field type.
kUNKNOWN = 8
};
@@ -849,22 +851,23 @@ public:
//!
//! \brief Plugin field attribute name
//!
- const char* name{nullptr};
+ AsciiChar const* name;
//!
//! \brief Plugin field attribute data
//!
- const void* data{nullptr};
+ void const* data;
//!
//! \brief Plugin field attribute type
//! \see PluginFieldType
//!
- PluginFieldType type{PluginFieldType::kUNKNOWN};
+ PluginFieldType type;
//!
//! \brief Number of data entries in the Plugin attribute
//!
- int32_t length{0};
+ int32_t length;
- PluginField(const char* name_ = nullptr, const void* data_ = nullptr, const PluginFieldType type_ = PluginFieldType::kUNKNOWN, int32_t length_ = 0)
+ PluginField(AsciiChar const* const name_ = nullptr, void const* const data_ = nullptr,
+ PluginFieldType const type_ = PluginFieldType::kUNKNOWN, int32_t const length_ = 0) noexcept
: name(name_)
, data(data_)
, type(type_)
@@ -873,10 +876,13 @@ public:
}
};
+//! Plugin field collection struct.
struct PluginFieldCollection
{
- int32_t nbFields; //!< Number of PluginField entries
- const PluginField* fields; //!< Pointer to PluginField entries
+ //! Number of PluginField entries.
+ int32_t nbFields;
+ //! Pointer to PluginField entries.
+ PluginField const* fields;
};
//!
@@ -893,7 +899,7 @@ public:
//!
//! \brief Return the version of the API the plugin creator was compiled with.
//!
- virtual int32_t getTensorRTVersion() const TRTNOEXCEPT
+ virtual int32_t getTensorRTVersion() const noexcept
{
return NV_TENSORRT_VERSION;
}
@@ -901,28 +907,28 @@ public:
//!
//! \brief Return the plugin name.
//!
- virtual const char* getPluginName() const TRTNOEXCEPT = 0;
+ virtual AsciiChar const* getPluginName() const noexcept = 0;
//!
//! \brief Return the plugin version.
//!
- virtual const char* getPluginVersion() const TRTNOEXCEPT = 0;
+ virtual AsciiChar const* getPluginVersion() const noexcept = 0;
//!
//! \brief Return a list of fields that needs to be passed to createPlugin.
//! \see PluginFieldCollection
//!
- virtual const PluginFieldCollection* getFieldNames() TRTNOEXCEPT = 0;
+ virtual PluginFieldCollection const* getFieldNames() noexcept = 0;
//!
//! \brief Return a plugin object. Return nullptr in case of error.
//!
- virtual IPluginV2* createPlugin(const char* name, const PluginFieldCollection* fc) TRTNOEXCEPT = 0;
+ virtual IPluginV2* createPlugin(AsciiChar const* name, PluginFieldCollection const* fc) noexcept = 0;
//!
//! \brief Called during deserialization of plugin layer. Return a plugin object.
//!
- virtual IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) TRTNOEXCEPT = 0;
+ virtual IPluginV2* deserializePlugin(AsciiChar const* name, void const* serialData, size_t serialLength) noexcept = 0;
//!
//! \brief Set the namespace of the plugin creator based on the plugin
@@ -930,14 +936,21 @@ public:
//!
//! \see IPluginRegistry::registerCreator()
//!
- virtual void setPluginNamespace(const char* pluginNamespace) TRTNOEXCEPT = 0;
+ virtual void setPluginNamespace(AsciiChar const* pluginNamespace) noexcept = 0;
//!
//! \brief Return the namespace of the plugin creator object.
//!
- virtual const char* getPluginNamespace() const TRTNOEXCEPT = 0;
+ virtual AsciiChar const* getPluginNamespace() const noexcept = 0;
- virtual ~IPluginCreator() {}
+ IPluginCreator() = default;
+ virtual ~IPluginCreator() = default;
+
+protected:
+ IPluginCreator(IPluginCreator const&) = default;
+ IPluginCreator(IPluginCreator&&) = default;
+ IPluginCreator& operator=(IPluginCreator const&) & = default;
+ IPluginCreator& operator=(IPluginCreator&&) & = default;
};
//!
@@ -954,6 +967,9 @@ public:
//!
//! \warning Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
//!
+//! \warning In the automotive safety context, be sure to call IPluginRegistry::setErrorRecorder() to register
+//! an error recorder with the registry before using other methods in the registry.
+//!
class IPluginRegistry
{
@@ -962,22 +978,30 @@ public:
//! \brief Register a plugin creator. Returns false if one with same type
//! is already registered.
//!
- virtual bool registerCreator(IPluginCreator& creator, const char* pluginNamespace) noexcept = 0;
+ virtual bool registerCreator(IPluginCreator& creator, AsciiChar const* const pluginNamespace) noexcept = 0;
//!
//! \brief Return all the registered plugin creators and the number of
//! registered plugin creators. Returns nullptr if none found.
//!
- virtual IPluginCreator* const* getPluginCreatorList(int32_t* numCreators) const noexcept = 0;
+ virtual IPluginCreator* const* getPluginCreatorList(int32_t* const numCreators) const noexcept = 0;
//!
- //! \brief Return plugin creator based on plugin type, version and
+ //! \brief Return plugin creator based on plugin name, version, and
//! namespace associated with plugin during network creation.
//!
- virtual IPluginCreator* getPluginCreator(const char* pluginType, const char* pluginVersion, const char* pluginNamespace = "") noexcept = 0;
+ virtual IPluginCreator* getPluginCreator(
+ AsciiChar const* const pluginName, AsciiChar const* const pluginVersion, AsciiChar const* const pluginNamespace = "") noexcept
+ = 0;
+
+ IPluginRegistry() = default;
+ IPluginRegistry(IPluginRegistry const&) = delete;
+ IPluginRegistry(IPluginRegistry&&) = delete;
+ IPluginRegistry& operator=(IPluginRegistry const&) & = delete;
+ IPluginRegistry& operator=(IPluginRegistry&&) & = delete;
protected:
- virtual ~IPluginRegistry() noexcept {}
+ virtual ~IPluginRegistry() noexcept = default;
public:
//!
@@ -990,12 +1014,12 @@ public:
//!
//! \param recorder The error recorder to register with this interface.
//
- //! \see getErrorRecorder
+ //! \see getErrorRecorder()
//!
- virtual void setErrorRecorder(IErrorRecorder* recorder) noexcept = 0;
+ virtual void setErrorRecorder(IErrorRecorder* const recorder) noexcept = 0;
//!
- //! \brief set the ErrorRecorder assigned to this interface.
+ //! \brief Set the ErrorRecorder assigned to this interface.
//!
//! Retrieves the assigned error recorder object for the given class. A default error recorder does not exist,
//! so a nullptr will be returned if setErrorRecorder has not been called, or an ErrorRecorder has not been
@@ -1003,27 +1027,40 @@ public:
//!
//! \return A pointer to the IErrorRecorder object that has been registered.
//!
- //! \see setErrorRecorder
+ //! \see setErrorRecorder()
//!
virtual IErrorRecorder* getErrorRecorder() const noexcept = 0;
+
+ //!
+ //! \brief Deregister a previously registered plugin creator.
+ //!
+ //! Since there may be a desire to limit the number of plugins,
+ //! this function provides a mechanism for removing plugin creators registered in TensorRT.
+ //! The plugin creator that is specified by \p creator is removed from TensorRT and no longer tracked.
+ //!
+ //! \return True if the plugin creator was deregistered, false if it was not found in the registry or otherwise
+ //! could
+ //! not be deregistered.
+ //!
+ virtual bool deregisterCreator(IPluginCreator const& creator) noexcept = 0;
};
-//!
-//! \enum TensorLocation
-//! \brief The location for tensor data storage, device or host.
-//!
-enum class TensorLocation : int32_t
+enum class AllocatorFlag : int32_t
{
- kDEVICE = 0, //!< Data stored on device.
- kHOST = 1, //!< Data stored on host.
+ kRESIZABLE = 0, //!< TensorRT may call realloc() on this allocation
};
-//! Maximum number of elements in TensorLocation enum. \see TensorLocation
+namespace impl
+{
+//! Maximum number of elements in AllocatorFlag enum. \see AllocatorFlag
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 2;
-}
+ static constexpr int32_t kVALUE = 1;
+};
+} // namespace impl
+
+using AllocatorFlags = uint32_t;
//!
//! \class IGpuAllocator
@@ -1034,10 +1071,10 @@ class IGpuAllocator
{
public:
//!
- //! A callback implemented by the application to handle acquisition of GPU memory.
+ //! A thread-safe callback implemented by the application to handle acquisition of GPU memory.
//!
//! \param size The size of the memory required.
- //! \param alignment The required alignment of memory. Alignment will zero
+ //! \param alignment The required alignment of memory. Alignment will be zero
//! or a power of 2 not exceeding the alignment guaranteed by cudaMalloc.
//! Thus this allocator can be safely implemented with cudaMalloc/cudaFree.
//! An alignment value of zero indicates any alignment is acceptable.
@@ -1047,22 +1084,69 @@ public:
//!
//! If an allocation request cannot be satisfied, nullptr should be returned.
//!
- virtual void* allocate(uint64_t size, uint64_t alignment, uint32_t flags) TRTNOEXCEPT = 0;
+ //! \note The implementation must guarantee thread safety for concurrent allocate/free/reallocate
+ //! requests.
+ //!
+ virtual void* allocate(uint64_t const size, uint64_t const alignment, AllocatorFlags const flags) noexcept = 0;
//!
- //! A callback implemented by the application to handle release of GPU memory.
+ //! A thread-safe callback implemented by the application to handle release of GPU memory.
//!
//! TensorRT may pass a nullptr to this function if it was previously returned by allocate().
//!
//! \param memory The acquired memory.
//!
- virtual void free(void* memory) TRTNOEXCEPT = 0;
+ //! \note The implementation must guarantee thread safety for concurrent allocate/free/reallocate
+ //! requests.
+ //!
+ virtual void free(void* const memory) noexcept = 0;
//!
//! Destructor declared virtual as general good practice for a class with virtual methods.
//! TensorRT never calls the destructor for an IGpuAllocator defined by the application.
//!
- virtual ~IGpuAllocator() {}
+ virtual ~IGpuAllocator() = default;
+ IGpuAllocator() = default;
+
+ //!
+ //! A thread-safe callback implemented by the application to resize an existing allocation.
+ //!
+ //! Only allocations which were allocated with AllocatorFlag::kRESIZABLE will be resized.
+ //!
+ //! Options are one of:
+ //! * resize in place leaving min(oldSize, newSize) bytes unchanged and return the original address
+ //! * move min(oldSize, newSize) bytes to a new location of sufficient size and return its address
+ //! * return nullptr, to indicate that the request could not be fulfilled.
+ //!
+ //! If nullptr is returned, TensorRT will assume that resize() is not implemented, and that the
+ //! allocation at baseAddr is still valid.
+ //!
+ //! This method is made available for use cases where delegating the resize
+ //! strategy to the application provides an opportunity to improve memory management.
+ //! One possible implementation is to allocate a large virtual device buffer and
+ //! progressively commit physical memory with cuMemMap. CU_MEM_ALLOC_GRANULARITY_RECOMMENDED
+ //! is suggested in this case.
+ //!
+ //! TensorRT may call realloc to increase the buffer by relatively small amounts.
+ //!
+ //! \param baseAddr the address of the original allocation.
+ //! \param alignment The alignment used by the original allocation.
+ //! \param newSize The new memory size required.
+ //! \return the address of the reallocated memory
+ //!
+ //! \note The implementation must guarantee thread safety for concurrent allocate/free/reallocate
+ //! requests.
+ //!
+ virtual void* reallocate(void* baseAddr, uint64_t alignment, uint64_t newSize) noexcept
+ {
+ return nullptr;
+ }
+
+protected:
+ IGpuAllocator(IGpuAllocator const&) = default;
+ IGpuAllocator(IGpuAllocator&&) = default;
+ IGpuAllocator& operator=(IGpuAllocator const&) & = default;
+ IGpuAllocator& operator=(IGpuAllocator&&) & = default;
};
//!
@@ -1070,8 +1154,9 @@ public:
//!
//! \brief Application-implemented logging interface for the builder, engine and runtime.
//!
-//! Note that although a logger is passed on creation to each instance of a IBuilder or safe::IRuntime interface, the logger is internally considered a singleton, and thus
-//! multiple instances of safe::IRuntime and/or IBuilder must all use the same logger.
+//! Note that although a logger is passed on creation to each instance of a IBuilder or IRuntime interfaces, the logger
+//! is internally considered a singleton, and thus multiple instances of IRuntime and/or IBuilder must all use the same
+//! logger.
//!
class ILogger
{
@@ -1083,11 +1168,16 @@ public:
//!
enum class Severity : int32_t
{
- kINTERNAL_ERROR = 0, //!< Internal error has occurred. Execution is unrecoverable.
- kERROR = 1, //!< Application error has occurred.
- kWARNING = 2, //!< Application error has been discovered. TensorRT has recovered or fallen back to a default.
- kINFO = 3, //!< Informational messages with instructional information.
- kVERBOSE = 4, //!< Verbose messages with debugging information.
+ //! An internal error has occurred. Execution is unrecoverable.
+ kINTERNAL_ERROR = 0,
+ //! An application error has occurred.
+ kERROR = 1,
+ //! An application error has been discovered, but TensorRT has recovered or fallen back to a default.
+ kWARNING = 2,
+ //! Informational messages with instructional information.
+ kINFO = 3,
+ //! Verbose messages with debugging information.
+ kVERBOSE = 4,
};
//!
@@ -1096,17 +1186,28 @@ public:
//! \param severity The severity of the message.
//! \param msg The log message, null terminated.
//!
- virtual void log(Severity severity, const char* msg) TRTNOEXCEPT = 0;
+ virtual void log(Severity severity, AsciiChar const* msg) noexcept = 0;
- virtual ~ILogger() {}
+ ILogger() = default;
+ virtual ~ILogger() = default;
+
+protected:
+ ILogger(ILogger const&) = default;
+ ILogger(ILogger&&) = default;
+ ILogger& operator=(ILogger const&) & = default;
+ ILogger& operator=(ILogger&&) & = default;
};
+namespace impl
+{
//! Maximum number of elements in ILogger::Severity enum. \see ILogger::Severity
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 5;
-}
+ //! Declaration of kVALUE that represents maximum number of elements in ILogger::Severity enum
+ static constexpr int32_t kVALUE = 5;
+};
+} // namespace impl
//!
//! \enum ErrorCode
@@ -1121,18 +1222,19 @@ enum class ErrorCode : int32_t
kSUCCESS = 0,
//!
- //! An error that does not fall into any other category. This error is included for forward compatibility
+ //! An error that does not fall into any other category. This error is included for forward compatibility.
//!
kUNSPECIFIED_ERROR = 1,
//!
- //! A non-recoverable TensorRT error occurred.
+ //! A non-recoverable TensorRT error occurred. TensorRT is in an invalid internal state when this error is
+ //! emitted and any further calls to TensorRT will result in undefined behavior.
//!
kINTERNAL_ERROR = 2,
//!
//! An argument passed to the function is invalid in isolation.
- //! This is a violation of the API contract
+ //! This is a violation of the API contract.
//!
kINVALID_ARGUMENT = 3,
@@ -1171,6 +1273,7 @@ enum class ErrorCode : int32_t
//! of this error are NaN squashing or integer overflow. In a dynamic system, the data can be thrown away and the
//! next frame can be processed or execution can be retried.
//! This is either a data corruption error, an input error, or a range error.
+ //! This is not used in safety but may be used in standard.
//!
kFAILED_COMPUTATION = 8,
@@ -1202,12 +1305,16 @@ enum class ErrorCode : int32_t
};
+namespace impl
+{
//! Maximum number of elements in ErrorCode enum. \see ErrorCode
template <>
-constexpr inline int32_t EnumMax()
+struct EnumMaxImpl
{
- return 11;
-}
+ //! Declaration of kVALUE
+ static constexpr int32_t kVALUE = 11;
+};
+} // namespace impl
//!
//! \class IErrorRecorder
@@ -1236,16 +1343,22 @@ public:
//!
//! A typedef of a c-style string for reporting error descriptions.
//!
- using ErrorDesc = const char*;
+ using ErrorDesc = char const*;
+
+ //!
+ //! The length limit for an error description, excluding the '\0' string terminator.
+ //!
+ static constexpr size_t kMAX_DESC_LENGTH = 127U;
//!
//! A typedef of a 32bit integer for reference counting.
//!
using RefCount = int32_t;
- virtual ~IErrorRecorder() noexcept {};
+ IErrorRecorder() = default;
+ virtual ~IErrorRecorder() noexcept = default;
- // Public API’s used to retrieve information from the error recorder.
+ // Public API used to retrieve information from the error recorder.
//!
//! \brief Return the number of errors
@@ -1314,10 +1427,10 @@ public:
//!
virtual void clear() noexcept = 0;
- // API’s used by TensorRT to report Error information to the application.
+ // API used by TensorRT to report Error information to the application.
//!
- //! \brief report an error to the error recorder with the corresponding enum and description.
+ //! \brief Report an error to the error recorder with the corresponding enum and description.
//!
//! \param val The error code enum that is being reported.
//! \param desc The string description of the error.
@@ -1357,54 +1470,21 @@ public:
//!
virtual RefCount decRefCount() noexcept = 0;
+protected:
+ IErrorRecorder(IErrorRecorder const&) = default;
+ IErrorRecorder(IErrorRecorder&&) = default;
+ IErrorRecorder& operator=(IErrorRecorder const&) & = default;
+ IErrorRecorder& operator=(IErrorRecorder&&) & = default;
+
}; // class IErrorRecorder
} // namespace nvinfer1
-//!
-//! Internal C entry point for creating safe::IRuntime.
-//! @private
-//!
-extern "C" TENSORRTAPI void* createSafeInferRuntime_INTERNAL(void* logger, int32_t version);
-
-//!
-//! \brief Return the logger object.
-//!
-extern "C" TENSORRTAPI nvinfer1::ILogger* getLogger();
-
//!
//! \brief Return the library version number.
//!
//! The format is as for TENSORRT_VERSION: (TENSORRT_MAJOR * 1000) + (TENSORRT_MINOR * 100) + TENSOR_PATCH.
//!
-extern "C" TENSORRTAPI int32_t getInferLibVersion();
-
-//!
-//! \brief Return the plugin registry
-//!
-extern "C" TENSORRTAPI nvinfer1::IPluginRegistry* getPluginRegistry();
-
-namespace nvinfer1
-{
-
-//!
-//! \brief Register the plugin creator to the registry
-//! The static registry object will be instantiated when the plugin library is
-//! loaded. This static object will register all creators available in the
-//! library to the registry.
-//!
-template
-class PluginRegistrar
-{
-public:
- PluginRegistrar() { getPluginRegistry()->registerCreator(instance, ""); }
-private:
- T instance{};
-};
-
-#define REGISTER_TENSORRT_PLUGIN(name) \
- static nvinfer1::PluginRegistrar pluginRegistrar##name {}
-
-} // namespace nvinfer1
+extern "C" TENSORRTAPI int32_t getInferLibVersion() noexcept;
#endif // NV_INFER_RUNTIME_COMMON_H
diff --git a/include/NvInferVersion.h b/include/NvInferVersion.h
index 807f3cf3..4779594f 100644
--- a/include/NvInferVersion.h
+++ b/include/NvInferVersion.h
@@ -23,13 +23,13 @@
#ifndef NV_INFER_VERSION_H
#define NV_INFER_VERSION_H
-#define NV_TENSORRT_MAJOR 7 //!< TensorRT major version.
-#define NV_TENSORRT_MINOR 2 //!< TensorRT minor version.
-#define NV_TENSORRT_PATCH 3 //!< TensorRT patch version.
-#define NV_TENSORRT_BUILD 4 //!< TensorRT build number.
+#define NV_TENSORRT_MAJOR 8 //!< TensorRT major version.
+#define NV_TENSORRT_MINOR 0 //!< TensorRT minor version.
+#define NV_TENSORRT_PATCH 1 //!< TensorRT patch version.
+#define NV_TENSORRT_BUILD 6 //!< TensorRT build number.
-#define NV_TENSORRT_SONAME_MAJOR 7 //!< Shared object library major version number.
-#define NV_TENSORRT_SONAME_MINOR 2 //!< Shared object library minor version number.
-#define NV_TENSORRT_SONAME_PATCH 3 //!< Shared object library patch version number.
+#define NV_TENSORRT_SONAME_MAJOR 8 //!< Shared object library major version number.
+#define NV_TENSORRT_SONAME_MINOR 0 //!< Shared object library minor version number.
+#define NV_TENSORRT_SONAME_PATCH 1 //!< Shared object library patch version number.
#endif // NV_INFER_VERSION_H
diff --git a/include/NvOnnxConfig.h b/include/NvOnnxConfig.h
index 46abeeaf..e3a17068 100644
--- a/include/NvOnnxConfig.h
+++ b/include/NvOnnxConfig.h
@@ -44,24 +44,23 @@ namespace nvonnxparser
//!
class IOnnxConfig
{
-protected:
- virtual ~IOnnxConfig() {}
-
public:
+ virtual ~IOnnxConfig() noexcept = default;
//!
//! \typedef Verbosity
//! \brief Defines Verbosity level.
//!
- typedef int Verbosity;
+ typedef int32_t Verbosity;
//!
//! \brief Set the Model Data Type.
//!
- //! Sets the Model DataType, one of the following: float -d 32 (default), half precision -d 16, and int8 -d 8 data types.
+ //! Sets the Model DataType, one of the following: float -d 32 (default), half precision -d 16, and int8 -d 8 data
+ //! types.
//!
//! \see getModelDtype()
//!
- virtual void setModelDtype(const nvinfer1::DataType) TRTNOEXCEPT = 0;
+ virtual void setModelDtype(const nvinfer1::DataType) noexcept = 0;
//!
//! \brief Get the Model Data Type.
@@ -70,7 +69,7 @@ public:
//!
//! \see setModelDtype() and #DataType
//!
- virtual nvinfer1::DataType getModelDtype() const TRTNOEXCEPT = 0;
+ virtual nvinfer1::DataType getModelDtype() const noexcept = 0;
//!
//! \brief Get the Model FileName.
@@ -79,7 +78,7 @@ public:
//!
//! \see setModelFileName()
//!
- virtual const char* getModelFileName() const TRTNOEXCEPT = 0;
+ virtual const char* getModelFileName() const noexcept = 0;
//!
//! \brief Set the Model File Name.
@@ -92,7 +91,7 @@ public:
//!
//! \see getModelFileName()
//!
- virtual void setModelFileName(const char* onnxFilename) TRTNOEXCEPT = 0;
+ virtual void setModelFileName(const char* onnxFilename) noexcept = 0;
//!
//! \brief Get the Verbosity Level.
@@ -101,27 +100,40 @@ public:
//!
//! \see addVerbosity(), reduceVerbosity()
//!
- virtual Verbosity getVerbosityLevel() const TRTNOEXCEPT = 0;
+ virtual Verbosity getVerbosityLevel() const noexcept = 0;
//!
//! \brief Increase the Verbosity Level.
//!
//! \return The Verbosity Level.
//!
- //! \see addVerbosity(), reduceVerbosity(), setVerbosity(Verbosity)
+ //! \see reduceVerbosity(), setVerbosity(Verbosity)
//!
- virtual void addVerbosity() TRTNOEXCEPT = 0; //!< Increase verbosity Level.
- virtual void reduceVerbosity() TRTNOEXCEPT = 0; //!< Decrease verbosity Level.
- virtual void setVerbosityLevel(Verbosity) TRTNOEXCEPT = 0; //!< Set to specific verbosity Level.
+ virtual void addVerbosity() noexcept = 0;
+
+ //!
+ //! \brief Reduce the Verbosity Level.
+ //!
+ //! \see addVerbosity(), setVerbosity(Verbosity)
+ //!
+ virtual void reduceVerbosity() noexcept = 0;
+
+ //!
+ //! \brief Set to specific verbosity Level.
+ //!
+ //! \see addVerbosity(), reduceVerbosity()
+ //!
+ virtual void setVerbosityLevel(Verbosity) noexcept = 0;
//!
//! \brief Returns the File Name of the Network Description as a Text File.
//!
- //! \return Return the name of the file containing the network description converted to a plain text, used for debugging purposes.
+ //! \return Return the name of the file containing the network description converted to a plain text, used for
+ //! debugging purposes.
//!
//! \see setTextFilename()
//!
- virtual const char* getTextFileName() const TRTNOEXCEPT = 0;
+ virtual const char* getTextFileName() const noexcept = 0;
//!
//! \brief Set the File Name of the Network Description as a Text File.
@@ -134,16 +146,17 @@ public:
//!
//! \see getTextFilename()
//!
- virtual void setTextFileName(const char* textFileName) TRTNOEXCEPT = 0;
+ virtual void setTextFileName(const char* textFileName) noexcept = 0;
//!
//! \brief Get the File Name of the Network Description as a Text File, including the weights.
//!
- //! \return Return the name of the file containing the network description converted to a plain text, used for debugging purposes.
+ //! \return Return the name of the file containing the network description converted to a plain text, used for
+ //! debugging purposes.
//!
//! \see setFullTextFilename()
//!
- virtual const char* getFullTextFileName() const TRTNOEXCEPT = 0;
+ virtual const char* getFullTextFileName() const noexcept = 0;
//!
//! \brief Set the File Name of the Network Description as a Text File, including the weights.
@@ -156,7 +169,7 @@ public:
//!
//! \see getFullTextFilename()
//!
- virtual void setFullTextFileName(const char* fullTextFileName) TRTNOEXCEPT = 0;
+ virtual void setFullTextFileName(const char* fullTextFileName) noexcept = 0;
//!
//! \brief Get whether the layer information will be printed.
@@ -165,19 +178,23 @@ public:
//!
//! \see setPrintLayerInfo()
//!
- virtual bool getPrintLayerInfo() const TRTNOEXCEPT = 0;
+ virtual bool getPrintLayerInfo() const noexcept = 0;
//!
//! \brief Set whether the layer information will be printed.
//!
//! \see getPrintLayerInfo()
//!
- virtual void setPrintLayerInfo(bool) TRTNOEXCEPT = 0;
+ virtual void setPrintLayerInfo(bool) noexcept = 0;
//!
//! \brief Destroy IOnnxConfig object.
//!
- virtual void destroy() TRTNOEXCEPT = 0;
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ //! \warning Calling destroy on a managed pointer will result in a double-free error.
+ //!
+ TRT_DEPRECATED virtual void destroy() noexcept = 0;
}; // class IOnnxConfig
diff --git a/include/NvUffParser.h b/include/NvUffParser.h
index 6d0996a7..51bd6e84 100644
--- a/include/NvUffParser.h
+++ b/include/NvUffParser.h
@@ -81,7 +81,7 @@ public:
FieldType type = FieldType::kUNKNOWN;
int32_t length = 1;
- FieldMap(const char* name, const void* data, const FieldType type, int32_t length = 1) TRTNOEXCEPT;
+ FieldMap(const char* name, const void* data, const FieldType type, int32_t length = 1);
};
struct FieldCollection
@@ -90,58 +90,6 @@ struct FieldCollection
const FieldMap* fields;
};
-//!
-//! \class IPluginFactory
-//!
-//! \brief Plugin factory used to configure plugins.
-//!
-class IPluginFactory
-{
-public:
- //!
- //! \brief A user implemented function that determines if a layer configuration is provided by an IPlugin.
- //!
- //! \param layerName Name of the layer which the user wishes to validate.
- //!
- virtual bool isPlugin(const char* layerName) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Creates a plugin.
- //!
- //! \param layerName Name of layer associated with the plugin.
- //! \param weights Weights used for the layer.
- //! \param nbWeights Number of weights.
- //! \param fc A collection of FieldMaps used as layer parameters for different plugin layers.
- //!
- //! \see FieldCollection
- //!
- virtual nvinfer1::IPlugin* createPlugin(const char* layerName, const nvinfer1::Weights* weights, int32_t nbWeights,
- const FieldCollection fc) TRTNOEXCEPT = 0;
-
- virtual ~IPluginFactory() {}
-};
-
-//!
-//! \class IPluginFactoryExt
-//!
-//! \brief Plugin factory used to configure plugins with added support for TRT versioning.
-//!
-class IPluginFactoryExt : public IPluginFactory
-{
-public:
- virtual int32_t getVersion() const TRTNOEXCEPT
- {
- return NV_TENSORRT_VERSION;
- }
-
- //!
- //! \brief A user implemented function that determines if a layer configuration is provided by an IPluginExt.
- //!
- //! \param layerName Name of the layer which the user wishes to validate.
- //!
- virtual bool isPluginExt(const char* layerName) TRTNOEXCEPT = 0;
-};
-
//!
//! \class IUffParser
//!
@@ -159,14 +107,14 @@ public:
//! \param inputDims Input dimensions.
//! \param inputOrder Input order on which the framework input was originally.
//!
- virtual bool registerInput(const char* inputName, nvinfer1::Dims inputDims, UffInputOrder inputOrder) TRTNOEXCEPT = 0;
+ virtual bool registerInput(const char* inputName, nvinfer1::Dims inputDims, UffInputOrder inputOrder) noexcept = 0;
//!
//! \brief Register an output name of a UFF network.
//!
//! \param outputName Output name.
//!
- virtual bool registerOutput(const char* outputName) TRTNOEXCEPT = 0;
+ virtual bool registerOutput(const char* outputName) noexcept = 0;
//!
//! \brief Parse a UFF file.
@@ -175,9 +123,9 @@ public:
//! \param network Network in which the UFFParser will fill the layers.
//! \param weightsType The type on which the weights will transformed in.
//!
- virtual bool parse(const char* file,
- nvinfer1::INetworkDefinition& network,
- nvinfer1::DataType weightsType=nvinfer1::DataType::kFLOAT) TRTNOEXCEPT = 0;
+ virtual bool parse(const char* file, nvinfer1::INetworkDefinition& network,
+ nvinfer1::DataType weightsType = nvinfer1::DataType::kFLOAT) noexcept
+ = 0;
//!
//! \brief Parse a UFF buffer, useful if the file already live in memory.
@@ -187,48 +135,36 @@ public:
//! \param network Network in which the UFFParser will fill the layers.
//! \param weightsType The type on which the weights will transformed in.
//!
- virtual bool parseBuffer(const char* buffer, std::size_t size,
- nvinfer1::INetworkDefinition& network,
- nvinfer1::DataType weightsType=nvinfer1::DataType::kFLOAT) TRTNOEXCEPT = 0;
+ virtual bool parseBuffer(const char* buffer, std::size_t size, nvinfer1::INetworkDefinition& network,
+ nvinfer1::DataType weightsType = nvinfer1::DataType::kFLOAT) noexcept
+ = 0;
- virtual void destroy() TRTNOEXCEPT = 0;
+ //!
+ //! \deprecated Deprecated interface will be removed in TensorRT 10.0.
+ //!
+ TRT_DEPRECATED virtual void destroy() noexcept = 0;
//!
//! \brief Return Version Major of the UFF.
//!
- virtual int32_t getUffRequiredVersionMajor() TRTNOEXCEPT = 0;
+ virtual int32_t getUffRequiredVersionMajor() noexcept = 0;
//!
//! \brief Return Version Minor of the UFF.
//!
- virtual int32_t getUffRequiredVersionMinor() TRTNOEXCEPT = 0;
+ virtual int32_t getUffRequiredVersionMinor() noexcept = 0;
//!
//! \brief Return Patch Version of the UFF.
//!
- virtual int32_t getUffRequiredVersionPatch() TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the IPluginFactory used to create the user defined plugins.
- //!
- //! \param factory Pointer to an instance of the user implmentation of IPluginFactory.
- //!
- virtual void setPluginFactory(IPluginFactory* factory) TRTNOEXCEPT = 0;
-
- //!
- //! \brief Set the IPluginFactoryExt used to create the user defined pluginExts.
- //!
- //! \param factory Pointer to an instance of the user implmentation of IPluginFactoryExt.
- //!
- virtual void setPluginFactoryExt(IPluginFactoryExt* factory) TRTNOEXCEPT = 0;
+ virtual int32_t getUffRequiredVersionPatch() noexcept = 0;
//!
//! \brief Set the namespace used to lookup and create plugins in the network.
//!
- virtual void setPluginNamespace(const char* libNamespace) TRTNOEXCEPT = 0;
+ virtual void setPluginNamespace(const char* libNamespace) noexcept = 0;
-protected:
- virtual ~IUffParser() {}
+ virtual ~IUffParser() noexcept = default;
public:
//!
@@ -239,23 +175,25 @@ public:
//! recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if
//! a recorder has been registered.
//!
+ //! If an error recorder is not set, messages will be sent to the global log stream.
+ //!
//! \param recorder The error recorder to register with this interface.
//
- //! \see getErrorRecorder
+ //! \see getErrorRecorder()
//!
- virtual void setErrorRecorder(nvinfer1::IErrorRecorder* recorder) TRTNOEXCEPT = 0;
+ virtual void setErrorRecorder(nvinfer1::IErrorRecorder* recorder) noexcept = 0;
//!
//! \brief get the ErrorRecorder assigned to this interface.
//!
- //! Retrieves the assigned error recorder object for the given class. A default error recorder does not exist,
- //! so a nullptr will be returned if setErrorRecorder has not been called.
+ //! Retrieves the assigned error recorder object for the given class. A
+ //! nullptr will be returned if setErrorRecorder has not been called.
//!
//! \return A pointer to the IErrorRecorder object that has been registered.
//!
- //! \see setErrorRecorder
+ //! \see setErrorRecorder()
//!
- virtual nvinfer1::IErrorRecorder* getErrorRecorder() const TRTNOEXCEPT = 0;
+ virtual nvinfer1::IErrorRecorder* getErrorRecorder() const noexcept = 0;
};
//!
@@ -265,22 +203,24 @@ public:
//!
//! \see nvuffparser::IUffParser
//!
-TENSORRTAPI IUffParser* createUffParser() TRTNOEXCEPT;
+//! \deprecated IUffParser will be removed in TensorRT 9.0. Plan to migrate your workflow to
+//! use nvonnxparser::IParser for deployment.
+//!
+TENSORRTAPI IUffParser* createUffParser() noexcept;
//!
//! \brief Shuts down protocol buffers library.
//!
//! \note No part of the protocol buffers library can be used after this function is called.
//!
-TENSORRTAPI void shutdownProtobufLibrary(void) TRTNOEXCEPT;
+TENSORRTAPI void shutdownProtobufLibrary(void) noexcept;
} // namespace nvuffparser
-
//!
//! Internal C entry point for creating IUffParser
//! @private
//!
-extern "C" TENSORRTAPI void* createNvUffParser_INTERNAL() TRTNOEXCEPT;
+extern "C" TENSORRTAPI void* createNvUffParser_INTERNAL() noexcept;
#endif /* !NV_UFF_PARSER_H */
diff --git a/include/NvUtils.h b/include/NvUtils.h
index ed6f51c2..4631b83c 100644
--- a/include/NvUtils.h
+++ b/include/NvUtils.h
@@ -72,8 +72,10 @@ namespace utils
//!
//! \return True on success, false on failure.
//!
-TENSORRTAPI bool reshapeWeights(
- const Weights& input, const int32_t* shape, const int32_t* shapeOrder, void* data, int32_t nbDims);
+//! \warning This file will be removed in TensorRT 10.0.
+//!
+TRT_DEPRECATED TENSORRTAPI bool reshapeWeights(
+ const Weights& input, int32_t const* shape, int32_t const* shapeOrder, void* data, int32_t nbDims) noexcept;
//!
//! \param input The input data to re-order.
@@ -116,7 +118,10 @@ TENSORRTAPI bool reshapeWeights(
//!
//! \see reshapeWeights()
//!
-TENSORRTAPI bool reorderSubBuffers(void* input, const int32_t* order, int32_t num, int32_t size);
+//! \warning This file will be removed in TensorRT 10.0.
+//!
+TRT_DEPRECATED TENSORRTAPI bool reorderSubBuffers(
+ void* input, int32_t const* order, int32_t num, int32_t size) noexcept;
//!
//! \param input The input data to transpose.
@@ -129,7 +134,10 @@ TENSORRTAPI bool reorderSubBuffers(void* input, const int32_t* order, int32_t nu
//!
//! \return True on success, false on failure.
//!
-TENSORRTAPI bool transposeSubBuffers(void* input, DataType type, int32_t num, int32_t height, int32_t width);
+//! \warning This file will be removed in TensorRT 10.0.
+//!
+TRT_DEPRECATED TENSORRTAPI bool transposeSubBuffers(
+ void* input, DataType type, int32_t num, int32_t height, int32_t width) noexcept;
} // namespace utils
} // namespace nvinfer1
diff --git a/parsers/caffe/NvCaffeParser.cpp b/parsers/caffe/NvCaffeParser.cpp
index 046b87f7..afa96bd2 100644
--- a/parsers/caffe/NvCaffeParser.cpp
+++ b/parsers/caffe/NvCaffeParser.cpp
@@ -19,17 +19,17 @@
using namespace nvcaffeparser1;
-void nvcaffeparser1::shutdownProtobufLibrary()
+void nvcaffeparser1::shutdownProtobufLibrary() noexcept
{
google::protobuf::ShutdownProtobufLibrary();
}
-extern "C" void* createNvCaffeParser_INTERNAL()
+extern "C" void* createNvCaffeParser_INTERNAL() noexcept
{
return nvcaffeparser1::createCaffeParser();
}
-ICaffeParser* nvcaffeparser1::createCaffeParser()
+ICaffeParser* nvcaffeparser1::createCaffeParser() noexcept
{
return new CaffeParser;
}
diff --git a/parsers/caffe/binaryProtoBlob.h b/parsers/caffe/binaryProtoBlob.h
index 8e4553cc..28cfa36f 100644
--- a/parsers/caffe/binaryProtoBlob.h
+++ b/parsers/caffe/binaryProtoBlob.h
@@ -26,41 +26,41 @@ namespace nvcaffeparser1
class BinaryProtoBlob : public IBinaryProtoBlob
{
public:
- BinaryProtoBlob(void* memory, nvinfer1::DataType type, nvinfer1::DimsNCHW dimensions)
+ BinaryProtoBlob(void* memory, nvinfer1::DataType type, nvinfer1::Dims4 dimensions)
: mMemory(memory)
, mDataType(type)
, mDimensions(dimensions)
{
}
- nvinfer1::DimsNCHW getDimensions() override
+ nvinfer1::Dims4 getDimensions() noexcept override
{
return mDimensions;
}
- nvinfer1::DataType getDataType() override
+ nvinfer1::DataType getDataType() noexcept override
{
return mDataType;
}
- const void* getData() override
+ const void* getData() noexcept override
{
return mMemory;
}
- void destroy() override
+ void destroy() noexcept override
{
delete this;
}
- ~BinaryProtoBlob() override
+ ~BinaryProtoBlob() noexcept override
{
free(mMemory);
}
void* mMemory;
nvinfer1::DataType mDataType;
- nvinfer1::DimsNCHW mDimensions;
+ nvinfer1::Dims4 mDimensions;
};
} // namespace nvcaffeparser1
#endif // TRT_CAFFE_PARSER_BINARY_PROTO_BLOB_H
diff --git a/parsers/caffe/blobNameToTensor.h b/parsers/caffe/blobNameToTensor.h
index 5c4e80a8..14b59913 100644
--- a/parsers/caffe/blobNameToTensor.h
+++ b/parsers/caffe/blobNameToTensor.h
@@ -33,7 +33,7 @@ public:
mMap[name] = tensor;
}
- nvinfer1::ITensor* find(const char* name) const override
+ nvinfer1::ITensor* find(const char* name) const noexcept override
{
auto p = mMap.find(name);
if (p == mMap.end())
diff --git a/parsers/caffe/caffeParser/caffeParser.cpp b/parsers/caffe/caffeParser/caffeParser.cpp
index 10600d9f..2c12cc2a 100644
--- a/parsers/caffe/caffeParser/caffeParser.cpp
+++ b/parsers/caffe/caffeParser/caffeParser.cpp
@@ -66,7 +66,7 @@ std::vector CaffeParser::parseNormalizeParam(const trtcaf
// If .caffemodel is not provided, need to randomize the weight
if (!weightFactory.isInitialized())
{
- int C = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions()).c();
+ int C = parserutils::getC(tensors[msg.bottom(0)]->getDimensions());
w.emplace_back(weightFactory.allocateWeights(C, std::normal_distribution(0.0F, 1.0F)));
}
else
@@ -313,7 +313,7 @@ const IBlobNameToTensor* CaffeParser::parseBuffers(const char* deployBuffer,
const char* modelBuffer,
std::size_t modelLength,
INetworkDefinition& network,
- DataType weightType)
+ DataType weightType) noexcept
{
mDeploy = std::unique_ptr(new trtcaffe::NetParameter);
google::protobuf::io::ArrayInputStream deployStream(deployBuffer, deployLength);
@@ -341,7 +341,7 @@ const IBlobNameToTensor* CaffeParser::parseBuffers(const char* deployBuffer,
const IBlobNameToTensor* CaffeParser::parse(const char* deployFile,
const char* modelFile,
INetworkDefinition& network,
- DataType weightType)
+ DataType weightType) noexcept
{
CHECK_NULL_RET_NULL(deployFile)
@@ -391,12 +391,12 @@ const IBlobNameToTensor* CaffeParser::parse(INetworkDefinition& network,
{
if (mDeploy->input_shape_size())
{
- dims = DimsCHW{(int) mDeploy->input_shape().Get(i).dim().Get(1), (int) mDeploy->input_shape().Get(i).dim().Get(2), (int) mDeploy->input_shape().Get(i).dim().Get(3)};
+ dims = Dims3{(int) mDeploy->input_shape().Get(i).dim().Get(1), (int) mDeploy->input_shape().Get(i).dim().Get(2), (int) mDeploy->input_shape().Get(i).dim().Get(3)};
}
else
{
// Deprecated, but still used in a lot of networks
- dims = DimsCHW{(int) mDeploy->input_dim().Get(i * 4 + 1), (int) mDeploy->input_dim().Get(i * 4 + 2), (int) mDeploy->input_dim().Get(i * 4 + 3)};
+ dims = Dims3{(int) mDeploy->input_dim().Get(i * 4 + 1), (int) mDeploy->input_dim().Get(i * 4 + 2), (int) mDeploy->input_dim().Get(i * 4 + 3)};
}
}
else
@@ -404,12 +404,12 @@ const IBlobNameToTensor* CaffeParser::parse(INetworkDefinition& network,
std::cout << "Warning, setting batch size to 1. Update the dimension after parsing due to using explicit batch size." << std::endl;
if (mDeploy->input_shape_size())
{
- dims = DimsNCHW{1, (int) mDeploy->input_shape().Get(i).dim().Get(1), (int) mDeploy->input_shape().Get(i).dim().Get(2), (int) mDeploy->input_shape().Get(i).dim().Get(3)};
+ dims = Dims4{1, (int) mDeploy->input_shape().Get(i).dim().Get(1), (int) mDeploy->input_shape().Get(i).dim().Get(2), (int) mDeploy->input_shape().Get(i).dim().Get(3)};
}
else
{
// Deprecated, but still used in a lot of networks
- dims = DimsNCHW{1, (int) mDeploy->input_dim().Get(i * 4 + 1), (int) mDeploy->input_dim().Get(i * 4 + 2), (int) mDeploy->input_dim().Get(i * 4 + 3)};
+ dims = Dims4{1, (int) mDeploy->input_dim().Get(i * 4 + 1), (int) mDeploy->input_dim().Get(i * 4 + 2), (int) mDeploy->input_dim().Get(i * 4 + 3)};
}
}
ITensor* tensor = network.addInput(mDeploy->input().Get(i).c_str(), DataType::kFLOAT, dims);
@@ -441,50 +441,10 @@ const IBlobNameToTensor* CaffeParser::parse(INetworkDefinition& network,
}
}
}
-
- // If there is a pluginFactory provided, use layer name matching to handle the plugin construction
- if (mPluginFactory && mPluginFactory->isPlugin(layerMsg.name().c_str()))
- {
- std::vector w = weights.getAllWeights(layerMsg.name());
- IPlugin* plugin = mPluginFactory->createPlugin(layerMsg.name().c_str(), w.empty() ? nullptr : &w[0], w.size());
- std::vector inputs;
- for (int i = 0, n = layerMsg.bottom_size(); i < n; i++)
- {
- inputs.push_back((*mBlobNameToTensor)[layerMsg.bottom(i)]);
- }
-
- bool isExt = mPluginFactoryIsExt && static_cast(mPluginFactory)->isPluginExt(layerMsg.name().c_str());
-
- ILayer* layer = isExt ? network.addPluginExt(&inputs[0], int(inputs.size()), *static_cast(plugin))
- : network.addPlugin(&inputs[0], int(inputs.size()), *plugin);
-
- layer->setName(layerMsg.name().c_str());
- if (plugin->getNbOutputs() != layerMsg.top_size())
- {
- std::cout << "Plugin layer output count is not equal to caffe output count" << std::endl;
- ok = false;
- }
- for (int i = 0, n = std::min(layer->getNbOutputs(), layerMsg.top_size()); i < n; i++)
- {
- (*mBlobNameToTensor)[layerMsg.top(i)] = layer->getOutput(i);
- }
-
- if (layer == nullptr)
- {
- std::cout << "error parsing layer type " << layerMsg.type() << " index " << i << std::endl;
- ok = false;
- }
-
- continue;
- }
if (getInferLibVersion() >= 5000)
{
if (mPluginFactoryV2 && mPluginFactoryV2->isPluginV2(layerMsg.name().c_str()))
{
- if (mPluginFactory)
- {
- RETURN_AND_LOG_ERROR(nullptr, "Both IPluginFactory and IPluginFactoryV2 are set. If using TensorRT 5.0 or later, switch to IPluginFactoryV2");
- }
std::vector w = weights.getAllWeights(layerMsg.name());
nvinfer1::IPluginV2* plugin = mPluginFactoryV2->createPlugin(layerMsg.name().c_str(), w.empty() ? nullptr : &w[0], w.size(), mPluginNamespace.c_str());
std::vector inputs;
@@ -596,14 +556,14 @@ const IBlobNameToTensor* CaffeParser::parse(INetworkDefinition& network,
Dims d;
if (network.hasImplicitBatchDimension())
{
- d = DimsCHW{(int) shape.dim().Get(1), (int) shape.dim().Get(2), (int) shape.dim().Get(3)};
+ d = Dims3{(int) shape.dim().Get(1), (int) shape.dim().Get(2), (int) shape.dim().Get(3)};
}
else
{
std::cout << "Warning, setting batch size to 1. Update the dimension after parsing due to "
"using explicit batch size."
<< std::endl;
- d = DimsNCHW{1, (int) shape.dim().Get(1), (int) shape.dim().Get(2), (int) shape.dim().Get(3)};
+ d = Dims4{1, (int) shape.dim().Get(1), (int) shape.dim().Get(2), (int) shape.dim().Get(3)};
}
ITensor* tensor = network.addInput(layerMsg.top(i).c_str(), DataType::kFLOAT, d);
(*mBlobNameToTensor)[layerMsg.top().Get(i)] = tensor;
@@ -651,7 +611,7 @@ const IBlobNameToTensor* CaffeParser::parse(INetworkDefinition& network,
return ok && weights.isOK() && mBlobNameToTensor->isOK() ? mBlobNameToTensor : nullptr;
}
-IBinaryProtoBlob* CaffeParser::parseBinaryProto(const char* fileName)
+IBinaryProtoBlob* CaffeParser::parseBinaryProto(const char* fileName) noexcept
{
CHECK_NULL_RET_NULL(fileName)
using namespace google::protobuf::io;
@@ -675,7 +635,7 @@ IBinaryProtoBlob* CaffeParser::parseBinaryProto(const char* fileName)
RETURN_AND_LOG_ERROR(nullptr, "parseBinaryProto: Could not parse mean file");
}
- DimsNCHW dims{1, 1, 1, 1};
+ Dims4 dims{1, 1, 1, 1};
if (blob.has_shape())
{
int size = blob.shape().dim_size(), s[4] = {1, 1, 1, 1};
@@ -684,14 +644,14 @@ IBinaryProtoBlob* CaffeParser::parseBinaryProto(const char* fileName)
assert(blob.shape().dim(i) < INT32_MAX);
s[i] = static_cast(blob.shape().dim(i));
}
- dims = DimsNCHW{s[0], s[1], s[2], s[3]};
+ dims = Dims4{s[0], s[1], s[2], s[3]};
}
else
{
- dims = DimsNCHW{blob.num(), blob.channels(), blob.height(), blob.width()};
+ dims = Dims4{blob.num(), blob.channels(), blob.height(), blob.width()};
}
- const int dataSize = dims.n() * dims.c() * dims.h() * dims.w();
+ const int dataSize = parserutils::volume(dims);
assert(dataSize > 0);
const trtcaffe::Type blobProtoDataType = CaffeWeightFactory::getBlobProtoDataType(blob);
diff --git a/parsers/caffe/caffeParser/caffeParser.h b/parsers/caffe/caffeParser/caffeParser.h
index 9be2fa04..6dc861b4 100644
--- a/parsers/caffe/caffeParser/caffeParser.h
+++ b/parsers/caffe/caffeParser/caffeParser.h
@@ -35,32 +35,25 @@ public:
const IBlobNameToTensor* parse(const char* deploy,
const char* model,
nvinfer1::INetworkDefinition& network,
- nvinfer1::DataType weightType) override;
+ nvinfer1::DataType weightType) noexcept override;
const IBlobNameToTensor* parseBuffers(const char* deployBuffer,
size_t deployLength,
const char* modelBuffer,
size_t modelLength,
nvinfer1::INetworkDefinition& network,
- nvinfer1::DataType weightType) override;
+ nvinfer1::DataType weightType) noexcept override;
- void setProtobufBufferSize(size_t size) override { mProtobufBufferSize = size; }
- void setPluginFactory(nvcaffeparser1::IPluginFactory* factory) override { mPluginFactory = factory; }
- void setPluginFactoryExt(nvcaffeparser1::IPluginFactoryExt* factory) override
- {
- mPluginFactory = factory;
- mPluginFactoryIsExt = true;
- }
-
- void setPluginFactoryV2(nvcaffeparser1::IPluginFactoryV2* factory) override { mPluginFactoryV2 = factory; }
- void setPluginNamespace(const char* libNamespace) override { mPluginNamespace = libNamespace; }
- IBinaryProtoBlob* parseBinaryProto(const char* fileName) override;
- void destroy() override { delete this; }
- void setErrorRecorder(nvinfer1::IErrorRecorder* recorder) override { (void)recorder; assert(!"TRT- Not implemented."); }
- nvinfer1::IErrorRecorder* getErrorRecorder() const override { assert(!"TRT- Not implemented."); return nullptr; }
+ void setProtobufBufferSize(size_t size) noexcept override { mProtobufBufferSize = size; }
+ void setPluginFactoryV2(nvcaffeparser1::IPluginFactoryV2* factory) noexcept override { mPluginFactoryV2 = factory; }
+ void setPluginNamespace(const char* libNamespace) noexcept override { mPluginNamespace = libNamespace; }
+ IBinaryProtoBlob* parseBinaryProto(const char* fileName) noexcept override;
+ void destroy() noexcept override { delete this; }
+ void setErrorRecorder(nvinfer1::IErrorRecorder* recorder) noexcept override { (void)recorder; assert(!"TRT- Not implemented."); }
+ nvinfer1::IErrorRecorder* getErrorRecorder() const noexcept override { assert(!"TRT- Not implemented."); return nullptr; }
private:
- ~CaffeParser() override;
+ ~CaffeParser() noexcept override;
std::vector parseNormalizeParam(const trtcaffe::LayerParameter& msg, CaffeWeightFactory& weightFactory, BlobNameToTensor& tensors);
std::vector parsePriorBoxParam(const trtcaffe::LayerParameter& msg, CaffeWeightFactory& weightFactory, BlobNameToTensor& tensors);
std::vector parseDetectionOutputParam(const trtcaffe::LayerParameter& msg, CaffeWeightFactory& weightFactory, BlobNameToTensor& tensors);
@@ -84,7 +77,6 @@ private:
std::vector mTmpAllocs;
BlobNameToTensor* mBlobNameToTensor{nullptr};
size_t mProtobufBufferSize{INT_MAX};
- nvcaffeparser1::IPluginFactory* mPluginFactory{nullptr};
nvcaffeparser1::IPluginFactoryV2* mPluginFactoryV2{nullptr};
bool mPluginFactoryIsExt{false};
std::vector mNewPlugins;
diff --git a/parsers/caffe/caffeParser/opParsers/parseBatchNorm.cpp b/parsers/caffe/caffeParser/opParsers/parseBatchNorm.cpp
index d493d13f..6142be74 100644
--- a/parsers/caffe/caffeParser/opParsers/parseBatchNorm.cpp
+++ b/parsers/caffe/caffeParser/opParsers/parseBatchNorm.cpp
@@ -87,7 +87,7 @@ ILayer* parseBatchNormalization(INetworkDefinition& network, const trtcaffe::Lay
const trtcaffe::BatchNormParameter& p = msg.batch_norm_param();
bool nvCaffe = weightFactory.getBlobsSize(msg.name()) == 5;
- int C = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions()).c();
+ int C = parserutils::getC(tensors[msg.bottom(0)]->getDimensions());
Weights mean{DataType::kFLOAT, nullptr, 0},
variance{DataType::kFLOAT, nullptr, 0},
diff --git a/parsers/caffe/caffeParser/opParsers/parseConv.cpp b/parsers/caffe/caffeParser/opParsers/parseConv.cpp
index 4bf7db62..2d59234c 100644
--- a/parsers/caffe/caffeParser/opParsers/parseConv.cpp
+++ b/parsers/caffe/caffeParser/opParsers/parseConv.cpp
@@ -32,7 +32,7 @@ ILayer* parseConvolution(INetworkDefinition& network, const trtcaffe::LayerParam
int kernelH = p.has_kernel_h() ? p.kernel_h() : p.kernel_size(0);
int kernelW = p.has_kernel_w() ? p.kernel_w() : p.kernel_size_size() > 1 ? p.kernel_size(1) : p.kernel_size(0);
- int C = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions()).c();
+ int C = parserutils::getC(tensors[msg.bottom(0)]->getDimensions());
int G = p.has_group() ? p.group() : 1;
auto CbyG = float(C / G * nbOutputs);
diff --git a/parsers/caffe/caffeParser/opParsers/parseCrop.cpp b/parsers/caffe/caffeParser/opParsers/parseCrop.cpp
index 8ba7efa2..4bbc1263 100644
--- a/parsers/caffe/caffeParser/opParsers/parseCrop.cpp
+++ b/parsers/caffe/caffeParser/opParsers/parseCrop.cpp
@@ -44,8 +44,8 @@ ILayer* parseCrop(INetworkDefinition& network, const trtcaffe::LayerParameter& m
// ONLY IMPLEMENT SPATIAL CROPPING
// IF CROP LAYER IS NOT SPATIAL CROP, ABORT
const trtcaffe::CropParameter& p = msg.crop_param();
- DimsCHW inputDims = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions());
- DimsCHW refDims = parserutils::getCHW(tensors[msg.bottom(1)]->getDimensions());
+ Dims3 inputDims = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions());
+ Dims3 refDims = parserutils::getCHW(tensors[msg.bottom(1)]->getDimensions());
bool hasAxis = p.has_axis(); // optional parameter
int axis = hasAxis ? p.axis() : 2; // default is 2 - spatial crop
axis = (axis < 0) ? 4 + axis : axis; // axis negative number correction
@@ -108,11 +108,11 @@ ILayer* parseCrop(INetworkDefinition& network, const trtcaffe::LayerParameter& m
// - ( inputDims.w() - refDims.w() - offsetWidth ) = -inputDims.w() + refDims.w() + offsetWidth
int prePadHeight = -offsetHeight;
int prePadWidth = -offsetWidth;
- int postPadHeight = -inputDims.h() + refDims.h() + offsetHeight;
- int postPadWidth = -inputDims.w() + refDims.w() + offsetWidth;
+ int postPadHeight = -inputDims.d[1] + refDims.d[1] + offsetHeight;
+ int postPadWidth = -inputDims.d[2] + refDims.d[2] + offsetWidth;
- DimsHW prePadding = DimsHW{prePadHeight, prePadWidth};
- DimsHW postPadding = DimsHW{postPadHeight, postPadWidth};
- return network.addPadding(*tensors[msg.bottom(0)], prePadding, postPadding);
+ Dims prePadding = parserutils::toDims(prePadHeight, prePadWidth);
+ Dims postPadding = parserutils::toDims(postPadHeight, postPadWidth);
+ return network.addPaddingNd(*tensors[msg.bottom(0)], prePadding, postPadding);
}
} //namespace nvcaffeparser1
diff --git a/parsers/caffe/caffeParser/opParsers/parseDeconv.cpp b/parsers/caffe/caffeParser/opParsers/parseDeconv.cpp
index e3cf1767..00229c41 100644
--- a/parsers/caffe/caffeParser/opParsers/parseDeconv.cpp
+++ b/parsers/caffe/caffeParser/opParsers/parseDeconv.cpp
@@ -40,7 +40,7 @@ ILayer* parseDeconvolution(INetworkDefinition& network, const trtcaffe::LayerPar
int kernelW = p.has_kernel_w() ? p.kernel_w() : p.kernel_size(0);
int kernelH = p.has_kernel_h() ? p.kernel_h() : p.kernel_size_size() > 1 ? p.kernel_size(1) : p.kernel_size(0);
- int C = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions()).c();
+ int C = parserutils::getC(tensors[msg.bottom(0)]->getDimensions());
float std_dev = 1.0F / sqrtf(kernelW * kernelH * sqrtf(C * nbOutputs));
Weights kernelWeights = weightFactory.isInitialized() ? weightFactory(msg.name(), WeightType::kGENERIC) : weightFactory.allocateWeights(kernelW * kernelH * C * nbOutputs / nbGroups, std::normal_distribution(0.0F, std_dev));
diff --git a/parsers/caffe/caffeParser/opParsers/parsePReLU.cpp b/parsers/caffe/caffeParser/opParsers/parsePReLU.cpp
index 48d58db3..f8ff0472 100644
--- a/parsers/caffe/caffeParser/opParsers/parsePReLU.cpp
+++ b/parsers/caffe/caffeParser/opParsers/parsePReLU.cpp
@@ -39,7 +39,7 @@ ILayer* parsePReLU(INetworkDefinition& network, const trtcaffe::LayerParameter&
}
int nWeights = channelShared ? 1 : inputDims.d[0]; // Caffe treats second input dimension as channels
- Dims slopesDims{inputDims.nbDims, {}, {}};
+ Dims slopesDims{inputDims.nbDims, {}};
std::fill(slopesDims.d, slopesDims.d + slopesDims.nbDims, 1);
slopesDims.d[0] = nWeights;
diff --git a/parsers/caffe/caffeParser/opParsers/parsePooling.cpp b/parsers/caffe/caffeParser/opParsers/parsePooling.cpp
index 634ddd06..2f0325fa 100644
--- a/parsers/caffe/caffeParser/opParsers/parsePooling.cpp
+++ b/parsers/caffe/caffeParser/opParsers/parsePooling.cpp
@@ -37,9 +37,9 @@ ILayer* parsePooling(INetworkDefinition& network, const trtcaffe::LayerParameter
int kernelH, kernelW;
if (p.has_global_pooling() && p.global_pooling())
{
- DimsCHW dims = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions());
- kernelH = dims.h();
- kernelW = dims.w();
+ Dims3 dims = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions());
+ kernelH = dims.d[1];
+ kernelW = dims.d[2];
}
else
{
diff --git a/parsers/caffe/caffeParser/opParsers/parseScale.cpp b/parsers/caffe/caffeParser/opParsers/parseScale.cpp
index b68ce689..3eefdb8b 100644
--- a/parsers/caffe/caffeParser/opParsers/parseScale.cpp
+++ b/parsers/caffe/caffeParser/opParsers/parseScale.cpp
@@ -28,7 +28,7 @@ ILayer* parseScale(INetworkDefinition& network, const trtcaffe::LayerParameter&
}
const trtcaffe::ScaleParameter& p = msg.scale_param();
- int C = parserutils::getCHW(tensors[msg.bottom(0)]->getDimensions()).c();
+ int C = parserutils::getC(tensors[msg.bottom(0)]->getDimensions());
Weights scale = weightFactory.isInitialized() ? weightFactory(msg.name(), WeightType::kGENERIC) : weightFactory.allocateWeights(C, std::uniform_real_distribution(0.9F, 1.1F));
Weights shift = !p.has_bias_term() || p.bias_term() ? (weightFactory.isInitialized() ? weightFactory(msg.name(), WeightType::kBIAS) : weightFactory.allocateWeights(C)) : weightFactory.getNullWeights();
diff --git a/parsers/common/parserUtils.h b/parsers/common/parserUtils.h
index c82478c1..8c03fbd9 100644
--- a/parsers/common/parserUtils.h
+++ b/parsers/common/parserUtils.h
@@ -101,26 +101,25 @@ inline std::ostream& operator<<(std::ostream& o, nvinfer1::DataType dt)
case nvinfer1::DataType::kFLOAT: o << "Float"; break;
case nvinfer1::DataType::kHALF: o << "Half"; break;
case nvinfer1::DataType::kINT8: o << "Int8"; break;
+ case nvinfer1::DataType::kBOOL: o << "Bool"; break;
}
return o;
}
-inline nvinfer1::DimsCHW getCHW(const nvinfer1::Dims& d)
+inline nvinfer1::Dims3 getCHW(const nvinfer1::Dims& d)
{
assert(d.nbDims >= 3);
- return nvinfer1::DimsCHW(d.d[d.nbDims - 3], d.d[d.nbDims - 2], d.d[d.nbDims - 1]);
+ return nvinfer1::Dims3(d.d[d.nbDims - 3], d.d[d.nbDims - 2], d.d[d.nbDims - 1]);
}
-inline nvinfer1::DimsCHW getCHWWithExpansion(const nvinfer1::Dims& d, int filler)
+inline int32_t getC(const nvinfer1::Dims& d)
{
- if (d.nbDims == 0)
- return nvinfer1::DimsCHW(filler, filler, filler);
- else if (d.nbDims == 1)
- return nvinfer1::DimsCHW(filler, filler, d.d[0]);
- else if (d.nbDims == 2)
- return nvinfer1::DimsCHW(filler, d.d[0], d.d[1]);
- else
- return nvinfer1::DimsCHW(d.d[d.nbDims - 3], d.d[d.nbDims - 2], d.d[d.nbDims - 1]);
+ return getCHW(d).d[0];
+}
+
+inline nvinfer1::Dims toDims(int32_t w, int32_t h) noexcept
+{
+ return nvinfer1::Dims{2, {w, h}};
}
inline int combineIndexDimensions(int batchSize, const nvinfer1::Dims& d)
diff --git a/plugin/CMakeLists.txt b/plugin/CMakeLists.txt
index 806c3ade..f29ed5dd 100644
--- a/plugin/CMakeLists.txt
+++ b/plugin/CMakeLists.txt
@@ -35,6 +35,7 @@ set(PLUGIN_LISTS
coordConvACPlugin
cropAndResizePlugin
detectionLayerPlugin
+ efficientNMSPlugin
flattenConcat
generateDetectionPlugin
gridAnchorPlugin
@@ -53,6 +54,7 @@ set(PLUGIN_LISTS
regionPlugin
reorgPlugin
resizeNearestPlugin
+ scatterPlugin
specialSlicePlugin
splitPlugin
)
@@ -70,7 +72,7 @@ if(BERT_GENCODES)
)
endif()
-include_directories(common common/kernels ../samples/common)
+include_directories(common common/kernels ${CMAKE_SOURCE_DIR}/samples/common)
foreach(PLUGIN_ITER ${PLUGIN_LISTS})
include_directories(${PLUGIN_ITER})
diff --git a/plugin/InferPlugin.cpp b/plugin/InferPlugin.cpp
index 0d97f439..5fe39c5b 100644
--- a/plugin/InferPlugin.cpp
+++ b/plugin/InferPlugin.cpp
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#include "NvInfer.h"
#include "NvInferPlugin.h"
#include "checkMacrosPlugin.h"
@@ -33,6 +32,7 @@ using namespace nvinfer1::plugin;
#include "coordConvACPlugin.h"
#include "cropAndResizePlugin.h"
#include "detectionLayerPlugin.h"
+#include "efficientNMSPlugin.h"
#include "flattenConcat.h"
#include "generateDetectionPlugin.h"
#include "gridAnchorPlugin.h"
@@ -53,6 +53,7 @@ using namespace nvinfer1::plugin;
#include "resizeNearestPlugin.h"
#include "specialSlicePlugin.h"
#include "split.h"
+#include "scatterPlugin.h"
using nvinfer1::plugin::RPROIParams;
@@ -169,6 +170,8 @@ extern "C"
initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
+ initializePlugin(logger, libNamespace);
+ initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
@@ -189,6 +192,7 @@ extern "C"
initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
+ initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
initializePlugin(logger, libNamespace);
return true;
diff --git a/plugin/README.md b/plugin/README.md
index beb66149..76742b03 100644
--- a/plugin/README.md
+++ b/plugin/README.md
@@ -11,6 +11,8 @@
| [coordConvACPlugin](coordConvACPlugin) | CoordConvAC | 1 |
| [cropAndResizePlugin](cropAndResizePlugin) | CropAndResize | 1 |
| [detectionLayerPlugin](detectionLayerPlugin) | DetectionLayer_TRT | 1 |
+| [efficientNMSPlugin](efficientNMSPlugin) | EfficientNMS_TRT | 1 |
+| [efficientNMSONNXPlugin](efficientNMSPlugin) | EfficientNMS_ONNX_TRT | 1 |
| [embLayerNormPlugin](embLayerNormPlugin) | CustomEmbLayerNormPluginDynamic | 1, 2 |
| [fcPlugin](fcPlugin) | CustomFCPluginDynamic | 1 |
| [flattenConcat](flattenConcat) | FlattenConcat_TRT | 1 |
@@ -33,6 +35,7 @@
| [regionPlugin](regionPlugin) | Region_TRT | 1 |
| [reorgPlugin](reorgPlugin) | Reorg_TRT | 1 |
| [resizeNearestPlugin](resizeNearestPlugin) | ResizeNearest_TRT | 1 |
+| [scatterPlugin](scatterPlugin) | ScatterND | 1 |
| [skipLayerNormPlugin](skipLayerNormPlugin) | CustomSkipLayerNormPluginDynamic | 1, 2, 3 |
| [specialSlicePlugin](specialSlicePlugin) | SpecialSlice_TRT | 1 |
| [splitPlugin](splitPlugin) | Split | 1 |
diff --git a/plugin/batchTilePlugin/batchTilePlugin.cpp b/plugin/batchTilePlugin/batchTilePlugin.cpp
index 6f3db20c..a28dc36c 100644
--- a/plugin/batchTilePlugin/batchTilePlugin.cpp
+++ b/plugin/batchTilePlugin/batchTilePlugin.cpp
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#include "batchTilePlugin.h"
#include
#include
@@ -65,78 +64,94 @@ BatchTilePlugin::BatchTilePlugin(const std::string name, const void* data, size_
assert(d == a + length);
}
-int BatchTilePlugin::getNbOutputs() const
+int BatchTilePlugin::getNbOutputs() const noexcept
{
return 1;
}
-Dims BatchTilePlugin::getOutputDimensions(int index, const Dims* inputs, int nbInputDims)
+Dims BatchTilePlugin::getOutputDimensions(int index, const Dims* inputs, int nbInputDims) noexcept
{
- assert(nbInputDims == 2);
- assert(index == 0);
- assert(inputs[1].nbDims == 4);
- return DimsCHW(inputs[1].d[1], inputs[1].d[2], inputs[1].d[3]);
+ try
+ {
+ assert(nbInputDims == 2);
+ assert(index == 0);
+ assert(inputs[1].nbDims == 4);
+ return Dims3(inputs[1].d[1], inputs[1].d[2], inputs[1].d[3]);
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return Dims{};
}
-int BatchTilePlugin::initialize()
+int BatchTilePlugin::initialize() noexcept
{
return STATUS_SUCCESS;
}
-size_t BatchTilePlugin::getWorkspaceSize(int) const
+size_t BatchTilePlugin::getWorkspaceSize(int) const noexcept
{
return 0;
}
-DataType BatchTilePlugin::getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const
+DataType BatchTilePlugin::getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const noexcept
{
assert(index == 0);
return DataType::kFLOAT;
}
-int BatchTilePlugin::enqueue(int batchSize, const void* const* inputs, void** outputs, void*, cudaStream_t stream)
+int BatchTilePlugin::enqueue(
+ int batchSize, const void* const* inputs, void* const* outputs, void*, cudaStream_t stream) noexcept
{
- char* output = reinterpret_cast(outputs[0]);
- // expand to batch size
- for (int i = 0; i < batchSize; i++)
+ try
{
- auto ret = cudaMemcpyAsync(output + i * mCopySize, inputs[1], mCopySize, cudaMemcpyDeviceToDevice, stream);
- if (ret != 0)
+ char* output = reinterpret_cast(outputs[0]);
+ // expand to batch size
+ for (int i = 0; i < batchSize; i++)
{
- std::cout << "Cuda failure: " << ret;
- abort();
+ auto ret = cudaMemcpyAsync(output + i * mCopySize, inputs[1], mCopySize, cudaMemcpyDeviceToDevice, stream);
+ if (ret != cudaSuccess)
+ {
+ return ret;
+ }
}
+ return STATUS_SUCCESS;
}
- return 0;
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return -1;
}
-void BatchTilePlugin::serialize(void* buffer) const
+void BatchTilePlugin::serialize(void* buffer) const noexcept
{
char *d = reinterpret_cast(buffer), *a = d;
writeToBuffer(d, mCopySize);
assert(d == a + getSerializationSize());
}
-void BatchTilePlugin::terminate() {}
+void BatchTilePlugin::terminate() noexcept {}
-size_t BatchTilePlugin::getSerializationSize() const
+size_t BatchTilePlugin::getSerializationSize() const noexcept
{
return sizeof(size_t);
}
-bool BatchTilePlugin::isOutputBroadcastAcrossBatch(int outputIndex, const bool* inputIsBroadcasted, int nbInputs) const
+bool BatchTilePlugin::isOutputBroadcastAcrossBatch(int outputIndex, const bool* inputIsBroadcasted, int nbInputs) const noexcept
{
return false;
}
-bool BatchTilePlugin::canBroadcastInputAcrossBatch(int inputIndex) const
+bool BatchTilePlugin::canBroadcastInputAcrossBatch(int inputIndex) const noexcept
{
return false;
}
void BatchTilePlugin::configurePlugin(const Dims* inputDims, int nbInputs, const Dims* outputDims, int nbOutputs,
const DataType* inputTypes, const DataType* outputTypes, const bool* inputIsBroadcast,
- const bool* outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize)
+ const bool* outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) noexcept
{
assert(nbOutputs == 1);
assert(inputDims[1].nbDims == 4);
@@ -144,38 +159,46 @@ void BatchTilePlugin::configurePlugin(const Dims* inputDims, int nbInputs, const
mCopySize = std::accumulate(inputDims[1].d, inputDims[1].d + 4, 1, std::multiplies()) * sizeof(float);
}
-bool BatchTilePlugin::supportsFormat(DataType type, PluginFormat format) const
+bool BatchTilePlugin::supportsFormat(DataType type, PluginFormat format) const noexcept
{
- return (type == DataType::kFLOAT && format == PluginFormat::kNCHW);
+ return (type == DataType::kFLOAT && format == PluginFormat::kLINEAR);
}
-const char* BatchTilePlugin::getPluginType() const
+const char* BatchTilePlugin::getPluginType() const noexcept
{
return BATCH_TILE_PLUGIN_NAME;
}
-const char* BatchTilePlugin::getPluginVersion() const
+const char* BatchTilePlugin::getPluginVersion() const noexcept
{
return BATCH_TILE_PLUGIN_VERSION;
}
-void BatchTilePlugin::destroy()
+void BatchTilePlugin::destroy() noexcept
{
delete this;
}
-IPluginV2Ext* BatchTilePlugin::clone() const
+IPluginV2Ext* BatchTilePlugin::clone() const noexcept
{
- auto* plugin = new BatchTilePlugin(mLayerName, mCopySize);
- plugin->setPluginNamespace(mNamespace.c_str());
- return plugin;
+ try
+ {
+ auto* plugin = new BatchTilePlugin(mLayerName, mCopySize);
+ plugin->setPluginNamespace(mNamespace.c_str());
+ return plugin;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
-void BatchTilePlugin::setPluginNamespace(const char* libNamespace)
+void BatchTilePlugin::setPluginNamespace(const char* libNamespace) noexcept
{
mNamespace = libNamespace;
}
-const char* BatchTilePlugin::getPluginNamespace() const
+const char* BatchTilePlugin::getPluginNamespace() const noexcept
{
return mNamespace.c_str();
}
@@ -186,29 +209,58 @@ BatchTilePluginCreator::BatchTilePluginCreator()
mFC.fields = nullptr;
}
-const char* BatchTilePluginCreator::getPluginName() const
+const char* BatchTilePluginCreator::getPluginName() const noexcept
{
return BATCH_TILE_PLUGIN_NAME;
}
-const char* BatchTilePluginCreator::getPluginVersion() const
+const char* BatchTilePluginCreator::getPluginVersion() const noexcept
{
return BATCH_TILE_PLUGIN_VERSION;
}
-const PluginFieldCollection* BatchTilePluginCreator::getFieldNames()
+const PluginFieldCollection* BatchTilePluginCreator::getFieldNames() noexcept
{
return &mFC;
}
-IPluginV2Ext* BatchTilePluginCreator::createPlugin(const char* name, const PluginFieldCollection* fc)
+IPluginV2Ext* BatchTilePluginCreator::createPlugin(const char* name, const PluginFieldCollection* fc) noexcept
{
- auto* plugin = new BatchTilePlugin(name);
- plugin->setPluginNamespace(mNamespace.c_str());
- return plugin;
+ try
+ {
+ auto* plugin = new BatchTilePlugin(name);
+ plugin->setPluginNamespace(mNamespace.c_str());
+ return plugin;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
-IPluginV2Ext* BatchTilePluginCreator::deserializePlugin(const char* name, const void* serialData, size_t serialLength)
+void BatchTilePluginCreator::setPluginNamespace(const char* libNamespace) noexcept
{
- return new BatchTilePlugin(name, serialData, serialLength);
+ try
+ {
+ mNamespace = libNamespace;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+}
+
+IPluginV2Ext* BatchTilePluginCreator::deserializePlugin(
+ const char* name, const void* serialData, size_t serialLength) noexcept
+{
+ try
+ {
+ return new BatchTilePlugin(name, serialData, serialLength);
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
diff --git a/plugin/batchTilePlugin/batchTilePlugin.h b/plugin/batchTilePlugin/batchTilePlugin.h
index 3464f523..d8f9ad6b 100644
--- a/plugin/batchTilePlugin/batchTilePlugin.h
+++ b/plugin/batchTilePlugin/batchTilePlugin.h
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#ifndef BATCHTILEPLUGIN_H
#define BATCHTILEPLUGIN_H
#include "NvInferPlugin.h"
@@ -35,45 +34,45 @@ public:
BatchTilePlugin() = delete;
- int getNbOutputs() const override;
+ int getNbOutputs() const noexcept override;
- Dims getOutputDimensions(int index, const Dims* inputs, int nbInputDims) override;
+ Dims getOutputDimensions(int index, const Dims* inputs, int nbInputDims) noexcept override;
- int initialize() override;
- void terminate() override;
+ int initialize() noexcept override;
+ void terminate() noexcept override;
- size_t getWorkspaceSize(int) const override;
+ size_t getWorkspaceSize(int) const noexcept override;
- int enqueue(
- int batchSize, const void* const* inputs, void** outputs, void* workspace, cudaStream_t stream) override;
+ int enqueue(int batchSize, const void* const* inputs, void* const* outputs, void* workspace,
+ cudaStream_t stream) noexcept override;
- DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const override;
+ DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const noexcept override;
- size_t getSerializationSize() const override;
+ size_t getSerializationSize() const noexcept override;
- void serialize(void* buffer) const override;
+ void serialize(void* buffer) const noexcept override;
- bool isOutputBroadcastAcrossBatch(int outputIndex, const bool* inputIsBroadcasted, int nbInputs) const override;
+ bool isOutputBroadcastAcrossBatch(int outputIndex, const bool* inputIsBroadcasted, int nbInputs) const noexcept override;
- bool canBroadcastInputAcrossBatch(int inputIndex) const override;
+ bool canBroadcastInputAcrossBatch(int inputIndex) const noexcept override;
void configurePlugin(const Dims* inputDims, int nbInputs, const Dims* outputDims, int nbOutputs,
const DataType* inputTypes, const DataType* outputTypes, const bool* inputIsBroadcast,
- const bool* outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) override;
+ const bool* outputIsBroadcast, PluginFormat floatFormat, int maxBatchSize) noexcept override;
- bool supportsFormat(DataType type, PluginFormat format) const override;
+ bool supportsFormat(DataType type, PluginFormat format) const noexcept override;
- const char* getPluginType() const override;
+ const char* getPluginType() const noexcept override;
- const char* getPluginVersion() const override;
+ const char* getPluginVersion() const noexcept override;
- void destroy() override;
+ void destroy() noexcept override;
- IPluginV2Ext* clone() const override;
+ IPluginV2Ext* clone() const noexcept override;
- void setPluginNamespace(const char* libNamespace) override;
+ void setPluginNamespace(const char* libNamespace) noexcept override;
- const char* getPluginNamespace() const override;
+ const char* getPluginNamespace() const noexcept override;
private:
const std::string mLayerName;
@@ -86,22 +85,19 @@ class BatchTilePluginCreator : public BaseCreator
public:
BatchTilePluginCreator();
- const char* getPluginName() const override;
+ const char* getPluginName() const noexcept override;
- const char* getPluginVersion() const override;
+ const char* getPluginVersion() const noexcept override;
- const PluginFieldCollection* getFieldNames() override;
+ const PluginFieldCollection* getFieldNames() noexcept override;
- IPluginV2Ext* createPlugin(const char* name, const PluginFieldCollection* fc) override;
+ IPluginV2Ext* createPlugin(const char* name, const PluginFieldCollection* fc) noexcept override;
- IPluginV2Ext* deserializePlugin(const char* name, const void* serialData, size_t serialLength) override;
+ IPluginV2Ext* deserializePlugin(const char* name, const void* serialData, size_t serialLength) noexcept override;
- void setPluginNamespace(const char* libNamespace) override
- {
- mNamespace = libNamespace;
- }
+ void setPluginNamespace(const char* libNamespace) noexcept override;
- const char* getPluginNamespace() const override
+ const char* getPluginNamespace() const noexcept override
{
return mNamespace.c_str();
}
diff --git a/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp b/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp
index d1074153..90ccf23f 100644
--- a/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp
+++ b/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp
@@ -35,15 +35,50 @@ const char* NMS_PLUGIN_VERSION{"1"};
const char* NMS_PLUGIN_NAMES[] = {"BatchedNMS_TRT", "BatchedNMSDynamic_TRT"};
} // namespace
+namespace nvinfer1
+{
+namespace plugin
+{
+template <>
+void write(char*& buffer, const NMSParameters& val)
+{
+ auto* param = reinterpret_cast(buffer);
+ param->shareLocation = val.shareLocation;
+ param->backgroundLabelId = val.backgroundLabelId;
+ param->numClasses = val.numClasses;
+ param->topK = val.topK;
+ param->keepTopK = val.keepTopK;
+ param->scoreThreshold = val.scoreThreshold;
+ param->iouThreshold = val.iouThreshold;
+ param->isNormalized = val.isNormalized;
+ buffer += sizeof(NMSParameters);
+}
+} // namespace plugin
+} // namespace nvinfer1
+
PluginFieldCollection BatchedNMSBasePluginCreator::mFC{};
std::vector BatchedNMSBasePluginCreator::mPluginAttributes;
-BatchedNMSPlugin::BatchedNMSPlugin(NMSParameters params) noexcept
- : param(params)
+static inline pluginStatus_t checkParams(const NMSParameters& param)
{
+ // NMS plugin supports maximum thread blocksize of 512 and upto 8 blocks at once.
+ constexpr int32_t maxTopK{512*8};
+ if (param.topK > maxTopK)
+ {
+ gLogError << "Invalid parameter: NMS topK (" << param.topK << ") exceeds limit (" << maxTopK << ")" << std::endl;
+ return STATUS_BAD_PARAM;
+ }
+
+ return STATUS_SUCCESS;
}
-BatchedNMSPlugin::BatchedNMSPlugin(const void* data, size_t length) noexcept
+BatchedNMSPlugin::BatchedNMSPlugin(NMSParameters params)
+ : param(params)
+{
+ mPluginStatus = checkParams(param);
+}
+
+BatchedNMSPlugin::BatchedNMSPlugin(const void* data, size_t length)
{
const char *d = reinterpret_cast(data), *a = d;
param = read(d);
@@ -54,14 +89,17 @@ BatchedNMSPlugin::BatchedNMSPlugin(const void* data, size_t length) noexcept
mPrecision = read(d);
mScoreBits = read(d);
ASSERT(d == a + length);
+
+ mPluginStatus = checkParams(param);
}
-BatchedNMSDynamicPlugin::BatchedNMSDynamicPlugin(NMSParameters params) noexcept
+BatchedNMSDynamicPlugin::BatchedNMSDynamicPlugin(NMSParameters params)
: param(params)
{
+ mPluginStatus = checkParams(param);
}
-BatchedNMSDynamicPlugin::BatchedNMSDynamicPlugin(const void* data, size_t length) noexcept
+BatchedNMSDynamicPlugin::BatchedNMSDynamicPlugin(const void* data, size_t length)
{
const char *d = reinterpret_cast(data), *a = d;
param = read(d);
@@ -72,6 +110,8 @@ BatchedNMSDynamicPlugin::BatchedNMSDynamicPlugin(const void* data, size_t length
mPrecision = read(d);
mScoreBits = read(d);
ASSERT(d == a + length);
+
+ mPluginStatus = checkParams(param);
}
int BatchedNMSPlugin::getNbOutputs() const noexcept
@@ -100,100 +140,116 @@ void BatchedNMSDynamicPlugin::terminate() noexcept {}
Dims BatchedNMSPlugin::getOutputDimensions(int index, const Dims* inputs, int nbInputDims) noexcept
{
- ASSERT(nbInputDims == 2);
- ASSERT(index >= 0 && index < this->getNbOutputs());
- ASSERT(inputs[0].nbDims == 3);
- ASSERT(inputs[1].nbDims == 2 || (inputs[1].nbDims == 3 && inputs[1].d[2] == 1));
- // boxesSize: number of box coordinates for one sample
- boxesSize = inputs[0].d[0] * inputs[0].d[1] * inputs[0].d[2];
- // scoresSize: number of scores for one sample
- scoresSize = inputs[1].d[0] * inputs[1].d[1];
- // num_detections
- if (index == 0)
+ try
{
- Dims dim0{};
- dim0.nbDims = 0;
- return dim0;
+ ASSERT(nbInputDims == 2);
+ ASSERT(index >= 0 && index < this->getNbOutputs());
+ ASSERT(inputs[0].nbDims == 3);
+ ASSERT(inputs[1].nbDims == 2 || (inputs[1].nbDims == 3 && inputs[1].d[2] == 1));
+ // boxesSize: number of box coordinates for one sample
+ boxesSize = inputs[0].d[0] * inputs[0].d[1] * inputs[0].d[2];
+ // scoresSize: number of scores for one sample
+ scoresSize = inputs[1].d[0] * inputs[1].d[1];
+ // num_detections
+ if (index == 0)
+ {
+ Dims dim0{};
+ dim0.nbDims = 0;
+ return dim0;
+ }
+ // nmsed_boxes
+ if (index == 1)
+ {
+ return DimsHW(param.keepTopK, 4);
+ }
+ // nmsed_scores or nmsed_classes
+ Dims dim1{};
+ dim1.nbDims = 1;
+ dim1.d[0] = param.keepTopK;
+ return dim1;
}
- // nmsed_boxes
- if (index == 1)
+ catch (const std::exception& e)
{
- return DimsHW(param.keepTopK, 4);
+ caughtError(e);
}
- // nmsed_scores or nmsed_classes
- Dims dim1{};
- dim1.nbDims = 1;
- dim1.d[0] = param.keepTopK;
- return dim1;
+ return Dims{};
}
DimsExprs BatchedNMSDynamicPlugin::getOutputDimensions(
int outputIndex, const DimsExprs* inputs, int nbInputs, IExprBuilder& exprBuilder) noexcept
{
- ASSERT(nbInputs == 2);
- ASSERT(outputIndex >= 0 && outputIndex < this->getNbOutputs());
-
- // Shape of boxes input should be
- // Constant shape: [batch_size, num_boxes, num_classes, 4] or [batch_size, num_boxes, 1, 4]
- // shareLocation == 0 or 1
- // or
- // Dynamic shape: some dimension values may be -1
- ASSERT(inputs[0].nbDims == 4);
-
- // Shape of scores input should be
- // Constant shape: [batch_size, num_boxes, num_classes] or [batch_size, num_boxes, num_classes, 1]
- // or
- // Dynamic shape: some dimension values may be -1
- ASSERT(inputs[1].nbDims == 3 || inputs[1].nbDims == 4);
-
- if (inputs[0].d[0]->isConstant() && inputs[0].d[1]->isConstant() && inputs[0].d[2]->isConstant()
- && inputs[0].d[3]->isConstant())
+ try
{
- boxesSize = exprBuilder
- .operation(DimensionOperation::kPROD,
- *exprBuilder.operation(DimensionOperation::kPROD, *inputs[0].d[1], *inputs[0].d[2]),
- *inputs[0].d[3])
- ->getConstantValue();
- }
+ ASSERT(nbInputs == 2);
+ ASSERT(outputIndex >= 0 && outputIndex < this->getNbOutputs());
- if (inputs[1].d[0]->isConstant() && inputs[1].d[1]->isConstant() && inputs[1].d[2]->isConstant())
- {
- scoresSize
- = exprBuilder.operation(DimensionOperation::kPROD, *inputs[1].d[1], *inputs[1].d[2])->getConstantValue();
- }
+ // Shape of boxes input should be
+ // Constant shape: [batch_size, num_boxes, num_classes, 4] or [batch_size, num_boxes, 1, 4]
+ // shareLocation == 0 or 1
+ // or
+ // Dynamic shape: some dimension values may be -1
+ ASSERT(inputs[0].nbDims == 4);
- DimsExprs out_dim;
- // num_detections
- if (outputIndex == 0)
- {
- out_dim.nbDims = 2;
- out_dim.d[0] = inputs[0].d[0];
- out_dim.d[1] = exprBuilder.constant(1);
- }
- // nmsed_boxes
- else if (outputIndex == 1)
- {
- out_dim.nbDims = 3;
- out_dim.d[0] = inputs[0].d[0];
- out_dim.d[1] = exprBuilder.constant(param.keepTopK);
- out_dim.d[2] = exprBuilder.constant(4);
- }
- // nmsed_scores
- else if (outputIndex == 2)
- {
- out_dim.nbDims = 2;
- out_dim.d[0] = inputs[0].d[0];
- out_dim.d[1] = exprBuilder.constant(param.keepTopK);
- }
- // nmsed_classes
- else
- {
- out_dim.nbDims = 2;
- out_dim.d[0] = inputs[0].d[0];
- out_dim.d[1] = exprBuilder.constant(param.keepTopK);
- }
+ // Shape of scores input should be
+ // Constant shape: [batch_size, num_boxes, num_classes] or [batch_size, num_boxes, num_classes, 1]
+ // or
+ // Dynamic shape: some dimension values may be -1
+ ASSERT(inputs[1].nbDims == 3 || inputs[1].nbDims == 4);
- return out_dim;
+ if (inputs[0].d[0]->isConstant() && inputs[0].d[1]->isConstant() && inputs[0].d[2]->isConstant()
+ && inputs[0].d[3]->isConstant())
+ {
+ boxesSize = exprBuilder
+ .operation(DimensionOperation::kPROD,
+ *exprBuilder.operation(DimensionOperation::kPROD, *inputs[0].d[1], *inputs[0].d[2]),
+ *inputs[0].d[3])
+ ->getConstantValue();
+ }
+
+ if (inputs[1].d[0]->isConstant() && inputs[1].d[1]->isConstant() && inputs[1].d[2]->isConstant())
+ {
+ scoresSize = exprBuilder.operation(DimensionOperation::kPROD, *inputs[1].d[1], *inputs[1].d[2])
+ ->getConstantValue();
+ }
+
+ DimsExprs out_dim;
+ // num_detections
+ if (outputIndex == 0)
+ {
+ out_dim.nbDims = 2;
+ out_dim.d[0] = inputs[0].d[0];
+ out_dim.d[1] = exprBuilder.constant(1);
+ }
+ // nmsed_boxes
+ else if (outputIndex == 1)
+ {
+ out_dim.nbDims = 3;
+ out_dim.d[0] = inputs[0].d[0];
+ out_dim.d[1] = exprBuilder.constant(param.keepTopK);
+ out_dim.d[2] = exprBuilder.constant(4);
+ }
+ // nmsed_scores
+ else if (outputIndex == 2)
+ {
+ out_dim.nbDims = 2;
+ out_dim.d[0] = inputs[0].d[0];
+ out_dim.d[1] = exprBuilder.constant(param.keepTopK);
+ }
+ // nmsed_classes
+ else
+ {
+ out_dim.nbDims = 2;
+ out_dim.d[0] = inputs[0].d[0];
+ out_dim.d[1] = exprBuilder.constant(param.keepTopK);
+ }
+
+ return out_dim;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return DimsExprs{};
}
size_t BatchedNMSPlugin::getWorkspaceSize(int maxBatchSize) const noexcept
@@ -210,41 +266,66 @@ size_t BatchedNMSDynamicPlugin::getWorkspaceSize(
}
int BatchedNMSPlugin::enqueue(
- int batchSize, const void* const* inputs, void** outputs, void* workspace, cudaStream_t stream) noexcept
+ int32_t batchSize, void const* const* inputs, void* const* outputs, void* workspace,
+ cudaStream_t stream) noexcept
{
- const void* const locData = inputs[0];
- const void* const confData = inputs[1];
+ try
+ {
+ const void* const locData = inputs[0];
+ const void* const confData = inputs[1];
- void* keepCount = outputs[0];
- void* nmsedBoxes = outputs[1];
- void* nmsedScores = outputs[2];
- void* nmsedClasses = outputs[3];
+ if (mPluginStatus != STATUS_SUCCESS)
+ {
+ return -1;
+ }
- pluginStatus_t status = nmsInference(stream, batchSize, boxesSize, scoresSize, param.shareLocation,
- param.backgroundLabelId, numPriors, param.numClasses, param.topK, param.keepTopK, param.scoreThreshold,
- param.iouThreshold, mPrecision, locData, mPrecision, confData, keepCount, nmsedBoxes, nmsedScores, nmsedClasses,
- workspace, param.isNormalized, false, mClipBoxes, mScoreBits);
- ASSERT(status == STATUS_SUCCESS);
- return 0;
+ void* keepCount = outputs[0];
+ void* nmsedBoxes = outputs[1];
+ void* nmsedScores = outputs[2];
+ void* nmsedClasses = outputs[3];
+
+ pluginStatus_t status = nmsInference(stream, batchSize, boxesSize, scoresSize, param.shareLocation,
+ param.backgroundLabelId, numPriors, param.numClasses, param.topK, param.keepTopK, param.scoreThreshold,
+ param.iouThreshold, mPrecision, locData, mPrecision, confData, keepCount, nmsedBoxes, nmsedScores, nmsedClasses,
+ workspace, param.isNormalized, false, mClipBoxes, mScoreBits);
+ return status == STATUS_SUCCESS ? 0 : -1;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return -1;
}
int BatchedNMSDynamicPlugin::enqueue(const PluginTensorDesc* inputDesc, const PluginTensorDesc* outputDesc,
const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept
{
- const void* const locData = inputs[0];
- const void* const confData = inputs[1];
+ try
+ {
+ const void* const locData = inputs[0];
+ const void* const confData = inputs[1];
- void* keepCount = outputs[0];
- void* nmsedBoxes = outputs[1];
- void* nmsedScores = outputs[2];
- void* nmsedClasses = outputs[3];
+ if (mPluginStatus != STATUS_SUCCESS)
+ {
+ return -1;
+ }
- pluginStatus_t status = nmsInference(stream, inputDesc[0].dims.d[0], boxesSize, scoresSize, param.shareLocation,
- param.backgroundLabelId, numPriors, param.numClasses, param.topK, param.keepTopK, param.scoreThreshold,
- param.iouThreshold, mPrecision, locData, mPrecision, confData, keepCount, nmsedBoxes, nmsedScores, nmsedClasses,
- workspace, param.isNormalized, false, mClipBoxes, mScoreBits);
- ASSERT(status == STATUS_SUCCESS);
- return 0;
+ void* keepCount = outputs[0];
+ void* nmsedBoxes = outputs[1];
+ void* nmsedScores = outputs[2];
+ void* nmsedClasses = outputs[3];
+
+ pluginStatus_t status = nmsInference(stream, inputDesc[0].dims.d[0], boxesSize, scoresSize, param.shareLocation,
+ param.backgroundLabelId, numPriors, param.numClasses, param.topK, param.keepTopK, param.scoreThreshold,
+ param.iouThreshold, mPrecision, locData, mPrecision, confData, keepCount, nmsedBoxes, nmsedScores, nmsedClasses,
+ workspace, param.isNormalized, false, mClipBoxes, mScoreBits);
+ return status;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return -1;
}
size_t BatchedNMSPlugin::getSerializationSize() const noexcept
@@ -289,60 +370,76 @@ void BatchedNMSPlugin::configurePlugin(const Dims* inputDims, int nbInputs, cons
const DataType* inputTypes, const DataType* outputTypes, const bool* inputIsBroadcast,
const bool* outputIsBroadcast, nvinfer1::PluginFormat format, int maxBatchSize) noexcept
{
- ASSERT(nbInputs == 2);
- ASSERT(nbOutputs == 4);
- ASSERT(inputDims[0].nbDims == 3);
- ASSERT(inputDims[1].nbDims == 2 || (inputDims[1].nbDims == 3 && inputDims[1].d[2] == 1));
- ASSERT(std::none_of(inputIsBroadcast, inputIsBroadcast + nbInputs, [](bool b) { return b; }));
- ASSERT(std::none_of(outputIsBroadcast, outputIsBroadcast + nbInputs, [](bool b) { return b; }));
+ try
+ {
+ ASSERT(nbInputs == 2);
+ ASSERT(nbOutputs == 4);
+ ASSERT(inputDims[0].nbDims == 3);
+ ASSERT(inputDims[1].nbDims == 2 || (inputDims[1].nbDims == 3 && inputDims[1].d[2] == 1));
+ ASSERT(std::none_of(inputIsBroadcast, inputIsBroadcast + nbInputs, [](bool b) { return b; }));
+ ASSERT(std::none_of(outputIsBroadcast, outputIsBroadcast + nbInputs, [](bool b) { return b; }));
- boxesSize = inputDims[0].d[0] * inputDims[0].d[1] * inputDims[0].d[2];
- scoresSize = inputDims[1].d[0] * inputDims[1].d[1];
- // num_boxes
- numPriors = inputDims[0].d[0];
- const int numLocClasses = param.shareLocation ? 1 : param.numClasses;
- // Third dimension of boxes must be either 1 or num_classes
- ASSERT(inputDims[0].d[1] == numLocClasses);
- ASSERT(inputDims[0].d[2] == 4);
- mPrecision = inputTypes[0];
+ boxesSize = inputDims[0].d[0] * inputDims[0].d[1] * inputDims[0].d[2];
+ scoresSize = inputDims[1].d[0] * inputDims[1].d[1];
+ // num_boxes
+ numPriors = inputDims[0].d[0];
+ const int numLocClasses = param.shareLocation ? 1 : param.numClasses;
+ // Third dimension of boxes must be either 1 or num_classes
+ ASSERT(inputDims[0].d[1] == numLocClasses);
+ ASSERT(inputDims[0].d[2] == 4);
+ mPrecision = inputTypes[0];
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
}
void BatchedNMSDynamicPlugin::configurePlugin(
const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out, int nbOutputs) noexcept
{
- ASSERT(nbInputs == 2);
- ASSERT(nbOutputs == 4);
+ try
+ {
+ ASSERT(nbInputs == 2);
+ ASSERT(nbOutputs == 4);
- // Shape of boxes input should be
- // Constant shape: [batch_size, num_boxes, num_classes, 4] or [batch_size, num_boxes, 1, 4]
- // shareLocation == 0 or 1
- const int numLocClasses = param.shareLocation ? 1 : param.numClasses;
- ASSERT(in[0].desc.dims.nbDims == 4);
- ASSERT(in[0].desc.dims.d[2] == numLocClasses);
- ASSERT(in[0].desc.dims.d[3] == 4);
+ // Shape of boxes input should be
+ // Constant shape: [batch_size, num_boxes, num_classes, 4] or [batch_size, num_boxes, 1, 4]
+ // shareLocation == 0 or 1
+ const int numLocClasses = param.shareLocation ? 1 : param.numClasses;
+ ASSERT(in[0].desc.dims.nbDims == 4);
+ ASSERT(in[0].desc.dims.d[2] == numLocClasses);
+ ASSERT(in[0].desc.dims.d[3] == 4);
- // Shape of scores input should be
- // Constant shape: [batch_size, num_boxes, num_classes] or [batch_size, num_boxes, num_classes, 1]
- ASSERT(in[1].desc.dims.nbDims == 3 || (in[1].desc.dims.nbDims == 4 && in[1].desc.dims.d[3] == 1));
+ // Shape of scores input should be
+ // Constant shape: [batch_size, num_boxes, num_classes] or [batch_size, num_boxes, num_classes, 1]
+ ASSERT(in[1].desc.dims.nbDims == 3 || (in[1].desc.dims.nbDims == 4 && in[1].desc.dims.d[3] == 1));
- boxesSize = in[0].desc.dims.d[1] * in[0].desc.dims.d[2] * in[0].desc.dims.d[3];
- scoresSize = in[1].desc.dims.d[1] * in[1].desc.dims.d[2];
- // num_boxes
- numPriors = in[0].desc.dims.d[1];
+ boxesSize = in[0].desc.dims.d[1] * in[0].desc.dims.d[2] * in[0].desc.dims.d[3];
+ scoresSize = in[1].desc.dims.d[1] * in[1].desc.dims.d[2];
+ // num_boxes
+ numPriors = in[0].desc.dims.d[1];
- mPrecision = in[0].desc.type;
+ mPrecision = in[0].desc.type;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
}
bool BatchedNMSPlugin::supportsFormat(DataType type, PluginFormat format) const noexcept
{
return ((type == DataType::kHALF || type == DataType::kFLOAT || type == DataType::kINT32)
- && format == PluginFormat::kNCHW);
+ && format == PluginFormat::kLINEAR);
}
bool BatchedNMSDynamicPlugin::supportsFormatCombination(
int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept
{
- ASSERT(0 <= pos && pos < 6);
+ ASSERT(nbInputs <= 2 && nbInputs >= 0);
+ ASSERT(nbOutputs <= 4 && nbOutputs >= 0);
+ ASSERT(pos < 6 && pos >= 0);
const auto* in = inOut;
const auto* out = inOut + nbInputs;
const bool consistentFloatPrecision = in[0].type == in[pos].type;
@@ -400,33 +497,56 @@ void BatchedNMSDynamicPlugin::destroy() noexcept
IPluginV2Ext* BatchedNMSPlugin::clone() const noexcept
{
- auto* plugin = new BatchedNMSPlugin(param);
- plugin->boxesSize = boxesSize;
- plugin->scoresSize = scoresSize;
- plugin->numPriors = numPriors;
- plugin->setPluginNamespace(mNamespace.c_str());
- plugin->setClipParam(mClipBoxes);
- plugin->mPrecision = mPrecision;
- plugin->setScoreBits(mScoreBits);
- return plugin;
+ try
+ {
+ auto* plugin = new BatchedNMSPlugin(param);
+ plugin->boxesSize = boxesSize;
+ plugin->scoresSize = scoresSize;
+ plugin->numPriors = numPriors;
+ plugin->setPluginNamespace(mNamespace.c_str());
+ plugin->setClipParam(mClipBoxes);
+ plugin->mPrecision = mPrecision;
+ plugin->setScoreBits(mScoreBits);
+ return plugin;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
IPluginV2DynamicExt* BatchedNMSDynamicPlugin::clone() const noexcept
{
- auto* plugin = new BatchedNMSDynamicPlugin(param);
- plugin->boxesSize = boxesSize;
- plugin->scoresSize = scoresSize;
- plugin->numPriors = numPriors;
- plugin->setPluginNamespace(mNamespace.c_str());
- plugin->setClipParam(mClipBoxes);
- plugin->mPrecision = mPrecision;
- plugin->setScoreBits(mScoreBits);
- return plugin;
+ try
+ {
+ auto* plugin = new BatchedNMSDynamicPlugin(param);
+ plugin->boxesSize = boxesSize;
+ plugin->scoresSize = scoresSize;
+ plugin->numPriors = numPriors;
+ plugin->setPluginNamespace(mNamespace.c_str());
+ plugin->setClipParam(mClipBoxes);
+ plugin->mPrecision = mPrecision;
+ plugin->setScoreBits(mScoreBits);
+ return plugin;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
void BatchedNMSPlugin::setPluginNamespace(const char* pluginNamespace) noexcept
{
- mNamespace = pluginNamespace;
+ try
+ {
+ mNamespace = pluginNamespace;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
}
const char* BatchedNMSPlugin::getPluginNamespace() const noexcept
@@ -436,7 +556,14 @@ const char* BatchedNMSPlugin::getPluginNamespace() const noexcept
void BatchedNMSDynamicPlugin::setPluginNamespace(const char* pluginNamespace) noexcept
{
- mNamespace = pluginNamespace;
+ try
+ {
+ mNamespace = pluginNamespace;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
}
const char* BatchedNMSDynamicPlugin::getPluginNamespace() const noexcept
@@ -484,8 +611,7 @@ void BatchedNMSDynamicPlugin::setScoreBits(int32_t scoreBits) noexcept
mScoreBits = scoreBits;
}
-bool BatchedNMSPlugin::isOutputBroadcastAcrossBatch(int outputIndex, const bool* inputIsBroadcasted, int nbInputs) const
- noexcept
+bool BatchedNMSPlugin::isOutputBroadcastAcrossBatch(int outputIndex, const bool* inputIsBroadcasted, int nbInputs) const noexcept
{
return false;
}
@@ -495,8 +621,7 @@ bool BatchedNMSPlugin::canBroadcastInputAcrossBatch(int inputIndex) const noexce
return false;
}
-BatchedNMSBasePluginCreator::BatchedNMSBasePluginCreator() noexcept
- : params{}
+BatchedNMSBasePluginCreator::BatchedNMSBasePluginCreator()
{
mPluginAttributes.clear();
mPluginAttributes.emplace_back(PluginField("shareLocation", nullptr, PluginFieldType::kINT32, 1));
@@ -513,19 +638,14 @@ BatchedNMSBasePluginCreator::BatchedNMSBasePluginCreator() noexcept
mFC.fields = mPluginAttributes.data();
}
-BatchedNMSPluginCreator::BatchedNMSPluginCreator() noexcept
+const char* BatchedNMSPluginCreator::getPluginName() const noexcept
{
- mPluginName = NMS_PLUGIN_NAMES[0];
+ return NMS_PLUGIN_NAMES[0];
}
-BatchedNMSDynamicPluginCreator::BatchedNMSDynamicPluginCreator() noexcept
+const char* BatchedNMSDynamicPluginCreator::getPluginName() const noexcept
{
- mPluginName = NMS_PLUGIN_NAMES[1];
-}
-
-const char* BatchedNMSBasePluginCreator::getPluginName() const noexcept
-{
- return mPluginName.c_str();
+ return NMS_PLUGIN_NAMES[1];
}
const char* BatchedNMSBasePluginCreator::getPluginVersion() const noexcept
@@ -540,147 +660,183 @@ const PluginFieldCollection* BatchedNMSBasePluginCreator::getFieldNames() noexce
IPluginV2Ext* BatchedNMSPluginCreator::createPlugin(const char* name, const PluginFieldCollection* fc) noexcept
{
- const PluginField* fields = fc->fields;
- mClipBoxes = true;
- mScoreBits = 16;
- for (int i = 0; i < fc->nbFields; ++i)
+ try
{
- const char* attrName = fields[i].name;
- if (!strcmp(attrName, "shareLocation"))
- {
- params.shareLocation = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "backgroundLabelId"))
- {
- ASSERT(fields[i].type == PluginFieldType::kINT32);
- params.backgroundLabelId = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "numClasses"))
- {
- ASSERT(fields[i].type == PluginFieldType::kINT32);
- params.numClasses = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "topK"))
- {
- ASSERT(fields[i].type == PluginFieldType::kINT32);
- params.topK = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "keepTopK"))
- {
- ASSERT(fields[i].type == PluginFieldType::kINT32);
- params.keepTopK = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "scoreThreshold"))
- {
- ASSERT(fields[i].type == PluginFieldType::kFLOAT32);
- params.scoreThreshold = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "iouThreshold"))
- {
- ASSERT(fields[i].type == PluginFieldType::kFLOAT32);
- params.iouThreshold = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "isNormalized"))
- {
- params.isNormalized = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "clipBoxes"))
- {
- mClipBoxes = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "scoreBits"))
- {
- mScoreBits = *(static_cast(fields[i].data));
- }
- }
+ NMSParameters params;
+ const PluginField* fields = fc->fields;
+ bool clipBoxes = true;
+ int32_t scoreBits = 16;
- auto* plugin = new BatchedNMSPlugin(params);
- plugin->setClipParam(mClipBoxes);
- plugin->setScoreBits(mScoreBits);
- plugin->setPluginNamespace(mNamespace.c_str());
- return plugin;
+ for (int i = 0; i < fc->nbFields; ++i)
+ {
+ const char* attrName = fields[i].name;
+ if (!strcmp(attrName, "shareLocation"))
+ {
+ params.shareLocation = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "backgroundLabelId"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kINT32);
+ params.backgroundLabelId = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "numClasses"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kINT32);
+ params.numClasses = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "topK"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kINT32);
+ params.topK = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "keepTopK"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kINT32);
+ params.keepTopK = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "scoreThreshold"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kFLOAT32);
+ params.scoreThreshold = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "iouThreshold"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kFLOAT32);
+ params.iouThreshold = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "isNormalized"))
+ {
+ params.isNormalized = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "clipBoxes"))
+ {
+ clipBoxes = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "scoreBits"))
+ {
+ scoreBits = *(static_cast(fields[i].data));
+ }
+ }
+
+ auto* plugin = new BatchedNMSPlugin(params);
+ plugin->setClipParam(clipBoxes);
+ plugin->setScoreBits(scoreBits);
+ plugin->setPluginNamespace(mNamespace.c_str());
+ return plugin;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
IPluginV2DynamicExt* BatchedNMSDynamicPluginCreator::createPlugin(
const char* name, const PluginFieldCollection* fc) noexcept
{
- const PluginField* fields = fc->fields;
- mClipBoxes = true;
- mScoreBits = 16;
- for (int i = 0; i < fc->nbFields; ++i)
+ try
{
- const char* attrName = fields[i].name;
- if (!strcmp(attrName, "shareLocation"))
- {
- params.shareLocation = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "backgroundLabelId"))
- {
- ASSERT(fields[i].type == PluginFieldType::kINT32);
- params.backgroundLabelId = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "numClasses"))
- {
- ASSERT(fields[i].type == PluginFieldType::kINT32);
- params.numClasses = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "topK"))
- {
- ASSERT(fields[i].type == PluginFieldType::kINT32);
- params.topK = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "keepTopK"))
- {
- ASSERT(fields[i].type == PluginFieldType::kINT32);
- params.keepTopK = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "scoreThreshold"))
- {
- ASSERT(fields[i].type == PluginFieldType::kFLOAT32);
- params.scoreThreshold = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "iouThreshold"))
- {
- ASSERT(fields[i].type == PluginFieldType::kFLOAT32);
- params.iouThreshold = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "isNormalized"))
- {
- params.isNormalized = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "clipBoxes"))
- {
- mClipBoxes = *(static_cast(fields[i].data));
- }
- else if (!strcmp(attrName, "scoreBits"))
- {
- mScoreBits = *(static_cast(fields[i].data));
- }
- }
+ NMSParameters params;
+ const PluginField* fields = fc->fields;
+ bool clipBoxes = true;
+ int32_t scoreBits = 16;
- auto* plugin = new BatchedNMSDynamicPlugin(params);
- plugin->setClipParam(mClipBoxes);
- plugin->setScoreBits(mScoreBits);
- plugin->setPluginNamespace(mNamespace.c_str());
- return plugin;
+ for (int i = 0; i < fc->nbFields; ++i)
+ {
+ const char* attrName = fields[i].name;
+ if (!strcmp(attrName, "shareLocation"))
+ {
+ params.shareLocation = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "backgroundLabelId"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kINT32);
+ params.backgroundLabelId = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "numClasses"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kINT32);
+ params.numClasses = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "topK"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kINT32);
+ params.topK = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "keepTopK"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kINT32);
+ params.keepTopK = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "scoreThreshold"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kFLOAT32);
+ params.scoreThreshold = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "iouThreshold"))
+ {
+ ASSERT(fields[i].type == PluginFieldType::kFLOAT32);
+ params.iouThreshold = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "isNormalized"))
+ {
+ params.isNormalized = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "clipBoxes"))
+ {
+ clipBoxes = *(static_cast(fields[i].data));
+ }
+ else if (!strcmp(attrName, "scoreBits"))
+ {
+ scoreBits = *(static_cast(fields[i].data));
+ }
+ }
+
+ auto* plugin = new BatchedNMSDynamicPlugin(params);
+ plugin->setClipParam(clipBoxes);
+ plugin->setScoreBits(scoreBits);
+ plugin->setPluginNamespace(mNamespace.c_str());
+ return plugin;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
IPluginV2Ext* BatchedNMSPluginCreator::deserializePlugin(
const char* name, const void* serialData, size_t serialLength) noexcept
{
- // This object will be deleted when the network is destroyed, which will
- // call NMS::destroy()
- auto* plugin = new BatchedNMSPlugin(serialData, serialLength);
- plugin->setPluginNamespace(mNamespace.c_str());
- return plugin;
+ try
+ {
+ // This object will be deleted when the network is destroyed, which will
+ // call NMS::destroy()
+ auto* plugin = new BatchedNMSPlugin(serialData, serialLength);
+ plugin->setPluginNamespace(mNamespace.c_str());
+ return plugin;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
IPluginV2DynamicExt* BatchedNMSDynamicPluginCreator::deserializePlugin(
const char* name, const void* serialData, size_t serialLength) noexcept
{
- // This object will be deleted when the network is destroyed, which will
- // call NMS::destroy()
- auto* plugin = new BatchedNMSDynamicPlugin(serialData, serialLength);
- plugin->setPluginNamespace(mNamespace.c_str());
- return plugin;
+ try
+ {
+ // This object will be deleted when the network is destroyed, which will
+ // call NMS::destroy()
+ auto* plugin = new BatchedNMSDynamicPlugin(serialData, serialLength);
+ plugin->setPluginNamespace(mNamespace.c_str());
+ return plugin;
+ }
+ catch (const std::exception& e)
+ {
+ caughtError(e);
+ }
+ return nullptr;
}
diff --git a/plugin/batchedNMSPlugin/batchedNMSPlugin.h b/plugin/batchedNMSPlugin/batchedNMSPlugin.h
index 4c4d82ea..2c441340 100644
--- a/plugin/batchedNMSPlugin/batchedNMSPlugin.h
+++ b/plugin/batchedNMSPlugin/batchedNMSPlugin.h
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#ifndef TRT_BATCHED_NMS_PLUGIN_H
#define TRT_BATCHED_NMS_PLUGIN_H
#include "gatherNMSOutputs.h"
@@ -32,9 +31,9 @@ namespace plugin
class BatchedNMSPlugin : public IPluginV2Ext
{
public:
- BatchedNMSPlugin(NMSParameters param) noexcept;
- BatchedNMSPlugin(const void* data, size_t length) noexcept;
- ~BatchedNMSPlugin() noexcept override = default;
+ BatchedNMSPlugin(NMSParameters param);
+ BatchedNMSPlugin(const void* data, size_t length);
+ ~BatchedNMSPlugin() override = default;
// IPluginV2 methods
const char* getPluginType() const noexcept override;
@@ -43,7 +42,7 @@ public:
Dims getOutputDimensions(int index, const Dims* inputs, int nbInputDims) noexcept override;
bool supportsFormat(DataType type, PluginFormat format) const noexcept override;
size_t getWorkspaceSize(int maxBatchSize) const noexcept override;
- int enqueue(int batchSize, const void* const* inputs, void** outputs, void* workspace,
+ int32_t enqueue(int32_t batchSize, void const* const* inputs, void* const* outputs, void* workspace,
cudaStream_t stream) noexcept override;
int initialize() noexcept override;
void terminate() noexcept override;
@@ -56,7 +55,7 @@ public:
void setScoreBits(int32_t scoreBits) noexcept;
// IPluginV2Ext methods
- nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputType, int nbInputs) const
+ nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const
noexcept override;
bool isOutputBroadcastAcrossBatch(int outputIndex, const bool* inputIsBroadcasted, int nbInputs) const
noexcept override;
@@ -75,14 +74,15 @@ private:
bool mClipBoxes{};
DataType mPrecision;
int32_t mScoreBits;
+ pluginStatus_t mPluginStatus{};
};
class BatchedNMSDynamicPlugin : public IPluginV2DynamicExt
{
public:
- BatchedNMSDynamicPlugin(NMSParameters param) noexcept;
- BatchedNMSDynamicPlugin(const void* data, size_t length) noexcept;
- ~BatchedNMSDynamicPlugin() noexcept override = default;
+ BatchedNMSDynamicPlugin(NMSParameters param);
+ BatchedNMSDynamicPlugin(const void* data, size_t length);
+ ~BatchedNMSDynamicPlugin() override = default;
// IPluginV2 methods
const char* getPluginType() const noexcept override;
@@ -99,19 +99,17 @@ public:
void setScoreBits(int32_t scoreBits) noexcept;
// IPluginV2Ext methods
- nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputType, int nbInputs) const
- noexcept override;
+ nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputType, int nbInputs) const noexcept override;
// IPluginV2DynamicExt methods
IPluginV2DynamicExt* clone() const noexcept override;
DimsExprs getOutputDimensions(
int outputIndex, const DimsExprs* inputs, int nbInputs, IExprBuilder& exprBuilder) noexcept override;
- bool supportsFormatCombination(
- int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept override;
- void configurePlugin(const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out,
- int nbOutputs) noexcept override;
- size_t getWorkspaceSize(const PluginTensorDesc* inputs, int nbInputs, const PluginTensorDesc* outputs,
- int nbOutputs) const noexcept override;
+ bool supportsFormatCombination(int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept override;
+ void configurePlugin(
+ const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out, int nbOutputs) noexcept override;
+ size_t getWorkspaceSize(
+ const PluginTensorDesc* inputs, int nbInputs, const PluginTensorDesc* outputs, int nbOutputs) const noexcept override;
int enqueue(const PluginTensorDesc* inputDesc, const PluginTensorDesc* outputDesc, const void* const* inputs,
void* const* outputs, void* workspace, cudaStream_t stream) noexcept override;
@@ -124,33 +122,27 @@ private:
bool mClipBoxes{};
DataType mPrecision;
int32_t mScoreBits;
+ pluginStatus_t mPluginStatus{};
};
class BatchedNMSBasePluginCreator : public BaseCreator
{
public:
- BatchedNMSBasePluginCreator() noexcept;
- ~BatchedNMSBasePluginCreator() noexcept override = default;
+ BatchedNMSBasePluginCreator();
+ ~BatchedNMSBasePluginCreator() override = default;
- const char* getPluginName() const noexcept override;
const char* getPluginVersion() const noexcept override;
const PluginFieldCollection* getFieldNames() noexcept override;
protected:
static PluginFieldCollection mFC;
- NMSParameters params;
static std::vector mPluginAttributes;
- bool mClipBoxes;
- int32_t mScoreBits;
- std::string mPluginName;
};
class BatchedNMSPluginCreator : public BatchedNMSBasePluginCreator
{
public:
- BatchedNMSPluginCreator() noexcept;
- ~BatchedNMSPluginCreator() noexcept override = default;
-
+ const char* getPluginName() const noexcept override;
IPluginV2Ext* createPlugin(const char* name, const PluginFieldCollection* fc) noexcept override;
IPluginV2Ext* deserializePlugin(const char* name, const void* serialData, size_t serialLength) noexcept override;
};
@@ -158,9 +150,7 @@ public:
class BatchedNMSDynamicPluginCreator : public BatchedNMSBasePluginCreator
{
public:
- BatchedNMSDynamicPluginCreator() noexcept;
- ~BatchedNMSDynamicPluginCreator() noexcept override = default;
-
+ const char* getPluginName() const noexcept override;
IPluginV2DynamicExt* createPlugin(const char* name, const PluginFieldCollection* fc) noexcept override;
IPluginV2DynamicExt* deserializePlugin(
const char* name, const void* serialData, size_t serialLength) noexcept override;
diff --git a/plugin/batchedNMSPlugin/gatherNMSOutputs.h b/plugin/batchedNMSPlugin/gatherNMSOutputs.h
index 5001f253..b6a8feaf 100644
--- a/plugin/batchedNMSPlugin/gatherNMSOutputs.h
+++ b/plugin/batchedNMSPlugin/gatherNMSOutputs.h
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#ifndef TRT_BATCHED_NMS_HELPER_H
#define TRT_BATCHED_NMS_HELPER_H
#include "plugin.h"
diff --git a/plugin/bertQKVToContextPlugin/fused_multihead_attention.h b/plugin/bertQKVToContextPlugin/fused_multihead_attention.h
index b797b862..fd29fd7c 100644
--- a/plugin/bertQKVToContextPlugin/fused_multihead_attention.h
+++ b/plugin/bertQKVToContextPlugin/fused_multihead_attention.h
@@ -100,31 +100,31 @@ extern unsigned char fused_multihead_attention_fp16_128_64_kernel_sm80_cu_o[];
extern unsigned char fused_multihead_attention_fp16_384_64_kernel_sm80_cu_o[];
extern unsigned char fused_multihead_attention_fp16_384_64_kernel_sm86_cu_o[];
-extern unsigned int fused_multihead_attention_fp16_64_64_kernel_sm75_cu_o_len;
-extern unsigned int fused_multihead_attention_fp16_96_64_kernel_sm75_cu_o_len;
-extern unsigned int fused_multihead_attention_fp16_64_64_kernel_sm80_cu_o_len;
-extern unsigned int fused_multihead_attention_fp16_96_64_kernel_sm80_cu_o_len;
-extern unsigned int fused_multihead_attention_fp16_128_64_kernel_sm75_cu_o_len;
-extern unsigned int fused_multihead_attention_fp16_384_64_kernel_sm75_cu_o_len;
-extern unsigned int fused_multihead_attention_int8_128_64_kernel_sm75_cu_o_len;
-extern unsigned int fused_multihead_attention_int8_384_64_kernel_sm75_cu_o_len;
-extern unsigned int fused_multihead_attention_int8_384_64_kernel_sm80_cu_o_len;
-extern unsigned int fused_multihead_attention_int8_128_64_kernel_sm80_cu_o_len;
-extern unsigned int fused_multihead_attention_fp16_128_64_kernel_sm80_cu_o_len;
-extern unsigned int fused_multihead_attention_fp16_384_64_kernel_sm80_cu_o_len;
-extern unsigned int fused_multihead_attention_fp16_384_64_kernel_sm86_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_64_64_kernel_sm75_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_96_64_kernel_sm75_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_64_64_kernel_sm80_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_96_64_kernel_sm80_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_128_64_kernel_sm75_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_384_64_kernel_sm75_cu_o_len;
+extern uint32_t fused_multihead_attention_int8_128_64_kernel_sm75_cu_o_len;
+extern uint32_t fused_multihead_attention_int8_384_64_kernel_sm75_cu_o_len;
+extern uint32_t fused_multihead_attention_int8_384_64_kernel_sm80_cu_o_len;
+extern uint32_t fused_multihead_attention_int8_128_64_kernel_sm80_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_128_64_kernel_sm80_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_384_64_kernel_sm80_cu_o_len;
+extern uint32_t fused_multihead_attention_fp16_384_64_kernel_sm86_cu_o_len;
static const struct FusedMultiHeadAttentionKernelMetaInfoV1
{
Data_type mDataType;
- unsigned int mS;
- unsigned int mD;
- unsigned int mSM;
+ uint32_t mS;
+ uint32_t mD;
+ uint32_t mSM;
const unsigned char* mCubin;
- unsigned int mCubinSize;
+ uint32_t mCubinSize;
const char* mFuncName;
- unsigned int mSharedMemBytes;
- unsigned int mThreadsPerCTA;
+ uint32_t mSharedMemBytes;
+ uint32_t mThreadsPerCTA;
} sMhaKernelMetaInfos[] = {
// Turing
{DATA_TYPE_FP16, 64, 64, kSM_75, fused_multihead_attention_fp16_64_64_kernel_sm75_cu_o,
@@ -196,7 +196,7 @@ class TFusedMultiHeadAttentionXMMAKernel
public:
using KernelMeta = TKernelMeta;
using KernelParam = TKernelParam;
- inline uint64_t hashID(unsigned int s, unsigned int d) const
+ inline uint64_t hashID(uint32_t s, uint32_t d) const
{
return (uint64_t) s << 32 | d;
}
@@ -205,8 +205,7 @@ public:
return hashID(kernelMeta.mS, kernelMeta.mD);
}
- TFusedMultiHeadAttentionXMMAKernel(
- const TKernelMeta* pMetaStart, unsigned int nMetaCount, Data_type type, unsigned int sm)
+ TFusedMultiHeadAttentionXMMAKernel(const TKernelMeta* pMetaStart, uint32_t nMetaCount, Data_type type, uint32_t sm)
: mDataType(type)
, mKernelMeta(pMetaStart)
, mKernelMetaCount(nMetaCount)
@@ -221,7 +220,7 @@ public:
return;
}
- for (unsigned int i = 0; i < mKernelMetaCount; ++i)
+ for (uint32_t i = 0; i < mKernelMetaCount; ++i)
{
const auto& kernelMeta = mKernelMeta[i];
if (kernelMeta.mSM == mSM && kernelMeta.mDataType == mDataType)
@@ -281,12 +280,12 @@ protected:
Data_type mDataType;
const TKernelMeta* mKernelMeta;
- unsigned int mKernelMetaCount;
- unsigned int mSM;
+ uint32_t mKernelMetaCount;
+ uint32_t mSM;
std::unordered_map mModules;
struct FusedMultiHeadAttentionKernelInfo
{
- unsigned int mMetaInfoIndex;
+ uint32_t mMetaInfoIndex;
CUfunction mDeviceFunction;
};
std::unordered_map mFunctions;
@@ -297,8 +296,8 @@ template
class TFusedMHAKernelFactory
{
public:
- const TFusedMHAKernelList* getXMMAKernels(const typename TFusedMHAKernelList::KernelMeta* pKernelList,
- unsigned int nbKernels, Data_type type, unsigned int sm)
+ const TFusedMHAKernelList* getXMMAKernels(
+ const typename TFusedMHAKernelList::KernelMeta* pKernelList, uint32_t nbKernels, Data_type type, uint32_t sm)
{
static std::mutex s_mutex;
std::lock_guard lg(s_mutex);
@@ -324,9 +323,16 @@ public:
private:
TFusedMHAKernelFactory() = default;
- inline uint64_t hashID(Data_type type, unsigned int sm) const
+ inline uint64_t hashID(Data_type type, uint32_t sm) const
{
- return (uint64_t) type << 32 | sm;
+ // use deviceID in hasID for multi GPU support before driver support context-less loading of cubin
+ int32_t deviceID{0};
+ cudaGetDevice(&deviceID);
+
+ ASSERT((deviceID & 0xFFFF) == deviceID);
+ ASSERT((type & 0xFFFF) == type);
+ ASSERT((sm & 0xFFFFFFFF) == sm);
+ return (uint64_t) type << 48 | (uint64_t) deviceID << 32 | sm;
}
std::unordered_map> mKernels;
@@ -336,7 +342,7 @@ using FusedMultiHeadAttentionXMMAKernel
= TFusedMultiHeadAttentionXMMAKernel;
using FusedMHAKernelFactory = TFusedMHAKernelFactory;
-inline const FusedMultiHeadAttentionXMMAKernel* getXMMAKernels(Data_type type, unsigned int sm)
+inline const FusedMultiHeadAttentionXMMAKernel* getXMMAKernels(Data_type type, uint32_t sm)
{
return FusedMHAKernelFactory::Get().getXMMAKernels(
sMhaKernelMetaInfos, sizeof(sMhaKernelMetaInfos) / sizeof(sMhaKernelMetaInfos[0]), type, sm);
diff --git a/plugin/bertQKVToContextPlugin/fused_multihead_attention_common.h b/plugin/bertQKVToContextPlugin/fused_multihead_attention_common.h
index 83214dc3..0e41db2e 100644
--- a/plugin/bertQKVToContextPlugin/fused_multihead_attention_common.h
+++ b/plugin/bertQKVToContextPlugin/fused_multihead_attention_common.h
@@ -16,7 +16,7 @@
#pragma once
-#include
+#include
namespace bert
{
enum Data_type
diff --git a/plugin/bertQKVToContextPlugin/fused_multihead_attention_v2.h b/plugin/bertQKVToContextPlugin/fused_multihead_attention_v2.h
index 46813dac..55a33abf 100644
--- a/plugin/bertQKVToContextPlugin/fused_multihead_attention_v2.h
+++ b/plugin/bertQKVToContextPlugin/fused_multihead_attention_v2.h
@@ -17,8 +17,8 @@
#pragma once
#include "fused_multihead_attention.h"
#include "fused_multihead_attention_common.h"
-#include
-#include
+#include
+#include
namespace bert
{
@@ -141,50 +141,50 @@ extern unsigned char fused_multihead_attention_v2_int8_384_64_kernel_sm75_cubin[
extern unsigned char fused_multihead_attention_v2_int8_384_64_kernel_sm80_cubin[];
extern unsigned char fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin[];
-extern unsigned int fused_multihead_attention_v2_fp16_128_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_128_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_128_64_kernel_sm86_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_256_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_256_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_256_64_kernel_sm86_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_384_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_384_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_384_64_kernel_sm86_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_64_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_64_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_64_64_kernel_sm86_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_96_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_96_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_fp16_96_64_kernel_sm86_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_128_64_kernel_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_128_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_128_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_128_64_kernel_sm86_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_192_64_kernel_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_192_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_192_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_256_64_kernel_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_256_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_256_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_384_64_kernel_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_384_64_kernel_sm75_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_384_64_kernel_sm80_cubin_len;
-extern unsigned int fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_128_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_128_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_128_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_256_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_256_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_256_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_384_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_384_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_384_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_64_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_64_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_64_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_96_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_96_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_fp16_96_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_128_64_kernel_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_128_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_128_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_128_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_192_64_kernel_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_192_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_192_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_256_64_kernel_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_256_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_256_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_384_64_kernel_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_384_64_kernel_sm75_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_384_64_kernel_sm80_cubin_len;
+extern uint32_t fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin_len;
static const struct FusedMultiHeadAttentionKernelMetaInfoV2
{
Data_type mDataType;
- unsigned int mS;
- unsigned int mD;
- unsigned int mSM;
+ uint32_t mS;
+ uint32_t mD;
+ uint32_t mSM;
const unsigned char* mCubin;
- unsigned int mCubinSize;
+ uint32_t mCubinSize;
const char* mFuncName;
- unsigned int mSharedMemBytes;
- unsigned int mThreadsPerCTA;
- unsigned int mUnrollStep;
+ uint32_t mSharedMemBytes;
+ uint32_t mThreadsPerCTA;
+ uint32_t mUnrollStep;
bool mInterleaved;
} sMhaKernelMetaInfosV2[] = {
// Xavier
@@ -391,6 +391,7 @@ static const struct FusedMultiHeadAttentionKernelMetaInfoV2
fused_multihead_attention_v2_fp16_384_64_kernel_sm86_cubin_len,
"fused_multihead_attention_v2_fp16_384_64_kernel_sm80", 65536, 256, 0, false},
+
{DATA_TYPE_INT8, 128, 64, kSM_86, fused_multihead_attention_v2_int8_128_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_128_64_kernel_sm86_cubin_len,
"fused_multihead_attention_v2_int8_128_64_kernel_sm80_interleaved_noloop", 20480, 128, 16, true},
@@ -405,40 +406,40 @@ static const struct FusedMultiHeadAttentionKernelMetaInfoV2
"fused_multihead_attention_v2_int8_128_64_kernel_sm80", 32768, 128, 0, false},
{DATA_TYPE_INT8, 192, 64, kSM_86, fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_192_64_kernel_sm80_interleaved_noloop", 28672, 128, 32, true},
+ "fused_multihead_attention_v2_int8_192_64_kernel_sm86_interleaved_noloop", 28672, 128, 32, true},
{DATA_TYPE_INT8, 192, 64, kSM_86, fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_192_64_kernel_sm80_noloop", 28672, 128, 32, false},
+ "fused_multihead_attention_v2_int8_192_64_kernel_sm86_noloop", 28672, 128, 32, false},
{DATA_TYPE_INT8, 192, 64, kSM_86, fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_192_64_kernel_sm80_interleaved", 32768, 128, 0, true},
+ "fused_multihead_attention_v2_int8_192_64_kernel_sm86_interleaved", 32768, 128, 0, true},
{DATA_TYPE_INT8, 192, 64, kSM_86, fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_192_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_192_64_kernel_sm80", 32768, 128, 0, false},
+ "fused_multihead_attention_v2_int8_192_64_kernel_sm86", 32768, 128, 0, false},
{DATA_TYPE_INT8, 256, 64, kSM_86, fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_256_64_kernel_sm80_interleaved_noloop", 36864, 128, 32, true},
+ "fused_multihead_attention_v2_int8_256_64_kernel_sm86_interleaved_noloop", 36864, 128, 32, true},
{DATA_TYPE_INT8, 256, 64, kSM_86, fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_256_64_kernel_sm80_noloop", 36864, 128, 32, false},
+ "fused_multihead_attention_v2_int8_256_64_kernel_sm86_noloop", 36864, 128, 32, false},
{DATA_TYPE_INT8, 256, 64, kSM_86, fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_256_64_kernel_sm80_interleaved", 36864, 128, 0, true},
+ "fused_multihead_attention_v2_int8_256_64_kernel_sm86_interleaved", 36864, 128, 0, true},
{DATA_TYPE_INT8, 256, 64, kSM_86, fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_256_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_256_64_kernel_sm80", 36864, 128, 0, false},
+ "fused_multihead_attention_v2_int8_256_64_kernel_sm86", 36864, 128, 0, false},
{DATA_TYPE_INT8, 384, 64, kSM_86, fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_384_64_kernel_sm80_interleaved_noloop", 53248, 128, 32, true},
+ "fused_multihead_attention_v2_int8_384_64_kernel_sm86_interleaved_noloop", 28672, 128, 32, true},
{DATA_TYPE_INT8, 384, 64, kSM_86, fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_384_64_kernel_sm80_noloop", 53248, 128, 32, false},
+ "fused_multihead_attention_v2_int8_384_64_kernel_sm86_noloop", 28672, 128, 32, false},
{DATA_TYPE_INT8, 384, 64, kSM_86, fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_384_64_kernel_sm80_interleaved", 51200, 128, 0, true},
+ "fused_multihead_attention_v2_int8_384_64_kernel_sm86_interleaved", 28672, 128, 0, true},
{DATA_TYPE_INT8, 384, 64, kSM_86, fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin,
fused_multihead_attention_v2_int8_384_64_kernel_sm86_cubin_len,
- "fused_multihead_attention_v2_int8_384_64_kernel_sm80", 53248, 128, 0, false},
+ "fused_multihead_attention_v2_int8_384_64_kernel_sm86", 28672, 128, 0, false},
#endif
};
@@ -447,14 +448,14 @@ class FusedMultiHeadAttentionXMMAKernelV2
Fused_multihead_attention_params_v2>
{
public:
- FusedMultiHeadAttentionXMMAKernelV2(const FusedMultiHeadAttentionKernelMetaInfoV2* pMetaStart,
- unsigned int nMetaCount, Data_type type, unsigned int sm)
+ FusedMultiHeadAttentionXMMAKernelV2(
+ const FusedMultiHeadAttentionKernelMetaInfoV2* pMetaStart, uint32_t nMetaCount, Data_type type, uint32_t sm)
: TFusedMultiHeadAttentionXMMAKernel(pMetaStart, nMetaCount, type, sm)
{
}
- inline uint64_t hashID(unsigned int s, bool interleaved, bool unroll) const
+ inline uint64_t hashID(uint32_t s, bool interleaved, bool unroll) const
{
return (uint64_t) s << 32 | (interleaved ? 2ull : 0ull) | (unroll ? 1ull : 0ull);
}
@@ -478,7 +479,7 @@ public:
{
const struct
{
- unsigned int mSM;
+ uint32_t mSM;
Data_type mDataType;
int mS;
int mMaxBatch;
@@ -506,7 +507,7 @@ public:
{kSM_86, bert::DATA_TYPE_INT8, 384, 8},
#endif
};
- for (unsigned int i = 0u; i < sizeof(unrollList) / sizeof(unrollList[0]); ++i)
+ for (uint32_t i = 0u; i < sizeof(unrollList) / sizeof(unrollList[0]); ++i)
{
if (mSM == unrollList[i].mSM && mDataType == unrollList[i].mDataType && params.s == unrollList[i].mS
&& params.b <= unrollList[i].mMaxBatch)
@@ -543,7 +544,7 @@ public:
using FusedMHAKernelFactoryV2 = TFusedMHAKernelFactory;
-inline const FusedMultiHeadAttentionXMMAKernelV2* getXMMAKernelsV2(Data_type type, unsigned int sm)
+inline const FusedMultiHeadAttentionXMMAKernelV2* getXMMAKernelsV2(Data_type type, uint32_t sm)
{
return FusedMHAKernelFactoryV2::Get().getXMMAKernels(
sMhaKernelMetaInfosV2, sizeof(sMhaKernelMetaInfosV2) / sizeof(sMhaKernelMetaInfosV2[0]), type, sm);
diff --git a/plugin/bertQKVToContextPlugin/qkvToContext.cu b/plugin/bertQKVToContextPlugin/qkvToContext.cu
index 14710371..24c90497 100644
--- a/plugin/bertQKVToContextPlugin/qkvToContext.cu
+++ b/plugin/bertQKVToContextPlugin/qkvToContext.cu
@@ -333,7 +333,8 @@ int computeMaskedScaledSoftmax(cudaStream_t stream, const int ld, const int B, c
return 0;
}
-std::pair tuneBatchedGemm(const int B, const int S, const int numHeads, const int headSize)
+std::pair tuneBatchedGemm(
+ const int B, const int S, const int numHeads, const int headSize, const int smVersion)
{
const int nruns = 500;
cublasHandle_t cublas;
@@ -378,6 +379,8 @@ std::pair tuneBatchedGemm(const int B, const int S, const int numHeads
int best2 = startAlgo;
float ms1 = 1000000;
float ms2 = 1000000;
+
+ ASSERT(smVersion >= kSM_53);
for (int a = startAlgo; a <= endAlgo; a++)
{
cublasGemmAlgo_t algo = static_cast(a);
@@ -439,12 +442,12 @@ template int computeMaskedScaledSoftmax(cudaStream_t stream, const int ld
template int computeMaskedScaledSoftmax(cudaStream_t stream, const int ld, const int B, const int N,
const float rsqrtHeadSize, const int* maskIdx, const half* input, half* output);
-size_t MHARunner::getSerializationSize() const
+size_t MHARunner::getSerializationSize() const noexcept
{
return sizeof(mS) + sizeof(mB);
}
-void MHARunner::serialize(void* buffer) const
+void MHARunner::serialize(void* buffer) const noexcept
{
serialize_value(&buffer, mS);
serialize_value(&buffer, mB);
@@ -457,11 +460,12 @@ void MHARunner::deserialize(const void* data, size_t length)
setup(mS, mB);
}
-UnfusedMHARunner::UnfusedMHARunner(const nvinfer1::DataType type, const int numHeads, const int headSize)
+UnfusedMHARunner::UnfusedMHARunner(const nvinfer1::DataType type, const int numHeads, const int headSize, const int sm)
: MHARunner(type, numHeads, headSize)
, mIsBestAlgoFound(false)
, mAlgoBatchedEx1(CUBLAS_GEMM_DEFAULT_TENSOR_OP)
, mAlgoBatchedEx2(CUBLAS_GEMM_DEFAULT_TENSOR_OP)
+ , mSm(sm)
{
CUBLASASSERT(cublasCreate(&mCublas));
}
@@ -471,12 +475,12 @@ UnfusedMHARunner::~UnfusedMHARunner()
CUBLASASSERT(cublasDestroy(mCublas));
}
-size_t UnfusedMHARunner::getSerializationSize() const
+size_t UnfusedMHARunner::getSerializationSize() const noexcept
{
return sizeof(mAlgoBatchedEx1) + sizeof(mAlgoBatchedEx2) + MHARunner::getSerializationSize();
}
-void UnfusedMHARunner::serialize(void* buffer) const
+void UnfusedMHARunner::serialize(void* buffer) const noexcept
{
serialize_value(&buffer, mAlgoBatchedEx1);
serialize_value(&buffer, mAlgoBatchedEx2);
@@ -496,7 +500,7 @@ void UnfusedMHARunner::setup(const int S, const int B)
MHARunner::setup(S, B);
if (mType == DataType::kHALF && !mIsBestAlgoFound)
{
- std::tie(mAlgoBatchedEx1, mAlgoBatchedEx2) = tuneBatchedGemm(B, S, mNumHeads, mHeadSize);
+ std::tie(mAlgoBatchedEx1, mAlgoBatchedEx2) = tuneBatchedGemm(B, S, mNumHeads, mHeadSize, mSm);
mIsBestAlgoFound = true;
gLogVerbose << "QKV Plugin - Selected Algos for batch gemms: " << mAlgoBatchedEx1 << ", " << mAlgoBatchedEx2
diff --git a/plugin/bertQKVToContextPlugin/qkvToContextInt8InterleavedPlugin.cpp b/plugin/bertQKVToContextPlugin/qkvToContextInt8InterleavedPlugin.cpp
index 51ab1d56..6c62f49f 100644
--- a/plugin/bertQKVToContextPlugin/qkvToContextInt8InterleavedPlugin.cpp
+++ b/plugin/bertQKVToContextPlugin/qkvToContextInt8InterleavedPlugin.cpp
@@ -14,9 +14,9 @@
* limitations under the License.
*/
-#include "qkvToContextInt8InterleavedPlugin.h"
#include "NvInfer.h"
#include "bertCommon.h"
+#include "qkvToContextInt8InterleavedPlugin.h"
#include "serialize.hpp"
#include
@@ -36,8 +36,8 @@ namespace bert
namespace
{
-static const char* QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_VERSION{"3"};
-static const char* QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_NAME{"CustomQKVToContextPluginDynamic"};
+const char* QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_VERSION{"3"};
+const char* QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_NAME{"CustomQKVToContextPluginDynamic"};
} // namespace
// Static class fields initialization
@@ -80,7 +80,7 @@ QKVToContextInterleavedPlugin::QKVToContextInterleavedPlugin(const std::string n
deserialize_value(&data, &length, &mDqProbs);
}
-int QKVToContextInterleavedPlugin::getSMVersion() const
+int QKVToContextInterleavedPlugin::getSMVersion() const noexcept
{
int device{-1};
CHECK(cudaGetDevice(&device));
@@ -90,7 +90,7 @@ int QKVToContextInterleavedPlugin::getSMVersion() const
}
// IPluginV2DynamicExt Methods
-nvinfer1::IPluginV2DynamicExt* QKVToContextInterleavedPlugin::clone() const
+nvinfer1::IPluginV2DynamicExt* QKVToContextInterleavedPlugin::clone() const noexcept
{
QKVToContextInterleavedPlugin* ret
= new QKVToContextInterleavedPlugin(mLayerName, mHiddenSize, mNumHeads, mDqProbs);
@@ -100,7 +100,7 @@ nvinfer1::IPluginV2DynamicExt* QKVToContextInterleavedPlugin::clone() const
}
DimsExprs QKVToContextInterleavedPlugin::getOutputDimensions(
- int outputIndex, const DimsExprs* inputs, int nbInputs, IExprBuilder& exprBuilder)
+ int outputIndex, const DimsExprs* inputs, int nbInputs, IExprBuilder& exprBuilder) noexcept
{
// Input SHAPE is 1x(3*N*H)xTotalx1 (NCHW)
// Output SHAPE is 1x(N*H)xTotalx1
@@ -111,12 +111,12 @@ DimsExprs QKVToContextInterleavedPlugin::getOutputDimensions(
DimsExprs output(inputs[IIDX]);
// output.d[0] = exprBuilder.constant(1);
// Divide last dim by three
- auto three = exprBuilder.constant(3);
+ const auto* three = exprBuilder.constant(3);
output.d[1] = exprBuilder.operation(DimensionOperation::kFLOOR_DIV, *inputs[IIDX].d[1], *three);
return output;
}
bool QKVToContextInterleavedPlugin::supportsFormatCombination(
- int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs)
+ int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept
{
assert(nbInputs == 3);
assert(nbOutputs == 1);
@@ -145,54 +145,54 @@ bool QKVToContextInterleavedPlugin::supportsFormatCombination(
}
void QKVToContextInterleavedPlugin::configurePlugin(
- const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out, int nbOutputs)
+ const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out, int nbOutputs) noexcept
{
}
size_t QKVToContextInterleavedPlugin::getWorkspaceSize(
- const PluginTensorDesc* inputs, int nbInputs, const PluginTensorDesc* outputs, int nbOutputs) const
+ const PluginTensorDesc* inputs, int nbInputs, const PluginTensorDesc* outputs, int nbOutputs) const noexcept
{
return 0;
}
// IPluginV2Ext Methods
DataType QKVToContextInterleavedPlugin::getOutputDataType(
- int index, const nvinfer1::DataType* inputTypes, int nbInputs) const
+ int index, const nvinfer1::DataType* inputTypes, int nbInputs) const noexcept
{
assert(index == 0);
return DataType::kINT8;
}
// IPluginV2 Methods
-const char* QKVToContextInterleavedPlugin::getPluginType() const
+const char* QKVToContextInterleavedPlugin::getPluginType() const noexcept
{
return QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_NAME;
}
-const char* QKVToContextInterleavedPlugin::getPluginVersion() const
+const char* QKVToContextInterleavedPlugin::getPluginVersion() const noexcept
{
return QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_VERSION;
}
-int QKVToContextInterleavedPlugin::getNbOutputs() const
+int QKVToContextInterleavedPlugin::getNbOutputs() const noexcept
{
return 1;
}
-int QKVToContextInterleavedPlugin::initialize()
+int QKVToContextInterleavedPlugin::initialize() noexcept
{
return 0;
}
-void QKVToContextInterleavedPlugin::terminate() {}
+void QKVToContextInterleavedPlugin::terminate() noexcept {}
-size_t QKVToContextInterleavedPlugin::getSerializationSize() const
+size_t QKVToContextInterleavedPlugin::getSerializationSize() const noexcept
{
return sizeof(mNumHeads) + sizeof(mHeadSize) + sizeof(mHiddenSize) + sizeof(mSM) + sizeof(mS) + sizeof(mB)
+ sizeof(mDqProbs);
}
-void QKVToContextInterleavedPlugin::serialize(void* buffer) const
+void QKVToContextInterleavedPlugin::serialize(void* buffer) const noexcept
{
serialize_value(&buffer, mNumHeads);
serialize_value(&buffer, mHeadSize);
@@ -203,23 +203,23 @@ void QKVToContextInterleavedPlugin::serialize(void* buffer) const
serialize_value(&buffer, mDqProbs);
}
-void QKVToContextInterleavedPlugin::destroy()
+void QKVToContextInterleavedPlugin::destroy() noexcept
{
delete this;
}
-void QKVToContextInterleavedPlugin::setPluginNamespace(const char* libNamespace)
+void QKVToContextInterleavedPlugin::setPluginNamespace(const char* libNamespace) noexcept
{
mNamespace = libNamespace;
}
-const char* QKVToContextInterleavedPlugin::getPluginNamespace() const
+const char* QKVToContextInterleavedPlugin::getPluginNamespace() const noexcept
{
return mNamespace.c_str();
}
int QKVToContextInterleavedPlugin::enqueue(const PluginTensorDesc* inputDesc, const PluginTensorDesc* outputDesc,
- const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream)
+ const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept
{
const int total = inputDesc[0].dims.d[2];
@@ -255,7 +255,7 @@ int QKVToContextInterleavedPlugin::enqueue(const PluginTensorDesc* inputDesc, co
float scaleBmm1 = scaleQkv * scaleQkv * 0.125; // 1 / sqrt(64)
float scaleBmm2 = mDqProbs * scaleQkv / scaleCtx;
- float scaleSoftmax = 1.f / mDqProbs;
+ float scaleSoftmax = 1.F / mDqProbs;
params.scale_bmm1 = reinterpret_cast(scaleBmm1);
params.scale_bmm2 = reinterpret_cast(scaleBmm2);
@@ -266,35 +266,38 @@ int QKVToContextInterleavedPlugin::enqueue(const PluginTensorDesc* inputDesc, co
params.use_int8_scale_max = true;
params.enable_i2f_trick
- = -double(1 << 22) * double(scaleBmm2) <= -128.f && double(1 << 22) * double(scaleBmm2) >= 127.f;
+ = -double(1 << 22) * double(scaleBmm2) <= -128.F && double(1 << 22) * double(scaleBmm2) >= 127.F;
mXmmaKernel->run(params, stream);
- CHECK(cudaPeekAtLastError());
- return 0;
+ return cudaPeekAtLastError();
}
QKVToContextInterleavedPluginCreator::QKVToContextInterleavedPluginCreator()
{
+ mPluginAttributes.emplace_back(PluginField("hidden_size", nullptr, PluginFieldType::kINT32, 1));
+ mPluginAttributes.emplace_back(PluginField("num_heads", nullptr, PluginFieldType::kINT32, 1));
+ mPluginAttributes.emplace_back(PluginField("dq_probs", nullptr, PluginFieldType::kFLOAT32, 1));
+
mFC.nbFields = mPluginAttributes.size();
mFC.fields = mPluginAttributes.data();
}
-const char* QKVToContextInterleavedPluginCreator::getPluginName() const
+const char* QKVToContextInterleavedPluginCreator::getPluginName() const noexcept
{
return QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_NAME;
}
-const char* QKVToContextInterleavedPluginCreator::getPluginVersion() const
+const char* QKVToContextInterleavedPluginCreator::getPluginVersion() const noexcept
{
return QKV_TO_CONTEXT_INTERLEAVED_PLUGIN_VERSION;
}
-const PluginFieldCollection* QKVToContextInterleavedPluginCreator::getFieldNames()
+const PluginFieldCollection* QKVToContextInterleavedPluginCreator::getFieldNames() noexcept
{
return &mFC;
}
-IPluginV2* QKVToContextInterleavedPluginCreator::createPlugin(const char* name, const PluginFieldCollection* fc)
+IPluginV2* QKVToContextInterleavedPluginCreator::createPlugin(const char* name, const PluginFieldCollection* fc) noexcept
{
int hiddenSize = 0;
int numHeads = 0;
@@ -325,17 +328,19 @@ IPluginV2* QKVToContextInterleavedPluginCreator::createPlugin(const char* name,
if (hiddenSize <= 0)
{
gLogError << "QKV: Invalid hiddenSize " << hiddenSize << std::endl;
+ return nullptr;
}
if (numHeads <= 0)
{
gLogError << "QKV: Invalid numHeads " << numHeads << std::endl;
+ return nullptr;
}
if (dqProbs < 0)
{
gLogInfo << "Using default scale factor\n";
- dqProbs = 1.f / 127.f;
+ dqProbs = 1.F / 127.F;
}
QKVToContextInterleavedPlugin* p = new QKVToContextInterleavedPlugin(name, hiddenSize, numHeads, dqProbs);
@@ -343,19 +348,19 @@ IPluginV2* QKVToContextInterleavedPluginCreator::createPlugin(const char* name,
}
IPluginV2* QKVToContextInterleavedPluginCreator::deserializePlugin(
- const char* name, const void* serialData, size_t serialLength)
+ const char* name, const void* serialData, size_t serialLength) noexcept
{
// This object will be deleted when the network is destroyed, which will
- // call QKVToContextInterleavedPlugin::destroy()
+ // call QKVToContextInterleavedPlugin::destroy() noexcept
return new QKVToContextInterleavedPlugin(name, serialData, serialLength);
}
-void QKVToContextInterleavedPluginCreator::setPluginNamespace(const char* libNamespace)
+void QKVToContextInterleavedPluginCreator::setPluginNamespace(const char* libNamespace) noexcept
{
mNamespace = libNamespace;
}
-const char* QKVToContextInterleavedPluginCreator::getPluginNamespace() const
+const char* QKVToContextInterleavedPluginCreator::getPluginNamespace() const noexcept
{
return mNamespace.c_str();
}
diff --git a/plugin/bertQKVToContextPlugin/qkvToContextInt8InterleavedPlugin.h b/plugin/bertQKVToContextPlugin/qkvToContextInt8InterleavedPlugin.h
index 060cd9d3..d7cb38eb 100644
--- a/plugin/bertQKVToContextPlugin/qkvToContextInt8InterleavedPlugin.h
+++ b/plugin/bertQKVToContextPlugin/qkvToContextInt8InterleavedPlugin.h
@@ -45,36 +45,37 @@ public:
QKVToContextInterleavedPlugin() = delete;
// IPluginV2DynamicExt Methods
- nvinfer1::IPluginV2DynamicExt* clone() const override;
- nvinfer1::DimsExprs getOutputDimensions(
- int outputIndex, const nvinfer1::DimsExprs* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) override;
+ nvinfer1::IPluginV2DynamicExt* clone() const noexcept override;
+ nvinfer1::DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs* inputs, int nbInputs,
+ nvinfer1::IExprBuilder& exprBuilder) noexcept override;
bool supportsFormatCombination(
- int pos, const nvinfer1::PluginTensorDesc* inOut, int nbInputs, int nbOutputs) override;
+ int pos, const nvinfer1::PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept override;
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc* in, int nbInputs,
- const nvinfer1::DynamicPluginTensorDesc* out, int nbOutputs) override;
+ const nvinfer1::DynamicPluginTensorDesc* out, int nbOutputs) noexcept override;
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc* inputs, int nbInputs,
- const nvinfer1::PluginTensorDesc* outputs, int nbOutputs) const override;
+ const nvinfer1::PluginTensorDesc* outputs, int nbOutputs) const noexcept override;
int enqueue(const nvinfer1::PluginTensorDesc* inputDesc, const nvinfer1::PluginTensorDesc* outputDesc,
- const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) override;
+ const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override;
// IPluginV2Ext Methods
- nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const override;
+ nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const
+ noexcept override;
// IPluginV2 Methods
- const char* getPluginType() const override;
- const char* getPluginVersion() const override;
- int getNbOutputs() const override;
- int initialize() override;
- void terminate() override;
- size_t getSerializationSize() const override;
- void serialize(void* buffer) const override;
- void destroy() override;
- void setPluginNamespace(const char* pluginNamespace) override;
- const char* getPluginNamespace() const override;
+ const char* getPluginType() const noexcept override;
+ const char* getPluginVersion() const noexcept override;
+ int getNbOutputs() const noexcept override;
+ int initialize() noexcept override;
+ void terminate() noexcept override;
+ size_t getSerializationSize() const noexcept override;
+ void serialize(void* buffer) const noexcept override;
+ void destroy() noexcept override;
+ void setPluginNamespace(const char* pluginNamespace) noexcept override;
+ const char* getPluginNamespace() const noexcept override;
protected:
- void createMHARunner();
- int getSMVersion() const;
+ void createMHARunner() noexcept;
+ int getSMVersion() const noexcept;
private:
const std::string mLayerName;
@@ -90,16 +91,6 @@ private:
const FusedMultiHeadAttentionXMMAKernelV2* mXmmaKernel;
float mDqProbs;
-
-protected:
- // To prevent compiler warnings.
- using nvinfer1::IPluginV2DynamicExt::canBroadcastInputAcrossBatch;
- using nvinfer1::IPluginV2DynamicExt::configurePlugin;
- using nvinfer1::IPluginV2DynamicExt::enqueue;
- using nvinfer1::IPluginV2DynamicExt::getOutputDimensions;
- using nvinfer1::IPluginV2DynamicExt::getWorkspaceSize;
- using nvinfer1::IPluginV2DynamicExt::isOutputBroadcastAcrossBatch;
- using nvinfer1::IPluginV2DynamicExt::supportsFormat;
};
class QKVToContextInterleavedPluginCreator : public nvinfer1::IPluginCreator
@@ -107,19 +98,19 @@ class QKVToContextInterleavedPluginCreator : public nvinfer1::IPluginCreator
public:
QKVToContextInterleavedPluginCreator();
- const char* getPluginName() const override;
+ const char* getPluginName() const noexcept override;
- const char* getPluginVersion() const override;
+ const char* getPluginVersion() const noexcept override;
- const nvinfer1::PluginFieldCollection* getFieldNames() override;
+ const nvinfer1::PluginFieldCollection* getFieldNames() noexcept override;
- nvinfer1::IPluginV2* createPlugin(const char* name, const nvinfer1::PluginFieldCollection* fc) override;
+ nvinfer1::IPluginV2* createPlugin(const char* name, const nvinfer1::PluginFieldCollection* fc) noexcept override;
- nvinfer1::IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) override;
+ nvinfer1::IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) noexcept override;
- void setPluginNamespace(const char* pluginNamespace) override;
+ void setPluginNamespace(const char* pluginNamespace) noexcept override;
- const char* getPluginNamespace() const override;
+ const char* getPluginNamespace() const noexcept override;
private:
static nvinfer1::PluginFieldCollection mFC;
diff --git a/plugin/bertQKVToContextPlugin/qkvToContextPlugin.cpp b/plugin/bertQKVToContextPlugin/qkvToContextPlugin.cpp
index 1285f2f8..9b6f018c 100644
--- a/plugin/bertQKVToContextPlugin/qkvToContextPlugin.cpp
+++ b/plugin/bertQKVToContextPlugin/qkvToContextPlugin.cpp
@@ -38,9 +38,9 @@ namespace bert
namespace
{
-static const char* QKV_TO_CONTEXT_PLUGIN_VERSION{"1"};
-static const char* QKV_TO_CONTEXT_VAR_SEQLEN_PLUGIN_VERSION{"2"};
-static const char* QKV_TO_CONTEXT_PLUGIN_NAME{"CustomQKVToContextPluginDynamic"};
+const char* QKV_TO_CONTEXT_PLUGIN_VERSION{"1"};
+const char* QKV_TO_CONTEXT_VAR_SEQLEN_PLUGIN_VERSION{"2"};
+const char* QKV_TO_CONTEXT_PLUGIN_NAME{"CustomQKVToContextPluginDynamic"};
} // namespace
// Static class fields initialization
@@ -118,12 +118,12 @@ void QKVToContextPluginDynamic::createMHARunner()
if (!unfusedDispatcher.get())
{
- unfusedDispatcher.reset(new UnfusedMHARunner(mType, mNumHeads, mHeadSize));
+ unfusedDispatcher.reset(new UnfusedMHARunner(mType, mNumHeads, mHeadSize, mSM));
}
}
// IPluginV2DynamicExt Methods
-nvinfer1::IPluginV2DynamicExt* QKVToContextPluginDynamic::clone() const
+nvinfer1::IPluginV2DynamicExt* QKVToContextPluginDynamic::clone() const noexcept
{
gLogVerbose << "QKV Clone" << std::endl;
@@ -148,19 +148,19 @@ nvinfer1::IPluginV2DynamicExt* QKVToContextPluginDynamic::clone() const
}
DimsExprs QKVToContextPluginDynamic::getOutputDimensions(
- int outputIndex, const DimsExprs* inputs, int nbInputs, IExprBuilder& exprBuilder)
+ int outputIndex, const DimsExprs* inputs, int nbInputs, IExprBuilder& exprBuilder) noexcept
{
// Input is BxSx3*N*H, output should be BxSxN*H
assert(outputIndex == 0);
// Copy over everything
DimsExprs output(inputs[IIDX]);
// Divide last dim by three
- auto three = exprBuilder.constant(3);
+ const auto* three = exprBuilder.constant(3);
output.d[HDIM] = exprBuilder.operation(DimensionOperation::kFLOOR_DIV, *inputs[IIDX].d[HDIM], *three);
return output;
}
bool QKVToContextPluginDynamic::supportsFormatCombination(
- int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs)
+ int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept
{
assert(pos >= 0);
assert(pos < 2 + mHasImask);
@@ -191,6 +191,16 @@ bool QKVToContextPluginDynamic::supportsFormatCombination(
return false;
}
}
+ if (mType == DataType::kHALF)
+ {
+ if (mSM < kSM_53)
+ {
+ gLogError
+ << "Half-precision floating-point is only supported on compute capability 5.3 and later for plugin "
+ << QKV_TO_CONTEXT_PLUGIN_NAME << std::endl;
+ return false;
+ }
+ }
if (pos == 0)
{
@@ -264,7 +274,7 @@ bool QKVToContextPluginDynamic::supportsFormatCombination(
return false;
}
void QKVToContextPluginDynamic::configurePlugin(
- const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out, int nbOutputs)
+ const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out, int nbOutputs) noexcept
{
assert(nbInputs == 1 + mHasImask);
assert(nbOutputs == 1);
@@ -331,7 +341,7 @@ void QKVToContextPluginDynamic::configurePlugin(
}
size_t QKVToContextPluginDynamic::getWorkspaceSize(
- const PluginTensorDesc* inputs, int nbInputs, const PluginTensorDesc* outputs, int nbOutputs) const
+ const PluginTensorDesc* inputs, int nbInputs, const PluginTensorDesc* outputs, int nbOutputs) const noexcept
{
// only unfused kernel need workspace, and we need larger workspace for larger sequence length
// we have already setup unfusedDispatcher with max sequence in configurePlugin
@@ -342,7 +352,7 @@ size_t QKVToContextPluginDynamic::getWorkspaceSize(
// IPluginV2Ext Methods
DataType QKVToContextPluginDynamic::getOutputDataType(
- int index, const nvinfer1::DataType* inputTypes, int nbInputs) const
+ int index, const nvinfer1::DataType* inputTypes, int nbInputs) const noexcept
{
assert(index == 0);
assert(inputTypes[0] == DataType::kFLOAT || inputTypes[0] == DataType::kHALF || inputTypes[0] == DataType::kINT8);
@@ -350,29 +360,29 @@ DataType QKVToContextPluginDynamic::getOutputDataType(
}
// IPluginV2 Methods
-const char* QKVToContextPluginDynamic::getPluginType() const
+const char* QKVToContextPluginDynamic::getPluginType() const noexcept
{
return QKV_TO_CONTEXT_PLUGIN_NAME;
}
-const char* QKVToContextPluginDynamic::getPluginVersion() const
+const char* QKVToContextPluginDynamic::getPluginVersion() const noexcept
{
return QKV_TO_CONTEXT_PLUGIN_VERSION;
}
-int QKVToContextPluginDynamic::getNbOutputs() const
+int QKVToContextPluginDynamic::getNbOutputs() const noexcept
{
return 1;
}
-int QKVToContextPluginDynamic::initialize()
+int QKVToContextPluginDynamic::initialize() noexcept
{
return 0;
}
-void QKVToContextPluginDynamic::terminate() {}
+void QKVToContextPluginDynamic::terminate() noexcept {}
-size_t QKVToContextPluginDynamic::getSerializationSize() const
+size_t QKVToContextPluginDynamic::getSerializationSize() const noexcept
{
ASSERT(unfusedDispatcher.get());
return sizeof(mNumHeads) + sizeof(mHeadSize) + sizeof(DataType) + sizeof(mHasImask) + sizeof(mHiddenSize)
@@ -380,7 +390,7 @@ size_t QKVToContextPluginDynamic::getSerializationSize() const
+ unfusedDispatcher->getSerializationSize();
}
-void QKVToContextPluginDynamic::serialize(void* buffer) const
+void QKVToContextPluginDynamic::serialize(void* buffer) const noexcept
{
serialize_value(&buffer, mType);
serialize_value(&buffer, mNumHeads);
@@ -405,23 +415,23 @@ void QKVToContextPluginDynamic::serialize(void* buffer) const
}
}
-void QKVToContextPluginDynamic::destroy()
+void QKVToContextPluginDynamic::destroy() noexcept
{
delete this;
}
-void QKVToContextPluginDynamic::setPluginNamespace(const char* libNamespace)
+void QKVToContextPluginDynamic::setPluginNamespace(const char* libNamespace) noexcept
{
mNamespace = libNamespace;
}
-const char* QKVToContextPluginDynamic::getPluginNamespace() const
+const char* QKVToContextPluginDynamic::getPluginNamespace() const noexcept
{
return mNamespace.c_str();
}
int QKVToContextPluginDynamic::enqueue(const PluginTensorDesc* inputDesc, const PluginTensorDesc* outputDesc,
- const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream)
+ const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept
{
assert(mS == inputDesc->dims.d[SDIM]);
assert(mB == inputDesc->dims.d[BDIM]);
@@ -446,26 +456,27 @@ QKVToContextPluginDynamicCreator::QKVToContextPluginDynamicCreator()
mPluginAttributes.emplace_back(PluginField("num_heads", nullptr, PluginFieldType::kINT32, 1));
mPluginAttributes.emplace_back(PluginField("has_mask", nullptr, PluginFieldType::kINT32, 1));
mPluginAttributes.emplace_back(PluginField("dq_probs", nullptr, PluginFieldType::kFLOAT32, 1));
+
mFC.nbFields = mPluginAttributes.size();
mFC.fields = mPluginAttributes.data();
}
-const char* QKVToContextPluginDynamicCreator::getPluginName() const
+const char* QKVToContextPluginDynamicCreator::getPluginName() const noexcept
{
return QKV_TO_CONTEXT_PLUGIN_NAME;
}
-const char* QKVToContextPluginDynamicCreator::getPluginVersion() const
+const char* QKVToContextPluginDynamicCreator::getPluginVersion() const noexcept
{
return QKV_TO_CONTEXT_PLUGIN_VERSION;
}
-const PluginFieldCollection* QKVToContextPluginDynamicCreator::getFieldNames()
+const PluginFieldCollection* QKVToContextPluginDynamicCreator::getFieldNames() noexcept
{
return &mFC;
}
-IPluginV2* QKVToContextPluginDynamicCreator::createPlugin(const char* name, const PluginFieldCollection* fc)
+IPluginV2* QKVToContextPluginDynamicCreator::createPlugin(const char* name, const PluginFieldCollection* fc) noexcept
{
gLogVerbose << "Creating QKV2ContextPlugin...\n";
@@ -510,16 +521,19 @@ IPluginV2* QKVToContextPluginDynamicCreator::createPlugin(const char* name, cons
if (typeId < 0 || typeId > 3)
{
gLogError << "QKV: Invalid TypeId " << typeId << std::endl;
+ return nullptr;
}
if (hiddenSize <= 0)
{
gLogError << "QKV: Invalid hiddenSize " << hiddenSize << std::endl;
+ return nullptr;
}
if (numHeads <= 0)
{
gLogError << "QKV: Invalid numHeads " << numHeads << std::endl;
+ return nullptr;
}
gLogVerbose << "Building the Plugin...\n";
@@ -527,7 +541,7 @@ IPluginV2* QKVToContextPluginDynamicCreator::createPlugin(const char* name, cons
if (type == DataType::kINT8 && dqProbs < 0)
{
gLogInfo << "Using default scale factor\n";
- dqProbs = 1.f / 127.f;
+ dqProbs = 1.F / 127.F;
}
QKVToContextPluginDynamic* p = new QKVToContextPluginDynamic(name, type, hiddenSize, numHeads, dqProbs, hasMask);
@@ -535,19 +549,19 @@ IPluginV2* QKVToContextPluginDynamicCreator::createPlugin(const char* name, cons
}
IPluginV2* QKVToContextPluginDynamicCreator::deserializePlugin(
- const char* name, const void* serialData, size_t serialLength)
+ const char* name, const void* serialData, size_t serialLength) noexcept
{
// This object will be deleted when the network is destroyed, which will
// call QKVToContextPluginDynamic::destroy()
return new QKVToContextPluginDynamic(name, serialData, serialLength);
}
-void QKVToContextPluginDynamicCreator::setPluginNamespace(const char* libNamespace)
+void QKVToContextPluginDynamicCreator::setPluginNamespace(const char* libNamespace) noexcept
{
mNamespace = libNamespace;
}
-const char* QKVToContextPluginDynamicCreator::getPluginNamespace() const
+const char* QKVToContextPluginDynamicCreator::getPluginNamespace() const noexcept
{
return mNamespace.c_str();
}
@@ -573,8 +587,7 @@ QKVToContextVarSeqlenPlugin::QKVToContextVarSeqlenPlugin(const std::string name,
{
// variable sequence length is only supported with the fused MHA kernels
// we should not override mS!
- assert((mSM == kSM_86 || mSM == kSM_80 || mSM == kSM_75 || mSM == kSM_72)
- && (type == DataType::kINT8 || type == DataType::kHALF)
+ assert((mSM == kSM_86 || mSM == kSM_80 || mSM == kSM_75 || mSM == kSM_72) && (type == DataType::kINT8 || type == DataType::kHALF)
&& "requesting maxSeqlen not compatible with GPU arch");
// the layout changes: SxB will be a combined \sum_i s_i and hdim will be the 2nd dimension instead of the third
mHdim = 1;
@@ -626,12 +639,12 @@ void QKVToContextVarSeqlenPlugin::createMHARunner()
else
{
assert(!mUseVarSeqlen);
- dispatcher.reset(new UnfusedMHARunner(mType, mNumHeads, mHeadSize));
+ dispatcher.reset(new UnfusedMHARunner(mType, mNumHeads, mHeadSize, mSM));
}
}
// IPluginV2DynamicExt Methods
-nvinfer1::IPluginV2DynamicExt* QKVToContextVarSeqlenPlugin::clone() const
+nvinfer1::IPluginV2DynamicExt* QKVToContextVarSeqlenPlugin::clone() const noexcept
{
gLogVerbose << "QKV Clone" << std::endl;
@@ -656,26 +669,26 @@ nvinfer1::IPluginV2DynamicExt* QKVToContextVarSeqlenPlugin::clone() const
}
DimsExprs QKVToContextVarSeqlenPlugin::getOutputDimensions(
- int outputIndex, const DimsExprs* inputs, int nbInputs, IExprBuilder& exprBuilder)
+ int outputIndex, const DimsExprs* inputs, int nbInputs, IExprBuilder& exprBuilder) noexcept
{
// Input is BxSx3*N*H, output should be BxSxN*H
assert(outputIndex == 0);
// Copy over everything
DimsExprs output(inputs[IIDX]);
// Divide last dim by three
- auto three = exprBuilder.constant(3);
+ const auto* three = exprBuilder.constant(3);
output.d[mHdim] = exprBuilder.operation(DimensionOperation::kFLOOR_DIV, *inputs[IIDX].d[mHdim], *three);
return output;
}
bool QKVToContextVarSeqlenPlugin::supportsFormatCombination(
- int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs)
+ int pos, const PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept
{
// we only support int8 IO in fused mha runner, and we only support fused mha runner on Turing and Ampere
if (mType == DataType::kINT8 && mSM != kSM_86 && mSM != kSM_80 && mSM != kSM_75 && mSM != kSM_72)
{
- gLogVerbose << "INT8 IO is only supported on Xavier, Turing and Ampere for plugin "
- << QKV_TO_CONTEXT_PLUGIN_NAME << std::endl;
+ gLogVerbose << "INT8 IO is only supported on Xavier, Turing and Ampere for plugin " << QKV_TO_CONTEXT_PLUGIN_NAME
+ << std::endl;
return false;
}
@@ -762,7 +775,7 @@ bool QKVToContextVarSeqlenPlugin::supportsFormatCombination(
}
void QKVToContextVarSeqlenPlugin::configurePlugin(
- const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out, int nbOutputs)
+ const DynamicPluginTensorDesc* in, int nbInputs, const DynamicPluginTensorDesc* out, int nbOutputs) noexcept
{
assert(nbInputs == 1 + mHasImask + 2 * mUseVarSeqlen);
assert(nbOutputs == 1);
@@ -807,14 +820,14 @@ void QKVToContextVarSeqlenPlugin::configurePlugin(
}
size_t QKVToContextVarSeqlenPlugin::getWorkspaceSize(
- const PluginTensorDesc* inputs, int nbInputs, const PluginTensorDesc* outputs, int nbOutputs) const
+ const PluginTensorDesc* inputs, int nbInputs, const PluginTensorDesc* outputs, int nbOutputs) const noexcept
{
return this->dispatcher->getWorkspaceSize();
}
// IPluginV2Ext Methods
DataType QKVToContextVarSeqlenPlugin::getOutputDataType(
- int index, const nvinfer1::DataType* inputTypes, int nbInputs) const
+ int index, const nvinfer1::DataType* inputTypes, int nbInputs) const noexcept
{
assert(index == 0);
assert(inputTypes[0] == DataType::kFLOAT || inputTypes[0] == DataType::kHALF || inputTypes[0] == DataType::kINT8);
@@ -822,36 +835,36 @@ DataType QKVToContextVarSeqlenPlugin::getOutputDataType(
}
// IPluginV2 Methods
-const char* QKVToContextVarSeqlenPlugin::getPluginType() const
+const char* QKVToContextVarSeqlenPlugin::getPluginType() const noexcept
{
return QKV_TO_CONTEXT_PLUGIN_NAME;
}
-const char* QKVToContextVarSeqlenPlugin::getPluginVersion() const
+const char* QKVToContextVarSeqlenPlugin::getPluginVersion() const noexcept
{
return QKV_TO_CONTEXT_VAR_SEQLEN_PLUGIN_VERSION;
}
-int QKVToContextVarSeqlenPlugin::getNbOutputs() const
+int QKVToContextVarSeqlenPlugin::getNbOutputs() const noexcept
{
return 1;
}
-int QKVToContextVarSeqlenPlugin::initialize()
+int QKVToContextVarSeqlenPlugin::initialize() noexcept
{
return 0;
}
-void QKVToContextVarSeqlenPlugin::terminate() {}
+void QKVToContextVarSeqlenPlugin::terminate() noexcept {}
-size_t QKVToContextVarSeqlenPlugin::getSerializationSize() const
+size_t QKVToContextVarSeqlenPlugin::getSerializationSize() const noexcept
{
return sizeof(mNumHeads) + sizeof(mHeadSize) + sizeof(DataType) + sizeof(mHasImask) + sizeof(mHiddenSize)
+ sizeof(mSM) + sizeof(mS) + sizeof(mB) + sizeof(mDqProbs) + dispatcher->getSerializationSize()
+ sizeof(mUseVarSeqlen) + sizeof(mHdim);
}
-void QKVToContextVarSeqlenPlugin::serialize(void* buffer) const
+void QKVToContextVarSeqlenPlugin::serialize(void* buffer) const noexcept
{
serialize_value(&buffer, mType);
serialize_value(&buffer, mNumHeads);
@@ -868,32 +881,31 @@ void QKVToContextVarSeqlenPlugin::serialize(void* buffer) const
dispatcher->serialize(buffer);
}
-void QKVToContextVarSeqlenPlugin::destroy()
+void QKVToContextVarSeqlenPlugin::destroy() noexcept
{
delete this;
}
-void QKVToContextVarSeqlenPlugin::setPluginNamespace(const char* libNamespace)
+void QKVToContextVarSeqlenPlugin::setPluginNamespace(const char* libNamespace) noexcept
{
mNamespace = libNamespace;
}
-const char* QKVToContextVarSeqlenPlugin::getPluginNamespace() const
+const char* QKVToContextVarSeqlenPlugin::getPluginNamespace() const noexcept
{
return mNamespace.c_str();
}
int QKVToContextVarSeqlenPlugin::enqueue(const nvinfer1::PluginTensorDesc* inputDesc,
const nvinfer1::PluginTensorDesc* outputDesc, const void* const* inputs, void* const* outputs, void* workspace,
- cudaStream_t stream)
+ cudaStream_t stream) noexcept
{
if (mUseVarSeqlen)
{
const int B = inputDesc[1].dims.d[0];
const int maxS = inputDesc[3].dims.d[0];
- ASSERT((maxS <= 384)
- && "No implementation for variable sequence length multi-head attention plugin with sequence > 384.");
+ ASSERT((maxS <= 384) && "No implementation for variable sequence length multi-head attention plugin with sequence > 384.");
int S = 384;
if (DataType::kHALF == mType && maxS <= 64)
@@ -911,10 +923,11 @@ int QKVToContextVarSeqlenPlugin::enqueue(const nvinfer1::PluginTensorDesc* input
else if (maxS <= 192)
{
S = 192;
- if (mType == DataType::kHALF)
+ if(mType == DataType::kHALF)
{
S = 256;
}
+
}
else if (maxS <= 256)
{
@@ -923,6 +936,7 @@ int QKVToContextVarSeqlenPlugin::enqueue(const nvinfer1::PluginTensorDesc* input
this->dispatcher->setup(S, B);
this->dispatcher->run(inputDesc, outputDesc, inputs, outputs, workspace, stream);
+ return cudaGetLastError();
}
else
{
@@ -931,10 +945,8 @@ int QKVToContextVarSeqlenPlugin::enqueue(const nvinfer1::PluginTensorDesc* input
const void* maskPtr = mHasImask ? inputs[1] : nullptr;
this->dispatcher->run(inputDesc[0], outputDesc[0], inputs[0], maskPtr, outputs[0], workspace, stream);
- return 0;
+ return cudaGetLastError();
}
-
- return 0;
}
QKVToContextVarSeqlenPluginCreator::QKVToContextVarSeqlenPluginCreator()
@@ -949,22 +961,22 @@ QKVToContextVarSeqlenPluginCreator::QKVToContextVarSeqlenPluginCreator()
mFC.fields = mPluginAttributes.data();
}
-const char* QKVToContextVarSeqlenPluginCreator::getPluginName() const
+const char* QKVToContextVarSeqlenPluginCreator::getPluginName() const noexcept
{
return QKV_TO_CONTEXT_PLUGIN_NAME;
}
-const char* QKVToContextVarSeqlenPluginCreator::getPluginVersion() const
+const char* QKVToContextVarSeqlenPluginCreator::getPluginVersion() const noexcept
{
return QKV_TO_CONTEXT_VAR_SEQLEN_PLUGIN_VERSION;
}
-const PluginFieldCollection* QKVToContextVarSeqlenPluginCreator::getFieldNames()
+const PluginFieldCollection* QKVToContextVarSeqlenPluginCreator::getFieldNames() noexcept
{
return &mFC;
}
-IPluginV2* QKVToContextVarSeqlenPluginCreator::createPlugin(const char* name, const PluginFieldCollection* fc)
+IPluginV2* QKVToContextVarSeqlenPluginCreator::createPlugin(const char* name, const PluginFieldCollection* fc) noexcept
{
gLogVerbose << "Creating QKV2ContextPlugin...\n";
@@ -1016,16 +1028,19 @@ IPluginV2* QKVToContextVarSeqlenPluginCreator::createPlugin(const char* name, co
if (typeId < 0 || typeId > 3)
{
gLogError << "QKV: Invalid TypeId " << typeId << std::endl;
+ return nullptr;
}
if (hiddenSize <= 0)
{
gLogError << "QKV: Invalid hiddenSize " << hiddenSize << std::endl;
+ return nullptr;
}
if (numHeads <= 0)
{
gLogError << "QKV: Invalid numHeads " << numHeads << std::endl;
+ return nullptr;
}
gLogVerbose << "Building the Plugin...\n";
@@ -1033,7 +1048,7 @@ IPluginV2* QKVToContextVarSeqlenPluginCreator::createPlugin(const char* name, co
if (type == DataType::kINT8 && dqProbs < 0)
{
gLogInfo << "Using default scale factor\n";
- dqProbs = 1.f / 127.f;
+ dqProbs = 1.F / 127.F;
}
QKVToContextVarSeqlenPlugin* p
@@ -1042,19 +1057,19 @@ IPluginV2* QKVToContextVarSeqlenPluginCreator::createPlugin(const char* name, co
}
IPluginV2* QKVToContextVarSeqlenPluginCreator::deserializePlugin(
- const char* name, const void* serialData, size_t serialLength)
+ const char* name, const void* serialData, size_t serialLength) noexcept
{
// This object will be deleted when the network is destroyed, which will
// call QKVToContextVarSeqlenPlugin::destroy()
return new QKVToContextVarSeqlenPlugin(name, serialData, serialLength);
}
-void QKVToContextVarSeqlenPluginCreator::setPluginNamespace(const char* libNamespace)
+void QKVToContextVarSeqlenPluginCreator::setPluginNamespace(const char* libNamespace) noexcept
{
mNamespace = libNamespace;
}
-const char* QKVToContextVarSeqlenPluginCreator::getPluginNamespace() const
+const char* QKVToContextVarSeqlenPluginCreator::getPluginNamespace() const noexcept
{
return mNamespace.c_str();
}
diff --git a/plugin/bertQKVToContextPlugin/qkvToContextPlugin.h b/plugin/bertQKVToContextPlugin/qkvToContextPlugin.h
index 43b3f9d8..6a0001d2 100644
--- a/plugin/bertQKVToContextPlugin/qkvToContextPlugin.h
+++ b/plugin/bertQKVToContextPlugin/qkvToContextPlugin.h
@@ -76,8 +76,8 @@ public:
const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream)
= 0;
- virtual size_t getSerializationSize() const;
- virtual void serialize(void* buffer) const;
+ virtual size_t getSerializationSize() const noexcept;
+ virtual void serialize(void* buffer) const noexcept;
virtual void deserialize(const void* data, size_t length);
virtual size_t getWorkspaceSize() const = 0;
@@ -129,32 +129,33 @@ public:
QKVToContextPluginDynamic() = delete;
// IPluginV2DynamicExt Methods
- nvinfer1::IPluginV2DynamicExt* clone() const override;
- nvinfer1::DimsExprs getOutputDimensions(
- int outputIndex, const nvinfer1::DimsExprs* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) override;
+ nvinfer1::IPluginV2DynamicExt* clone() const noexcept override;
+ nvinfer1::DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs* inputs, int nbInputs,
+ nvinfer1::IExprBuilder& exprBuilder) noexcept override;
bool supportsFormatCombination(
- int pos, const nvinfer1::PluginTensorDesc* inOut, int nbInputs, int nbOutputs) override;
+ int pos, const nvinfer1::PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept override;
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc* in, int nbInputs,
- const nvinfer1::DynamicPluginTensorDesc* out, int nbOutputs) override;
+ const nvinfer1::DynamicPluginTensorDesc* out, int nbOutputs) noexcept override;
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc* inputs, int nbInputs,
- const nvinfer1::PluginTensorDesc* outputs, int nbOutputs) const override;
+ const nvinfer1::PluginTensorDesc* outputs, int nbOutputs) const noexcept override;
int enqueue(const nvinfer1::PluginTensorDesc* inputDesc, const nvinfer1::PluginTensorDesc* outputDesc,
- const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) override;
+ const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override;
// IPluginV2Ext Methods
- nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const override;
+ nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const
+ noexcept override;
// IPluginV2 Methods
- const char* getPluginType() const override;
- const char* getPluginVersion() const override;
- int getNbOutputs() const override;
- int initialize() override;
- void terminate() override;
- size_t getSerializationSize() const override;
- void serialize(void* buffer) const override;
- void destroy() override;
- void setPluginNamespace(const char* pluginNamespace) override;
- const char* getPluginNamespace() const override;
+ const char* getPluginType() const noexcept override;
+ const char* getPluginVersion() const noexcept override;
+ int getNbOutputs() const noexcept override;
+ int initialize() noexcept override;
+ void terminate() noexcept override;
+ size_t getSerializationSize() const noexcept override;
+ void serialize(void* buffer) const noexcept override;
+ void destroy() noexcept override;
+ void setPluginNamespace(const char* pluginNamespace) noexcept override;
+ const char* getPluginNamespace() const noexcept override;
protected:
void createMHARunner();
@@ -177,18 +178,7 @@ private:
int mNumHeads;
bool mHasImask;
nvinfer1::DataType mType;
-
float mDqProbs;
-
-protected:
- // To prevent compiler warnings.
- using nvinfer1::IPluginV2DynamicExt::canBroadcastInputAcrossBatch;
- using nvinfer1::IPluginV2DynamicExt::configurePlugin;
- using nvinfer1::IPluginV2DynamicExt::enqueue;
- using nvinfer1::IPluginV2DynamicExt::getOutputDimensions;
- using nvinfer1::IPluginV2DynamicExt::getWorkspaceSize;
- using nvinfer1::IPluginV2DynamicExt::isOutputBroadcastAcrossBatch;
- using nvinfer1::IPluginV2DynamicExt::supportsFormat;
};
class QKVToContextPluginDynamicCreator : public nvinfer1::IPluginCreator
@@ -196,19 +186,19 @@ class QKVToContextPluginDynamicCreator : public nvinfer1::IPluginCreator
public:
QKVToContextPluginDynamicCreator();
- const char* getPluginName() const override;
+ const char* getPluginName() const noexcept override;
- const char* getPluginVersion() const override;
+ const char* getPluginVersion() const noexcept override;
- const nvinfer1::PluginFieldCollection* getFieldNames() override;
+ const nvinfer1::PluginFieldCollection* getFieldNames() noexcept override;
- nvinfer1::IPluginV2* createPlugin(const char* name, const nvinfer1::PluginFieldCollection* fc) override;
+ nvinfer1::IPluginV2* createPlugin(const char* name, const nvinfer1::PluginFieldCollection* fc) noexcept override;
- nvinfer1::IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) override;
+ nvinfer1::IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) noexcept override;
- void setPluginNamespace(const char* pluginNamespace) override;
+ void setPluginNamespace(const char* pluginNamespace) noexcept override;
- const char* getPluginNamespace() const override;
+ const char* getPluginNamespace() const noexcept override;
private:
static nvinfer1::PluginFieldCollection mFC;
@@ -229,32 +219,33 @@ public:
QKVToContextVarSeqlenPlugin() = delete;
// IPluginV2DynamicExt Methods
- nvinfer1::IPluginV2DynamicExt* clone() const override;
- nvinfer1::DimsExprs getOutputDimensions(
- int outputIndex, const nvinfer1::DimsExprs* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) override;
+ nvinfer1::IPluginV2DynamicExt* clone() const noexcept override;
+ nvinfer1::DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs* inputs, int nbInputs,
+ nvinfer1::IExprBuilder& exprBuilder) noexcept override;
bool supportsFormatCombination(
- int pos, const nvinfer1::PluginTensorDesc* inOut, int nbInputs, int nbOutputs) override;
+ int pos, const nvinfer1::PluginTensorDesc* inOut, int nbInputs, int nbOutputs) noexcept override;
void configurePlugin(const nvinfer1::DynamicPluginTensorDesc* in, int nbInputs,
- const nvinfer1::DynamicPluginTensorDesc* out, int nbOutputs) override;
+ const nvinfer1::DynamicPluginTensorDesc* out, int nbOutputs) noexcept override;
size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc* inputs, int nbInputs,
- const nvinfer1::PluginTensorDesc* outputs, int nbOutputs) const override;
+ const nvinfer1::PluginTensorDesc* outputs, int nbOutputs) const noexcept override;
int enqueue(const nvinfer1::PluginTensorDesc* inputDesc, const nvinfer1::PluginTensorDesc* outputDesc,
- const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) override;
+ const void* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override;
// IPluginV2Ext Methods
- nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const override;
+ nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType* inputTypes, int nbInputs) const
+ noexcept override;
// IPluginV2 Methods
- const char* getPluginType() const override;
- const char* getPluginVersion() const override;
- int getNbOutputs() const override;
- int initialize() override;
- void terminate() override;
- size_t getSerializationSize() const override;
- void serialize(void* buffer) const override;
- void destroy() override;
- void setPluginNamespace(const char* pluginNamespace) override;
- const char* getPluginNamespace() const override;
+ const char* getPluginType() const noexcept override;
+ const char* getPluginVersion() const noexcept override;
+ int getNbOutputs() const noexcept override;
+ int initialize() noexcept override;
+ void terminate() noexcept override;
+ size_t getSerializationSize() const noexcept override;
+ void serialize(void* buffer) const noexcept override;
+ void destroy() noexcept override;
+ void setPluginNamespace(const char* pluginNamespace) noexcept override;
+ const char* getPluginNamespace() const noexcept override;
protected:
void createMHARunner();
@@ -278,16 +269,6 @@ private:
int mHdim;
bool mUseVarSeqlen;
-
-protected:
- // To prevent compiler warnings.
- using nvinfer1::IPluginV2DynamicExt::canBroadcastInputAcrossBatch;
- using nvinfer1::IPluginV2DynamicExt::configurePlugin;
- using nvinfer1::IPluginV2DynamicExt::enqueue;
- using nvinfer1::IPluginV2DynamicExt::getOutputDimensions;
- using nvinfer1::IPluginV2DynamicExt::getWorkspaceSize;
- using nvinfer1::IPluginV2DynamicExt::isOutputBroadcastAcrossBatch;
- using nvinfer1::IPluginV2DynamicExt::supportsFormat;
};
class QKVToContextVarSeqlenPluginCreator : public nvinfer1::IPluginCreator
@@ -295,19 +276,19 @@ class QKVToContextVarSeqlenPluginCreator : public nvinfer1::IPluginCreator
public:
QKVToContextVarSeqlenPluginCreator();
- const char* getPluginName() const override;
+ const char* getPluginName() const noexcept override;
- const char* getPluginVersion() const override;
+ const char* getPluginVersion() const noexcept override;
- const nvinfer1::PluginFieldCollection* getFieldNames() override;
+ const nvinfer1::PluginFieldCollection* getFieldNames() noexcept override;
- nvinfer1::IPluginV2* createPlugin(const char* name, const nvinfer1::PluginFieldCollection* fc) override;
+ nvinfer1::IPluginV2* createPlugin(const char* name, const nvinfer1::PluginFieldCollection* fc) noexcept override;
- nvinfer1::IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) override;
+ nvinfer1::IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) noexcept override;
- void setPluginNamespace(const char* pluginNamespace) override;
+ void setPluginNamespace(const char* pluginNamespace) noexcept override;
- const char* getPluginNamespace() const override;
+ const char* getPluginNamespace() const noexcept override;
private:
static nvinfer1::PluginFieldCollection mFC;
@@ -318,7 +299,7 @@ private:
class UnfusedMHARunner : public MHARunner
{
public:
- UnfusedMHARunner(const nvinfer1::DataType type, const int numHeads, const int headSize);
+ UnfusedMHARunner(const nvinfer1::DataType type, const int numHeads, const int headSize, const int smVersion);
virtual ~UnfusedMHARunner();
virtual void setup(const int S, const int B) override;
@@ -331,8 +312,8 @@ public:
size_t getWorkspaceSize() const override;
- size_t getSerializationSize() const override;
- void serialize(void* buffer) const override;
+ size_t getSerializationSize() const noexcept override;
+ void serialize(void* buffer) const noexcept override;
void deserialize(const void* data, size_t length) override;
bool isValid(int s) const override;
@@ -341,6 +322,7 @@ private:
int mAlgoBatchedEx1;
int mAlgoBatchedEx2;
cublasHandle_t mCublas;
+ int mSm;
};
class FusedMHARunnerFP16 : public MHARunner
diff --git a/plugin/common/bboxUtils.h b/plugin/common/bboxUtils.h
index 88b0a0d4..ff5686d7 100644
--- a/plugin/common/bboxUtils.h
+++ b/plugin/common/bboxUtils.h
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#ifndef TRT_BBOX_UTILS_H
#define TRT_BBOX_UTILS_H
diff --git a/plugin/common/bertCommon.h b/plugin/common/bertCommon.h
index cab8f989..88128218 100644
--- a/plugin/common/bertCommon.h
+++ b/plugin/common/bertCommon.h
@@ -44,6 +44,8 @@ constexpr uint32_t BDIM = 1; // batch dimension
constexpr uint32_t SDIM = 0; // seq len dimension
constexpr uint32_t HDIM = 2; // hidden dimension
+constexpr int32_t kSM_53 = 53;
+constexpr int32_t kSM_70 = 70;
constexpr int32_t kSM_72 = 72;
constexpr int32_t kSM_75 = 75;
constexpr int32_t kSM_80 = 80;
@@ -107,7 +109,7 @@ inline int getMHAMaskPackedSize(int smVersion, nvinfer1::DataType dataType, int
return packedSize;
}
-inline unsigned int getElementSize(nvinfer1::DataType t)
+inline uint32_t getElementSize(nvinfer1::DataType t) noexcept
{
switch (t)
{
@@ -117,7 +119,6 @@ inline unsigned int getElementSize(nvinfer1::DataType t)
case nvinfer1::DataType::kBOOL:
case nvinfer1::DataType::kINT8: return 1;
}
- throw std::runtime_error("Invalid DataType.");
return 0;
}
@@ -362,7 +363,7 @@ struct WeightsWithOwnership : public nvinfer1::Weights
}
}
- void convertAndCopy(const char*& srcBuf, size_t count, nvinfer1::DataType type)
+ void convertAndCopy(const char*& srcBuf, size_t count, nvinfer1::DataType type) noexcept
{
this->type = type;
this->count = count;
diff --git a/plugin/common/checkMacrosPlugin.h b/plugin/common/checkMacrosPlugin.h
index 83ed64d2..fe778c3b 100644
--- a/plugin/common/checkMacrosPlugin.h
+++ b/plugin/common/checkMacrosPlugin.h
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#ifndef CHECK_MACROS_PLUGIN_H
#define CHECK_MACROS_PLUGIN_H
@@ -55,8 +54,7 @@ class LogStream : public std::ostream
Buf buffer;
public:
- LogStream()
- : std::ostream(&buffer){};
+ LogStream() : std::ostream(&buffer) {};
};
extern LogStream gLogError;
@@ -84,10 +82,7 @@ public:
{
}
virtual void log(std::ostream& logStream) const;
- void setMessage(const char* msg)
- {
- message = msg;
- }
+ void setMessage(const char* msg) { message = msg; }
protected:
const char* file{nullptr};
@@ -125,6 +120,11 @@ public:
}
};
+
+inline void caughtError(const std::exception& e)
+{
+ gLogError << e.what() << std::endl;
+}
} // namespace plugin
} // namespace nvinfer1
@@ -150,24 +150,24 @@ public:
} \
}
-#define API_CHECK_WEIGHTS(Name) \
- API_CHECK((Name).values != nullptr); \
- API_CHECK((Name).count > 0); \
+#define API_CHECK_WEIGHTS(Name) \
+ API_CHECK((Name).values != nullptr); \
+ API_CHECK((Name).count > 0); \
API_CHECK(int((Name).type) >= 0 && int((Name).type) < EnumMax());
-#define API_CHECK_WEIGHTS0(Name) \
- API_CHECK((Name).count >= 0); \
- API_CHECK((Name).count > 0 ? ((Name).values != nullptr) : ((Name).values == nullptr)); \
+#define API_CHECK_WEIGHTS0(Name) \
+ API_CHECK((Name).count >= 0); \
+ API_CHECK((Name).count > 0 ? ((Name).values != nullptr) : ((Name).values == nullptr)); \
API_CHECK(int((Name).type) >= 0 && int((Name).type) < EnumMax());
-#define API_CHECK_WEIGHTS_RETVAL(Name, retval) \
- API_CHECK_RETVAL((Name).values != nullptr, retval); \
- API_CHECK_RETVAL((Name).count > 0, retval); \
+#define API_CHECK_WEIGHTS_RETVAL(Name, retval) \
+ API_CHECK_RETVAL((Name).values != nullptr, retval); \
+ API_CHECK_RETVAL((Name).count > 0, retval); \
API_CHECK_RETVAL(int((Name).type) >= 0 && int((Name).type) < EnumMax(), retval);
-#define API_CHECK_WEIGHTS0_RETVAL(Name, retval) \
- API_CHECK_RETVAL((Name).count >= 0, retval); \
- API_CHECK_RETVAL((Name).count > 0 ? ((Name).values != nullptr) : ((Name).values == nullptr), retval); \
+#define API_CHECK_WEIGHTS0_RETVAL(Name, retval) \
+ API_CHECK_RETVAL((Name).count >= 0, retval); \
+ API_CHECK_RETVAL((Name).count > 0 ? ((Name).values != nullptr) : ((Name).values == nullptr), retval); \
API_CHECK_RETVAL(int((Name).type) >= 0 && int((Name).type) < EnumMax(), retval);
#define API_CHECK_NULL(param) API_CHECK((param) != nullptr)
@@ -178,6 +178,26 @@ public:
#define API_CHECK_ENUM_RANGE_RETVAL(Type, val, retval) \
API_CHECK_RETVAL(int(val) >= 0 && int(val) < EnumMax(), retval)
+#define CHECK_CUDA(call) \
+ do \
+ { \
+ cudaError_t status = call; \
+ if (status != cudaSuccess) \
+ { \
+ return status; \
+ } \
+ } while (0)
+
+#define CHECK_CUDNN(call) \
+ do \
+ { \
+ cudnnStatus_t status = call; \
+ if (status != CUDNN_STATUS_SUCCESS) \
+ { \
+ return status; \
+ } \
+ } while (0)
+
#define CUBLASASSERTMSG(status_, msg) \
{ \
auto s_ = status_; \
diff --git a/plugin/common/common.cuh b/plugin/common/common.cuh
index afecf56b..03e1c468 100644
--- a/plugin/common/common.cuh
+++ b/plugin/common/common.cuh
@@ -438,9 +438,9 @@ __device__ inline float myExp(const float x)
return __expf(x);
}
-static inline __device__ uint32_t float4_to_char4(float x,
- float y,
- float z,
+static inline __device__ uint32_t float4_to_char4(float x,
+ float y,
+ float z,
float w) {
uint32_t dst;
#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 720
diff --git a/plugin/common/cub_helper.h b/plugin/common/cub_helper.h
index d0742a7a..7e3dc962 100644
--- a/plugin/common/cub_helper.h
+++ b/plugin/common/cub_helper.h
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#include "kernel.h"
template
size_t cubSortPairsWorkspaceSize(int num_items, int num_segments)
diff --git a/plugin/common/cudaDriverWrapper.cpp b/plugin/common/cudaDriverWrapper.cpp
index 5728a65b..cbef9876 100644
--- a/plugin/common/cudaDriverWrapper.cpp
+++ b/plugin/common/cudaDriverWrapper.cpp
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#define CUDA_LIB_NAME "cuda"
#if defined(_WIN32)
@@ -33,8 +32,9 @@
#include "cudaDriverWrapper.h"
#include "plugin.h"
+#include
+#include
#include
-#include
using namespace nvinfer1;
@@ -43,24 +43,24 @@ CUDADriverWrapper::CUDADriverWrapper()
handle = dllOpen(CUDA_LIB_NAME);
ASSERT(handle != nullptr);
- auto load_sym = [](void* handle, const char* name) {
+ auto load_sym = [](void* handle, const char *name) {
void* ret = dllGetSym(handle, name);
ASSERT(ret != nullptr);
return ret;
};
- *(void**) (&_cuGetErrorName) = load_sym(handle, "cuGetErrorName");
- *(void**) (&_cuFuncSetAttribute) = load_sym(handle, "cuFuncSetAttribute");
- *(void**) (&_cuLinkComplete) = load_sym(handle, "cuLinkComplete");
- *(void**) (&_cuModuleUnload) = load_sym(handle, "cuModuleUnload");
- *(void**) (&_cuLinkDestroy) = load_sym(handle, "cuLinkDestroy");
- *(void**) (&_cuModuleLoadData) = load_sym(handle, "cuModuleLoadData");
- *(void**) (&_cuLinkCreate) = load_sym(handle, "cuLinkCreate_v2");
- *(void**) (&_cuModuleGetFunction) = load_sym(handle, "cuModuleGetFunction");
- *(void**) (&_cuLinkAddFile) = load_sym(handle, "cuLinkAddFile_v2");
- *(void**) (&_cuLinkAddData) = load_sym(handle, "cuLinkAddData_v2");
- *(void**) (&_cuLaunchCooperativeKernel) = load_sym(handle, "cuLaunchCooperativeKernel");
- *(void**) (&_cuLaunchKernel) = load_sym(handle, "cuLaunchKernel");
+ *(void**)(&_cuGetErrorName) = load_sym(handle, "cuGetErrorName");
+ *(void**)(&_cuFuncSetAttribute) = load_sym(handle, "cuFuncSetAttribute");
+ *(void**)(&_cuLinkComplete) = load_sym(handle, "cuLinkComplete");
+ *(void**)(&_cuModuleUnload) = load_sym(handle, "cuModuleUnload");
+ *(void**)(&_cuLinkDestroy) = load_sym(handle, "cuLinkDestroy");
+ *(void**)(&_cuModuleLoadData) = load_sym(handle, "cuModuleLoadData");
+ *(void**)(&_cuLinkCreate) = load_sym(handle, "cuLinkCreate_v2");
+ *(void**)(&_cuModuleGetFunction) = load_sym(handle, "cuModuleGetFunction");
+ *(void**)(&_cuLinkAddFile) = load_sym(handle, "cuLinkAddFile_v2");
+ *(void**)(&_cuLinkAddData) = load_sym(handle, "cuLinkAddData_v2");
+ *(void**)(&_cuLaunchCooperativeKernel) = load_sym(handle, "cuLaunchCooperativeKernel");
+ *(void**)(&_cuLaunchKernel) = load_sym(handle, "cuLaunchKernel");
}
CUDADriverWrapper::~CUDADriverWrapper()
@@ -99,7 +99,7 @@ CUresult CUDADriverWrapper::cuModuleLoadData(CUmodule* module, const void* image
}
CUresult CUDADriverWrapper::cuLinkCreate(
- unsigned int numOptions, CUjit_option* options, void** optionValues, CUlinkState* stateOut) const
+ uint32_t numOptions, CUjit_option* options, void** optionValues, CUlinkState* stateOut) const
{
return (*_cuLinkCreate)(numOptions, options, optionValues, stateOut);
}
@@ -109,29 +109,29 @@ CUresult CUDADriverWrapper::cuModuleGetFunction(CUfunction* hfunc, CUmodule hmod
return (*_cuModuleGetFunction)(hfunc, hmod, name);
}
-CUresult CUDADriverWrapper::cuLinkAddFile(CUlinkState state, CUjitInputType type, const char* path,
- unsigned int numOptions, CUjit_option* options, void** optionValues) const
+CUresult CUDADriverWrapper::cuLinkAddFile(CUlinkState state, CUjitInputType type, const char* path, uint32_t numOptions,
+ CUjit_option* options, void** optionValues) const
{
return (*_cuLinkAddFile)(state, type, path, numOptions, options, optionValues);
}
CUresult CUDADriverWrapper::cuLinkAddData(CUlinkState state, CUjitInputType type, void* data, size_t size,
- const char* name, unsigned int numOptions, CUjit_option* options, void** optionValues) const
+ const char* name, uint32_t numOptions, CUjit_option* options, void** optionValues) const
{
return (*_cuLinkAddData)(state, type, data, size, name, numOptions, options, optionValues);
}
-CUresult CUDADriverWrapper::cuLaunchCooperativeKernel(CUfunction f, unsigned int gridDimX, unsigned int gridDimY,
- unsigned int gridDimZ, unsigned int blockDimX, unsigned int blockDimY, unsigned int blockDimZ,
- unsigned int sharedMemBytes, CUstream hStream, void** kernelParams) const
+CUresult CUDADriverWrapper::cuLaunchCooperativeKernel(CUfunction f, uint32_t gridDimX, uint32_t gridDimY,
+ uint32_t gridDimZ, uint32_t blockDimX, uint32_t blockDimY, uint32_t blockDimZ, uint32_t sharedMemBytes,
+ CUstream hStream, void** kernelParams) const
{
return (*_cuLaunchCooperativeKernel)(
f, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY, blockDimZ, sharedMemBytes, hStream, kernelParams);
}
-CUresult CUDADriverWrapper::cuLaunchKernel(CUfunction f, unsigned int gridDimX, unsigned int gridDimY,
- unsigned int gridDimZ, unsigned int blockDimX, unsigned int blockDimY, unsigned int blockDimZ,
- unsigned int sharedMemBytes, CUstream hStream, void** kernelParams, void** extra) const
+CUresult CUDADriverWrapper::cuLaunchKernel(CUfunction f, uint32_t gridDimX, uint32_t gridDimY, uint32_t gridDimZ,
+ uint32_t blockDimX, uint32_t blockDimY, uint32_t blockDimZ, uint32_t sharedMemBytes, CUstream hStream,
+ void** kernelParams, void** extra) const
{
return (*_cuLaunchKernel)(
f, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY, blockDimZ, sharedMemBytes, hStream, kernelParams, extra);
diff --git a/plugin/common/cudaDriverWrapper.h b/plugin/common/cudaDriverWrapper.h
index 6e03000e..2dd3e23d 100644
--- a/plugin/common/cudaDriverWrapper.h
+++ b/plugin/common/cudaDriverWrapper.h
@@ -17,6 +17,7 @@
#ifndef CUDA_DRIVER_WRAPPER_H
#define CUDA_DRIVER_WRAPPER_H
+#include
#include
#include
@@ -46,24 +47,23 @@ public:
CUresult cuModuleLoadData(CUmodule* module, const void* image) const;
- CUresult cuLinkCreate(
- unsigned int numOptions, CUjit_option* options, void** optionValues, CUlinkState* stateOut) const;
+ CUresult cuLinkCreate(uint32_t numOptions, CUjit_option* options, void** optionValues, CUlinkState* stateOut) const;
CUresult cuModuleGetFunction(CUfunction* hfunc, CUmodule hmod, const char* name) const;
- CUresult cuLinkAddFile(CUlinkState state, CUjitInputType type, const char* path, unsigned int numOptions,
+ CUresult cuLinkAddFile(CUlinkState state, CUjitInputType type, const char* path, uint32_t numOptions,
CUjit_option* options, void** optionValues) const;
CUresult cuLinkAddData(CUlinkState state, CUjitInputType type, void* data, size_t size, const char* name,
- unsigned int numOptions, CUjit_option* options, void** optionValues) const;
+ uint32_t numOptions, CUjit_option* options, void** optionValues) const;
- CUresult cuLaunchCooperativeKernel(CUfunction f, unsigned int gridDimX, unsigned int gridDimY,
- unsigned int gridDimZ, unsigned int blockDimX, unsigned int blockDimY, unsigned int blockDimZ,
- unsigned int sharedMemBytes, CUstream hStream, void** kernelParams) const;
+ CUresult cuLaunchCooperativeKernel(CUfunction f, uint32_t gridDimX, uint32_t gridDimY, uint32_t gridDimZ,
+ uint32_t blockDimX, uint32_t blockDimY, uint32_t blockDimZ, uint32_t sharedMemBytes, CUstream hStream,
+ void** kernelParams) const;
- CUresult cuLaunchKernel(CUfunction f, unsigned int gridDimX, unsigned int gridDimY, unsigned int gridDimZ,
- unsigned int blockDimX, unsigned int blockDimY, unsigned int blockDimZ, unsigned int sharedMemBytes,
- CUstream hStream, void** kernelParams, void** extra) const;
+ CUresult cuLaunchKernel(CUfunction f, uint32_t gridDimX, uint32_t gridDimY, uint32_t gridDimZ, uint32_t blockDimX,
+ uint32_t blockDimY, uint32_t blockDimZ, uint32_t sharedMemBytes, CUstream hStream, void** kernelParams,
+ void** extra) const;
private:
void* handle;
@@ -80,9 +80,9 @@ private:
CUlinkState, CUjitInputType, void*, size_t, const char*, unsigned int, CUjit_option*, void**);
CUresult (*_cuLaunchCooperativeKernel)(CUfunction, unsigned int, unsigned int, unsigned int, unsigned int,
unsigned int, unsigned int, unsigned int, CUstream, void**);
- CUresult (*_cuLaunchKernel)(CUfunction f, unsigned int gridDimX, unsigned int gridDimY, unsigned int gridDimZ,
- unsigned int blockDimX, unsigned int blockDimY, unsigned int blockDimZ, unsigned int sharedMemBytes,
- CUstream hStream, void** kernelParams, void** extra);
+ CUresult (*_cuLaunchKernel)(CUfunction f, uint32_t gridDimX, uint32_t gridDimY, uint32_t gridDimZ,
+ uint32_t blockDimX, uint32_t blockDimY, uint32_t blockDimZ, uint32_t sharedMemBytes, CUstream hStream,
+ void** kernelParams, void** extra);
};
inline void cuErrCheck_(CUresult stat, const CUDADriverWrapper& wrap, const char* file, int line)
diff --git a/plugin/common/kernels/kernel.h b/plugin/common/kernels/kernel.h
index 45cbee0a..b6e4f261 100644
--- a/plugin/common/kernels/kernel.h
+++ b/plugin/common/kernels/kernel.h
@@ -91,6 +91,8 @@ size_t normalizePluginWorkspaceSize(bool acrossSpatial, int C, int H, int W);
pluginStatus_t normalizeInference(cudaStream_t stream, cublasHandle_t handle, bool acrossSpatial, bool channelShared,
int N, int C, int H, int W, float eps, const void* scale, const void* inputData, void* outputData, void* workspace);
+pluginStatus_t scatterNDInference(cudaStream_t stream, int* outputDims, int nOutputDims, int sliceRank, int nRows, int rowSize, int CopySize, int sizeOfElementInBytes, const void* index, const void* updates, const void* data, void* output, void* workspace);
+
pluginStatus_t priorBoxInference(cudaStream_t stream, PriorBoxParameters param, int H, int W, int numPriors,
int numAspectRatios, const void* minSize, const void* maxSize, const void* aspectRatios, void* outputData);
diff --git a/plugin/common/kernels/maskRCNNKernels.h b/plugin/common/kernels/maskRCNNKernels.h
index 2b6fd19a..3408df33 100644
--- a/plugin/common/kernels/maskRCNNKernels.h
+++ b/plugin/common/kernels/maskRCNNKernels.h
@@ -100,7 +100,7 @@ struct RefineDetectionWorkSpace
nvinfer1::DimsHW sortClassScoreDims;
nvinfer1::DimsHW sortClassLabelDims;
nvinfer1::DimsHW sortClassSampleIdxDims;
- nvinfer1::Dims sortClassValidCountDims = {1, {1, 0}, {nvinfer1::DimensionType::kINDEX}};
+ nvinfer1::Dims sortClassValidCountDims = {1, {1, 0}};
nvinfer1::DimsHW sortClassPosDims;
nvinfer1::DimsHW sortNMSMarkDims;
@@ -132,7 +132,7 @@ struct ProposalWorkSpace
nvinfer1::DimsHW sortClassScoreDims;
nvinfer1::DimsHW sortClassLabelDims;
nvinfer1::DimsHW sortClassSampleIdxDims;
- nvinfer1::Dims sortClassValidCountDims = {1, {1, 0}, {nvinfer1::DimensionType::kINDEX}};
+ nvinfer1::Dims sortClassValidCountDims = {1, {1, 0}};
nvinfer1::DimsHW sortClassPosDims;
nvinfer1::DimsHW sortNMSMarkDims;
@@ -167,7 +167,7 @@ struct MultilevelProposeROIWorkSpace
nvinfer1::DimsHW sortClassScoreDims;
nvinfer1::DimsHW sortClassLabelDims;
nvinfer1::DimsHW sortClassSampleIdxDims;
- nvinfer1::Dims sortClassValidCountDims = {1, {1, 0}, {nvinfer1::DimensionType::kINDEX}};
+ nvinfer1::Dims sortClassValidCountDims = {1, {1, 0}};
nvinfer1::DimsHW sortClassPosDims;
nvinfer1::DimsHW sortNMSMarkDims;
diff --git a/plugin/common/kernels/reducedMathPlugin.h b/plugin/common/kernels/reducedMathPlugin.h
index 12bb8b99..d047b6c7 100644
--- a/plugin/common/kernels/reducedMathPlugin.h
+++ b/plugin/common/kernels/reducedMathPlugin.h
@@ -30,7 +30,7 @@ namespace plugin
namespace detail
{
-void find_divisor(int denom, unsigned int& mul_coeff, unsigned int& shift_coeff);
+void findDivisor(int denom, unsigned int& mul_coeff, unsigned int& shift_coeff);
__host__ __device__ __forceinline__ unsigned int umulhi(unsigned int x, unsigned int y)
{
@@ -58,7 +58,7 @@ public:
__host__ __forceinline__ reduced_divisor(int _y)
: y(_y)
{
- detail::find_divisor(y, mul_coeff, shift_coeff);
+ detail::findDivisor(y, mul_coeff, shift_coeff);
}
__host__ __device__ __forceinline__ reduced_divisor(unsigned _mul_coeff, unsigned _shift_coeff, int _y)
: mul_coeff(_mul_coeff)
@@ -68,7 +68,7 @@ public:
}
__host__ __device__ __forceinline__ int div(int x) const
{
- // if dividing by 1, then find_divisor wouldn't have worked because
+ // if dividing by 1, then findDivisor wouldn't have worked because
// mul_coeff would have had to be 2^32, which can't be represented,
// so we have to special case that one.
return (y != 1) ? detail::umulhi((unsigned int) x, mul_coeff) >> shift_coeff : x;
diff --git a/plugin/common/nmsUtils.h b/plugin/common/nmsUtils.h
index 71baf547..1434bcb6 100644
--- a/plugin/common/nmsUtils.h
+++ b/plugin/common/nmsUtils.h
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#ifndef TRT_NMS_UTILS_H
#define TRT_NMS_UTILS_H
diff --git a/plugin/common/plugin.h b/plugin/common/plugin.h
index d2a2e87e..27a1fb7b 100644
--- a/plugin/common/plugin.h
+++ b/plugin/common/plugin.h
@@ -13,7 +13,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
#ifndef TRT_PLUGIN_H
#define TRT_PLUGIN_H
#include "checkMacrosPlugin.h"
@@ -45,15 +44,9 @@ namespace plugin
class BasePlugin : public IPluginV2
{
protected:
- void setPluginNamespace(const char* libNamespace) override
- {
- mNamespace = libNamespace;
- }
+ void setPluginNamespace(const char* libNamespace) noexcept override { mNamespace = libNamespace; }
- const char* getPluginNamespace() const override
- {
- return mNamespace.c_str();
- }
+ const char* getPluginNamespace() const noexcept override { return mNamespace.c_str(); }
std::string mNamespace;
};
@@ -61,12 +54,12 @@ protected:
class BaseCreator : public IPluginCreator
{
public:
- void setPluginNamespace(const char* libNamespace) override
+ void setPluginNamespace(const char* libNamespace) noexcept override
{
mNamespace = libNamespace;
}
- const char* getPluginNamespace() const override
+ const char* getPluginNamespace() const noexcept override
{
return mNamespace.c_str();
}
diff --git a/plugin/common/pluginLogging.h b/plugin/common/pluginLogging.h
deleted file mode 100644
index 91ee254d..00000000
--- a/plugin/common/pluginLogging.h
+++ /dev/null
@@ -1,529 +0,0 @@
-/*
- * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#ifndef PLUGIN_LOGGING_H
-#define PLUGIN_LOGGING_H
-
-#include "NvInferRuntimeCommon.h"
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-
-using Severity = nvinfer1::ILogger::Severity;
-
-class LogStreamConsumerBuffer : public std::stringbuf
-{
-public:
- LogStreamConsumerBuffer(std::ostream& stream, const std::string& prefix, bool shouldLog)
- : mOutput(stream)
- , mPrefix(prefix)
- , mShouldLog(shouldLog)
- {
- }
-
- LogStreamConsumerBuffer(LogStreamConsumerBuffer&& other)
- : mOutput(other.mOutput)
- {
- }
-
- ~LogStreamConsumerBuffer()
- {
- // std::streambuf::pbase() gives a pointer to the beginning of the buffered part of the output sequence
- // std::streambuf::pptr() gives a pointer to the current position of the output sequence
- // if the pointer to the beginning is not equal to the pointer to the current position,
- // call putOutput() to log the output to the stream
- if (pbase() != pptr())
- {
- putOutput();
- }
- }
-
- // synchronizes the stream buffer and returns 0 on success
- // synchronizing the stream buffer consists of inserting the buffer contents into the stream,
- // resetting the buffer and flushing the stream
- virtual int sync()
- {
- putOutput();
- return 0;
- }
-
- void putOutput()
- {
- if (mShouldLog)
- {
- // prepend timestamp
- std::time_t timestamp = std::time(nullptr);
- tm* tm_local = std::localtime(×tamp);
- std::cout << "[";
- std::cout << std::setw(2) << std::setfill('0') << 1 + tm_local->tm_mon << "/";
- std::cout << std::setw(2) << std::setfill('0') << tm_local->tm_mday << "/";
- std::cout << std::setw(4) << std::setfill('0') << 1900 + tm_local->tm_year << "-";
- std::cout << std::setw(2) << std::setfill('0') << tm_local->tm_hour << ":";
- std::cout << std::setw(2) << std::setfill('0') << tm_local->tm_min << ":";
- std::cout << std::setw(2) << std::setfill('0') << tm_local->tm_sec << "] ";
- // std::stringbuf::str() gets the string contents of the buffer
- // insert the buffer contents pre-appended by the appropriate prefix into the stream
- mOutput << mPrefix << str();
- // set the buffer to empty
- str("");
- // flush the stream
- mOutput.flush();
- }
- }
-
- void setShouldLog(bool shouldLog)
- {
- mShouldLog = shouldLog;
- }
-
-private:
- std::ostream& mOutput;
- std::string mPrefix;
- bool mShouldLog;
-};
-
-//!
-//! \class LogStreamConsumerBase
-//! \brief Convenience object used to initialize LogStreamConsumerBuffer before std::ostream in LogStreamConsumer
-//!
-class LogStreamConsumerBase
-{
-public:
- LogStreamConsumerBase(std::ostream& stream, const std::string& prefix, bool shouldLog)
- : mBuffer(stream, prefix, shouldLog)
- {
- }
-
-protected:
- std::mutex mLogMutex;
- LogStreamConsumerBuffer mBuffer;
-};
-
-//!
-//! \class LogStreamConsumer
-//! \brief Convenience object used to facilitate use of C++ stream syntax when logging messages.
-//! Order of base classes is LogStreamConsumerBase and then std::ostream.
-//! This is because the LogStreamConsumerBase class is used to initialize the LogStreamConsumerBuffer member field
-//! in LogStreamConsumer and then the address of the buffer is passed to std::ostream.
-//! This is necessary to prevent the address of an uninitialized buffer from being passed to std::ostream.
-//! Please do not change the order of the parent classes.
-//!
-class LogStreamConsumer : protected LogStreamConsumerBase, public std::ostream
-{
-public:
- //! \brief Creates a LogStreamConsumer which logs messages with level severity.
- //! Reportable severity determines if the messages are severe enough to be logged.
- LogStreamConsumer(Severity reportableSeverity, Severity severity)
- : LogStreamConsumerBase(severityOstream(severity), severityPrefix(severity), severity <= reportableSeverity)
- , std::ostream(&mBuffer) // links the stream buffer with the stream
- , mShouldLog(severity <= reportableSeverity)
- , mSeverity(severity)
- {
- }
-
- LogStreamConsumer(LogStreamConsumer&& other)
- : LogStreamConsumerBase(severityOstream(other.mSeverity), severityPrefix(other.mSeverity), other.mShouldLog)
- , std::ostream(&mBuffer) // links the stream buffer with the stream
- , mShouldLog(other.mShouldLog)
- , mSeverity(other.mSeverity)
- {
- }
-
- void setReportableSeverity(Severity reportableSeverity)
- {
- mShouldLog = mSeverity <= reportableSeverity;
- mBuffer.setShouldLog(mShouldLog);
- }
-
- std::mutex& getMutex()
- {
- return mLogMutex;
- }
-
-private:
- static std::ostream& severityOstream(Severity severity)
- {
- return severity >= Severity::kINFO ? std::cout : std::cerr;
- }
-
- static std::string severityPrefix(Severity severity)
- {
- switch (severity)
- {
- case Severity::kINTERNAL_ERROR: return "[F] ";
- case Severity::kERROR: return "[E] ";
- case Severity::kWARNING: return "[W] ";
- case Severity::kINFO: return "[I] ";
- case Severity::kVERBOSE: return "[V] ";
- default: assert(0); return "";
- }
- }
-
- bool mShouldLog;
- Severity mSeverity;
-};
-
-// Use mutex to protect multi-stream write to buffer
-template
-LogStreamConsumer& operator<<(LogStreamConsumer& logger, const T& obj)
-{
- std::lock_guard