asfiyab-nvidia
365 lines
16 KiB
Plaintext
365 lines
16 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2021 NVIDIA Corporation. All Rights Reserved.\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# http://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License.\n",
|
|
"# ==============================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<img src=\"https://upload.wikimedia.org/wikipedia/en/6/6d/Nvidia_image_logo.svg\" style=\"width: 90px; float: right;\">\n",
|
|
"\n",
|
|
"# QA Inference on BERT using TensorRT"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. Overview\n",
|
|
"\n",
|
|
"Bidirectional Embedding Representations from Transformers (BERT), is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. \n",
|
|
"\n",
|
|
"The original paper can be found here: https://arxiv.org/abs/1810.04805.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 1.a Learning objectives\n",
|
|
"\n",
|
|
"This notebook demonstrates:\n",
|
|
"- Inference on Question Answering (QA) task with BERT Base/Large model\n",
|
|
"- The use fine-tuned NVIDIA BERT models\n",
|
|
"- Use of BERT model with TRT"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Requirements\n",
|
|
"\n",
|
|
"Please refer to the ReadMe file"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. BERT Inference: Question Answering\n",
|
|
"\n",
|
|
"We can run inference on a fine-tuned BERT model for tasks like Question Answering.\n",
|
|
"\n",
|
|
"Here we use a BERT model fine-tuned on a [SQuaD 2.0 Dataset](https://rajpurkar.github.io/SQuAD-explorer/) which contains 100,000+ question-answer pairs on 500+ articles combined with over 50,000 new, unanswerable questions."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 3.a Paragraph and Queries\n",
|
|
"\n",
|
|
"The paragraph and the questions can be customized by changing the text below. Note that when using models with small sequence lengths, you should use a shorter paragraph:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Paragraph:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"paragraph_text = \"The Apollo program, also known as Project Apollo, was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans on the Moon from 1969 to 1972. First conceived during Dwight D. Eisenhower's administration as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was later dedicated to President John F. Kennedy's national goal of landing a man on the Moon and returning him safely to the Earth by the end of the 1960s, which he proposed in a May 25, 1961, address to Congress. Project Mercury was followed by the two-man Project Gemini. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the two-man Gemini program which ran concurrently with it from 1962 to 1966. Gemini missions developed some of the space travel techniques that were necessary for the success of the Apollo missions. Apollo used Saturn family rockets as launch vehicles. Apollo/Saturn vehicles were also used for an Apollo Applications Program, which consisted of Skylab, a space station that supported three manned missions in 1973-74, and the Apollo-Soyuz Test Project, a joint Earth orbit mission with the Soviet Union in 1975.\"\n",
|
|
"\n",
|
|
"# Short paragraph version for BERT models with max sequence length of 128\n",
|
|
"short_paragraph_text = \"The Apollo program was the third United States human spaceflight program. First conceived as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was dedicated to President John F. Kennedy's national goal of landing a man on the Moon. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972 followed by the Apollo-Soyuz Test Project a joint Earth orbit mission with the Soviet Union in 1975.\"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Question:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"question_text = \"What project put the first Americans into space?\"\n",
|
|
"#question_text = \"What year did the first manned Apollo flight occur?\"\n",
|
|
"#question_text = \"What President is credited with the original notion of putting Americans in space?\"\n",
|
|
"#question_text = \"Who did the U.S. collaborate with on an Earth orbit mission in 1975?\"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"In this example we ask our BERT model questions related to the following paragraph:\n",
|
|
"\n",
|
|
"**The Apollo Program**\n",
|
|
"_\"The Apollo program, also known as Project Apollo, was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans on the Moon from 1969 to 1972. First conceived during Dwight D. Eisenhower's administration as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was later dedicated to President John F. Kennedy's national goal of landing a man on the Moon and returning him safely to the Earth by the end of the 1960s, which he proposed in a May 25, 1961, address to Congress. Project Mercury was followed by the two-man Project Gemini. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the two-man Gemini program which ran concurrently with it from 1962 to 1966. Gemini missions developed some of the space travel techniques that were necessary for the success of the Apollo missions. Apollo used Saturn family rockets as launch vehicles. Apollo/Saturn vehicles were also used for an Apollo Applications Program, which consisted of Skylab, a space station that supported three manned missions in 1973-74, and the Apollo-Soyuz Test Project, a joint Earth orbit mission with the Soviet Union in 1975.\"_\n",
|
|
"\n",
|
|
"The questions and relative answers expected are shown below:\n",
|
|
"\n",
|
|
" - **Q1:** \"What project put the first Americans into space?\" \n",
|
|
" - **A1:** \"Project Mercury\"\n",
|
|
" - **Q2:** \"What program was created to carry out these projects and missions?\"\n",
|
|
" - **A2:** \"The Apollo program\"\n",
|
|
" - **Q3:** \"What year did the first manned Apollo flight occur?\"\n",
|
|
" - **A3:** \"1968\"\n",
|
|
" - **Q4:** \"What President is credited with the original notion of putting Americans in space?\"\n",
|
|
" - **A4:** \"John F. Kennedy\"\n",
|
|
" - **Q5:** \"Who did the U.S. collaborate with on an Earth orbit mission in 1975?\"\n",
|
|
" - **A5:** \"Soviet Union\"\n",
|
|
" - **Q6:** \"How long did Project Apollo run?\"\n",
|
|
" - **A6:** \"1961 to 1972\"\n",
|
|
" - **Q7:** \"What program helped develop space travel techniques that Project Apollo used?\"\n",
|
|
" - **A7:** \"Gemini Mission\"\n",
|
|
" - **Q8:** \"What space station supported three manned missions in 1973-1974?\"\n",
|
|
" - **A8:** \"Skylab\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Data Preprocessing\n",
|
|
"Let's convert the paragraph and the question to BERT input with the help of the tokenizer:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import helpers.data_processing as dp\n",
|
|
"import helpers.tokenization as tokenization\n",
|
|
"\n",
|
|
"tokenizer = tokenization.FullTokenizer(vocab_file=\"/workspace/TensorRT/demo/BERT/models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt\", do_lower_case=True)\n",
|
|
"\n",
|
|
"# The maximum number of tokens for the question. Questions longer than this will be truncated to this length.\n",
|
|
"max_query_length = 64\n",
|
|
"\n",
|
|
"# When splitting up a long document into chunks, how much stride to take between chunks.\n",
|
|
"doc_stride = 128\n",
|
|
"\n",
|
|
"# The maximum total input sequence length after WordPiece tokenization. \n",
|
|
"# Sequences longer than this will be truncated, and sequences shorter \n",
|
|
"max_seq_length = 128\n",
|
|
"\n",
|
|
"# Extract tokens from the paragraph\n",
|
|
"doc_tokens = dp.convert_doc_tokens(short_paragraph_text)\n",
|
|
"\n",
|
|
"# Extract features from the paragraph and question\n",
|
|
"features = dp.convert_example_to_features(doc_tokens, question_text, tokenizer, max_seq_length, doc_stride, max_query_length)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## TensorRT Inference"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import tensorrt as trt\n",
|
|
"TRT_LOGGER = trt.Logger(trt.Logger.INFO)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import ctypes\n",
|
|
"import os\n",
|
|
"\n",
|
|
"ctypes.CDLL(\"libnvinfer_plugin.so\", mode=ctypes.RTLD_GLOBAL)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from helpers.cuda_utils import cuda_call, CudaStreamContext, memcpy_host_to_device, memcpy_device_to_host\n",
|
|
"from cuda.bindings import driver as cuda, runtime as cudart\n",
|
|
"import collections\n",
|
|
"import numpy as np\n",
|
|
"import time\n",
|
|
"\n",
|
|
"# Load the BERT-Large Engine\n",
|
|
"with open(\"/workspace/TensorRT/demo/BERT/engines/bert_large_128.engine\", \"rb\") as f, \\\n",
|
|
" trt.Runtime(TRT_LOGGER) as runtime, \\\n",
|
|
" runtime.deserialize_cuda_engine(f.read()) as engine, \\\n",
|
|
" engine.create_execution_context() as context:\n",
|
|
"\n",
|
|
" # We always use batch size 1.\n",
|
|
" input_shape = (1, max_seq_length)\n",
|
|
" input_nbytes = trt.volume(input_shape) * trt.int32.itemsize\n",
|
|
" \n",
|
|
" # Allocate device memory for inputs.\n",
|
|
" d_inputs = [cuda_call(cudart.cudaMalloc(input_nbytes)) for binding in range(3)]\n",
|
|
"\n",
|
|
" # Specify input shapes. These must be within the min/max bounds of the active profile (0th profile in this case)\n",
|
|
" # Note that input shapes can be specified on a per-inference basis, but in this case, we only have a single shape.\n",
|
|
" for binding in range(3):\n",
|
|
" tensor_name = engine.get_tensor_name(binding)\n",
|
|
" context.set_input_shape(tensor_name, input_shape)\n",
|
|
" assert context.all_binding_shapes_specified\n",
|
|
"\n",
|
|
" # Allocate output buffer by querying the size from the context. This may be different for different input shapes.\n",
|
|
" h_output = np.empty(tuple(context.get_tensor_shape(engine.get_tensor_name(3))), dtype=np.float32)\n",
|
|
" cuda_call(cudart.cudaHostRegister(h_output, h_output.nbytes, 0))\n",
|
|
" d_output = cuda_call(cudart.cudaMalloc(h_output.nbytes))\n",
|
|
"\n",
|
|
" with CudaStreamContext() as stream:\n",
|
|
" print(\"\\nRunning Inference...\")\n",
|
|
"\n",
|
|
" _NetworkOutput = collections.namedtuple( # pylint: disable=invalid-name\n",
|
|
" \"NetworkOutput\",\n",
|
|
" [\"start_logits\", \"end_logits\", \"feature_index\"])\n",
|
|
" networkOutputs = []\n",
|
|
"\n",
|
|
" eval_time_elapsed = 0\n",
|
|
" for feature_index, feature in enumerate(features):\n",
|
|
" # Copy inputs\n",
|
|
" input_ids = np.ascontiguousarray(feature.input_ids.ravel())\n",
|
|
" segment_ids = np.ascontiguousarray(feature.segment_ids.ravel())\n",
|
|
" input_mask = np.ascontiguousarray(feature.input_mask.ravel())\n",
|
|
"\n",
|
|
" eval_start_time = time.time()\n",
|
|
" memcpy_host_to_device_async(d_inputs[0], input_ids, stream.stream)\n",
|
|
" memcpy_host_to_device_async(d_inputs[1], segment_ids, stream.stream)\n",
|
|
" memcpy_host_to_device_async(d_inputs[2], input_mask, stream.stream)\n",
|
|
"\n",
|
|
" # Setup tensor address\n",
|
|
" bindings = [int(d_inputs[i]) for i in range(3)] + [int(d_output)]\n",
|
|
"\n",
|
|
" for i in range(engine.num_io_tensors):\n",
|
|
" context.set_tensor_address(engine.get_tensor_name(i), bindings[i])\n",
|
|
"\n",
|
|
" # Run inference\n",
|
|
" context.execute_async_v3(stream_handle=stream.stream)\n",
|
|
" # Synchronize the stream\n",
|
|
" stream.synchronize()\n",
|
|
" eval_time_elapsed += (time.time() - eval_start_time)\n",
|
|
"\n",
|
|
" # Transfer predictions back from GPU\n",
|
|
" memcpy_device_to_host_async(h_output, d_output, stream.stream)\n",
|
|
" stream.synchronize()\n",
|
|
"\n",
|
|
" for index, batch in enumerate(h_output):\n",
|
|
" # Data Post-processing\n",
|
|
" networkOutputs.append(_NetworkOutput(\n",
|
|
" start_logits = np.array(batch.squeeze()[:, 0]),\n",
|
|
" end_logits = np.array(batch.squeeze()[:, 1]),\n",
|
|
" feature_index = feature_index\n",
|
|
" ))\n",
|
|
"\n",
|
|
" eval_time_elapsed /= len(features)\n",
|
|
" \n",
|
|
" print(\"-----------------------------\")\n",
|
|
" print(\"Running Inference at {:.3f} Sentences/Sec\".format(1.0/eval_time_elapsed))\n",
|
|
" print(\"-----------------------------\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Data Post-Processing"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now that we have the inference results let's extract the actual answer to our question"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
" # The total number of n-best predictions to generate in the nbest_predictions.json output file\n",
|
|
" n_best_size = 20\n",
|
|
"\n",
|
|
" # The maximum length of an answer that can be generated. This is needed \n",
|
|
" # because the start and end predictions are not conditioned on one another\n",
|
|
" max_answer_length = 30\n",
|
|
"\n",
|
|
" prediction, nbest_json, scores_diff_json = dp.get_predictions(doc_tokens, features,\n",
|
|
" networkOutputs, n_best_size, max_answer_length)\n",
|
|
" \n",
|
|
" for index, output in enumerate(networkOutputs):\n",
|
|
" print(\"Processing output\")\n",
|
|
" print(\"Answer: '{}'\".format(prediction))\n",
|
|
" print(\"with prob: {:.3f}%\".format(nbest_json[0]['probability'] * 100.0))\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|