This reverts commit 970d88ee18 due to broken blog links
This commit is contained in:
Zhaoqi
@@ -12,17 +12,16 @@ Amazon SageMaker examples are divided in two repositories:
|
||||
|
||||
- [Sagemaker Example Community repository](https://github.com/aws/amazon-sagemaker-examples-community) is another SageMaker repository which contains additional examples and reference solutions, beyond the examples showcased in the [official repository](https://github.com/aws/amazon-sagemaker-examples). This repository is maintained by community of engineers and solution architects at AWS.
|
||||
|
||||
## Planning to submit a PR to this repository? Read this first:
|
||||
|
||||
## Planning to submit a PR to this repository? read this first:
|
||||
- This repository will only accept notebooks/examples which demonstrate a feature of SageMaker, not yet covered anywhere in this repository. PR submitters are requested to check this before submitting the PR to avoid getting it rejected.
|
||||
|
||||
- If you still would like to contribute your example, please submit a PR to [Sagemaker Example Community repository](https://github.com/aws/amazon-sagemaker-examples-community) instead.
|
||||
|
||||
|
||||
## :hammer_and_wrench: Setup
|
||||
|
||||
The quickest setup to run example notebooks includes:
|
||||
|
||||
- An [AWS Account](http://docs.aws.amazon.com/sagemaker/latest/dg/gs-account.html)
|
||||
- An [AWS account](http://docs.aws.amazon.com/sagemaker/latest/dg/gs-account.html)
|
||||
- Proper [IAM User and Role](http://docs.aws.amazon.com/sagemaker/latest/dg/authentication-and-access-control.html) setup
|
||||
- An [Amazon SageMaker Notebook Instance](http://docs.aws.amazon.com/sagemaker/latest/dg/gs-setup-working-env.html)
|
||||
- An [S3 bucket](http://docs.aws.amazon.com/sagemaker/latest/dg/gs-config-permissions.html)
|
||||
@@ -38,6 +37,22 @@ As of February 7, 2022, the default branch is named "main". See our [announcemen
|
||||
|
||||
## :notebook: Examples
|
||||
|
||||
### Introduction to geospatial capabilities
|
||||
|
||||
These examples introduce SageMaker geospatial capabilities which makes it easy to build, train, and deploy ML models using geospatial data.
|
||||
|
||||
- [How to use SageMaker Processing with geospatial image](sagemaker-geospatial/processing-geospatial-ndvi/geospatial-processing-ndvi-intro.ipynb) shows how to compute the normalized difference vegetation index (NDVI) which indicates health and density of vegetation using SageMaker Processing and satellite imagery
|
||||
- [Monitoring Lake Drought with SageMaker Geospatial Capabilities](sagemaker-geospatial/lake-mead-drought-monitoring) shows how to monitor Lake Mead drought using SageMaker geospatial capabilities.
|
||||
- [Digital Farming with Amazon SageMaker Geospatial Capabilities](sagemaker-geospatial/digital-farming-pipelines) shows how geospatial capabilities can help accelerating, optimizing, and easing the processing of the geospatial data for the Digital Farming use cases.
|
||||
- [Assess wildfire damage with Amazon SageMaker Geospatial Capabilities](sagemaker-geospatial/dixie-wildfire-damage-assessment/dixie-wildfire-damage-assessment.ipynb) demonstrates how Amazon SageMaker geospatial capabilities can be used to identify and assess vegetation loss caused by the Dixie wildfire in Northern California.
|
||||
- [Monitoring Glacier Melting with SageMaker Geospatial Capabilities](sagemaker-geospatial/mount-shasta-glacier-melting-monitoring) shows how to monitor glacier melting at Mount Shasta using SageMaker geospatial capabilities.
|
||||
- [Monitoring of methane (CH4) emission point sources using Amazon SageMaker Geospatial Capabilities](sagemaker-geospatial/methane-emission-monitoring/monitor_methane_ch4_emission_point_sources.ipynb) demonstrates how methane emissions can be detected by using open data Satellite imagery (Sentinel-2).
|
||||
- [Segmenting aerial imagery using geospatial GPU notebook](sagemaker-geospatial/segment-aerial-naip/segment_naip_geospatial_notebook.ipynb) shows how to use the geospatial GPU notebook with open-source libraries to perform segmentation on aerial imagery.
|
||||
- [Perform Sentinel-1 InSAR using ESA SNAP Toolkit](sagemaker-geospatial/sentinel1-insar-snap/sentinel1_insar_kumamoto.ipynb) shows how the SNAP toolkit can be used within Amazon SageMaker geospatial capabilities to create interferograms on Sentinel-1 SAR data.
|
||||
- [How to use Vector Enrichment Jobs for Map Matching](sagemaker-geospatial/vector-enrichment-map-matching/vector-enrichment-map-matching.ipynb) shows how to use vector enrichtment operations with Amazon SageMaker Geospatial capabilities to snap GPS coordinates to road segments.
|
||||
- [How to use Vector Enrichment Jobs for Reverse Geocoding](sagemaker-geospatial/vector-enrichment-reverse-geocoding/vector-enrichment-reverse-geocoding.ipynb) shows how to use Amazon SageMaker Geospatial capabilities for reverse geocoding to obtain human readable addresses from data with latitude/longitude information.
|
||||
- [Building geospatial pipelines with SageMaker Pipelines](sagemaker-geospatial/geospatial-processing-pipeline/geospatial_pipeline_processing.ipynb) shows how a geospatial data processing workflow can be automated by using Amazon SageMaker Pipelines.
|
||||
|
||||
### Introduction to Ground Truth Labeling Jobs
|
||||
|
||||
These examples provide quick walkthroughs to get you up and running with the labeling job workflow for Amazon SageMaker Ground Truth.
|
||||
@@ -54,15 +69,20 @@ These examples provide quick walkthroughs to get you up and running with the lab
|
||||
These examples provide a gentle introduction to machine learning concepts as they are applied in practical use cases across a variety of sectors.
|
||||
|
||||
- [Predicting Customer Churn](introduction_to_applying_machine_learning/xgboost_customer_churn) uses customer interaction and service usage data to find those most likely to churn, and then walks through the cost/benefit trade-offs of providing retention incentives. This uses Amazon SageMaker's implementation of [XGBoost](https://github.com/dmlc/xgboost) to create a highly predictive model.
|
||||
- [Cancer Prediction](introduction_to_applying_machine_learning/breast_cancer_prediction) predicts Breast Cancer based on features derived from images, using SageMaker's Linear Learner.
|
||||
- [Ensembling](introduction_to_applying_machine_learning/ensemble_modeling) predicts income using two Amazon SageMaker models to show the advantages in ensembling.
|
||||
- [Video Game Sales](introduction_to_applying_machine_learning/video_game_sales) develops a binary prediction model for the success of video games based on review scores.
|
||||
- [MXNet Gluon Recommender System](introduction_to_applying_machine_learning/gluon_recommender_system) uses neural network embeddings for non-linear matrix factorization to predict user movie ratings on Amazon digital reviews.
|
||||
- [Fair Linear Learner](introduction_to_applying_machine_learning/fair_linear_learner) is an example of an effective way to create fair linear models with respect to sensitive features.
|
||||
- [Population Segmentation of US Census Data using PCA and Kmeans](introduction_to_applying_machine_learning/US-census_population_segmentation_PCA_Kmeans) analyzes US census data and reduces dimensionality using PCA then clusters US counties using KMeans to identify segments of similar counties.
|
||||
- [Document Embedding using Object2Vec](introduction_to_applying_machine_learning/object2vec_document_embedding) is an example to embed a large collection of documents in a common low-dimensional space, so that the semantic distances between these documents are preserved.
|
||||
- [Traffic violations forecasting using DeepAR](introduction_to_applying_machine_learning/deepar_chicago_traffic_violations) is an example to use daily traffic violation data to predict pattern and seasonality to use Amazon DeepAR alogorithm.
|
||||
- [Visual Inspection Automation with Pre-trained Amazon SageMaker Models](introduction_to_applying_machine_learning/visual_object_detection) is an example for fine-tuning pre-trained Amazon Sagemaker models on a target dataset.
|
||||
- [Create SageMaker Models Using the PyTorch Model Zoo](introduction_to_applying_machine_learning/sagemaker_pytorch_model_zoo) contains an example notebook to create a SageMaker model leveraging the PyTorch Model Zoo and visualize the results.
|
||||
- [Deep Demand Forecasting](introduction_to_applying_machine_learning/deep_demand_forecasting) provides an end-to-end solution for Demand Forecasting task using three state-of-the-art time series algorithms LSTNet, Prophet, and SageMaker DeepAR, which are available in GluonTS and Amazon SageMaker.
|
||||
- [Fraud Detection Using Graph Neural Networks](introduction_to_applying_machine_learning/fraud_detection_using_graph_neural_networks) is an example to identify fraudulent transactions from transaction and user identity datasets.
|
||||
- [Identify key insights from textual document](introduction_to_applying_machine_learning/identify_key_insights_from_textual_document) contains comphrensive notebooks for five natural language processing tasks Document Summarization, Text Classification, Question and Answering, Name Entity Recognition, and Semantic Relation Extracion.
|
||||
- [Synthetic Churn Prediction with Text](introduction_to_applying_machine_learning/synthetic_churn_prediction_with_text) contains an example notebook to train, deploy and use a churn prediction model that processed numerical, categorical and textual features to make its prediction.
|
||||
- [Credit Card Fraud Detector](introduction_to_applying_machine_learning/credit_card_fraud_detector) is an example of the core of a credit card fraud detection system using SageMaker with Random Cut Forest and XGBoost.
|
||||
- [Churn Prediction Multimodality of Text and Tabular](introduction_to_applying_machine_learning/churn_prediction_multimodality_of_text_and_tabular) is an example notebook to train and deploy a churn prediction model that uses state-of-the-art natural language processing model to find useful signals in text. In addition to textual inputs, this model uses traditional structured data inputs such as numerical and categorical fields.
|
||||
|
||||
@@ -70,11 +90,16 @@ These examples provide a gentle introduction to machine learning concepts as the
|
||||
|
||||
These examples introduce SageMaker's hyperparameter tuning functionality which helps deliver the best possible predictions by running a large number of training jobs to determine which hyperparameter values are the most impactful.
|
||||
|
||||
- [TensorFlow Tuning](hyperparameter_tuning/tensorflow2_mnist) shows how to use SageMaker hyperparameter tuning with the pre-built TensorFlow container and MNIST dataset.
|
||||
- [XGBoost Tuning](hyperparameter_tuning/xgboost_direct_marketing) shows how to use SageMaker hyperparameter tuning to improve your model fit.
|
||||
- [BlazingText Tuning](hyperparameter_tuning/blazingtext_text_classification_20_newsgroups) shows how to use SageMaker hyperparameter tuning with the BlazingText built-in algorithm and 20_newsgroups dataset..
|
||||
- [TensorFlow Tuning](hyperparameter_tuning/tensorflow_mnist) shows how to use SageMaker hyperparameter tuning with the pre-built TensorFlow container and MNIST dataset.
|
||||
- [MXNet Tuning](hyperparameter_tuning/mxnet_mnist) shows how to use SageMaker hyperparameter tuning with the pre-built MXNet container and MNIST dataset.
|
||||
- [HuggingFace Tuning](hyperparameter_tuning/huggingface_multiclass_text_classification_20_newsgroups) shows how to use SageMaker hyperparameter tuning with the pre-built HuggingFace container and 20_newsgroups dataset.
|
||||
- [Keras BYO Tuning](hyperparameter_tuning/keras_bring_your_own) shows how to use SageMaker hyperparameter tuning with a custom container running a Keras convolutional network on CIFAR-10 data.
|
||||
- [R BYO Tuning](hyperparameter_tuning/r_bring_your_own) shows how to use SageMaker hyperparameter tuning with the custom container from the [Bring Your Own R Algorithm](advanced_functionality/r_bring_your_own) example.
|
||||
- [Analyzing Results](hyperparameter_tuning/analyze_results) is a shared notebook that can be used after each of the above notebooks to provide analysis on how training jobs with different hyperparameters performed.
|
||||
- [Model tuning for distributed training](hyperparameter_tuning/model_tuning_for_distributed_training) shows how to use SageMaker hyperparameter tuning with Hyperband strategy for optimizing model in distributed training.
|
||||
- [Neural Architecture Search for Large Language Models](hyperparameter_tuning/neural_architecture_search_llm) shows how to prune fine-tuned large language models via neural architecture search.
|
||||
|
||||
### SageMaker Autopilot
|
||||
|
||||
@@ -82,7 +107,7 @@ These examples introduce SageMaker Autopilot. Autopilot automatically performs f
|
||||
|
||||
- [Customer Churn AutoML](autopilot/) shows how to use SageMaker Autopilot to automatically train a model for the [Predicting Customer Churn](introduction_to_applying_machine_learning/xgboost_customer_churn) task.
|
||||
- [Targeted Direct Marketing AutoML](autopilot/) shows how to use SageMaker Autopilot to automatically train a model.
|
||||
- [Housing Prices AutoML](autopilot/autopilot_california_housing.ipynb) shows how to use SageMaker Autopilot for a linear regression problem (predict housing prices).
|
||||
- [Housing Prices AutoML](sagemaker-autopilot/housing_prices) shows how to use SageMaker Autopilot for a linear regression problem (predict housing prices).
|
||||
- [Portfolio Churn Prediction with Amazon SageMaker Autopilot and Neo4j](autopilot/sagemaker_autopilot_neo4j_portfolio_churn.ipynb) shows how to use SageMaker Autopilot with graph embeddings to predict investment portfolio churn.
|
||||
- [Move Amazon SageMaker Autopilot ML models from experimentation to production using Amazon SageMaker Pipelines](autopilot/sagemaker-autopilot-pipelines) shows how to use SageMaker Autopilot in combination with SageMaker Pipelines for end-to-end AutoML training automation.
|
||||
- [Amazon SageMaker Autopilot models to serverless endpoints](autopilot/autopilot-serverless-inference) shows how to deploy Autopilot generated models to serverless endpoints.
|
||||
@@ -98,10 +123,14 @@ These examples provide quick walkthroughs to get you up and running with Amazon
|
||||
- [Neural Topic Model (NTM)](introduction_to_amazon_algorithms/ntm_synthetic) uses Amazon SageMaker Neural Topic Model (NTM) to uncover topics in documents from a synthetic data source, where topic distributions are known.
|
||||
- [Principal Components Analysis (PCA)](introduction_to_amazon_algorithms/pca_mnist) uses Amazon SageMaker PCA to calculate eigendigits from MNIST.
|
||||
- [Seq2Seq](introduction_to_amazon_algorithms/seq2seq_translation_en-de) uses the Amazon SageMaker Seq2Seq algorithm that's built on top of [Sockeye](https://github.com/awslabs/sockeye), which is a sequence-to-sequence framework for Neural Machine Translation based on MXNet. Seq2Seq implements state-of-the-art encoder-decoder architectures which can also be used for tasks like Abstractive Summarization in addition to Machine Translation. This notebook shows translation from English to German text.
|
||||
- [Image Classification](introduction_to_amazon_algorithms/imageclassification_caltech) includes full training and transfer learning examples of Amazon SageMaker's Image Classification algorithm. This uses a ResNet deep convolutional neural network to classify images from the caltech dataset.
|
||||
- [XGBoost for regression](introduction_to_amazon_algorithms/xgboost_abalone) predicts the age of abalone ([Abalone dataset](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html)) using regression from Amazon SageMaker's implementation of [XGBoost](https://github.com/dmlc/xgboost).
|
||||
- [XGBoost for multi-class classification](introduction_to_amazon_algorithms/xgboost_mnist) uses Amazon SageMaker's implementation of [XGBoost](https://github.com/dmlc/xgboost) to classify handwritten digits from the MNIST dataset as one of the ten digits using a multi-class classifier. Both single machine and distributed use-cases are presented.
|
||||
- [DeepAR for time series forecasting](introduction_to_amazon_algorithms/deepar_synthetic) illustrates how to use the Amazon SageMaker DeepAR algorithm for time series forecasting on a synthetically generated data set.
|
||||
- [BlazingText Word2Vec](introduction_to_amazon_algorithms/blazingtext_word2vec_text8) generates Word2Vec embeddings from a cleaned text dump of Wikipedia articles using SageMaker's fast and scalable BlazingText implementation.
|
||||
- [Object detection for bird images](introduction_to_amazon_algorithms/object_detection_birds) demonstrates how to use the Amazon SageMaker Object Detection algorithm with a public dataset of Bird images.
|
||||
- [Object2Vec for movie recommendation](introduction_to_amazon_algorithms/object2vec_movie_recommendation) demonstrates how Object2Vec can be used to model data consisting of pairs of singleton tokens using movie recommendation as a running example.
|
||||
- [Object2Vec for multi-label classification](introduction_to_amazon_algorithms/object2vec_multilabel_genre_classification) shows how ObjectToVec algorithm can train on data consisting of pairs of sequences and singleton tokens using the setting of genre prediction of movies based on their plot descriptions.
|
||||
- [Object2Vec for sentence similarity](introduction_to_amazon_algorithms/object2vec_sentence_similarity) explains how to train Object2Vec using sequence pairs as input using sentence similarity analysis as the application.
|
||||
- [IP Insights for suspicious logins](introduction_to_amazon_algorithms/ipinsights_login) shows how to train IP Insights on a login events for a web server to identify suspicious login attempts.
|
||||
- [Semantic Segmentation](introduction_to_amazon_algorithms/semantic_segmentation_pascalvoc) shows how to train a semantic segmentation algorithm using the Amazon SageMaker Semantic Segmentation algorithm. It also demonstrates how to host the model and produce segmentation masks and probability of segmentation.
|
||||
@@ -118,21 +147,31 @@ These examples provide quick walkthroughs to get you up and running with Amazon
|
||||
- [JumpStart Upscaling](introduction_to_amazon_algorithms/jumpstart_upscaling) demonstrates how to enhance image quality with Stable Diffusion models in JumpStart.
|
||||
- [JumpStart Inpainting](introduction_to_amazon_algorithms/jumpstart_inpainting) demonstrates how to inpaint an image with Stable Diffusion models in JumpStart.
|
||||
- [In-context learning with AlexaTM 20B](introduction_to_amazon_algorithms/jumpstart_alexatm20b) demonstrates how to use AlexaTM 20B for in-context-learning in JumpStart.
|
||||
|
||||
### Amazon SageMaker RL
|
||||
|
||||
The following provide examples demonstrating different capabilities of Amazon SageMaker RL.
|
||||
|
||||
- [Cartpole using Coach](reinforcement_learning/rl_cartpole_coach) demonstrates the simplest usecase of Amazon SageMaker RL using Intel's RL Coach.
|
||||
- [AWS DeepRacer](reinforcement_learning/rl_deepracer_robomaker_coach_gazebo) demonstrates AWS DeepRacer trainig using RL Coach in the Gazebo environment.
|
||||
- [HVAC using EnergyPlus](reinforcement_learning/rl_hvac_coach_energyplus) demonstrates the training of HVAC systems using the EnergyPlus environment.
|
||||
- [Knapsack Problem](reinforcement_learning/rl_knapsack_coach_custom) demonstrates how to solve the knapsack problem using a custom environment.
|
||||
- [Mountain Car](reinforcement_learning/rl_mountain_car_coach_gymEnv) Mountain car is a classic RL problem. This notebook explains how to solve this using the OpenAI Gym environment.
|
||||
- [Distributed Neural Network Compression](reinforcement_learning/rl_network_compression_ray_custom) This notebook explains how to compress ResNets using RL, using a custom environment and the RLLib toolkit.
|
||||
- [Portfolio Management](reinforcement_learning/rl_portfolio_management_coach_customEnv) This notebook uses a custom Gym environment to manage multiple financial investments.
|
||||
- [Autoscaling](reinforcement_learning/rl_predictive_autoscaling_coach_customEnv) demonstrates how to adjust load depending on demand. This uses RL Coach and a custom environment.
|
||||
- [Roboschool](reinforcement_learning/rl_roboschool_ray) is an open source physics simulator that is commonly used to train RL policies for robotic systems. This notebook demonstrates training a few agents using it.
|
||||
- [Stable Baselines](reinforcement_learning/rl_roboschool_stable_baselines) In this notebook example, we will make the HalfCheetah agent learn to walk using the stable-baselines, which are a set of improved implementations of Reinforcement Learning (RL) algorithms based on OpenAI Baselines.
|
||||
- [Travelling Salesman](reinforcement_learning/rl_traveling_salesman_vehicle_routing_coach) is a classic NP hard problem, which this notebook solves with AWS SageMaker RL.
|
||||
- [Tic-tac-toe](reinforcement_learning/rl_tic_tac_toe_coach_customEnv) is a simple implementation of a custom Gym environment to train and deploy an RL agent in Coach that then plays tic-tac-toe interactively in a Jupyter Notebook.
|
||||
- [Unity Game Agent](reinforcement_learning/rl_unity_ray) shows how to use RL algorithms to train an agent to play Unity3D game.
|
||||
|
||||
### Scientific Details of Algorithms
|
||||
|
||||
These examples provide more thorough mathematical treatment on a select group of algorithms.
|
||||
|
||||
- [Streaming Median](scientific_details_of_algorithms/streaming_median) sequentially introduces concepts used in streaming algorithms, which many SageMaker algorithms rely on to deliver speed and scalability.
|
||||
- [Latent Dirichlet Allocation (LDA)](scientific_details_of_algorithms/lda_topic_modeling) dives into Amazon SageMaker's spectral decomposition approach to LDA.
|
||||
- [Linear Learner features](scientific_details_of_algorithms/linear_learner_class_weights_loss_functions) shows how to use the class weights and loss functions features of the SageMaker Linear Learner algorithm to improve performance on a credit card fraud prediction task
|
||||
|
||||
### Amazon SageMaker Debugger
|
||||
|
||||
@@ -140,13 +179,24 @@ These examples provide and introduction to SageMaker Debugger which allows debug
|
||||
|
||||
- [Using a built-in rule with TensorFlow](sagemaker-debugger/tensorflow_builtin_rule/)
|
||||
- [Using a custom rule with TensorFlow Keras](sagemaker-debugger/tensorflow_keras_custom_rule/)
|
||||
- [Interactive tensor analysis in notebook with MXNet](sagemaker-debugger/mnist_tensor_analysis/)
|
||||
- [Visualizing Debugging Tensors of MXNet training](sagemaker-debugger/mnist_tensor_plot/)
|
||||
- [Real-time analysis in notebook with MXNet](sagemaker-debugger/mxnet_realtime_analysis/)
|
||||
- [Using a built in rule with XGBoost](sagemaker-debugger/xgboost_builtin_rules/)
|
||||
- [Real-time analysis in notebook with XGBoost](sagemaker-debugger/xgboost_realtime_analysis/)
|
||||
- [Using SageMaker Debugger with Managed Spot Training and MXNet](sagemaker-debugger/mxnet_spot_training/)
|
||||
- [Reacting to CloudWatch Events from Rules to take an action based on status with TensorFlow](sagemaker-debugger/tensorflow_action_on_rule/)
|
||||
- [Using SageMaker Debugger with a custom PyTorch container](sagemaker-debugger/pytorch_custom_container/)
|
||||
|
||||
### Amazon SageMaker Distributed Training
|
||||
|
||||
These examples provide an introduction to SageMaker Distributed Training Libraries for data parallelism and model parallelism. The libraries are optimized for the SageMaker training environment, help adapt your distributed training jobs to SageMaker, and improve training speed and throughput.
|
||||
More examples for models such as BERT and YOLOv5 can be found in [distributed_training/](https://github.com/aws/amazon-sagemaker-examples/tree/main/training/distributed_training).
|
||||
|
||||
- [Train GPT-2 with Sharded Data Parallel](https://github.com/aws/amazon-sagemaker-examples/tree/main/training/distributed_training/pytorch/model_parallel/gpt2/smp-train-gpt-simple-sharded-data-parallel.ipynb) shows how to train GPT-2 with near-linear scaling using Sharded Data Parallelism technique in SageMaker Model Parallelism Library.
|
||||
- [Train EleutherAI GPT-J with Model Parallel](https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/model_parallel/gpt-j/11_train_gptj_smp_tensor_parallel_notebook.ipynb) shows how to train EleutherAI GPT-J with PyTorch and Tensor Parallelism technique in the SageMaker Model Parallelism Library.
|
||||
- [Train MaskRCNN with Data Parallel](https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/data_parallel/maskrcnn/pytorch_smdataparallel_maskrcnn_demo.ipynb) shows how to train MaskRCNN with PyTorch and SageMaker Data Parallelism Library.
|
||||
|
||||
### Amazon SageMaker Smart Sifting
|
||||
|
||||
These examples provide an Introduction to Smart Sifting library. Smart Sifting is a framework to speed up training of PyTorch models. The framework implements a set of algorithms that filter out inconsequential training examples during training, reducing the computational cost and accelerating the training process. It is configuration-driven and extensible, allowing users to add custom logic to transform their training examples into a filterable format. Smart sifting provides a generic utility for any DNN model, and can reduce the training cost by up to 35% in infrastructure cost.
|
||||
@@ -158,8 +208,9 @@ These examples provide an Introduction to Smart Sifting library. Smart Sifting i
|
||||
|
||||
These examples provide an introduction to SageMaker Clarify which provides machine learning developers with greater visibility into their training data and models so they can identify and limit bias and explain predictions.
|
||||
|
||||
- [Fairness and Explainability with SageMaker Clarify](sagemaker-clarify/fairness_and_explainability) shows how to use SageMaker Clarify Processor API to measure the pre-training bias of a dataset and post-training bias of a model, and explain the importance of the input features on the model's decision.
|
||||
- [Amazon SageMaker Clarify Model Monitors](sagemaker_model_monitor/fairness_and_explainability) shows how to use SageMaker Clarify Model Monitor API to schedule bias monitor to monitor predictions for bias drift on a regular basis, and schedule explainability monitor to monitor predictions for feature attribution drift on a regular basis.
|
||||
* [Fairness and Explainability with SageMaker Clarify](sagemaker-clarify/fairness_and_explainability) shows how to use SageMaker Clarify Processor API to measure the pre-training bias of a dataset and post-training bias of a model, and explain the importance of the input features on the model's decision.
|
||||
* [TimeSeries Explainability with SageMaker Clarify](sagemaker-clarify/time_series_deepar) shows how to use SageMaker Clarify Processor API to explain the importance of the input features on the time-series model's decision.
|
||||
* [Amazon SageMaker Clarify Model Monitors](sagemaker_model_monitor/fairness_and_explainability) shows how to use SageMaker Clarify Model Monitor API to schedule bias monitor to monitor predictions for bias drift on a regular basis, and schedule explainability monitor to monitor predictions for feature attribution drift on a regular basis.
|
||||
|
||||
### Publishing content from RStudio on Amazon SageMaker to RStudio Connect
|
||||
|
||||
@@ -173,32 +224,45 @@ These examples show you how to run R examples, and publish applications in RStud
|
||||
|
||||
These examples showcase unique functionality available in Amazon SageMaker. They cover a broad range of topics and utilize a variety of methods, but aim to provide the user with sufficient insight or inspiration to develop within Amazon SageMaker.
|
||||
|
||||
- [Data Distribution Types](advanced_functionality/data_distribution_types) showcases the difference between two methods for sending data from S3 to Amazon SageMaker Training instances. This has particular implication for scalability and accuracy of distributed training.
|
||||
- [Distributed Training and Batch Transform with Sentiment Classification](advanced_functionality/sentiment_parallel_batch) shows how to use SageMaker Distributed Data Parallelism, SageMaker Debugger, and distrubted SageMaker Batch Transform on a HuggingFace Estimator, in a sentiment classification use case.
|
||||
- [Encrypting Your Data](advanced_functionality/handling_kms_encrypted_data) shows how to use Server Side KMS encrypted data with Amazon SageMaker training. The IAM role used for S3 access needs to have permissions to encrypt and decrypt data with the KMS key.
|
||||
- [Using Parquet Data](advanced_functionality/parquet_to_recordio_protobuf) shows how to bring [Parquet](https://parquet.apache.org/) data sitting in S3 into an Amazon SageMaker Notebook and convert it into the recordIO-protobuf format that many SageMaker algorithms consume.
|
||||
- [Connecting to Redshift](advanced_functionality/working_with_redshift_data) demonstrates how to copy data from Redshift to S3 and vice-versa without leaving Amazon SageMaker Notebooks.
|
||||
- [Bring Your Own XGBoost Model](advanced_functionality/xgboost_bring_your_own_model) shows how to use Amazon SageMaker Algorithms containers to bring a pre-trained model to a realtime hosted endpoint without ever needing to think about REST APIs.
|
||||
- [Bring Your Own k-means Model](advanced_functionality/kmeans_bring_your_own_model) shows how to take a model that's been fit elsewhere and use Amazon SageMaker Algorithms containers to host it.
|
||||
- [Bring Your Own R Algorithm](advanced_functionality/r_bring_your_own) shows how to bring your own algorithm container to Amazon SageMaker using the R language.
|
||||
- [Installing the R Kernel](advanced_functionality/install_r_kernel) shows how to install the R kernel into an Amazon SageMaker Notebook Instance.
|
||||
- [Bring Your Own scikit Algorithm](advanced_functionality/scikit_bring_your_own) provides a detailed walkthrough on how to package a scikit learn algorithm for training and production-ready hosting.
|
||||
- [Bring Your Own MXNet Model](advanced_functionality/mxnet_mnist_byom) shows how to bring a model trained anywhere using MXNet into Amazon SageMaker.
|
||||
- [Bring Your Own TensorFlow Model](advanced_functionality/tensorflow_iris_byom) shows how to bring a model trained anywhere using TensorFlow into Amazon SageMaker.
|
||||
- [Bring Your Own Model train and deploy BERTopic](advanced_functionality/pytorch_extend_container_train_deploy_bertopic) shows how to bring a model through an external library, how to train it and deploy it into Amazon SageMaker by extending the pytorch base containers.
|
||||
- [Experiment Management Capabilities with Search](advanced_functionality/search) shows how to organize Training Jobs into projects, and track relationships between Models, Endpoints, and Training Jobs.
|
||||
- [Host Multiple Models with Your Own Algorithm](advanced_functionality/multi_model_bring_your_own) shows how to deploy multiple models to a realtime hosted endpoint with your own custom algorithm.
|
||||
- [Host Multiple Models with XGBoost](advanced_functionality/multi_model_xgboost_home_value) shows how to deploy multiple models to a realtime hosted endpoint using a multi-model enabled XGBoost container.
|
||||
- [Host Multiple Models with SKLearn](advanced_functionality/multi_model_sklearn_home_value) shows how to deploy multiple models to a realtime hosted endpoint using a multi-model enabled SKLearn container.
|
||||
- [Host Multimodal HuggingFace Model](advanced_functionality/huggingface_deploy_instructpix2pix) shows how to host an instruction based image editing model from HuggingFace as a SageMaker endpoint using single core or multi-core GPU based instances. Inference Recommender is used to run load tests and compare the performance of instances.
|
||||
- [SageMaker Training and Inference with Script Mode](sagemaker-script-mode) shows how to use custom training and inference scripts, similar to those you would use outside of SageMaker, with SageMaker's prebuilt containers for various frameworks like Scikit-learn, PyTorch, and XGBoost.
|
||||
- [Host Models with NVidia Triton Server](sagemaker-triton) shows how to deploy models to a realtime hosted endpoint using [Triton](https://developer.nvidia.com/nvidia-triton-inference-server) as the model inference server.
|
||||
- [Heterogenous Clusters Training in TensorFlow or PyTorch](training/heterogeneous-clusters/README.md) shows how to train using TensorFlow tf.data.service (distributed data pipeline) or Pytorch (with gRPC) on top of Amazon SageMaker Heterogenous clusters to overcome CPU bottlenecks by including different instance types (GPU/CPU) in the same training job.
|
||||
- [Heterogenous Clusters Training in TensorFlow or PyTorch ](training/heterogeneous-clusters/README.md) shows how to train using TensorFlow tf.data.service (distributed data pipeline) or Pytorch (with gRPC) on top of Amazon SageMaker Heterogenous clusters to overcome CPU bottlenecks by including different instance types (GPU/CPU) in the same training job.
|
||||
|
||||
### Amazon SageMaker Neo Compilation Jobs
|
||||
|
||||
These examples provide an introduction to how to use Neo to compile and optimize deep learning models.
|
||||
|
||||
- [GluonCV SSD Mobilenet](sagemaker_neo_compilation_jobs/gluoncv_ssd_mobilenet) shows how to train GluonCV SSD MobileNet and use Amazon SageMaker Neo to compile and optimize the trained model.
|
||||
- [Image Classification](sagemaker_neo_compilation_jobs/imageclassification_caltech) Adapts from [image classification](introduction_to_amazon_algorithms/imageclassification_caltech) including Neo API and comparison against the uncompiled baseline.
|
||||
- [MNIST with MXNet](sagemaker_neo_compilation_jobs/mxnet_mnist) Adapts from [MXNet MNIST](sagemaker-python-sdk/mxnet_mnist) including Neo API and comparison against the uncompiled baseline.
|
||||
- [Deploying pre-trained PyTorch vision models](sagemaker_neo_compilation_jobs/pytorch_torchvision) shows how to use Amazon SageMaker Neo to compile and optimize pre-trained PyTorch models from TorchVision.
|
||||
- [Distributed TensorFlow](sagemaker_neo_compilation_jobs/tensorflow_distributed_mnist) includes Neo API and comparison against the uncompiled baseline.
|
||||
- [Predicting Customer Churn](sagemaker_neo_compilation_jobs/xgboost_customer_churn) Adapts from [XGBoost customer churn](introduction_to_applying_machine_learning/xgboost_customer_churn) including Neo API and comparison against the uncompiled baseline.
|
||||
|
||||
### Amazon SageMaker Processing
|
||||
|
||||
These examples show you how to use SageMaker Processing jobs to run data processing workloads.
|
||||
|
||||
- [Scikit-Learn Data Processing and Model Evaluation](sagemaker_processing/scikit_learn_data_processing_and_model_evaluation) shows how to use SageMaker Processing and the Scikit-Learn container to run data preprocessing and model evaluation workloads.
|
||||
- [Feature transformation with Amazon SageMaker Processing and SparkML](sagemaker_processing/feature_transformation_with_sagemaker_processing) shows how to use SageMaker Processing to run data processing workloads using SparkML prior to training.
|
||||
- [Feature transformation with Amazon SageMaker Processing and Dask](sagemaker_processing/feature_transformation_with_sagemaker_processing_dask) shows how to use SageMaker Processing to transform data using Dask distributed clusters
|
||||
- [Distributed Data Processing using Apache Spark and SageMaker Processing](sagemaker_processing/spark_distributed_data_processing) shows how to use the built-in Spark container on SageMaker Processing using the SageMaker Python SDK.
|
||||
|
||||
@@ -220,10 +284,17 @@ These examples show you how to use [SageMaker Pipelines](https://aws.amazon.com/
|
||||
|
||||
These examples show you how to train and host in pre-built deep learning framework containers using the SageMaker Python SDK.
|
||||
|
||||
- [Chainer CIFAR-10](sagemaker-python-sdk/chainer_cifar10) trains a VGG image classification network on CIFAR-10 using Chainer (both single machine and multi-machine versions are included)
|
||||
- [Chainer MNIST](sagemaker-python-sdk/chainer_mnist) trains a basic neural network on MNIST using Chainer (shows how to use local mode)
|
||||
- [Chainer sentiment analysis](sagemaker-python-sdk/chainer_sentiment_analysis) trains a LSTM network with embeddings to predict text sentiment using Chainer
|
||||
- [IRIS with Scikit-learn](sagemaker-python-sdk/scikit_learn_iris) trains a Scikit-learn classifier on IRIS data
|
||||
- [Model Registry and Batch Transform with Scikit-learn](sagemaker-python-sdk/scikit_learn_model_registry_batch_transform) trains a Scikit-learn Random Forest model, registers it in Model Registry, and runs a Batch Transform Job.
|
||||
- [MNIST with MXNet Gluon](sagemaker-python-sdk/mxnet_gluon_mnist) trains a basic neural network on the MNIST handwritten digit dataset using MXNet Gluon
|
||||
- [MNIST with MXNet](sagemaker-python-sdk/mxnet_mnist) trains a basic neural network on the MNIST handwritten digit data using MXNet's symbolic syntax
|
||||
- [Sentiment Analysis with MXNet Gluon](sagemaker-python-sdk/mxnet_gluon_sentiment) trains a text classifier using embeddings with MXNet Gluon
|
||||
- [TensorFlow training and serving](sagemaker-python-sdk/tensorflow_script_mode_training_and_serving) trains a basic neural network on MNIST
|
||||
- [TensorFlow with Horovod](sagemaker-python-sdk/tensorflow_script_mode_horovod) trains on MNIST using Horovod for distributed training
|
||||
- [TensorFlow using shell commands](sagemaker-python-sdk/tensorflow_script_mode_using_shell_commands) shows how to use a shell script for the container's entry point
|
||||
|
||||
#### Pre-Built Machine Learning Framework Containers
|
||||
|
||||
@@ -242,9 +313,9 @@ These examples show how to use Amazon SageMaker for model training, hosting, and
|
||||
### Using Amazon SageMaker with Amazon Keyspaces (for Apache Cassandra)
|
||||
|
||||
These examples show how to use Amazon SageMaker to read data from [Amazon Keyspaces](https://docs.aws.amazon.com/keyspaces/).
|
||||
|
||||
- [Train Machine Learning Models using Amazon Keyspaces as a Data Source](ingest_data/sagemaker-keyspaces)
|
||||
|
||||
|
||||
### AWS Marketplace
|
||||
|
||||
#### Create algorithms/model packages for listing in AWS Marketplace for machine learning.
|
||||
@@ -260,7 +331,7 @@ Once you have created an algorithm or a model package to be listed in the AWS Ma
|
||||
- [Curate your AWS Marketplace model package listing and sample notebook](aws_marketplace/curating_aws_marketplace_listing_and_sample_notebook/ModelPackage) provides instructions on how to craft a sample notebook to be associated with your listing and how to curate a good AWS Marketplace listing that makes it easy for AWS customers to consume your model package.
|
||||
- [Curate your AWS Marketplace algorithm listing and sample notebook](aws_marketplace/curating_aws_marketplace_listing_and_sample_notebook/Algorithm) provides instructions on how to craft a sample notebook to be associated with your listing and how to curate a good AWS Marketplace listing that makes it easy for your customers to consume your algorithm.
|
||||
|
||||
#### Use algorithms, data, and model packages from AWS Marketplace
|
||||
#### Use algorithms, data, and model packages from AWS Marketplace.
|
||||
|
||||
These examples show you how to use model-packages and algorithms from AWS Marketplace and dataset products from AWS Data Exchange, for machine learning.
|
||||
|
||||
@@ -268,6 +339,7 @@ These examples show you how to use model-packages and algorithms from AWS Market
|
||||
- [Using Algorithm From AWS Marketplace](aws_marketplace/using_algorithms/amazon_demo_product) provides a detailed walkthrough on how to use Algorithm with the enhanced SageMaker Train/Transform/Hosting/Tuning APIs by choosing a canonical product listed on AWS Marketplace.
|
||||
- [Using AutoML algorithm](aws_marketplace/using_algorithms/automl) provides a detailed walkthrough on how to use AutoML algorithm from AWS Marketplace.
|
||||
- [Using Model Packages](aws_marketplace/using_model_packages)
|
||||
- [Using Model Packages From AWS Marketplace](aws_marketplace/using_model_packages/generic_sample_notebook) is a generic notebook which provides sample code snippets you can modify and use for performing inference on Model Packages from AWS Marketplace, using Amazon SageMaker.
|
||||
- [Using Amazon Demo product From AWS Marketplace](aws_marketplace/using_model_packages/amazon_demo_product) provides a detailed walkthrough on how to use Model Package entities with the enhanced SageMaker Transform/Hosting APIs by choosing a canonical product listed on AWS Marketplace.
|
||||
- [Using models for extracting vehicle metadata](aws_marketplace/using_model_packages/auto_insurance) provides a detailed walkthrough on how to use pre-trained models from AWS Marketplace for extracting metadata for a sample use-case of auto-insurance claim processing.
|
||||
- [Using models for identifying non-compliance at a workplace](aws_marketplace/using_model_packages/improving_industrial_workplace_safety) provides a detailed walkthrough on how to use pre-trained models from AWS Marketplace for extracting metadata for a sample use-case of generating summary reports for identifying non-compliance at a construction/industrial workplace.
|
||||
@@ -275,21 +347,17 @@ These examples show you how to use model-packages and algorithms from AWS Market
|
||||
- [Amazon Augmented AI with AWS Marketplace ML models](aws_marketplace/using_model_packages/amazon_augmented_ai_with_aws_marketplace_ml_models) will show you how to use AWS Marketplace pre-trained ML models with Amazon Augmented AI to implement human-in-loop workflow reviews with your ML model predictions.
|
||||
- [Monitoring data quality in third-party models from AWS Marketplace](aws_marketplace/using_model_packages/data_quality_monitoring) will show you how to perform Data Quality monitoring on a pre-trained third-party model from AWS Marketplace.
|
||||
- [Evaluating ML models from AWS Marketplace for person counting use case](aws_marketplace/using_model_packages/evaluating_aws_marketplace_models_for_person_counting_use_case) will show you how to use two AWS Marketplace GluonCV pre-trained ML models for person counting use case and evaluate each model for performance in different types of crowd images.
|
||||
- [Preprocessing audio data using a pre-trained machine learning model](aws_marketplace/using_model_packages/preprocessing-audio-data-using-a-machine-learning-model) demonstrates the usage of a pre-trained audio track separation model to create synthetic features and improve an acoustic classification model.
|
||||
- [Preprocessing audio data using a pre-trained machine learning model](using_model_packages/preprocessing-audio-data-using-a-machine-learning-model) demonstrates the usage of a pre-trained audio track separation model to create synthetic features and improve an acoustic classification model.
|
||||
- [Using Dataset Products](aws_marketplace/using_data)
|
||||
- [Using Dataset Product from AWS Data Exchange with ML model from AWS Marketplace](aws_marketplace/using_data/using_data_with_ml_model) is a sample notebook which shows how a dataset from AWS Data Exchange can be used with an ML Model Package from AWS Marketplace.
|
||||
- [Using Shutterstock Image Datasets to train Image Classification Models](aws_marketplace/using_data/image_classification_with_shutterstock_image_datasets) provides a detailed walkthrough on how to use the [Free Sample: Images & Metadata of “Whole Foods” Shoppers](https://aws.amazon.com/marketplace/pp/prodview-y6xuddt42fmbu?qid=1623195111604&sr=0-1&ref_=srh_res_product_title#offers) from Shutterstock's Image Datasets to train a multi-label image classification model using Shutterstock's pre-labeled image assets. You can learn more about this implementation [from this blog post](https://aws.amazon.com/blogs/awsmarketplace/using-shutterstocks-image-datasets-to-train-your-computer-vision-models/).
|
||||
|
||||
### Using Amazon SageMaker for Generative AI use cases
|
||||
### Using Amazon SageMaker for Generative AI use cases.
|
||||
|
||||
These examples show you how to use AWS services for Generative AI use cases.
|
||||
|
||||
- Text-to-image
|
||||
- [Fine-tune Stable Diffusion XL model with Kohya](use-cases/text-to-image-fine-tuning) Provides an automated solution to create the necessary components to fine-tune a custom Stable Diffusion XL model.
|
||||
|
||||
### Archived
|
||||
|
||||
This folder houses legacy, low-viewed, and duplicate notebooks, with a 6-month grace period before deletion. If you believe a notebook has been moved into this folder in error, please submit a PR with justification.
|
||||
|
||||
## :balance_scale: License
|
||||
|
||||
@@ -301,4 +369,3 @@ For more details, please take a look at the [LICENSE](https://github.com/aws/ama
|
||||
Although we're extremely excited to receive contributions from the community, we're still working on the best mechanism to take in examples from external sources. Please bear with us in the short-term if pull requests take longer than expected or are closed.
|
||||
Please read our [contributing guidelines](https://github.com/aws/amazon-sagemaker-examples/blob/master/CONTRIBUTING.md)
|
||||
if you'd like to open an issue or submit a pull request.
|
||||
|
||||
|
||||
@@ -0,0 +1,458 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Train an MNIST model with PyTorch\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"MNIST is a widely used dataset for handwritten digit classification. It consists of 70,000 labeled 28x28 pixel grayscale images of hand-written digits. The dataset is split into 60,000 training images and 10,000 test images. There are 10 classes (one for each of the 10 digits). This tutorial shows how to train and test an MNIST model on SageMaker using PyTorch. \n",
|
||||
"\n",
|
||||
"## Runtime\n",
|
||||
"\n",
|
||||
"This notebook takes approximately 5 minutes to run.\n",
|
||||
"\n",
|
||||
"## Contents\n",
|
||||
"\n",
|
||||
"1. [PyTorch Estimator](#PyTorch-Estimator)\n",
|
||||
"1. [Implement the entry point for training](#Implement-the-entry-point-for-training)\n",
|
||||
"1. [Set hyperparameters](#Set-hyperparameters)\n",
|
||||
"1. [Set up channels for the training and testing data](#Set-up-channels-for-the-training-and-testing-data)\n",
|
||||
"1. [Run the training script on SageMaker](#Run-the-training-script-on-SageMaker)\n",
|
||||
"1. [Inspect and store model data](#Inspect-and-store-model-data)\n",
|
||||
"1. [Test and debug the entry point before executing the training container](#Test-and-debug-the-entry-point-before-executing-the-training-container)\n",
|
||||
"1. [Conclusion](#Conclusion)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import json\n",
|
||||
"\n",
|
||||
"import sagemaker\n",
|
||||
"from sagemaker.pytorch import PyTorch\n",
|
||||
"from sagemaker import get_execution_role\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"sess = sagemaker.Session()\n",
|
||||
"region = sess.boto_region_name\n",
|
||||
"\n",
|
||||
"role = get_execution_role()\n",
|
||||
"\n",
|
||||
"output_path = \"s3://\" + sess.default_bucket() + \"/DEMO-mnist\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## PyTorch Estimator\n",
|
||||
"\n",
|
||||
"The `PyTorch` class allows you to run your training script on SageMaker\n",
|
||||
"infrastracture in a containerized environment. In this notebook, we\n",
|
||||
"refer to this container as *training container*. \n",
|
||||
"\n",
|
||||
"You need to configure\n",
|
||||
"it with the following parameters to set up the environment:\n",
|
||||
"\n",
|
||||
"- `entry_point`: A user-defined Python file used by the training container as the \n",
|
||||
"instructions for training. We further discuss this file in the next subsection.\n",
|
||||
"\n",
|
||||
"- `role`: An IAM role to make AWS service requests\n",
|
||||
"\n",
|
||||
"- `instance_type`: The type of SageMaker instance to run your training script. \n",
|
||||
"Set it to `local` if you want to run the training job on \n",
|
||||
"the SageMaker instance you are using to run this notebook\n",
|
||||
"\n",
|
||||
"- `instance_count`: The number of instances to run your training job on. \n",
|
||||
"Multiple instances are needed for distributed training.\n",
|
||||
"\n",
|
||||
"- `output_path`: \n",
|
||||
"S3 bucket URI to save training output (model artifacts and output files)\n",
|
||||
"\n",
|
||||
"- `framework_version`: The version of PyTorch to use\n",
|
||||
"\n",
|
||||
"- `py_version`: The Python version to use\n",
|
||||
"\n",
|
||||
"For more information, see the [EstimatorBase API reference](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html#sagemaker.estimator.EstimatorBase)\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Implement the entry point for training\n",
|
||||
"\n",
|
||||
"The entry point for training is a Python script that provides all \n",
|
||||
"the code for training a PyTorch model. It is used by the SageMaker \n",
|
||||
"PyTorch Estimator (`PyTorch` class above) as the entry point for running the training job.\n",
|
||||
"\n",
|
||||
"Under the hood, SageMaker PyTorch Estimator creates a docker image\n",
|
||||
"with runtime environemnts \n",
|
||||
"specified by the parameters you provide to initiate the\n",
|
||||
"estimator class, and it injects the training script into the \n",
|
||||
"docker image as the entry point to run the container.\n",
|
||||
"\n",
|
||||
"In the rest of the notebook, we use *training image* to refer to the \n",
|
||||
"docker image specified by the PyTorch Estimator and *training container*\n",
|
||||
"to refer to the container that runs the training image. \n",
|
||||
"\n",
|
||||
"This means your training script is very similar to a training script\n",
|
||||
"you might run outside Amazon SageMaker, but it can access the useful environment \n",
|
||||
"variables provided by the training image. See [the complete list of environment variables](https://github.com/aws/sagemaker-training-toolkit/blob/master/ENVIRONMENT_VARIABLES.md) for a complete \n",
|
||||
"description of all environment variables your training script\n",
|
||||
"can access. \n",
|
||||
"\n",
|
||||
"In this example, we use the training script `code/train.py`\n",
|
||||
"as the entry point for our PyTorch Estimator.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pygmentize 'code/train.py'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Set hyperparameters\n",
|
||||
"\n",
|
||||
"In addition, the PyTorch estimator allows you to parse command line arguments\n",
|
||||
"to your training script via `hyperparameters`.\n",
|
||||
"\n",
|
||||
"Note: local mode is not supported in SageMaker Studio. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set local_mode to True to run the training script on the machine that runs this notebook\n",
|
||||
"\n",
|
||||
"local_mode = False\n",
|
||||
"\n",
|
||||
"if local_mode:\n",
|
||||
" instance_type = \"local\"\n",
|
||||
"else:\n",
|
||||
" instance_type = \"ml.c4.xlarge\"\n",
|
||||
"\n",
|
||||
"est = PyTorch(\n",
|
||||
" entry_point=\"train.py\",\n",
|
||||
" source_dir=\"code\", # directory of your training script\n",
|
||||
" role=role,\n",
|
||||
" framework_version=\"1.5.0\",\n",
|
||||
" py_version=\"py3\",\n",
|
||||
" instance_type=instance_type,\n",
|
||||
" instance_count=1,\n",
|
||||
" volume_size=250,\n",
|
||||
" output_path=output_path,\n",
|
||||
" hyperparameters={\"batch-size\": 128, \"epochs\": 1, \"learning-rate\": 1e-3, \"log-interval\": 100},\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The training container executes your training script like:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"python train.py --batch-size 100 --epochs 1 --learning-rate 1e-3 --log-interval 100\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Set up channels for the training and testing data\n",
|
||||
"\n",
|
||||
"Tell the `PyTorch` estimator where to find the training and \n",
|
||||
"testing data. It can be a path to an S3 bucket, or a path\n",
|
||||
"in your local file system if you use local mode. In this example,\n",
|
||||
"we download the MNIST data from a public S3 bucket and upload it \n",
|
||||
"to your default bucket. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import logging\n",
|
||||
"import boto3\n",
|
||||
"from botocore.exceptions import ClientError\n",
|
||||
"\n",
|
||||
"# Download training and testing data from a public S3 bucket\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def download_from_s3(data_dir=\"./data\", train=True):\n",
|
||||
" \"\"\"Download MNIST dataset and convert it to numpy array\n",
|
||||
"\n",
|
||||
" Args:\n",
|
||||
" data_dir (str): directory to save the data\n",
|
||||
" train (bool): download training set\n",
|
||||
"\n",
|
||||
" Returns:\n",
|
||||
" None\n",
|
||||
" \"\"\"\n",
|
||||
"\n",
|
||||
" if not os.path.exists(data_dir):\n",
|
||||
" os.makedirs(data_dir)\n",
|
||||
"\n",
|
||||
" if train:\n",
|
||||
" images_file = \"train-images-idx3-ubyte.gz\"\n",
|
||||
" labels_file = \"train-labels-idx1-ubyte.gz\"\n",
|
||||
" else:\n",
|
||||
" images_file = \"t10k-images-idx3-ubyte.gz\"\n",
|
||||
" labels_file = \"t10k-labels-idx1-ubyte.gz\"\n",
|
||||
"\n",
|
||||
" # download objects\n",
|
||||
" s3 = boto3.client(\"s3\")\n",
|
||||
" bucket = f\"sagemaker-example-files-prod-{region}\"\n",
|
||||
" for obj in [images_file, labels_file]:\n",
|
||||
" key = os.path.join(\"datasets/image/MNIST\", obj)\n",
|
||||
" dest = os.path.join(data_dir, obj)\n",
|
||||
" if not os.path.exists(dest):\n",
|
||||
" s3.download_file(bucket, key, dest)\n",
|
||||
" return\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"download_from_s3(\"./data\", True)\n",
|
||||
"download_from_s3(\"./data\", False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Upload to the default bucket\n",
|
||||
"\n",
|
||||
"prefix = \"DEMO-mnist\"\n",
|
||||
"bucket = sess.default_bucket()\n",
|
||||
"loc = sess.upload_data(path=\"./data\", bucket=bucket, key_prefix=prefix)\n",
|
||||
"\n",
|
||||
"channels = {\"training\": loc, \"testing\": loc}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The keys of the `channels` dictionary are passed to the training image,\n",
|
||||
"and it creates the environment variable `SM_CHANNEL_<key name>`. \n",
|
||||
"\n",
|
||||
"In this example, `SM_CHANNEL_TRAINING` and `SM_CHANNEL_TESTING` are created in the training image (see \n",
|
||||
"how `code/train.py` accesses these variables). For more information,\n",
|
||||
"see: [SM_CHANNEL_{channel_name}](https://github.com/aws/sagemaker-training-toolkit/blob/master/ENVIRONMENT_VARIABLES.md#sm_channel_channel_name).\n",
|
||||
"\n",
|
||||
"If you want, you can create a channel for validation:\n",
|
||||
"```\n",
|
||||
"channels = {\n",
|
||||
" 'training': train_data_loc,\n",
|
||||
" 'validation': val_data_loc,\n",
|
||||
" 'test': test_data_loc\n",
|
||||
"}\n",
|
||||
"```\n",
|
||||
"You can then access this channel within your training script via\n",
|
||||
"`SM_CHANNEL_VALIDATION`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Run the training script on SageMaker\n",
|
||||
"Now, the training container has everything to execute your training\n",
|
||||
"script. Start the container by calling the `fit()` method."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"est.fit(inputs=channels)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Inspect and store model data\n",
|
||||
"\n",
|
||||
"Now, the training is finished, and the model artifact has been saved in \n",
|
||||
"the `output_path`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pt_mnist_model_data = est.model_data\n",
|
||||
"print(\"Model artifact saved at:\\n\", pt_mnist_model_data)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We store the variable `pt_mnist_model_data` in the current notebook kernel."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%store pt_mnist_model_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Test and debug the entry point before executing the training container\n",
|
||||
"\n",
|
||||
"The entry point `code/train.py` can be executed in the training container. \n",
|
||||
"When you develop your own training script, it is a good practice to simulate the container environment \n",
|
||||
"in the local shell and test it before sending it to SageMaker, because debugging in a containerized environment\n",
|
||||
"is rather cumbersome. The following script shows how you can test your training script:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pygmentize code/test_train.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Conclusion\n",
|
||||
"\n",
|
||||
"In this notebook, we trained a PyTorch model on the MNIST dataset by fitting a SageMaker estimator. For next steps on how to deploy the trained model and perform inference, see [Deploy a Trained PyTorch Model](https://sagemaker-examples.readthedocs.io/en/latest/frameworks/pytorch/get_started_mnist_deploy.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Notebook CI Test Results\n",
|
||||
"\n",
|
||||
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (PyTorch 1.13 Python 3.9 CPU Optimized)",
|
||||
"language": "python",
|
||||
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/pytorch-1.13-cpu-py39"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.16"
|
||||
},
|
||||
"notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,460 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Train an MNIST model with TensorFlow\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"MNIST is a widely-used dataset for handwritten digit classification. It consists of 70,000 labeled 28x28 pixel grayscale images of hand-written digits. The dataset is split into 60,000 training images and 10,000 test images. There are 10 classes (one for each of the 10 digits). This tutorial will show how to train a TensorFlow V2 model on MNIST model on SageMaker.\n",
|
||||
"\n",
|
||||
"## Runtime\n",
|
||||
"\n",
|
||||
"This notebook takes approximately 5 minutes to run.\n",
|
||||
"\n",
|
||||
"## Contents\n",
|
||||
"\n",
|
||||
"1. [TensorFlow Estimator](#TensorFlow-Estimator)\n",
|
||||
"1. [Implement the training entry point](#Implement-the-training-entry-point)\n",
|
||||
"1. [Set hyperparameters](#Set-hyperparameters)\n",
|
||||
"1. [Set up channels for training and testing data](#Set-up-channels-for-training-and-testing-data)\n",
|
||||
"1. [Run the training script on SageMaker](#Run-the-training-script-on-SageMaker)\n",
|
||||
"1. [Inspect and store model data](#Inspect-and-store-model-data)\n",
|
||||
"1. [Test and debug the entry point before running the training container](#Test-and-debug-the-entry-point-before-running-the-training-container)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import json\n",
|
||||
"\n",
|
||||
"import sagemaker\n",
|
||||
"from sagemaker.tensorflow import TensorFlow\n",
|
||||
"from sagemaker import get_execution_role\n",
|
||||
"\n",
|
||||
"sess = sagemaker.Session()\n",
|
||||
"\n",
|
||||
"role = get_execution_role()\n",
|
||||
"\n",
|
||||
"output_path = \"s3://\" + sess.default_bucket() + \"/DEMO-tensorflow/mnist\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## TensorFlow Estimator\n",
|
||||
"\n",
|
||||
"The `TensorFlow` class allows you to run your training script on SageMaker\n",
|
||||
"infrastracture in a containerized environment. In this notebook, we\n",
|
||||
"refer to this container as the \"training container.\" \n",
|
||||
"\n",
|
||||
"Configure it with the following parameters to set up the environment:\n",
|
||||
"\n",
|
||||
"- `entry_point`: A user-defined Python file used by the training container as the instructions for training. We will further discuss this file in the next subsection.\n",
|
||||
"\n",
|
||||
"- `role`: An IAM role to make AWS service requests\n",
|
||||
"\n",
|
||||
"- `instance_type`: The type of SageMaker instance to run your training script. Set it to `local` if you want to run the training job on the SageMaker instance you are using to run this notebook.\n",
|
||||
"\n",
|
||||
"- `model_dir`: S3 bucket URI where the checkpoint data and models can be exported to during training (default: None). \n",
|
||||
"To disable having model_dir passed to your training script, set `model_dir`=False\n",
|
||||
"\n",
|
||||
"- `instance_count`: The number of instances to run your training job on. Multiple instances are needed for distributed training.\n",
|
||||
"\n",
|
||||
"- `output_path`: the S3 bucket URI to save training output (model artifacts and output files).\n",
|
||||
"\n",
|
||||
"- `framework_version`: The TensorFlow version to use.\n",
|
||||
"\n",
|
||||
"- `py_version`: The Python version to use.\n",
|
||||
"\n",
|
||||
"For more information, see the [EstimatorBase API reference](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html#sagemaker.estimator.EstimatorBase).\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Implement the training entry point\n",
|
||||
"\n",
|
||||
"The entry point for training is a Python script that provides all \n",
|
||||
"the code for training a TensorFlow model. It is used by the SageMaker \n",
|
||||
"TensorFlow Estimator (`TensorFlow` class above) as the entry point for running the training job.\n",
|
||||
"\n",
|
||||
"Under the hood, SageMaker TensorFlow Estimator downloads a docker image\n",
|
||||
"with runtime environments \n",
|
||||
"specified by the parameters to initiate the\n",
|
||||
"estimator class and it injects the training script into the \n",
|
||||
"docker image as the entry point to run the container.\n",
|
||||
"\n",
|
||||
"In the rest of the notebook, we use *training image* to refer to the \n",
|
||||
"docker image specified by the TensorFlow Estimator and *training container*\n",
|
||||
"to refer to the container that runs the training image. \n",
|
||||
"\n",
|
||||
"This means your training script is very similar to a training script\n",
|
||||
"you might run outside Amazon SageMaker, but it can access the useful environment \n",
|
||||
"variables provided by the training image. See [the complete list of environment variables](https://github.com/aws/sagemaker-training-toolkit/blob/master/ENVIRONMENT_VARIABLES.md) for a complete \n",
|
||||
"description of all environment variables your training script\n",
|
||||
"can access. \n",
|
||||
"\n",
|
||||
"In this example, we use the training script `code/train.py`\n",
|
||||
"as the entry point for our TensorFlow Estimator. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pygmentize 'code/train.py'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Set hyperparameters\n",
|
||||
"\n",
|
||||
"In addition, the TensorFlow estimator allows you to parse command line arguments\n",
|
||||
"to your training script via `hyperparameters`.\n",
|
||||
"\n",
|
||||
"<span style=\"color:red\"> Note: local mode is not supported in SageMaker Studio. </span>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set local_mode to be True if you want to run the training script on the machine that runs this notebook\n",
|
||||
"\n",
|
||||
"local_mode = False\n",
|
||||
"\n",
|
||||
"if local_mode:\n",
|
||||
" instance_type = \"local\"\n",
|
||||
"else:\n",
|
||||
" instance_type = \"ml.c4.xlarge\"\n",
|
||||
"\n",
|
||||
"est = TensorFlow(\n",
|
||||
" entry_point=\"train.py\",\n",
|
||||
" source_dir=\"code\", # directory of your training script\n",
|
||||
" role=role,\n",
|
||||
" framework_version=\"2.3.1\",\n",
|
||||
" model_dir=False, # don't pass --model_dir to your training script\n",
|
||||
" py_version=\"py37\",\n",
|
||||
" instance_type=instance_type,\n",
|
||||
" instance_count=1,\n",
|
||||
" volume_size=250,\n",
|
||||
" output_path=output_path,\n",
|
||||
" hyperparameters={\n",
|
||||
" \"batch-size\": 512,\n",
|
||||
" \"epochs\": 1,\n",
|
||||
" \"learning-rate\": 1e-3,\n",
|
||||
" \"beta_1\": 0.9,\n",
|
||||
" \"beta_2\": 0.999,\n",
|
||||
" },\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The training container runs your training script like:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"python train.py --batch-size 32 --epochs 1 --learning-rate 0.001 --beta_1 0.9 --beta_2 0.999\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Set up channels for training and testing data\n",
|
||||
"\n",
|
||||
"Tell `TensorFlow` estimator where to find the training and \n",
|
||||
"testing data. It can be a path to an S3 bucket, or a path\n",
|
||||
"in your local file system if you use local mode. In this example,\n",
|
||||
"we download the MNIST data from a public S3 bucket and upload it \n",
|
||||
"to your default bucket. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import logging\n",
|
||||
"import boto3\n",
|
||||
"from botocore.exceptions import ClientError\n",
|
||||
"\n",
|
||||
"# Download training and testing data from a public S3 bucket\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def download_from_s3(data_dir=\"./data\", train=True):\n",
|
||||
" \"\"\"Download MNIST dataset and convert it to numpy array\n",
|
||||
"\n",
|
||||
" Args:\n",
|
||||
" data_dir (str): directory to save the data\n",
|
||||
" train (bool): download training set\n",
|
||||
"\n",
|
||||
" Returns:\n",
|
||||
" None\n",
|
||||
" \"\"\"\n",
|
||||
"\n",
|
||||
" if not os.path.exists(data_dir):\n",
|
||||
" os.makedirs(data_dir)\n",
|
||||
"\n",
|
||||
" if train:\n",
|
||||
" images_file = \"train-images-idx3-ubyte.gz\"\n",
|
||||
" labels_file = \"train-labels-idx1-ubyte.gz\"\n",
|
||||
" else:\n",
|
||||
" images_file = \"t10k-images-idx3-ubyte.gz\"\n",
|
||||
" labels_file = \"t10k-labels-idx1-ubyte.gz\"\n",
|
||||
"\n",
|
||||
" # download objects\n",
|
||||
" s3 = boto3.client(\"s3\")\n",
|
||||
" bucket = f\"sagemaker-example-files-prod-{boto3.session.Session().region_name}\"\n",
|
||||
" for obj in [images_file, labels_file]:\n",
|
||||
" key = os.path.join(\"datasets/image/MNIST\", obj)\n",
|
||||
" dest = os.path.join(data_dir, obj)\n",
|
||||
" if not os.path.exists(dest):\n",
|
||||
" s3.download_file(bucket, key, dest)\n",
|
||||
" return\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"download_from_s3(\"./data\", True)\n",
|
||||
"download_from_s3(\"./data\", False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Upload to the default bucket\n",
|
||||
"\n",
|
||||
"prefix = \"DEMO-mnist\"\n",
|
||||
"bucket = sess.default_bucket()\n",
|
||||
"loc = sess.upload_data(path=\"./data\", bucket=bucket, key_prefix=prefix)\n",
|
||||
"\n",
|
||||
"channels = {\"training\": loc, \"testing\": loc}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The keys of the `channels` dictionary are passed to the training image,\n",
|
||||
"and it creates the environment variable `SM_CHANNEL_<key name>`. \n",
|
||||
"\n",
|
||||
"In this example, `SM_CHANNEL_TRAINING` and `SM_CHANNEL_TESTING` are created in the training image (see \n",
|
||||
"how `code/train.py` accesses these variables). For more information,\n",
|
||||
"see: [SM_CHANNEL_{channel_name}](https://github.com/aws/sagemaker-training-toolkit/blob/master/ENVIRONMENT_VARIABLES.md#sm_channel_channel_name).\n",
|
||||
"\n",
|
||||
"If you want, you can create a channel for validation:\n",
|
||||
"```\n",
|
||||
"channels = {\n",
|
||||
" 'training': train_data_loc,\n",
|
||||
" 'validation': val_data_loc,\n",
|
||||
" 'test': test_data_loc\n",
|
||||
"}\n",
|
||||
"```\n",
|
||||
"You can then access this channel within your training script via\n",
|
||||
"`SM_CHANNEL_VALIDATION`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Run the training script on SageMaker\n",
|
||||
"Now, the training container has everything to run your training\n",
|
||||
"script. Start the container by calling the `fit()` method."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"est.fit(inputs=channels)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Inspect and store model data\n",
|
||||
"\n",
|
||||
"Now, the training is finished, and the model artifact has been saved in \n",
|
||||
"the `output_path`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tf_mnist_model_data = est.model_data\n",
|
||||
"print(\"Model artifact saved at:\\n\", tf_mnist_model_data)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We store the variable `tf_mnist_model_data` in the current notebook kernel. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%store tf_mnist_model_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Test and debug the entry point before running the training container\n",
|
||||
"\n",
|
||||
"The entry point `code/train.py` provided here has been tested and it can be runs in the training container. \n",
|
||||
"When you develop your own training script, it is a good practice to simulate the container environment \n",
|
||||
"in the local shell and test it before sending it to SageMaker, because debugging in a containerized environment\n",
|
||||
"is rather cumbersome. The following script shows how you can test your training script:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pygmentize code/test_train.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Conclusion\n",
|
||||
"\n",
|
||||
"In this notebook, we trained a TensorFlow model on the MNIST dataset by fitting a SageMaker estimator. For next steps on how to deploy the trained model and perform inference, see [Deploy a Trained TensorFlow V2 Model](https://sagemaker-examples.readthedocs.io/en/latest/frameworks/tensorflow/get_started_mnist_deploy.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Notebook CI Test Results\n",
|
||||
"\n",
|
||||
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (TensorFlow 2.10.0 Python 3.9 CPU Optimized)",
|
||||
"language": "python",
|
||||
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/tensorflow-2.10.1-cpu-py39-ubuntu20.04-sagemaker-v1.2"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.16"
|
||||
},
|
||||
"notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,454 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Hyperparameter Tuning with the SageMaker TensorFlow Container\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"This tutorial focuses on how to create a convolutional neural network model to train the [MNIST dataset](http://yann.lecun.com/exdb/mnist/) using the SageMaker TensorFlow container. It leverages hyperparameter tuning to run multiple training jobs with different hyperparameter combinations, to find the one with the best model training result.\n",
|
||||
"\n",
|
||||
"## Runtime\n",
|
||||
"\n",
|
||||
"This notebook takes approximately 10 minutes to run.\n",
|
||||
"\n",
|
||||
"## Contents\n",
|
||||
"\n",
|
||||
"1. [Set Up the Environment](#Set-Up-the-Environment)\n",
|
||||
"1. [Data](#Data)\n",
|
||||
"1. [Run a TensorFlow Training Job](#Run-a-TensorFlow-Training-Job)\n",
|
||||
"1. [Set Up Channels for Training and Testing Data](#Set-Up-Channels-for-Training-and-Testing-Data)\n",
|
||||
"1. [Run a Hyperparameter Tuning Job](#Run-a-Hyperparameter-Tuning-Job)\n",
|
||||
"1. [Deploy the Best Model](#Deploy-the-Best-Model)\n",
|
||||
"1. [Evaluate](#Evaluate)\n",
|
||||
"1. [Cleanup](#Cleanup)\n",
|
||||
"\n",
|
||||
"## Set Up the Environment \n",
|
||||
"Set up a few things before starting the workflow:\n",
|
||||
"\n",
|
||||
"1. A boto3 session object to manage interactions with the Amazon SageMaker APIs. \n",
|
||||
"2. An execution role which is passed to SageMaker to access your AWS resources."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import json\n",
|
||||
"\n",
|
||||
"import sagemaker\n",
|
||||
"from sagemaker.tensorflow import TensorFlow\n",
|
||||
"from sagemaker import get_execution_role\n",
|
||||
"\n",
|
||||
"sess = sagemaker.Session()\n",
|
||||
"region = sess.boto_region_name\n",
|
||||
"role = get_execution_role()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Data\n",
|
||||
"Download the MNIST data from a public S3 bucket and save it in a temporary directory."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import logging\n",
|
||||
"import boto3\n",
|
||||
"from botocore.exceptions import ClientError\n",
|
||||
"\n",
|
||||
"public_bucket = f\"sagemaker-example-files-prod-{region}\"\n",
|
||||
"local_data_dir = \"/tmp/data\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Download training and testing data from a public S3 bucket\n",
|
||||
"def download_from_s3(data_dir=\"/tmp/data\", train=True):\n",
|
||||
" \"\"\"Download MNIST dataset and convert it to numpy array\n",
|
||||
"\n",
|
||||
" Args:\n",
|
||||
" data_dir (str): directory to save the data\n",
|
||||
" train (bool): download training set\n",
|
||||
"\n",
|
||||
" Returns:\n",
|
||||
" None\n",
|
||||
" \"\"\"\n",
|
||||
" # project root\n",
|
||||
" if not os.path.exists(data_dir):\n",
|
||||
" os.makedirs(data_dir)\n",
|
||||
"\n",
|
||||
" if train:\n",
|
||||
" images_file = \"train-images-idx3-ubyte.gz\"\n",
|
||||
" labels_file = \"train-labels-idx1-ubyte.gz\"\n",
|
||||
" else:\n",
|
||||
" images_file = \"t10k-images-idx3-ubyte.gz\"\n",
|
||||
" labels_file = \"t10k-labels-idx1-ubyte.gz\"\n",
|
||||
"\n",
|
||||
" # download objects\n",
|
||||
" s3 = boto3.client(\"s3\")\n",
|
||||
" bucket = public_bucket\n",
|
||||
" for obj in [images_file, labels_file]:\n",
|
||||
" key = os.path.join(\"datasets/image/MNIST\", obj)\n",
|
||||
" dest = os.path.join(data_dir, obj)\n",
|
||||
" if not os.path.exists(dest):\n",
|
||||
" s3.download_file(bucket, key, dest)\n",
|
||||
" return\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"download_from_s3(local_data_dir, True)\n",
|
||||
"download_from_s3(local_data_dir, False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Run a TensorFlow Training Job\n",
|
||||
"A TensorFlow training job is defined by using the `TensorFlow` estimator class. It lets you run your training script on SageMaker infrastructure in a containerized environment. For more information on how to instantiate it, see the example [Train an MNIST model with TensorFlow](https://sagemaker-examples.readthedocs.io/en/latest/frameworks/tensorflow/get_started_mnist_train.html#TensorFlow-Estimator)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"est = TensorFlow(\n",
|
||||
" entry_point=\"train.py\",\n",
|
||||
" source_dir=\"code\", # directory of your training script\n",
|
||||
" role=role,\n",
|
||||
" framework_version=\"2.3.1\",\n",
|
||||
" model_dir=\"/opt/ml/model\",\n",
|
||||
" py_version=\"py37\",\n",
|
||||
" instance_type=\"ml.m5.4xlarge\",\n",
|
||||
" instance_count=1,\n",
|
||||
" volume_size=250,\n",
|
||||
" hyperparameters={\n",
|
||||
" \"batch-size\": 512,\n",
|
||||
" \"epochs\": 4,\n",
|
||||
" },\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Set Up Channels for Training and Testing Data\n",
|
||||
"Upload the MNIST data to the default bucket of your AWS account and pass the S3 URI as the channels of training and testing data for the `TensorFlow` estimator class. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prefix = \"mnist\"\n",
|
||||
"bucket = sess.default_bucket()\n",
|
||||
"loc = sess.upload_data(path=local_data_dir, bucket=bucket, key_prefix=prefix)\n",
|
||||
"\n",
|
||||
"channels = {\"training\": loc, \"testing\": loc}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Run a Hyperparameter Tuning Job\n",
|
||||
"Now that you have set up the training job and the input data channels, you are ready to train the model with hyperparameter search.\n",
|
||||
"\n",
|
||||
"Set up the hyperparameter tuning job with the following steps:\n",
|
||||
"* Define the ranges of hyperparameters we plan to tune. In this example, we tune the learning rate.\n",
|
||||
"* Define the objective metric for the tuning job to optimize.\n",
|
||||
"* Create a hyperparameter tuner with the above setting, as well as tuning resource configurations.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"For a typical ML model, there are three kinds of hyperparamters:\n",
|
||||
"\n",
|
||||
"- Categorical parameters need to take one value from a discrete set. We define this by passing the list of possible values to `CategoricalParameter(list)`\n",
|
||||
"- Continuous parameters can take any real number value between the minimum and maximum value, defined by `ContinuousParameter(min, max)`\n",
|
||||
"- Integer parameters can take any integer value between the minimum and maximum value, defined by `IntegerParameter(min, max)`\n",
|
||||
"\n",
|
||||
"Learning rate is a continuous variable, so we define its range\n",
|
||||
"by `ContinuousParameter`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker.tuner import ContinuousParameter, HyperparameterTuner\n",
|
||||
"\n",
|
||||
"hyperparamter_range = {\"learning-rate\": ContinuousParameter(1e-4, 1e-3)}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Next we specify the objective metric that we'd like to tune and its definition, which includes the regular expression (regex) needed to extract that metric from the CloudWatch logs of the training job. In this particular case, our script emits average loss value and we use it as the objective metric. We set the `objective_type` to `Minimize`, so that hyperparameter tuning seeks to minimize the objective metric when searching for the best hyperparameter value."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"objective_metric_name = \"average test loss\"\n",
|
||||
"objective_type = \"Minimize\"\n",
|
||||
"metric_definitions = [\n",
|
||||
" {\n",
|
||||
" \"Name\": \"average test loss\",\n",
|
||||
" \"Regex\": \"Test Loss: ([0-9\\\\.]+)\",\n",
|
||||
" }\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now, you'll create a `HyperparameterTuner` object. It takes the following parameters:\n",
|
||||
"- The `TensorFlow` estimator you previously created.\n",
|
||||
"- Your hyperparameter ranges.\n",
|
||||
"- Objective metric name and definition.\n",
|
||||
"- Tuning resource configurations such as the number of training jobs to run in total, and how many training jobs to run in parallel."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tuner = HyperparameterTuner(\n",
|
||||
" est,\n",
|
||||
" objective_metric_name,\n",
|
||||
" hyperparamter_range,\n",
|
||||
" metric_definitions,\n",
|
||||
" max_jobs=3,\n",
|
||||
" max_parallel_jobs=3,\n",
|
||||
" objective_type=objective_type,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"tuner.fit(inputs=channels)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Deploy the Best Model\n",
|
||||
"After training with hyperparameter optimization, you can deploy the best-performing model (by the objective metric you defined) to a SageMaker endpoint. For more information about deploying a model to a SageMaker endpoint, see the example [Deploy a Trained TensorFlow V2 Model](https://sagemaker-examples.readthedocs.io/en/latest/frameworks/tensorflow/get_started_mnist_deploy.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"predictor = tuner.deploy(initial_instance_count=1, instance_type=\"ml.m5.xlarge\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Evaluate\n",
|
||||
"Now, you can evaluate the best-performing model by invoking the endpoint with the MNIST test set. The test data needs to be readily consumable by the model, so we arrange them into the correct shape that is accepted by a TensorFlow model. We also normalize them so that the pixel values have mean 0 and standard deviation 1, since this is the convention used to train the model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import random\n",
|
||||
"import gzip\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"%matplotlib inline\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"images_file = \"t10k-images-idx3-ubyte.gz\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def read_mnist(data_dir, images_file):\n",
|
||||
" \"\"\"Byte string to numpy arrays\"\"\"\n",
|
||||
" with gzip.open(os.path.join(data_dir, images_file), \"rb\") as f:\n",
|
||||
" images = np.frombuffer(f.read(), np.uint8, offset=16).reshape(-1, 28, 28)\n",
|
||||
" return images\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"X = read_mnist(local_data_dir, images_file)\n",
|
||||
"\n",
|
||||
"# randomly sample 16 images to inspect\n",
|
||||
"mask = random.sample(range(X.shape[0]), 16)\n",
|
||||
"samples = X[mask]\n",
|
||||
"\n",
|
||||
"# plot the images\n",
|
||||
"fig, axs = plt.subplots(nrows=1, ncols=16, figsize=(16, 1))\n",
|
||||
"\n",
|
||||
"for i, splt in enumerate(axs):\n",
|
||||
" splt.imshow(samples[i])\n",
|
||||
"\n",
|
||||
"# preprocess the data to be consumed by the model\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def normalize(x, axis):\n",
|
||||
" eps = np.finfo(float).eps\n",
|
||||
"\n",
|
||||
" mean = np.mean(x, axis=axis, keepdims=True)\n",
|
||||
" # avoid division by zero\n",
|
||||
" std = np.std(x, axis=axis, keepdims=True) + eps\n",
|
||||
" return (x - mean) / std\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"samples = normalize(samples, axis=(1, 2))\n",
|
||||
"samples = np.expand_dims(samples, axis=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"predictions = predictor.predict(samples)[\"predictions\"]\n",
|
||||
"\n",
|
||||
"# softmax to logit\n",
|
||||
"predictions = np.array(predictions, dtype=np.float32)\n",
|
||||
"predictions = np.argmax(predictions, axis=1)\n",
|
||||
"\n",
|
||||
"print(\"Predictions: \", *predictions)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Cleanup\n",
|
||||
"If you do not plan to continue using the endpoint, delete it to free up resources."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"predictor.delete_endpoint()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Notebook CI Test Results\n",
|
||||
"\n",
|
||||
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (TensorFlow 2.10.0 Python 3.9 CPU Optimized)",
|
||||
"language": "python",
|
||||
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/tensorflow-2.10.1-cpu-py39-ubuntu20.04-sagemaker-v1.2"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.16"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
|
Before Width: | Height: | Size: 249 KiB After Width: | Height: | Size: 249 KiB |
|
Before Width: | Height: | Size: 6.5 KiB After Width: | Height: | Size: 6.5 KiB |
|
Before Width: | Height: | Size: 6.8 KiB After Width: | Height: | Size: 6.8 KiB |
|
Before Width: | Height: | Size: 346 KiB After Width: | Height: | Size: 346 KiB |
|
Before Width: | Height: | Size: 257 KiB After Width: | Height: | Size: 257 KiB |
|
Before Width: | Height: | Size: 498 KiB After Width: | Height: | Size: 498 KiB |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user