Creating SageMaker Autopilot/Pipelines example. (#3627)
* Creating SageMaker Autopilot/Pipelines example. * Applying black code formatter to notebook. Co-authored-by: atqy <95724753+atqy@users.noreply.github.com>
This commit is contained in:
Marcelo Aberle
@@ -77,6 +77,7 @@ These examples introduce SageMaker Autopilot. Autopilot automatically performs f
|
||||
- [Targeted Direct Marketing AutoML](autopilot/) shows how to use SageMaker Autopilot to automatically train a model.
|
||||
- [Housing Prices AutoML](sagemaker-autopilot/housing_prices) shows how to use SageMaker Autopilot for a linear regression problem (predict housing prices).
|
||||
- [Portfolio Churn Prediction with Amazon SageMaker Autopilot and Neo4j](autopilot/sagemaker_autopilot_neo4j_portfolio_churn.ipynb) shows how to use SageMaker Autopilot with graph embeddings to predict investment portfolio churn.
|
||||
- [Move Amazon SageMaker Autopilot ML models from experimentation to production using Amazon SageMaker Pipelines](autopilot/sagemaker-autopilot-pipelines) shows how to use SageMaker Autopilot in combination with SageMaker Pipelines for end-to-end AutoML training automation.
|
||||
|
||||
### Introduction to Amazon Algorithms
|
||||
|
||||
|
||||
@@ -0,0 +1,615 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "37904665-4371-4e02-89b6-7134cdc8a086",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Move Amazon SageMaker Autopilot ML models from experimentation to production using Amazon SageMaker Pipelines"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2bcea118",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[Amazon SageMaker Autopilot](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-automate-model-development.html) automatically builds, trains, and tunes the best custom machine learning (ML) models based on your data. It’s an automated machine learning (AutoML) solution that eliminates the heavy lifting of handwritten ML models that requires ML expertise. Data scientists need to only provide a tabular dataset and select the target column to predict, and Autopilot automatically infers the problem type, performs data preprocessing and feature engineering, selects the algorithms and training mode, and explores different configurations to find the best ML model. Then you can directly deploy the model to an [Amazon SageMaker](https://aws.amazon.com/sagemaker/) endpoint or iterate on the recommended solutions to further improve the model quality.\n",
|
||||
"\n",
|
||||
"Although Autopilot eliminates the heavy lifting of building ML models, MLOps engineers still have to create, automate, and manage end-to-end ML workflows. [SageMaker Pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) helps you automate the different steps of the ML lifecycle, including data preprocessing, training, tuning and evaluating ML models, and deploying them. \n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to leverage SageMaker Autopilot as part of a SageMaker Pipelines end-to-end AutoML training workflow. This notebook has successfully been run using SageMaker Studio with the Amazon Linux 2, Jupyter Lab 3 platform identifier. When running this notebook with older versions of SageMaker Studio or a SageMaker Notebook Instance, the *boto3* and/or *sagemaker* packages might need to be upgraded."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a6dd89e7-1457-4e49-b9a9-35ffc34a5037",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Imports"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2f127fbc-8caf-4130-b53e-f4085d51608c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import boto3\n",
|
||||
"import os\n",
|
||||
"import pandas as pd\n",
|
||||
"import sagemaker\n",
|
||||
"import time\n",
|
||||
"from datetime import datetime\n",
|
||||
"from sagemaker import ModelPackage\n",
|
||||
"from sagemaker.image_uris import retrieve\n",
|
||||
"from sagemaker.lambda_helper import Lambda\n",
|
||||
"from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
|
||||
"from sagemaker.sklearn.processing import SKLearnProcessor\n",
|
||||
"from sagemaker.workflow.callback_step import CallbackStep\n",
|
||||
"from sagemaker.workflow.lambda_step import LambdaStep\n",
|
||||
"from sagemaker.workflow.parameters import ParameterInteger, ParameterString\n",
|
||||
"from sagemaker.workflow.pipeline import Pipeline\n",
|
||||
"from sagemaker.workflow.steps import ProcessingStep, CacheConfig"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "78f54649-7693-471c-a16c-e84bcb0fb67d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Initialization"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7f3b02e9-962b-448a-af69-161e6a22d883",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"boto_session = boto3.session.Session()\n",
|
||||
"aws_region = boto_session.region_name\n",
|
||||
"sagemaker_client = boto_session.client(\"sagemaker\")\n",
|
||||
"lambda_client = boto_session.client(\"lambda\")\n",
|
||||
"sagemaker_session = sagemaker.session.Session(\n",
|
||||
" boto_session=boto_session, sagemaker_client=sagemaker_client\n",
|
||||
")\n",
|
||||
"sqs_client = boto_session.client(\n",
|
||||
" \"sqs\",\n",
|
||||
" region_name=aws_region,\n",
|
||||
" endpoint_url=f\"https://sqs.{aws_region}.amazonaws.com\",\n",
|
||||
")\n",
|
||||
"DATASET_PATH = os.path.join(\"data\", \"diabetic\", \"data\", \"diabetic_transformed.csv\")\n",
|
||||
"BUCKET_NAME = sagemaker_session.default_bucket()\n",
|
||||
"PROCESSING_JOB_LOCAL_BASE_PATH = \"/opt/ml/processing\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3356ec3c-8d29-44b7-96a8-e97c88a5e913",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## IAM permissions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e6f8e24a-c189-4cb5-bd9c-fc1ffd3f0ccb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For demo purposes, this notebook simplifies the IAM permissions configuration when [creating required IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) that can be assumed by the SageMaker and Lambda services. The following managed policies are sufficient to run this notebook but should be further scoped down to improve security (least privilege principle).\n",
|
||||
"- Lambda Execution Role:\n",
|
||||
" - AmazonSageMakerFullAccess\n",
|
||||
" - AmazonSQSFullAccess\n",
|
||||
"- SageMaker Execution Role:\n",
|
||||
" - AmazonSageMakerFullAccess\n",
|
||||
" - AWSLambda_FullAccess\n",
|
||||
" - AmazonSQSFullAccess"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "30a66279",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO: need to replace the lambda execution role name by its actual value\n",
|
||||
"lambda_execution_role_name = \"\"\n",
|
||||
"aws_account_id = boto3.client(\"sts\").get_caller_identity().get(\"Account\")\n",
|
||||
"LAMBDA_EXECUTION_ROLE_ARN = f\"arn:aws:iam::{aws_account_id}:role/{lambda_execution_role_name}\" # to be assumed by the Lambda service\n",
|
||||
"SAGEMAKER_EXECUTION_ROLE_ARN = (\n",
|
||||
" sagemaker.get_execution_role()\n",
|
||||
") # to be assumed by the SageMaker service"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b27c3d3a-5753-4bad-8f94-b3e1985d7d5f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## SageMaker Pipelines parameters"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c2571f7a-b71d-41b9-ae5a-4b7fb9ad8c09",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cache_config = CacheConfig(enable_caching=False)\n",
|
||||
"autopilot_job_name = ParameterString(\n",
|
||||
" name=\"AutopilotJobName\",\n",
|
||||
" default_value=\"autopilot-\" + datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\"),\n",
|
||||
")\n",
|
||||
"model_package_name = ParameterString(\n",
|
||||
" name=\"ModelPackageName\",\n",
|
||||
" default_value=autopilot_job_name.default_value + \"-model-package\",\n",
|
||||
")\n",
|
||||
"target_attribute_name = ParameterString(name=\"TargetAttributeName\", default_value=\"readmitted\")\n",
|
||||
"train_val_dataset_s3_path = ParameterString(\n",
|
||||
" name=\"TrainValDatasetS3Path\",\n",
|
||||
" default_value=sagemaker.s3.s3_path_join(\n",
|
||||
" \"s3://\", BUCKET_NAME, autopilot_job_name.default_value, \"data\", \"train_val.csv\"\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"x_test_s3_path = ParameterString(\n",
|
||||
" name=\"XTestDatasetS3Path\",\n",
|
||||
" default_value=sagemaker.s3.s3_path_join(\n",
|
||||
" \"s3://\", BUCKET_NAME, autopilot_job_name.default_value, \"data\", \"x_test.csv\"\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"y_test_s3_path = ParameterString(\n",
|
||||
" name=\"YTestDatasetS3Path\",\n",
|
||||
" default_value=sagemaker.s3.s3_path_join(\n",
|
||||
" \"s3://\", BUCKET_NAME, autopilot_job_name.default_value, \"data\", \"y_test.csv\"\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"max_autopilot_candidates = ParameterInteger(name=\"MaxAutopilotCandidates\", default_value=16)\n",
|
||||
"max_autopilot_job_runtime = ParameterInteger(\n",
|
||||
" name=\"MaxAutoMLJobRuntimeInSeconds\", default_value=7200 # 2 hours\n",
|
||||
")\n",
|
||||
"max_autopilot_training_job_runtime = ParameterInteger(\n",
|
||||
" name=\"MaxRuntimePerTrainingJobInSeconds\", default_value=3600\n",
|
||||
") # 1 hour\n",
|
||||
"instance_count = ParameterInteger(name=\"InstanceCount\", default_value=1)\n",
|
||||
"instance_type = ParameterString(name=\"InstanceType\", default_value=\"ml.m5.xlarge\")\n",
|
||||
"model_approval_status = ParameterString(name=\"ModelApprovalStatus\", default_value=\"Approved\")\n",
|
||||
"batch_transform_output_s3_path = ParameterString(\n",
|
||||
" name=\"BatchTransformOutputS3Path\",\n",
|
||||
" default_value=sagemaker.s3.s3_path_join(\n",
|
||||
" \"s3://\", BUCKET_NAME, autopilot_job_name.default_value, \"batch-transform-output\"\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"training_output_s3_path = ParameterString(\n",
|
||||
" name=\"TrainingOutputS3Path\",\n",
|
||||
" default_value=sagemaker.s3.s3_path_join(\n",
|
||||
" \"s3://\", BUCKET_NAME, autopilot_job_name.default_value, \"training-output\"\n",
|
||||
" ),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "56139cf4-3c64-484d-b5ec-03c366190533",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "55f00ed3-c835-452b-ae6b-5fa5ec21146a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We use a publicly available hospital readmission dataset - diabetic patients’ dataset to predict re-admission of\n",
|
||||
"diabetic patients within 30 days post discharge. This is a multi-class classification problem since the readmission\n",
|
||||
"options are either \"< 30\" if the patient is readmitted within 30 days, \"> 30\" if the patient is readmitted after 30\n",
|
||||
"days or \"no\" for no record of readmission.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1c7dc0df-0332-48c5-bae0-56e295e359f4",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!mkdir data\n",
|
||||
"!wget https://static.us-east-1.prod.workshops.aws/public/d56bf7ad-9738-4edf-9be0-f03cd22d8cf2/static/resources/hcls/diabetic.zip -nc -O data/data.zip\n",
|
||||
"!unzip -o data/data.zip -d data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b1188bc9-b428-46c8-80c0-167acf11855d",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = pd.read_csv(DATASET_PATH)\n",
|
||||
"train_val_data = data.sample(frac=0.8)\n",
|
||||
"test_data = data.drop(train_val_data.index)\n",
|
||||
"train_val_data.to_csv(train_val_dataset_s3_path.default_value, index=False, header=True)\n",
|
||||
"test_data.drop(target_attribute_name.default_value, axis=1).to_csv(\n",
|
||||
" x_test_s3_path.default_value, index=False, header=False\n",
|
||||
")\n",
|
||||
"test_data[target_attribute_name.default_value].to_csv(\n",
|
||||
" y_test_s3_path.default_value, index=False, header=True\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6f5b23dc-28eb-4b88-b88e-e5b185dd73f5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## First pipeline step: start Autopilot job"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4bc14c97-290b-4ece-ac49-06355424abd8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This pipeline step uses a [Lambda step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda) which runs a serverless Lambda function we create. The Lambda function in the *start_autopilot_job.py* script creates a [SageMaker Autopilot job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_auto_ml_job)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d88cd9f3-457b-4c82-ac39-bcd1bb21adf5",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"lambda_start_autopilot_job = Lambda(\n",
|
||||
" function_name=\"StartSagemakerAutopilotJob\",\n",
|
||||
" execution_role_arn=LAMBDA_EXECUTION_ROLE_ARN,\n",
|
||||
" script=\"start_autopilot_job.py\",\n",
|
||||
" handler=\"start_autopilot_job.lambda_handler\",\n",
|
||||
" session=sagemaker_session,\n",
|
||||
")\n",
|
||||
"lambda_start_autopilot_job.upsert()\n",
|
||||
"step_start_autopilot_job = LambdaStep(\n",
|
||||
" name=\"StartAutopilotJobStep\",\n",
|
||||
" lambda_func=lambda_start_autopilot_job,\n",
|
||||
" inputs={\n",
|
||||
" \"TrainValDatasetS3Path\": train_val_dataset_s3_path.default_value,\n",
|
||||
" \"MaxCandidates\": max_autopilot_candidates.default_value,\n",
|
||||
" \"MaxRuntimePerTrainingJobInSeconds\": max_autopilot_training_job_runtime.default_value,\n",
|
||||
" \"MaxAutoMLJobRuntimeInSeconds\": max_autopilot_job_runtime.default_value,\n",
|
||||
" \"TargetAttributeName\": target_attribute_name.default_value,\n",
|
||||
" \"TrainingOutputS3Path\": training_output_s3_path.default_value,\n",
|
||||
" \"AutopilotJobName\": autopilot_job_name,\n",
|
||||
" \"ProblemType\": \"MulticlassClassification\",\n",
|
||||
" \"AutopilotExecutionRoleArn\": SAGEMAKER_EXECUTION_ROLE_ARN,\n",
|
||||
" \"AutopilotObjectiveMetricName\": \"F1macro\",\n",
|
||||
" \"AutopilotMode\": \"ENSEMBLING\",\n",
|
||||
" },\n",
|
||||
" cache_config=cache_config,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "565dcb52-32b5-49e6-90a2-c934ce811d5b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Second pipeline step: check Autopilot job status"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "23526ce5-9c66-4c17-b055-87ff89ccaeba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The step repeatedly keeps track of the training job status by leveraging a separate Lambda function in *check_autopilot_job_status.py* until the Autopilot training job’s completion."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ffc544c1-34bc-4421-bc4a-5e9eac6a9bb0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"lambda_check_autopilot_job_status_function_name = \"CheckSagemakerAutopilotJobStatus\"\n",
|
||||
"lambda_check_autopilot_job_status = Lambda(\n",
|
||||
" function_name=lambda_check_autopilot_job_status_function_name,\n",
|
||||
" execution_role_arn=LAMBDA_EXECUTION_ROLE_ARN,\n",
|
||||
" script=\"check_autopilot_job_status.py\",\n",
|
||||
" handler=\"check_autopilot_job_status.lambda_handler\",\n",
|
||||
" session=sagemaker_session,\n",
|
||||
" timeout=15,\n",
|
||||
")\n",
|
||||
"lambda_check_autopilot_job_status.upsert()\n",
|
||||
"queue_url = sqs_client.create_queue(\n",
|
||||
" QueueName=\"AutopilotSagemakerPipelinesSqsCallback\",\n",
|
||||
" Attributes={\"DelaySeconds\": \"5\", \"VisibilityTimeout\": \"300\"},\n",
|
||||
")[\n",
|
||||
" \"QueueUrl\"\n",
|
||||
"] # 5 minutes timeout\n",
|
||||
"# Add event source mapping\n",
|
||||
"try:\n",
|
||||
" response = lambda_client.create_event_source_mapping(\n",
|
||||
" EventSourceArn=sqs_client.get_queue_attributes(\n",
|
||||
" QueueUrl=queue_url, AttributeNames=[\"QueueArn\"]\n",
|
||||
" )[\"Attributes\"][\"QueueArn\"],\n",
|
||||
" FunctionName=lambda_check_autopilot_job_status_function_name,\n",
|
||||
" Enabled=True,\n",
|
||||
" BatchSize=1,\n",
|
||||
" )\n",
|
||||
"except lambda_client.exceptions.ResourceConflictException:\n",
|
||||
" pass\n",
|
||||
"step_check_autopilot_job_status_callback = CallbackStep(\n",
|
||||
" name=\"CheckAutopilotJobStatusCallbackStep\",\n",
|
||||
" sqs_queue_url=queue_url,\n",
|
||||
" inputs={\"AutopilotJobName\": autopilot_job_name},\n",
|
||||
" outputs=[],\n",
|
||||
" depends_on=[step_start_autopilot_job],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bce3a94-9910-4d93-8584-8b14e5482e23",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Third pipeline step: evaluate Autopilot model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "673e9c1d-7d3d-44af-bc9f-df12011d35c8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The [SageMaker Processing step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) launches a [SageMaker Batch Transform Job](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) to evaluate the trained SageMaker Autopilot model against an evaluation dataset and generates the performance metrics evaluation report and model explainability metrics."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "251c1d49-63d5-4d72-8928-d67af2b771c5",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"processing_evaluation = SKLearnProcessor(\n",
|
||||
" role=SAGEMAKER_EXECUTION_ROLE_ARN,\n",
|
||||
" framework_version=\"1.0-1\",\n",
|
||||
" instance_count=instance_count.default_value,\n",
|
||||
" instance_type=instance_type.default_value,\n",
|
||||
" sagemaker_session=sagemaker_session,\n",
|
||||
")\n",
|
||||
"step_autopilot_model_evaluation = ProcessingStep(\n",
|
||||
" name=\"EvaluateBestAutopilotModelStep\",\n",
|
||||
" job_arguments=[\n",
|
||||
" \"--autopilot-job-name\",\n",
|
||||
" autopilot_job_name,\n",
|
||||
" \"--aws-region\",\n",
|
||||
" aws_region,\n",
|
||||
" \"--batch-transform-output-s3-path\",\n",
|
||||
" batch_transform_output_s3_path.default_value,\n",
|
||||
" \"--instance-type\",\n",
|
||||
" instance_type.default_value,\n",
|
||||
" \"--instance-count\",\n",
|
||||
" str(instance_count.default_value),\n",
|
||||
" \"--local-base-path\",\n",
|
||||
" PROCESSING_JOB_LOCAL_BASE_PATH,\n",
|
||||
" \"--sagemaker-execution-role-arn\",\n",
|
||||
" SAGEMAKER_EXECUTION_ROLE_ARN,\n",
|
||||
" \"--x-test-s3-path\",\n",
|
||||
" x_test_s3_path.default_value,\n",
|
||||
" \"--y-test-file-name\",\n",
|
||||
" y_test_s3_path.default_value.split(\"/\")[-1],\n",
|
||||
" ],\n",
|
||||
" processor=processing_evaluation,\n",
|
||||
" code=\"evaluate_autopilot_model.py\",\n",
|
||||
" depends_on=[step_check_autopilot_job_status_callback],\n",
|
||||
" inputs=[\n",
|
||||
" ProcessingInput(\n",
|
||||
" input_name=\"LabelsTestDataset\",\n",
|
||||
" source=y_test_s3_path.default_value,\n",
|
||||
" destination=os.path.join(PROCESSING_JOB_LOCAL_BASE_PATH, \"data\"),\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
" outputs=[\n",
|
||||
" ProcessingOutput(\n",
|
||||
" output_name=\"EvaluationReport\",\n",
|
||||
" source=os.path.join(PROCESSING_JOB_LOCAL_BASE_PATH, \"evaluation_report\"),\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
" cache_config=cache_config,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4ac7aba3-47a9-477c-a5a0-5b3f631d82e3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Fourth pipeline step: register Autopilot model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7f4312c8-25ef-4acb-8861-c5d318234691",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Using a Lambda step, the Lambda function in *register_autopilot_job.py* registers the SageMaker Autopilot model to the SageMaker Model Registry using the evaluation report obtained in the previous SageMaker Processing step. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "199eb3ba-64ce-4514-b927-b89aff42e66d",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"lambda_register_autopilot_model = Lambda(\n",
|
||||
" function_name=\"RegisterSagemakerAutopilotModel\",\n",
|
||||
" execution_role_arn=LAMBDA_EXECUTION_ROLE_ARN,\n",
|
||||
" script=\"register_autopilot_model.py\",\n",
|
||||
" handler=\"register_autopilot_model.lambda_handler\",\n",
|
||||
" session=sagemaker_session,\n",
|
||||
" timeout=15,\n",
|
||||
")\n",
|
||||
"lambda_register_autopilot_model.upsert()\n",
|
||||
"step_register_autopilot_model = LambdaStep(\n",
|
||||
" name=\"RegisterAutopilotModelStep\",\n",
|
||||
" lambda_func=lambda_register_autopilot_model,\n",
|
||||
" inputs={\n",
|
||||
" \"AutopilotJobName\": autopilot_job_name,\n",
|
||||
" \"EvaluationReportS3Path\": step_autopilot_model_evaluation.properties.ProcessingOutputConfig.Outputs[\n",
|
||||
" \"EvaluationReport\"\n",
|
||||
" ].S3Output.S3Uri,\n",
|
||||
" \"ModelPackageName\": model_package_name.default_value,\n",
|
||||
" \"ModelApprovalStatus\": model_approval_status.default_value,\n",
|
||||
" \"InstanceType\": instance_type.default_value,\n",
|
||||
" },\n",
|
||||
" cache_config=cache_config,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d10d3a2b-15d3-4841-ba44-cb4763444191",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Create and run pipeline"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1e567b30-6202-4ad1-876f-b0a433b3a92b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Once the pipeline steps are defined, we combine them into a SageMaker Pipeline. The steps are run in sequential order. The pipeline executes all of the steps for an AutoML job leveraging SageMaker Autopilot and SageMaker Pipelines for training, model evaluation and model registration."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2ef27d1d-a4e2-4a4b-92c4-2878602bed20",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pipeline = Pipeline(\n",
|
||||
" name=\"autopilot-demo-pipeline\",\n",
|
||||
" parameters=[\n",
|
||||
" autopilot_job_name,\n",
|
||||
" target_attribute_name,\n",
|
||||
" train_val_dataset_s3_path,\n",
|
||||
" x_test_s3_path,\n",
|
||||
" y_test_s3_path,\n",
|
||||
" max_autopilot_candidates,\n",
|
||||
" max_autopilot_job_runtime,\n",
|
||||
" max_autopilot_training_job_runtime,\n",
|
||||
" instance_count,\n",
|
||||
" instance_type,\n",
|
||||
" model_approval_status,\n",
|
||||
" ],\n",
|
||||
" steps=[\n",
|
||||
" step_start_autopilot_job,\n",
|
||||
" step_check_autopilot_job_status_callback,\n",
|
||||
" step_autopilot_model_evaluation,\n",
|
||||
" step_register_autopilot_model,\n",
|
||||
" ],\n",
|
||||
" sagemaker_session=sagemaker_session,\n",
|
||||
")\n",
|
||||
"pipeline.upsert(role_arn=SAGEMAKER_EXECUTION_ROLE_ARN)\n",
|
||||
"pipeline_execution = pipeline.start()\n",
|
||||
"pipeline_execution.wait(delay=20, max_attempts=24 * 60 * 3) # max wait: 24 hours"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a1b9621a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Deploy the Autopilot Model Endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "113f3dfc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[Deploying](https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html) the previously registered best Autopilot model from the ML training pipeline to a SageMaker Endpoint."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "21f1a5d8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = ModelPackage(\n",
|
||||
" role=SAGEMAKER_EXECUTION_ROLE_ARN,\n",
|
||||
" model_package_arn=model_package_name.default_value,\n",
|
||||
" sagemaker_session=sagemaker_session,\n",
|
||||
")\n",
|
||||
"while (\n",
|
||||
" sagemaker_client.describe_model_package(ModelPackageName=model_package_name.default_value)[\n",
|
||||
" \"ModelPackageStatus\"\n",
|
||||
" ]\n",
|
||||
" != \"Completed\"\n",
|
||||
"):\n",
|
||||
" time.sleep(10)\n",
|
||||
"model.deploy(\n",
|
||||
" initial_instance_count=instance_count.default_value,\n",
|
||||
" instance_type=instance_type.default_value,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "30c7985e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Cleanup"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0d331c75",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_client.delete_endpoint(EndpointName=model.endpoint_name)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.10.4 64-bit ('venv3104')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.4"
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "9b940e11c83395c293f3c0e2c599e38d216720ee98e9a77da40826c04a31739a"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
import boto3
|
||||
import json
|
||||
import logging
|
||||
|
||||
sagemaker_client = boto3.client("sagemaker")
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
try:
|
||||
payload = json.loads(event["Records"][0]["body"])
|
||||
callback_token = payload["token"]
|
||||
autopilot_job = sagemaker_client.describe_auto_ml_job(
|
||||
AutoMLJobName=payload["arguments"]["AutopilotJobName"]
|
||||
)
|
||||
autopilot_job_status = autopilot_job["AutoMLJobStatus"]
|
||||
if autopilot_job_status == "Completed":
|
||||
sagemaker_client.send_pipeline_execution_step_success(
|
||||
CallbackToken=callback_token
|
||||
)
|
||||
elif autopilot_job_status in ["InProgress", "Stopping"]:
|
||||
raise ValueError("Autopilot training not finished yet. Retrying later...")
|
||||
else:
|
||||
sagemaker_client.send_pipeline_execution_step_failure(
|
||||
CallbackToken=callback_token,
|
||||
FailureReason=autopilot_job.get(
|
||||
"FailureReason",
|
||||
f"Autopilot training job (status: {autopilot_job_status}) failed to finish.",
|
||||
),
|
||||
)
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logging.exception(e)
|
||||
sagemaker_client.send_pipeline_execution_step_failure(
|
||||
CallbackToken=callback_token,
|
||||
FailureReason=str(e),
|
||||
)
|
||||
@@ -0,0 +1,110 @@
|
||||
import argparse
|
||||
import boto3
|
||||
import json
|
||||
import os
|
||||
import pandas as pd
|
||||
import random
|
||||
import string
|
||||
import time
|
||||
from sklearn.metrics import f1_score, precision_score, recall_score
|
||||
from urllib.parse import urlparse
|
||||
|
||||
RANDOM_SUFFIX = "".join(random.choices(string.ascii_lowercase, k=8))
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--autopilot-job-name", type=str)
|
||||
parser.add_argument("--aws-region", type=str)
|
||||
parser.add_argument("--x-test-s3-path", type=str)
|
||||
parser.add_argument("--y-test-file-name", type=str)
|
||||
parser.add_argument("--batch-transform-output-s3-path", type=str)
|
||||
parser.add_argument("--instance-type", type=str)
|
||||
parser.add_argument("--instance-count", type=int)
|
||||
parser.add_argument("--local-base-path", type=str)
|
||||
parser.add_argument("--sagemaker-execution-role-arn", type=str)
|
||||
args = parser.parse_args()
|
||||
|
||||
boto_session = boto3.session.Session(region_name=args.aws_region)
|
||||
s3_client = boto_session.client("s3")
|
||||
sagemaker_client = boto_session.client("sagemaker")
|
||||
|
||||
# Create model
|
||||
model_name = args.autopilot_job_name + RANDOM_SUFFIX
|
||||
response = sagemaker_client.create_model(
|
||||
ModelName=model_name,
|
||||
Containers=sagemaker_client.describe_auto_ml_job(
|
||||
AutoMLJobName=args.autopilot_job_name
|
||||
)["BestCandidate"]["InferenceContainers"],
|
||||
ExecutionRoleArn=args.sagemaker_execution_role_arn,
|
||||
)
|
||||
|
||||
# Create batch transform job
|
||||
batch_transform_job_name = args.autopilot_job_name + RANDOM_SUFFIX
|
||||
response = sagemaker_client.create_transform_job(
|
||||
TransformJobName=batch_transform_job_name,
|
||||
ModelName=model_name,
|
||||
TransformInput={
|
||||
"DataSource": {
|
||||
"S3DataSource": {
|
||||
"S3DataType": "S3Prefix",
|
||||
"S3Uri": args.x_test_s3_path,
|
||||
}
|
||||
},
|
||||
"ContentType": "text/csv",
|
||||
"SplitType": "Line",
|
||||
},
|
||||
TransformOutput={
|
||||
"S3OutputPath": args.batch_transform_output_s3_path,
|
||||
"AssembleWith": "Line",
|
||||
},
|
||||
TransformResources={
|
||||
"InstanceType": args.instance_type,
|
||||
"InstanceCount": args.instance_count,
|
||||
},
|
||||
)
|
||||
|
||||
# Wait for the batch transform job to finish
|
||||
while (
|
||||
sagemaker_client.describe_transform_job(TransformJobName=batch_transform_job_name)[
|
||||
"TransformJobStatus"
|
||||
]
|
||||
== "InProgress"
|
||||
):
|
||||
time.sleep(10)
|
||||
|
||||
# Download batch transform results
|
||||
x_test_file_name = args.x_test_s3_path.split("/")[-1]
|
||||
predictions_s3_path = os.path.join(
|
||||
args.batch_transform_output_s3_path, x_test_file_name + ".out"
|
||||
)
|
||||
o = urlparse(predictions_s3_path)
|
||||
s3_client.download_file(
|
||||
Bucket=o.netloc, Key=o.path.strip("/"), Filename="predictions.csv"
|
||||
)
|
||||
|
||||
# Create best model evaluation report
|
||||
y_pred = pd.read_csv("predictions.csv", header=0).iloc[:, 0]
|
||||
y_true = pd.read_csv(
|
||||
os.path.join(args.local_base_path, "data", args.y_test_file_name), header=1
|
||||
)
|
||||
evaluation_report = {
|
||||
"multiclass_classification_metrics": {
|
||||
"weighted_f1": {
|
||||
"value": f1_score(y_pred, y_true, average="weighted"),
|
||||
"standard_deviation": "NaN",
|
||||
},
|
||||
"weighted_precision": {
|
||||
"value": precision_score(y_pred, y_true, average="weighted"),
|
||||
"standard_deviation": "NaN",
|
||||
},
|
||||
"weighted_recall": {
|
||||
"value": recall_score(y_pred, y_true, average="weighted"),
|
||||
"standard_deviation": "NaN",
|
||||
},
|
||||
},
|
||||
}
|
||||
evaluation_report_path = os.path.join(
|
||||
args.local_base_path, "evaluation_report", "evaluation_report.json"
|
||||
)
|
||||
os.makedirs(os.path.dirname(evaluation_report_path), exist_ok=True)
|
||||
with open(evaluation_report_path, "w") as f:
|
||||
f.write(json.dumps(evaluation_report))
|
||||
@@ -0,0 +1,62 @@
|
||||
import boto3
|
||||
import os
|
||||
from botocore.exceptions import ClientError
|
||||
from urllib.parse import urlparse
|
||||
|
||||
s3_client = boto3.client("s3")
|
||||
sagemaker_client = boto3.client("sagemaker")
|
||||
|
||||
|
||||
def get_explainability_report_json_s3_path(s3_path):
|
||||
o = urlparse(s3_path)
|
||||
bucket_name = o.netloc
|
||||
s3_prefix = o.path.strip("/")
|
||||
paginator = s3_client.get_paginator("list_objects_v2")
|
||||
response = paginator.paginate(
|
||||
Bucket=bucket_name, Prefix=s3_prefix, PaginationConfig={"PageSize": 1}
|
||||
)
|
||||
for page in response:
|
||||
files = page.get("Contents")
|
||||
for file in files:
|
||||
if "analysis.json" in file["Key"]:
|
||||
return os.path.join("s3://", bucket_name, file["Key"])
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
# Get the explainability results from the Autopilot job
|
||||
autopilot_job = sagemaker_client.describe_auto_ml_job(
|
||||
AutoMLJobName=event["AutopilotJobName"]
|
||||
)
|
||||
explainability_report_s3_path = autopilot_job["BestCandidate"][
|
||||
"CandidateProperties"
|
||||
]["CandidateArtifactLocations"]["Explainability"]
|
||||
autopilot_job["BestCandidate"]["InferenceContainers"][0].pop("Environment")
|
||||
sagemaker_client.create_model_package(
|
||||
ModelPackageName=event["ModelPackageName"],
|
||||
InferenceSpecification={
|
||||
"Containers": autopilot_job["BestCandidate"]["InferenceContainers"],
|
||||
"SupportedContentTypes": ["text/csv"],
|
||||
"SupportedResponseMIMETypes": ["text/csv"],
|
||||
"SupportedTransformInstanceTypes": [event["InstanceType"]],
|
||||
"SupportedRealtimeInferenceInstanceTypes": [event["InstanceType"]],
|
||||
},
|
||||
ModelApprovalStatus=event["ModelApprovalStatus"],
|
||||
ModelMetrics={
|
||||
"ModelQuality": {
|
||||
"Statistics": {
|
||||
"ContentType": ".json",
|
||||
"S3Uri": os.path.join(
|
||||
event["EvaluationReportS3Path"], "evaluation_report.json"
|
||||
),
|
||||
},
|
||||
},
|
||||
"Explainability": {
|
||||
"Report": {
|
||||
"ContentType": ".json",
|
||||
"S3Uri": get_explainability_report_json_s3_path(
|
||||
explainability_report_s3_path
|
||||
),
|
||||
}
|
||||
},
|
||||
},
|
||||
)
|
||||
@@ -0,0 +1,51 @@
|
||||
import sys
|
||||
from pip._internal import main
|
||||
|
||||
# Upgrading boto3 to the newest release to be able to use the latest SageMaker features
|
||||
main(
|
||||
[
|
||||
"install",
|
||||
"-I",
|
||||
"-q",
|
||||
"boto3",
|
||||
"--target",
|
||||
"/tmp/",
|
||||
"--no-cache-dir",
|
||||
"--disable-pip-version-check",
|
||||
]
|
||||
)
|
||||
sys.path.insert(0, "/tmp/")
|
||||
import boto3
|
||||
|
||||
sagemaker_client = boto3.client("sagemaker")
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
sagemaker_client.create_auto_ml_job(
|
||||
AutoMLJobName=event["AutopilotJobName"],
|
||||
InputDataConfig=[
|
||||
{
|
||||
"DataSource": {
|
||||
"S3DataSource": {
|
||||
"S3DataType": "S3Prefix",
|
||||
"S3Uri": event["TrainValDatasetS3Path"],
|
||||
}
|
||||
},
|
||||
"TargetAttributeName": event["TargetAttributeName"],
|
||||
}
|
||||
],
|
||||
OutputDataConfig={"S3OutputPath": event["TrainingOutputS3Path"]},
|
||||
ProblemType=event["ProblemType"],
|
||||
AutoMLJobObjective={"MetricName": event["AutopilotObjectiveMetricName"]},
|
||||
AutoMLJobConfig={
|
||||
"CompletionCriteria": {
|
||||
"MaxCandidates": event["MaxCandidates"],
|
||||
"MaxRuntimePerTrainingJobInSeconds": event[
|
||||
"MaxRuntimePerTrainingJobInSeconds"
|
||||
],
|
||||
"MaxAutoMLJobRuntimeInSeconds": event["MaxAutoMLJobRuntimeInSeconds"],
|
||||
},
|
||||
"Mode": event["AutopilotMode"],
|
||||
},
|
||||
RoleArn=event["AutopilotExecutionRoleArn"],
|
||||
)
|
||||
Reference in New Issue
Block a user