Tutorials/Boston Housing - XGBoost (Hyperparameter Tuning) - Low Level.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Boston Housing Prices\n",
    "\n",
    "## Using XGBoost in SageMaker\n",
    "\n",
    "_Deep Learning Nanodegree Program | Deployment_\n",
    "\n",
    "---\n",
    "\n",
    "As an introduction to using SageMaker's Low Level Python API we will look at a relatively simple problem. Namely, we will use the [Boston Housing Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) to predict the median value of a home in the area of Boston Mass.\n",
    "\n",
    "The documentation reference for the API used in this notebook is the [SageMaker Developer's Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/)\n",
    "\n",
    "## General Outline\n",
    "\n",
    "Typically, when using a notebook instance with SageMaker, you will proceed through the following steps. Of course, not every step will need to be done with each project. Also, there is quite a lot of room for variation in many of the steps, as you will see throughout these lessons.\n",
    "\n",
    "1. Download or otherwise retrieve the data.\n",
    "2. Process / Prepare the data.\n",
    "3. Upload the processed data to S3.\n",
    "4. Train a chosen model.\n",
    "5. Test the trained model (typically using a batch transform job).\n",
    "6. Deploy the trained model.\n",
    "7. Use the deployed model.\n",
    "\n",
    "In this notebook we will only be covering steps 1 through 5 as we just want to get a feel for using SageMaker. In later notebooks we will talk about deploying a trained model in much more detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 0: Setting up the notebook\n",
    "\n",
    "We begin by setting up all of the necessary bits required to run our notebook. To start that means loading all of the Python modules we will need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "\n",
    "import time\n",
    "from time import gmtime, strftime\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import load_boston\n",
    "import sklearn.model_selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the modules above, we need to import the various bits of SageMaker that we will be using. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "# This is an object that represents the SageMaker session that we are currently operating in. This\n",
    "# object contains some useful information that we will need to access later such as our region.\n",
    "session = sagemaker.Session()\n",
    "\n",
    "# This is an object that represents the IAM role that we are currently assigned. When we construct\n",
    "# and launch the training job later we will need to tell it what IAM role it should have. Since our\n",
    "# use case is relatively simple we will simply assign the training job the role we currently have.\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Downloading the data\n",
    "\n",
    "Fortunately, this dataset can be retrieved using sklearn and so this step is relatively straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "boston = load_boston()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Preparing and splitting the data\n",
    "\n",
    "Given that this is clean tabular data, we don't need to do any processing. However, we do need to split the rows in the dataset up into train, test and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we package up the input data and the target variable (the median value) as pandas dataframes. This\n",
    "# will make saving the data to a file a little easier later on.\n",
    "\n",
    "X_bos_pd = pd.DataFrame(boston.data, columns=boston.feature_names)\n",
    "Y_bos_pd = pd.DataFrame(boston.target)\n",
    "\n",
    "# We split the dataset into 2/3 training and 1/3 testing sets.\n",
    "X_train, X_test, Y_train, Y_test = sklearn.model_selection.train_test_split(X_bos_pd, Y_bos_pd, test_size=0.33)\n",
    "\n",
    "# Then we split the training set further into 2/3 training and 1/3 validation sets.\n",
    "X_train, X_val, Y_train, Y_val = sklearn.model_selection.train_test_split(X_train, Y_train, test_size=0.33)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Uploading the data files to S3\n",
    "\n",
    "When a training job is constructed using SageMaker, a container is executed which performs the training operation. This container is given access to data that is stored in S3. This means that we need to upload the data we want to use for training to S3. In addition, when we perform a batch transform job, SageMaker expects the input data to be stored on S3. We can use the SageMaker API to do this and hide some of the details.\n",
    "\n",
    "### Save the data locally\n",
    "\n",
    "First we need to create the test, train and validation csv files which we will then upload to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is our local data directory. We need to make sure that it exists.\n",
    "data_dir = '../data/boston'\n",
    "if not os.path.exists(data_dir):\n",
    "    os.makedirs(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We use pandas to save our test, train and validation data to csv files. Note that we make sure not to include header\n",
    "# information or an index as this is required by the built in algorithms provided by Amazon. Also, for the train and\n",
    "# validation data, it is assumed that the first entry in each row is the target variable.\n",
    "\n",
    "X_test.to_csv(os.path.join(data_dir, 'test.csv'), header=False, index=False)\n",
    "\n",
    "pd.concat([Y_val, X_val], axis=1).to_csv(os.path.join(data_dir, 'validation.csv'), header=False, index=False)\n",
    "pd.concat([Y_train, X_train], axis=1).to_csv(os.path.join(data_dir, 'train.csv'), header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload to S3\n",
    "\n",
    "Since we are currently running inside of a SageMaker session, we can use the object which represents this session to upload our data to the 'default' S3 bucket. Note that it is good practice to provide a custom prefix (essentially an S3 folder) to make sure that you don't accidentally interfere with data uploaded from some other notebook or project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Created S3 bucket: sagemaker-us-east-1-440180731255\n"
     ]
    }
   ],
   "source": [
    "prefix = 'boston-xgboost-LL'\n",
    "\n",
    "test_location = session.upload_data(os.path.join(data_dir, 'test.csv'), key_prefix=prefix)\n",
    "val_location = session.upload_data(os.path.join(data_dir, 'validation.csv'), key_prefix=prefix)\n",
    "train_location = session.upload_data(os.path.join(data_dir, 'train.csv'), key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Train and construct the XGBoost model\n",
    "\n",
    "Now that we have the training and validation data uploaded to S3, we can construct our XGBoost model and train it. Unlike in the previous notebooks, instead of training a single model, we will use SageMakers hyperparameter tuning functionality to train multiple models and use the one that performs the best on the validation set.\n",
    "\n",
    "### Set up the training job\n",
    "\n",
    "First, we will set up a training job for our model. This is very similar to the way in which we constructed the training job in previous notebooks. Essentially this describes the *base* training job from which SageMaker will create refinements by changing some hyperparameters during the hyperparameter tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will need to know the name of the container that we want to use for training. SageMaker provides\n",
    "# a nice utility method to construct this for us.\n",
    "container = get_image_uri(session.boto_region_name, 'xgboost')\n",
    "\n",
    "# We now specify the parameters we wish to use for our training job\n",
    "training_params = {}\n",
    "\n",
    "# We need to specify the permissions that this training job will have. For our purposes we can use\n",
    "# the same permissions that our current SageMaker session has.\n",
    "training_params['RoleArn'] = role\n",
    "\n",
    "# Here we describe the algorithm we wish to use. The most important part is the container which\n",
    "# contains the training code.\n",
    "training_params['AlgorithmSpecification'] = {\n",
    "    \"TrainingImage\": container,\n",
    "    \"TrainingInputMode\": \"File\"\n",
    "}\n",
    "\n",
    "# We also need to say where we would like the resulting model artifacst stored.\n",
    "training_params['OutputDataConfig'] = {\n",
    "    \"S3OutputPath\": \"s3://\" + session.default_bucket() + \"/\" + prefix + \"/output\"\n",
    "}\n",
    "\n",
    "# We also need to set some parameters for the training job itself. Namely we need to describe what sort of\n",
    "# compute instance we wish to use along with a stopping condition to handle the case that there is\n",
    "# some sort of error and the training script doesn't terminate.\n",
    "training_params['ResourceConfig'] = {\n",
    "    \"InstanceCount\": 1,\n",
    "    \"InstanceType\": \"ml.m4.xlarge\",\n",
    "    \"VolumeSizeInGB\": 5\n",
    "}\n",
    "    \n",
    "training_params['StoppingCondition'] = {\n",
    "    \"MaxRuntimeInSeconds\": 86400\n",
    "}\n",
    "\n",
    "# Next we set the algorithm specific hyperparameters. You may wish to change these to see what effect\n",
    "# there is on the resulting model.\n",
    "training_params['StaticHyperParameters'] = {\n",
    "    \"gamma\": \"4\",\n",
    "    \"subsample\": \"0.8\",\n",
    "    \"objective\": \"reg:linear\",\n",
    "    \"early_stopping_rounds\": \"10\",\n",
    "    \"num_round\": \"200\"\n",
    "}\n",
    "\n",
    "# Now we need to tell SageMaker where the data should be retrieved from.\n",
    "training_params['InputDataConfig'] = [\n",
    "    {\n",
    "        \"ChannelName\": \"train\",\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"S3DataType\": \"S3Prefix\",\n",
    "                \"S3Uri\": train_location,\n",
    "                \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "            }\n",
    "        },\n",
    "        \"ContentType\": \"csv\",\n",
    "        \"CompressionType\": \"None\"\n",
    "    },\n",
    "    {\n",
    "        \"ChannelName\": \"validation\",\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"S3DataType\": \"S3Prefix\",\n",
    "                \"S3Uri\": val_location,\n",
    "                \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "            }\n",
    "        },\n",
    "        \"ContentType\": \"csv\",\n",
    "        \"CompressionType\": \"None\"\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up the tuning job\n",
    "\n",
    "Now that the *base* training job has been set up, we can describe the tuning job that we would like SageMaker to perform. In particular, like in the high level notebook, we will specify which hyperparameters we wish SageMaker to change and what range of values they may take on. In addition we need to specify how many models should be trained and how many should be trained in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to construct a dictionary which specifies the tuning job we want SageMaker to perform\n",
    "tuning_job_config = {\n",
    "    # First we specify which hyperparameters we want SageMaker to be able to vary,\n",
    "    # and we specify the type and range of the hyperparameters.\n",
    "    \"ParameterRanges\": {\n",
    "    \"CategoricalParameterRanges\": [],\n",
    "    \"ContinuousParameterRanges\": [\n",
    "        {\n",
    "            \"MaxValue\": \"0.5\",\n",
    "            \"MinValue\": \"0.05\",\n",
    "            \"Name\": \"eta\"\n",
    "        },\n",
    "    ],\n",
    "    \"IntegerParameterRanges\": [\n",
    "        {\n",
    "            \"MaxValue\": \"12\",\n",
    "            \"MinValue\": \"3\",\n",
    "            \"Name\": \"max_depth\"\n",
    "        },\n",
    "        {\n",
    "            \"MaxValue\": \"8\",\n",
    "            \"MinValue\": \"2\",\n",
    "            \"Name\": \"min_child_weight\"\n",
    "        }\n",
    "    ]},\n",
    "    # We also need to specify how many models should be fit and how many can be fit in parallel\n",
    "    \"ResourceLimits\": {\n",
    "        \"MaxNumberOfTrainingJobs\": 20,\n",
    "        \"MaxParallelTrainingJobs\": 3\n",
    "    },\n",
    "    # Here we specify how SageMaker should update the hyperparameters as new models are fit\n",
    "    \"Strategy\": \"Bayesian\",\n",
    "    # And lastly we need to specify how we'd like to determine which models are better or worse\n",
    "    \"HyperParameterTuningJobObjective\": {\n",
    "        \"MetricName\": \"validation:rmse\",\n",
    "        \"Type\": \"Minimize\"\n",
    "    }\n",
    "  }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Execute the tuning job\n",
    "\n",
    "Now that we've built the data structures that describe the tuning job we want SageMaker to execute, it is time to actually start the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'HyperParameterTuningJobArn': 'arn:aws:sagemaker:us-east-1:440180731255:hyper-parameter-tuning-job/tuning-job2018-10-11-06-27-07',\n",
       " 'ResponseMetadata': {'RequestId': 'fd725770-5711-41f9-bf9d-0873f6fbab13',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'fd725770-5711-41f9-bf9d-0873f6fbab13',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '130',\n",
       "   'date': 'Thu, 11 Oct 2018 06:27:07 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First we need to choose a name for the job. This is useful for if we want to recall information about our\n",
    "# tuning job at a later date. Note that SageMaker requires a tuning job name and that the name needs to\n",
    "# be unique, which we accomplish by appending the current timestamp.\n",
    "tuning_job_name = \"tuning-job\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "# And now we ask SageMaker to create (and execute) the training job\n",
    "session.sagemaker_client.create_hyper_parameter_tuning_job(HyperParameterTuningJobName = tuning_job_name,\n",
    "                                                           HyperParameterTuningJobConfig = tuning_job_config,\n",
    "                                                           TrainingJobDefinition = training_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tuning job has now been created by SageMaker and is currently running. Since we need the output of the tuning job, we may wish to wait until it has finished. We can do so by asking SageMaker to output the logs generated by the tuning job and continue doing so until the job terminates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "..................................................................................................................................................................................................................................................................!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'HyperParameterTuningJobName': 'tuning-job2018-10-11-06-27-07',\n",
       " 'HyperParameterTuningJobArn': 'arn:aws:sagemaker:us-east-1:440180731255:hyper-parameter-tuning-job/tuning-job2018-10-11-06-27-07',\n",
       " 'HyperParameterTuningJobConfig': {'Strategy': 'Bayesian',\n",
       "  'HyperParameterTuningJobObjective': {'Type': 'Minimize',\n",
       "   'MetricName': 'validation:rmse'},\n",
       "  'ResourceLimits': {'MaxNumberOfTrainingJobs': 20,\n",
       "   'MaxParallelTrainingJobs': 3},\n",
       "  'ParameterRanges': {'IntegerParameterRanges': [{'Name': 'max_depth',\n",
       "     'MinValue': '3',\n",
       "     'MaxValue': '12'},\n",
       "    {'Name': 'min_child_weight', 'MinValue': '2', 'MaxValue': '8'}],\n",
       "   'ContinuousParameterRanges': [{'Name': 'eta',\n",
       "     'MinValue': '0.05',\n",
       "     'MaxValue': '0.5'}],\n",
       "   'CategoricalParameterRanges': []}},\n",
       " 'TrainingJobDefinition': {'StaticHyperParameters': {'_tuning_objective_metric': 'validation:rmse',\n",
       "   'early_stopping_rounds': '10',\n",
       "   'gamma': '4',\n",
       "   'num_round': '200',\n",
       "   'objective': 'reg:linear',\n",
       "   'subsample': '0.8'},\n",
       "  'AlgorithmSpecification': {'TrainingImage': '811284229777.dkr.ecr.us-east-1.amazonaws.com/xgboost:1',\n",
       "   'TrainingInputMode': 'File',\n",
       "   'MetricDefinitions': [{'Name': 'train:mae',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\]#011train-mae:(\\\\S+).*'},\n",
       "    {'Name': 'validation:auc',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-auc:(\\\\S+)'},\n",
       "    {'Name': 'train:merror',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\]#011train-merror:(\\\\S+).*'},\n",
       "    {'Name': 'train:auc', 'Regex': '.*\\\\[[0-9]+\\\\]#011train-auc:(\\\\S+).*'},\n",
       "    {'Name': 'validation:mae',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-mae:(\\\\S+)'},\n",
       "    {'Name': 'validation:error',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-error:(\\\\S+)'},\n",
       "    {'Name': 'validation:merror',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-merror:(\\\\S+)'},\n",
       "    {'Name': 'validation:logloss',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-logloss:(\\\\S+)'},\n",
       "    {'Name': 'train:rmse', 'Regex': '.*\\\\[[0-9]+\\\\]#011train-rmse:(\\\\S+).*'},\n",
       "    {'Name': 'train:logloss',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\]#011train-logloss:(\\\\S+).*'},\n",
       "    {'Name': 'train:mlogloss',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\]#011train-mlogloss:(\\\\S+).*'},\n",
       "    {'Name': 'validation:rmse',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-rmse:(\\\\S+)'},\n",
       "    {'Name': 'validation:ndcg',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-ndcg:(\\\\S+)'},\n",
       "    {'Name': 'train:error', 'Regex': '.*\\\\[[0-9]+\\\\]#011train-error:(\\\\S+).*'},\n",
       "    {'Name': 'validation:mlogloss',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-mlogloss:(\\\\S+)'},\n",
       "    {'Name': 'train:ndcg', 'Regex': '.*\\\\[[0-9]+\\\\]#011train-ndcg:(\\\\S+).*'},\n",
       "    {'Name': 'train:map', 'Regex': '.*\\\\[[0-9]+\\\\]#011train-map:(\\\\S+).*'},\n",
       "    {'Name': 'validation:map',\n",
       "     'Regex': '.*\\\\[[0-9]+\\\\].*#011validation-map:(\\\\S+)'}]},\n",
       "  'RoleArn': 'arn:aws:iam::440180731255:role/service-role/AmazonSageMaker-ExecutionRole-20181011T002316',\n",
       "  'InputDataConfig': [{'ChannelName': 'train',\n",
       "    'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n",
       "      'S3Uri': 's3://sagemaker-us-east-1-440180731255/boston-xgboost-LL/train.csv',\n",
       "      'S3DataDistributionType': 'FullyReplicated'}},\n",
       "    'ContentType': 'csv',\n",
       "    'CompressionType': 'None'},\n",
       "   {'ChannelName': 'validation',\n",
       "    'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n",
       "      'S3Uri': 's3://sagemaker-us-east-1-440180731255/boston-xgboost-LL/validation.csv',\n",
       "      'S3DataDistributionType': 'FullyReplicated'}},\n",
       "    'ContentType': 'csv',\n",
       "    'CompressionType': 'None'}],\n",
       "  'OutputDataConfig': {'S3OutputPath': 's3://sagemaker-us-east-1-440180731255/boston-xgboost-LL/output'},\n",
       "  'ResourceConfig': {'InstanceType': 'ml.m4.xlarge',\n",
       "   'InstanceCount': 1,\n",
       "   'VolumeSizeInGB': 5},\n",
       "  'StoppingCondition': {'MaxRuntimeInSeconds': 86400}},\n",
       " 'HyperParameterTuningJobStatus': 'Completed',\n",
       " 'CreationTime': datetime.datetime(2018, 10, 11, 6, 27, 8, 98000, tzinfo=tzlocal()),\n",
       " 'HyperParameterTuningEndTime': datetime.datetime(2018, 10, 11, 6, 49, 22, 424000, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2018, 10, 11, 6, 49, 22, 424000, tzinfo=tzlocal()),\n",
       " 'TrainingJobStatusCounters': {'Completed': 20,\n",
       "  'InProgress': 0,\n",
       "  'RetryableError': 0,\n",
       "  'NonRetryableError': 0,\n",
       "  'Stopped': 0},\n",
       " 'ObjectiveStatusCounters': {'Succeeded': 20, 'Pending': 0, 'Failed': 0},\n",
       " 'BestTrainingJob': {'TrainingJobName': 'tuning-job2018-10-11-06-27-07-007-a6563e06',\n",
       "  'TrainingJobArn': 'arn:aws:sagemaker:us-east-1:440180731255:training-job/tuning-job2018-10-11-06-27-07-007-a6563e06',\n",
       "  'CreationTime': datetime.datetime(2018, 10, 11, 6, 33, 7, tzinfo=tzlocal()),\n",
       "  'TrainingStartTime': datetime.datetime(2018, 10, 11, 6, 35, 19, tzinfo=tzlocal()),\n",
       "  'TrainingEndTime': datetime.datetime(2018, 10, 11, 6, 36, 2, tzinfo=tzlocal()),\n",
       "  'TrainingJobStatus': 'Completed',\n",
       "  'TunedHyperParameters': {'eta': '0.09351784490432244',\n",
       "   'max_depth': '12',\n",
       "   'min_child_weight': '2'},\n",
       "  'FinalHyperParameterTuningJobObjectiveMetric': {'MetricName': 'validation:rmse',\n",
       "   'Value': 2.953089952468872},\n",
       "  'ObjectiveStatus': 'Succeeded'},\n",
       " 'ResponseMetadata': {'RequestId': '623d54cb-e42f-4801-843b-892ac4b67477',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '623d54cb-e42f-4801-843b-892ac4b67477',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '4715',\n",
       "   'date': 'Thu, 11 Oct 2018 06:49:24 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.wait_for_tuning_job(tuning_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the model\n",
    "\n",
    "Now that the tuning job has finished, SageMaker has fit a number of models, the results of which are stored in a data structure which we can access using the name of the tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuning_job_info = session.sagemaker_client.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=tuning_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Among the pieces of information included in the `tuning_job_info` object is the name of the training job which performed best out of all of the models that SageMaker fit to our data. Using this training job name we can get access to the resulting model artifacts, from which we can construct a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We begin by asking SageMaker to describe for us the results of the best training job. The data\n",
    "# structure returned contains a lot more information than we currently need, try checking it out\n",
    "# yourself in more detail.\n",
    "best_training_job_name = tuning_job_info['BestTrainingJob']['TrainingJobName']\n",
    "training_job_info = session.sagemaker_client.describe_training_job(TrainingJobName=best_training_job_name)\n",
    "\n",
    "model_artifacts = training_job_info['ModelArtifacts']['S3ModelArtifacts']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just like when we created a training job, the model name must be unique\n",
    "model_name = best_training_job_name + \"-model\"\n",
    "\n",
    "# We also need to tell SageMaker which container should be used for inference and where it should\n",
    "# retrieve the model artifacts from. In our case, the xgboost container that we used for training\n",
    "# can also be used for inference.\n",
    "primary_container = {\n",
    "    \"Image\": container,\n",
    "    \"ModelDataUrl\": model_artifacts\n",
    "}\n",
    "\n",
    "# And lastly we construct the SageMaker model\n",
    "model_info = session.sagemaker_client.create_model(\n",
    "                                ModelName = model_name,\n",
    "                                ExecutionRoleArn = role,\n",
    "                                PrimaryContainer = primary_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Testing the model\n",
    "\n",
    "Now that we have fit our model to the training data, using the validation data to avoid overfitting, we can test our model. To do this we will make use of SageMaker's Batch Transform functionality. In other words, we need to set up and execute a batch transform job, similar to the way that we constructed the training job earlier.\n",
    "\n",
    "### Set up the batch transform job\n",
    "\n",
    "Just like when we were training our model, we first need to provide some information in the form of a data structure that describes the batch transform job which we wish to execute.\n",
    "\n",
    "We will only be using some of the options available here but to see some of the additional options please see the SageMaker documentation for [creating a batch transform job](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTransformJob.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just like in each of the previous steps, we need to make sure to name our job and the name should be unique.\n",
    "transform_job_name = 'boston-xgboost-batch-transform-' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "# Now we construct the data structure which will describe the batch transform job.\n",
    "transform_request = \\\n",
    "{\n",
    "    \"TransformJobName\": transform_job_name,\n",
    "    \n",
    "    # This is the name of the model that we created earlier.\n",
    "    \"ModelName\": model_name,\n",
    "    \n",
    "    # This describes how many compute instances should be used at once. If you happen to be doing a very large\n",
    "    # batch transform job it may be worth running multiple compute instances at once.\n",
    "    \"MaxConcurrentTransforms\": 1,\n",
    "    \n",
    "    # This says how big each individual request sent to the model should be, at most. One of the things that\n",
    "    # SageMaker does in the background is to split our data up into chunks so that each chunks stays under\n",
    "    # this size limit.\n",
    "    \"MaxPayloadInMB\": 6,\n",
    "    \n",
    "    # Sometimes we may want to send only a single sample to our endpoint at a time, however in this case each of\n",
    "    # the chunks that we send should contain multiple samples of our input data.\n",
    "    \"BatchStrategy\": \"MultiRecord\",\n",
    "    \n",
    "    # This next object describes where the output data should be stored. Some of the more advanced options which\n",
    "    # we don't cover here also describe how SageMaker should collect output from various batches.\n",
    "    \"TransformOutput\": {\n",
    "        \"S3OutputPath\": \"s3://{}/{}/batch-bransform/\".format(session.default_bucket(),prefix)\n",
    "    },\n",
    "    \n",
    "    # Here we describe our input data. Of course, we need to tell SageMaker where on S3 our input data is stored, in\n",
    "    # addition we need to detail the characteristics of our input data. In particular, since SageMaker may need to\n",
    "    # split our data up into chunks, it needs to know how the individual samples in our data file appear. In our\n",
    "    # case each line is its own sample and so we set the split type to 'line'. We also need to tell SageMaker what\n",
    "    # type of data is being sent, in this case csv, so that it can properly serialize the data.\n",
    "    \"TransformInput\": {\n",
    "        \"ContentType\": \"text/csv\",\n",
    "        \"SplitType\": \"Line\",\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"S3DataType\": \"S3Prefix\",\n",
    "                \"S3Uri\": test_location,\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    \n",
    "    # And lastly we tell SageMaker what sort of compute instance we would like it to use.\n",
    "    \"TransformResources\": {\n",
    "            \"InstanceType\": \"ml.m4.xlarge\",\n",
    "            \"InstanceCount\": 1\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Execute the batch transform job\n",
    "\n",
    "Now that we have created the request data structure, it is time to as SageMaker to set up and run our batch transform job. Just like in the previous steps, SageMaker performs these tasks in the background so that if we want to wait for the transform job to terminate (and ensure the job is progressing) we can ask SageMaker to wait of the transform job to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_response = session.sagemaker_client.create_transform_job(**transform_request)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".....................................!\n"
     ]
    }
   ],
   "source": [
    "transform_desc = session.wait_for_transform_job(transform_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyze the results\n",
    "\n",
    "Now that the transform job has completed, the results are stored on S3 as we requested. Since we'd like to do a bit of analysis in the notebook we can use some notebook magic to copy the resulting output from S3 and save it locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_output = \"s3://{}/{}/batch-bransform/\".format(session.default_bucket(),prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 2.3 KiB/2.3 KiB (17.9 KiB/s) with 1 file(s) remaining\r",
      "download: s3://sagemaker-us-east-1-440180731255/boston-xgboost-LL/batch-bransform/test.csv.out to ../data/boston/test.csv.out\r\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $transform_output $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how well our model works we can create a simple scatter plot between the predicted and actual values. If the model was completely accurate the resulting scatter plot would look like the line $x=y$. As we can see, our model seems to have done okay but there is room for improvement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = pd.read_csv(os.path.join(data_dir, 'test.csv.out'), header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Median Price vs Predicted Price')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X2cHFWd7/HPN5MBJoBMgODCQAgggiCQSBB9ZXUVUVhBjaIiPiyuXFnddRXRYHC5CyisUbziutd1xQdgBSE8GQEfgGtAXHZBJ+YJFngpgsAkC1EyKmSAyeR3/6jqpNPph+qZru6e7u/79ZpXuquqq07XZOpU/c45v6OIwMzMuteUVhfAzMxayxWBmVmXc0VgZtblXBGYmXU5VwRmZl3OFYGZWZdzRWA1SQpJL0pf/5uk/90GZZop6WlJPa0uS7NJekTSsenrT0v6ZhOO+RpJjzdwf/dJek2j9mcT44qgg6QXiOcl7V6yfEV6MZ810WNExIci4rMT3U+p9EKzKb24/0nSg5L+uko5Ho2InSJirNFlmShJl6W/h6clPSXpNkkH53GsiPiniPhfGct0QR5lSPcfkp5Jv/OQpC9Vq6Qj4tCIuCOv8lh9XBF0noeBUwpvJB0G9LWuOHVZExE7AS8APgV8Q9IhpRtJmtr0ktXvC+l32Rt4Eris3EaT5LtkdUT6nV8HvBv4YOkGHfZ9O4Yrgs7zHeCvit6fCvx78QaStpf0RUmPSnoiDff0Fa1fIGmtpDWSPlDy2c13lpKmS7pZ0jpJ69PXexdte4ekz0q6K73Lv7X0aaWcSCwB1gOHSJqV3nGeJulRYGnRsqnpsXaVdGla5vWSlhSV48T0qWhY0n9KOrzccdPz8MWSZd+XdGb6+lPp3W7hieV1Gb7LBuC7wEvTfZwn6TpJV0j6I/B+SVMkLZT0kKTfS7pG0q5FZXifpN+m6/6hpHznSbqi6P2fp99xWNJjkt4v6XTgPcBZ6R37Tem2e0m6Pv39PSzpo0X76Ut/1+sl/TdwVK3vWvSdHwB+VvSdH0nP3SrgGUlTS8JbPWmI66H03C6TtE+67uD0ieqp9Jy/M2s5LDtXBJ3nbuAFkl6SPpqfDFxRss3ngRcDs4EXAQPAPwJIOh74JPB64EDg2CrHmgJcCuwLzARGgP9bss27gb8G9gC2S/ddVXphfCvQD6wuWvUXwEuA48p87DvANODQ9FgXp/t6GfBt4G+A3YCvAzdK2r7MPr4LnCxJ6WenA28ArpZ0EPAR4KiI2DktwyMZvstOJBfh5UWL3wJcl36/K4GPAvPT77cXSQX41fTzhwBfA96XrtuN5Cmj3LFmAj8C/gWYQfL7XRERl6TH+UIaTnuTpCnATcBKkt//64AzJBXO7bnAAenPcSQ3FJmkZX5VyXc+BTgB6I+IjSUfOTNd/0aSp8EPABsk7QjcRvJ72SPd5l8lHZq1LJZRRPinQ35ILkzHAucAnwOOJ/lDmgoEMAsQ8AxwQNHnXgk8nL7+NrCoaN2L08++KH1/GXBBhePPBtYXvb8DOKfo/d8CP67w2dcAm4Bh4ClgBfCudN2stAz7F21fWDYV2DP97PQy+/0a8NmSZQ8Cf1FmWwGPAq9O338QWJq+fhFJiOdYoLfG7+Ey4Nn0u/wPcGPhfAPnAXeWbH8/8Lqi93sCo+l3+0fg6qJ1OwLPA8cW7e+K9PXZwPeqlOmCovdHA4+WbHM2cGn6+jfA8UXrTgcer/KdA/gjSSX2EHABMKXo/+UHyv1fLfp9vKXMPk8Gflay7OvAua3+W+u0H8frOtN3gDuB/SgJC5HcKU4DlqU3vpBcAAsNe3sBy4q2/22lg0iaRnLnfTwwPV28s6Se2NKI+z9FH9kA7FSl3GsiouzdbuqxCsv3AZ6KiPVl1u0LnCrp74uWbUfyPbcSESHpapI7zztJnmauSNf9WtIZJBfeQyXdApwZEWsqlOmLEXFOxu+xL/A9SZuKlo0BL0zLuXn7iHhG0u8r7HcfkotwFvsCe0kaLlrWQxLSofS4VPl/UORlEfHrCusq/e6gcrn3BY4uKeNUkv/f1kAODXWgiPgtSaPxG4EbSlb/jiSEc2hE9Kc/u0TSyAewluQPs2BmlUN9AjgIODoiXgC8Ol2uyh+ZkEqpch8DdpXUX2HdhUXftT8ipkXEVRX2dRXwdkn7ktw1X7/54BHfjYg/J7lABUmIrRHf4zHgL0vKuENEDFHy+0gr390q7PcxklBO1mM+XHLMnSPijen6ev4fZFEtzXGlcj8G/LSkjDtFxIcnWBYr4Yqgc50GHBMRzxQvjIhNwDeAiyXtASBpoCg2fA1JA+Yh6UXn3CrH2JmkUhlOGzerbZubiFhLEhv/17QBu1dSoVL6BvAhSUcrsaOkEyTtXGFfy4F1wDeBWyJiGEDSQZKOSdsWniX53o3quvpvwIVp5YOkGZLekq67DjgxbQTeDvgMlf9urwSOlfTOtEF2N0mz03VPAPsXbftz4I9pI25f2mD7UkmFRuFrgLPT87k3UPxE1WjfBD4r6cD0d3S4pN2Am4EXp43lvenPUZJekmNZupIrgg4VEQ9FxGCF1Z8Cfg3craTnyv8jubMnIn4EfBlYmm6ztMphvkzSNfV3JI3UP25M6cflfSRx9QdIYvlnAKTn4IMkjdjrSb7T+2vs6yqStoDvFi3bHlhE8l3/h6Tx8tMNKvs/k7Qj3CrpTyTn8ui0/PcBf5eWZW36HcoO7IqIR0meAj/BlnaWI9LV3yLpgTUsaUkaunsTSbvOw+n3+iawS7r9+SThoIeBW8k3HPMlkornVpJ2hm8BfRHxJ5LG+ncBa0jO++dJfhfWQIrwxDRmZt3MTwRmZl3OFYGZWZdzRWBm1uVcEZiZdblJMaBs9913j1mzZrW6GGZmk8qyZct+FxEzam03KSqCWbNmMThYqSekmZmVIynLiHCHhszMup0rAjOzLueKwMysy7kiMDPrcq4IzMy63KToNWRm7W/J8iEuuuVB1gyPsFd/HwuOO4j5cwZaXazMJnv5J8IVgZlN2JLlQ5x9w2pGRpPM3EPDI5x9QzLL6GS4mE728k+UQ0NmNmEX3fLg5otowcjoGBfd8mCLSlSfyV7+iXJFYGYTtmZ4pK7l7Wayl3+iHBoyswnbq7+PoTIXzb36+1pQmvo1q/xZ2yGa3V7hJwIzm7AFxx1EX2/PVsv6entYcNxBLSpRfZpR/kI7xNDwCMGWdogly4fGtV0juSIwswmbP2eAz73tMAb6+xAw0N/H59522KRpaG1G+bO2Q7SivcKhITNriPlzBibNhb+cvMuftR2iFe0VfiIwM2uCSu0NpcuzbtdIuVcEknokLZd0c/r+MkkPS1qR/szOuwxmZq2WtR2iFe0tzQgNfQy4H3hB0bIFEXFdE45tZraVVo0gLhyj1rGzbtdIuVYEkvYGTgAuBM7M81hmZrW0egRxu7aj5B0a+jJwFrCpZPmFklZJuljS9uU+KOl0SYOSBtetW5dzMc2sG0yGEcRLlg+x4LqVW3UfXXDdysnZfVTSicCTEbGsZNXZwMHAUcCuwKfKfT4iLomIuRExd8aMmlNumpnVNBlGEJ9/032MjsVWy0bHgvNvui+3Y+b5RDAPeLOkR4CrgWMkXRERayPxHHAp8PIcy2BmtlkreuTUa/2G0bqWN0JuFUFEnB0Re0fELOBdwNKIeK+kPQEkCZgP3JtXGczMik2kR86S5UPMW7SU/Rb+gHmLluYaqmm2Vgwou1LSDEDACuBDLSiDmXWh8fbIaWYjc39fL8Mj29799/f1NvQ4xRQRtbdqsblz58bg4GCri2FmXWreoqVlk9IN9Pdx18JjGnqsJcuHWHDtSkY3bbk2904RF73jiLorHUnLImJure2cYsLMrIZmNjJ33DgCM7NO0Ow0280eb+BcQ2ZmNUz2NNu1+InAzKyGVoRrmskVgZlZBu2aHqIRHBoyM+tyrgjMzLqcKwIzsy7nisDMrMu5IjAz63KuCMzMupwrAjOzLudxBGZmbabZ8yq7IjAzayOtmFfZoSEzszbSinmVXRGYmbWRVsyr7IrAzKyNtGJe5dwrAkk9kpZLujl9v5+keyT9StJiSdvlXQYzs8miFSmvm/FE8DHg/qL3nwcujogDgfXAaU0og5lZ21iyfIh5i5ay38IfMG/RUpYsH9q8bv6cAT73tsMY6O9DJNNhfu5th03eXkOS9gZOAC4EzpQk4Bjg3ekmlwPnAV/LsxxmZu0iS6+gTpuh7MvAWcCm9P1uwHBEbEzfPw6U/baSTpc0KGlw3bp1ORfTzLpJtTvyvLWiV1AtuVUEkk4EnoyIZcWLy2wa5T4fEZdExNyImDtjxoxcymhm3adwRz40PEKw5Y68WZVBK3oF1ZLnE8E84M2SHgGuJgkJfRnol1QISe0NrMmxDGZmW2n1HXkregXVkltFEBFnR8TeETELeBewNCLeA9wOvD3d7FTg+3mVwcysVKvvyFvRK6iWVqSY+BRwtaQLgOXAt1pQBjOrU7Pz3+Rlr/4+hspc9Jt1R144Z+10LhVRNkTfVubOnRuDg4OtLoZZS7TDBbi0pwskd7F5d2vMw2T4Lo36nUtaFhFza23nkcVmbazVDZsFrY6rN1Ir+unXoxW/c2cfNWtj1S7AzbxwtTqu3mjN7qdfj1b8zl0RmLWxPC/A9YQfWh1X7yZOOmdmW8mrq2G94Yd27OnSqToy6ZyZjV9eF+B6Y/7tHlfvJK2odB0aMmtjeXU1HE/4oZ3j6p2kFd1LXRGYtbk8LsCO+be3Tks6Z2ZtyDF/K+YnArMu1MjwQzsMeLOJcUVg1qUaEX7Iklvf2p8rAjPbrN67+3YZ8GYT44rAzIDx3d132ojjbuXGYjMDxpdPqB1z61v9XBGYGTC+u3v3PuoMrgjMDBjf3b1HHHcGtxGYGZDc3ZfL01+4u6/UkOwRx5NfbhWBpB2AO4Ht0+NcFxHnSroM+AvgD+mm74+IFXmVw8yyqTa2wN1EO1ueTwTPAcdExNOSeoH/kPSjdN2CiLgux2Ob2ThUurt3N9HOlltFEMkcmE+nb3vTn/afF9OsAzR6tK+7iTZXs0dr59pYLKlH0grgSeC2iLgnXXWhpFWSLpa0fYXPni5pUNLgunXr8iymWUfJY6pDdxNtnlZMVZlrRRARYxExG9gbeLmklwJnAwcDRwG7Ap+q8NlLImJuRMydMWNGnsU06yh5zC/sbqLN04r5oZvSfTQihoE7gOMjYm0kngMuBV7ejDKYdYs8wjjuJto8rQjD1WwjkCTgPcD+EfEZSTOBP4uIn9f43AxgNCKGJfUBxwKfl7RnRKxN9zsfuHfiX8PMCvKaa8DdRJujFXNFZHki+FfglcAp6fs/AV/N8Lk9gdslrQJ+QdJGcDNwpaTVwGpgd+CCukttZhU5jDO5tetUlUdHxMskLQeIiPWStqv1oYhYBcwps/yY+otpZlm1YqpDa5x2napyVFIPadfPNOSzKbcSmXWQVk3a4jDO5Nbs31+WiuArwPeAPSRdCLwdOCfXUpl1AI/GtcmiZkUQEVdKWga8DhAwPyLuz71kZpOcR+PaZJGl19ArgPsi4qvp+50lHV00OMzMymjVaFzPIWz1ytJr6GtsSRUB8Ey6zMyqaMVo3FaMSrXJL0tFoDRvEAARsQmnrzarqRXdAFsxKtUmvywVwW8kfVRSb/rzMeA3eRfMbLJrxWhcJ4ez8chyZ/8hkp5D55B0If0JcHqehTLrFM3uBph1VKrbEaxYzSeCiHgyIt4VEXtExAsj4t0R8WQzCmdm9ckSjnI7gpWq+EQg6ayI+IKkf6HMPAIR8dFcS2ZmdcsyKtXdWttfs5/YqoWGCmMFBnM7upk1XK1wlNsR2lsrBiJWrAgi4qY0tcRLI2JBLkc3s6ZrRXZLy64VT2xV2wgiYgw4Mpcjm1lLODtpe2vL+QiA5ZJuBK4lGUwGQETckFupzCw35doRXnvwDC665UE+vniFexG1WCue2LJUBLsCvweK00cH4IrAbJIqbkdwcrz2suC4g7b6fUB7zEewICJ+l1sJzKyl3IuovbTVfASS3gR8m2Q+gk3AOyPiP3MriZm1hHsRtZ9mD0Ss1lh8IfCqiNgLOAn4XD07lrSDpJ9LWinpPknnp8v3k3SPpF9JWpxltjOzTrVk+RDzFi1lv4U/YN6ipS0Z1NWK5HjWXqpVBBsj4gGANOX0znXu+zngmIg4ApgNHJ+mtP48cHFEHAisB06rv9hmk1+7jPB1LyKr1kawh6QzK72PiC9V23GasbSQvro3/QmSRud3p8svB87Daa2tC7VLbN5zHFu1iuAbbP0UUPq+pnRA2jLgRcBXgYeA4YjYmG7yOFD2f5uk00mT282cObOew5pNChONzTcyDYHnOO5u1UYWnz/RnacD0mZL6ieZ9/gl5Tar8NlLgEsA5s6dW3Ybs3ZS74V5Iv3F3eXTGinLfAQTFhHDwB3AK4B+SYUKaG9gTTPKYJan8cT7JxKb9wQ01ki5VQSSZqRPAkjqA44lSWR3O/D2dLNTge/nVQazZhnPhXkiE9e4y6c1Up5TTu4JXJ62E0wBromImyX9N3C1pAuA5cC3ciyDWVOM98I83ti8E8dZI1UbUHZmpXWQqdfQKmBOmeW/AV6etYBmlbTTLFvNvjC3Ig2Bda5qoaGd05+5wIdJevcMkExdeUj+RTOrrF364Bc0uy9+K+ZDts6lpLt/lQ2kW4GTIuJP6fudgWsj4vgmlA9Ieg0NDnp+HEssWT7EJ65ZyViZ/7sD/X3ctfCYMp/KXzs9oZgBSFoWEXNrbZeljWAm8HzR++eBWeMsl9mEFJ4EylUC0NrGUvfFt8kqS0XwHeDnkr5H0uf/rcC/51oqswrK9c4p5sZSs/rVrAgi4kJJPwJelS7664hYnm+xzMqrdsfvxlKz8cnafXQa8MeIuDQdH7BfRDycZ8HMyqnUO6dHyqWx1HF/6wY1KwJJ55L0HDoIuJQkedwVwLx8i2a2rUrdJvOqBMabxqFcBQJO7GbtKcsTwVtJxgP8EiAi1qQ9h8yarpmZMsebHbRcBbLg2pUgGB2LzcucG8jaRZaK4PmICEkBIGnHnMtkVlWzeueMd7RwuQpkdNO2vZw8HaS1iywVwTWSvk6SLO6DwAeAb+ZbLLP65BGKGe9o4Xq6sDo3kLWDLL2Gvijp9cAfSdoJ/jEibsu9ZGYZ5RWKGW8ah0oVSKVtzVqtZvZRSZ+PiNsiYkFEfDIibpP0+WYUziyLSqGYQiVQUG+a5vGmcSiXbqKcRnR3bYc5j23yyxIaej3wqZJlf1lmmVlL5BmKqbc9ohCiGhkdQ2w761Jh2UCGUFWtrquenMYapVr20Q8DfwscIGlV0aqdgf/Mu2BmWbVLKKb0wlwuCUahEqiVDynLRb5d5jy2ya9aaOi7wJtIJo55U9HPkRHxniaUzSyTZoZiqqmV/qIgy1NJloluPDmNNUrFiiAi/hARjwD/DDwVEb+NiN8Co5KOblYBzWopjuVDEn4pNX1ab+5pmrNegLM8lWS5yFfajxugrV5Zpqr8GvB00ftn0mVVSdpH0u2S7pd0n6SPpcvPkzQkaUX688bxFd1si/lzBrhr4TEM9PeVDclM225q7uGSLBfgrE8lWS7yzZ4DwTpXlopAUTRpQURsIlsj80bgExHxEpJJ6/9OUmFCm4sjYnb688O6S21WQaU76aHhkVx71SxZPsSG5zdus7x3ipg+rbfuyWOyXOQ9OY01SpYL+m8kfZQtTwF/C/ym1ociYi2wNn39J0n3k8xwZpabag3HQ8MjnLF4BWffsIpnRzdVHWRWT7K50obdgv6+Xs5786HjujBnTaXhORCsEbJUBB8CvgKcQ9Lp4SfA6fUcRNIsknxF95Akq/uIpL8CBkmeGtaX+czphePMnDmznsNZl1qyfIhnntv2rrzUyOgmoHJ3y3q7ZVZqJN5x+4mFo3yRt2apOVXlhA8g7QT8FLgwIm6Q9ELgdySVymeBPSPiA9X24akqrVildBLl7sqzKO3OOW/R0rJPFf19vey4/dRt7tD3W/iDsu0SAh5edELd5TFrlAlPVSnprIj4gqR/oUyX6Ij4aIZC9ALXA1dGxA3p554oWv8N4OZa+7HuVnzh36Wvl2ee37hN6ojtp04ZVyVQ2EexSu0MwyOjDI+MbnVcGH9OosnCczJ0vmqhofvTf8d1Ky5JwLeA+yPiS0XL90zbDyBJcX3vePZv3aE0TFO4EBcbGR0bdyVQfJzCxS3rALWR0THOWLyC/r5eenu0VUqLTum949HL3aFiRRARN6X/Xj7Ofc8D3geslrQiXfZp4BRJs0meMh4B/mac+7cukHWQViV9vVM2twnUOk7hwlYu2Vw1wyOjm3sHDW8Y7ai7Zo9e7g7VQkM3UX6UPAAR8eZqO46I/6D82B53F7XMJjJKtq+3h5OOHGDxLx7bJgFdteOU67Gz4fmNrN+w7dNIweimYNp2U1n+j28Yd3nbkUcvd4dqoaEvpv++DfgzkukpAU4huZM3y90ufb1lw0FZfO5th3HRLQ/WrARg23h+aY+dSl1Ei3XixbHT2z8sUS009FMASZ+NiFcXrbpJ0p25l8wMeH7j+MJC06f1ctEtD2aK9fdOUc14fvFTQqV9Fl8cO6WBdbxzMtjkkmVk8QxJ+xfeSNoPmJFfkcy22JAhvl9qiuDpZzdmqgT6+3q56B1HZLpIF9JYfPnk2VVH/RaeHoaGRwi2NLBOxrkCPHq5O2QZUPZx4A5JhdHEs3ADr7WxoPwcwQV9vT0TupjNnzPA4G+f4qp7HmMsgh6Jk44c2OqpoZMaWD2wrfNlmaryx5IOBA5OFz0QEc/lWyxrZ80Me0yf1lu1kbacamMks0wIU8uS5UNcv2yIsfRAYxFcv2yIufvuyvw5A25gtUkny1SV04AFwEciYiUwU9KJuZfMmqae6Q6bHfY4902HMqVc37NxKIwgnmilVWuuAKeHtskmSxvBpcDzwCvT948DF+RWImuqei/sWSZMaaT5cwb40jtn09/XO6H9NLKBs9Ydv9ND22STpSI4ICK+AIwCRMQI5ccH2CRU74W9FWGP+XMGWHHuG3ikzrw9PVIuDZy17vjdwGqTTZbG4ucl9ZEOLpN0AOA2gg5R74W9Ur/yXfp6mbdo6TbtBvWmc6617UDG9A/FDcKF/X588YqGtGlk6VLpBlabTLI8EZwL/BjYR9KVJGmoz8q1VNY09cazy4U9eqeIZ57fuE146ZwlqzOHnc5ZspqPL16x1bZnLF7BnM/cutX2WeYnLp6WcsnyIRZcu3Kr/S64duWE2jR8x2+dpmoa6jRx3N7ABpJZxgTcHRG/a07xEk5DnZ9yI2Zrda8svXOvlH6hR9rcs6ZYcdrnJcuHOP+m+6r2DCotT7XPTOudwj+97fDN284+/9ayI5P7+3pZcW5npYMwK5U1DXXN+QjSHR3ZsJKNgyuCfE20O2ilfPyVFPL0F+7Wq/X5L5g+rXerPD6V5gyA5Allpx2mMrxhtGq56m1zMJtsJjwfQZG7JR0VEb9oQLksg2anJ5hoPLtSu0GlJ4IguZA/9cxzmSoBgPUbRjlnyWpuf2Ada9IwTyWjm6LusQdm3SxLG8FrSSqDhyStkrRa0qq8C9atJmN6gnJxewGv2H96xXj+0PBIpvTQxa64+9HN52Wipk+bWHdUs06SpSL4S2B/4BjgTcCJ6b+Wg2b302+E+XMGOOnIga36FAfwy0f/wElHDjDQhgOpTjh8z1YXwaxtVKwIJO0g6QySUcXHA0MR8dvCT9NK2GUma3qC2x9Yt82d+sjoGLc/sI67Fh7TdgNPrl821NZPWWbNVK2N4HKSQWQ/I3kqOAT4WDMK1c0mU/734raMSuGaoeER9lv4gyRW1IiYTonp03p5+tmNVdsayh16MieBM2u0aqGhQyLivRHxdeDtwKvq2bGkfSTdLul+SfdJ+li6fFdJt0n6Vfrv9AmUv+NMlvQEpW0Z1QTVE8GNV+8UEVE+02jxqOJKh273pyyzZqn2RLC520VEbEyGFNRlI/CJiPilpJ2BZZJuA94P/CQiFklaCCwEPlXvzjtVuWkSWzGpSa2eSxOdS7ghVH4ye4BNETycdg+t1NW0HZ+yzFqhWkVwhKQ/pq8F9KXvBUREvKDajiNiLbA2ff0nSfcDA8BbgNekm10O3IErgq00Oz1B6UX/tQfP4PplQ5sv9IWeS4WyQXvcTY+ORcUuqsUXec+yZVZdtakqq4/jr4OkWcAc4B7ghWklQUSslbRHhc+cDpwOMHPmzEYVpetluehfefejNWPqldoymm0sgr7enpp5f6D1T1lm7armyOIJH0DaCfgpcGFE3CBpOCL6i9avj4iq7QQeWdwY5dJJ1NOGWxgRXGlfrVAYD1AYQNbf18t5bz7UF3kzso8szjKOYCKF6AWuB66MiBvSxU9I2jNdvyfwZJ5lsC3KxfXruQ0oDrcUJ17Lm4B5B+y6bbK7HvH0s1vnOXpuY/1zHJt1u9wqgjRh3beA+yPiS0WrbgROTV+fCnw/rzLY1iYa1x8aHmHWwh9szghabTL33p7GjBzokXjPK2Zy5QdfuU3Gzx23m7pNj6GR0THOv+m+hhzbrFtkyTU0XvOA9wGrJa1Il30aWARcI+k04FHgHTmWwYrUE9ef1juF6TtuX3b79RtGWXDdSmDrhu3SGPy1g49y10NPZTreFGCXdH7i4nBV6XzAxSGf/Rb+oOy+1m8Y3VxR5aXZ+aDM8pTbE0FE/EdEKCIOj4jZ6c8PI+L3EfG6iDgw/TfblcImLEsu/4KR0U3ctfCYiqGf0bHYnPai0kXxkd9nfwLZBEzbbmrZfv+VUmxU6/6ZZ0qOcvmgPr54BbMyzPls1o7yfCKwNlPuzr3SXAJTJJYsH6oaTlozPMI5S1Zv1cuouKtpvaGoWscqteC4gzhj8YoyW+fbvbVaW0u5rrZm7S73XkON4F5D+anW+6evt4cdeqdUTOk8fVpvxZz/PRI77zC14oCvcgpPH+XCUcWT2RSrNPFMpe0bIcv8C3ke3yyrtug1ZK2zZPkQ8xYtZb8a4YpC75+eMiPHR0bHiEhSOZTq7RHPjY5VvCAcVF14AAANu0lEQVSORfDM8xvLfracQt//elNsnPfmQ5uekiPLiOR2GHBnlpUrgg5UaU6Dc5asLls5zJ8zwKYKT4Z/GBnlonccQX/flvz906f1cvJR+7ChxnwCo2PBTjtM3aqnz7wDdt2m0pk+rZftp07h44tXcNEtD25OXZ1lPuBWzB+cpa3F6StsMnFoqANVyq1TOniseC7gSp/p7+tlx+2nbtMQXG2qyNJjPlxlSsjxzJncDgoN5EPDI1XPq1krOTTUxSqFJar1xil3l9s7RTzz/Mays6VlDX3UujOejBPxAJvHUDyy6AQuPnl2U59IzBrNvYY6UD3jBQoX9Kw9igoX6SzHyBKrn6wT8RRrdpJAs0ZzRdAhivvy90/rpXeKMk0MX5o2IsuArTXDI1x88mwWXLeS0bEtx5gi2KUv6UmUdZDVZJqIx6xTuSLoAKVx9vUbRuntEf19vfxhZJQpFVI1QxLuOeDsH3LK0ftwwfzDtlpX8yJdssseiXPfVD3hW63sp+AU0WbN5jaCDlAuzj46Fuy4/VQeXnRCxR5BBWMRXHH3o5yzZPVWy6t15bzolge3eeIY3RRVY/vlejNdv2yorl5CZtZ4fiLoALXi7FnbDK6657Gtngqq5fH/+DhG9FZqGC5McG9mreGKoAnyTlBW6UIfwJzP3MqzGecMKBc+qtQQWumYu/T1Mm/R0rLftRMahs06kSuCnJXG78ebi6ZcbP32B9axZniEXfp66e3RVg23BZXSQ5RTbnRxJeWmfyx0Ny2kfCj9rm4YNmtPbiPIWSP6yZeLrV9x96Ob3w+PjDI6FmTM5kBfb/lf+ylH75O5TOVG9O60w9RtKqNaYxXcMGzWen4iyFkjwiHlKpNyMvQWBeDZ0U289xUzueqexxiLZAL4cr2Gaqmnu2lhe/DcwWbtxhVBzhoRDml0DL1/Wi8XzD+s7gt/LVm+qwdfmbWfPKeq/LakJyXdW7TsPElDklakP2/M6/jtohHhkEbH0J9+dmMuk6c49GM2OeXZRnAZcHyZ5RcXz1iW4/HbQiOyY9YzsxgkieIKxyvX/lurv38tlVJctyITqJlNXG6hoYi4U9KsvPY/mZQLh9TTpbRcbL3auIDhkVF23H4qF588e1z9/aup1QvKoR+zyacVbQQfkfRXwCDwiYhY34IytNR4upSWXmBrpYEu7LM/nRC+1HjDTdV6QbkCMJucmt199GvAAcBsYC3wfyptKOl0SYOSBtetW9es8mWe2WsiJtqldMnyIZ565rma2xVmGGtk3N6Dwsw6T1Mrgoh4IiLGImIT8A3g5VW2vSQi5kbE3BkzZjSlfJVm9mp0ZTCRi+mS5UMsuHYlIzVmByv4w8hoQ+P2lZ4kPCjMbPJqamhI0p4RsTZ9+1bg3mrbN1uzwh7VulnWajsol+yt1rHqjdtXK0O5EcXuGWQ2ueXZffQq4L+AgyQ9Luk04AuSVktaBbwW+Hhexx+PZoU9KnWzfO3BM2o+kdRblnov0LWeitwzyKzz5Nlr6JQyi7+V1/EaoVm5cCqNsM3yRFLP7GPjkaUM7hlk1lk8srhIM8Me5S6mWbp6LjjuIBZcuzJzeKjeBHduDDbrPk46V6TVYY8sDbHz5wxw0TuOoL+vd/OyHbfrqfiLrDfBnRuDzbqPnwhKtDLskfWJpNIAtTMaMHjMjcFm3cdPBG1kIk8k8+ck0z2WU8/dfKufisys+RQ15rNtB3Pnzo3BwcFWF6PtlY5YhuRufiIX8rxnVzOz/EhaFhFza23nJ4JJrHQUNNDQu/lmDbAzs9ZyG8EkVSlf0efedljDJoJ3XiGz7uCKYJxaHTJpxkXaXUnNuoNDQ+PQDiGTSoPKGnmRdldSs+7gimAcGjEh/UQsWT5EpXnqG3mR9oxjZt3BoaFxyCNkUk+o6aJbHqRcXy9Rf26hajzZvFl3cEUwDo3OSVTvRDWVKpyosP1EOK+QWedzaGgcGh0yqTfUVKnCqTSgzMysGlcE49Do0bf1hpocuzezRnJoaJwaGTKpN9Tk2L2ZNZIrgjYwnkRvjt2bWaO4ImgDvsM3s1bKrSKQ9G3gRODJiHhpumxXYDEwC3gEeGdErM+rDJOJ7/DNrFXybCy+DDi+ZNlC4CcRcSDwk/R91ypNGudkbmbWCrlVBBFxJ/BUyeK3AJenry8H5ud1/HbXDmkqzMyg+d1HXxgRawHSf/eotKGk0yUNShpct25d0wrYLK1OU2FmVtC2jcURcQlwCSQT00x0f63OFlrKmT3NrF00+4ngCUl7AqT/PtmMg7ZjGMaZPc2sXTS7IrgRODV9fSrw/WYctB3DMB4dbGbtIs/uo1cBrwF2l/Q4cC6wCLhG0mnAo8A78jp+sVaHYQphqaHhEXokxiIY6O/jpCMHuP2BdW0TrjKz7pRbRRARp1RY9bq8jllJo7OF1qM0s+hYJM0dQ8MjXL9saEI5iszMGqErks61MgxTLixV0OrwlJkZtHGvoYkq7SXUqjBMrfCTewmZWat1ZEVQbqKXVoVhKoWlitebmbVSR4aG2qmXULmwVIF7CZlZO+jIJ4JW9xIqVpxZtLTXkHsJmVk76MiKoJW9hMpxZlEza2cdGRryYC0zs+w68onAE72YmWXXkRUBOBxjZpZVR4aGzMwsO1cEZmZdzhWBmVmXc0VgZtblXBGYmXU5RUx4FsjcSVoH/LbV5Zig3YHftboQbcTnYwufi635fGwx0XOxb0TMqLXRpKgIOoGkwYiY2+pytAufjy18Lrbm87FFs86FQ0NmZl3OFYGZWZdzRdA8l7S6AG3G52MLn4ut+Xxs0ZRz4TYCM7Mu5ycCM7Mu54rAzKzLuSLIgaRvS3pS0r1Fy3aVdJukX6X/Tm9lGZtF0j6Sbpd0v6T7JH0sXd6t52MHST+XtDI9H+eny/eTdE96PhZL2q7VZW0WST2Slku6OX3fzefiEUmrJa2QNJguy/1vxRVBPi4Dji9ZthD4SUQcCPwkfd8NNgKfiIiXAK8A/k7SIXTv+XgOOCYijgBmA8dLegXweeDi9HysB05rYRmb7WPA/UXvu/lcALw2ImYXjR/I/W/FFUEOIuJO4KmSxW8BLk9fXw7Mb2qhWiQi1kbEL9PXfyL5gx+ge89HRMTT6dve9CeAY4Dr0uVdcz4k7Q2cAHwzfS+69FxUkfvfiiuC5nlhRKyF5OII7NHi8jSdpFnAHOAeuvh8pKGQFcCTwG3AQ8BwRGxMN3mcpLLsBl8GzgI2pe93o3vPBSQ3BbdKWibp9HRZ7n8rHTtDmbUXSTsB1wNnRMQfkxu/7hQRY8BsSf3A94CXlNusuaVqPkknAk9GxDJJryksLrNpx5+LIvMiYo2kPYDbJD3QjIP6iaB5npC0J0D675MtLk/TSOolqQSujIgb0sVdez4KImIYuIOk7aRfUuHGbG9gTavK1UTzgDdLegS4miQk9GW681wAEBFr0n+fJLlJeDlN+FtxRdA8NwKnpq9PBb7fwrI0TRrz/RZwf0R8qWhVt56PGemTAJL6gGNJ2k1uB96ebtYV5yMizo6IvSNiFvAuYGlEvIcuPBcAknaUtHPhNfAG4F6a8LfikcU5kHQV8BqSFLJPAOcCS4BrgJnAo8A7IqK0QbnjSPpz4GfAarbEgT9N0k7QjefjcJIGvx6SG7FrIuIzkvYnuSveFVgOvDcinmtdSZsrDQ19MiJO7NZzkX7v76VvpwLfjYgLJe1Gzn8rrgjMzLqcQ0NmZl3OFYGZWZdzRWBm1uVcEZiZdTlXBGZmXc4VgU16kkLSd4reT5W0rpDNso793CFpbvr6h4X+/hMs2/vTsqyQ9N+SPlhhu7mSvjLR45mNh1NMWCd4BnippL6IGAFeDwxNZIcR8caGlCyxOCI+kqYNuE/SjRHxRGGlpKkRMQgMNvCYZpn5icA6xY9IslgCnAJcVViRjtj8tqRfpHnv35Iu75N0taRVkhYDfUWfeUTS7unrJWkSsPuKEoEh6WlJF6ZzC9wt6YXVCpimDXgI2FfSeZIukXQr8O+SXlOUj38nSZemeelXSTopXf4GSf8l6ZeSrk3zN5lNmCsC6xRXA++StANwOMnI5YJ/IElfcBTwWuCidAj/h4ENEXE4cCFwZIV9fyAijgTmAh9NR3oC7Ajcnc4tcCdQNuxTkI4c3R/4dbroSOAtEfHukk3/N/CHiDgsLdvStFI6Bzg2Il5G8vRwZrXjmWXl0JB1hIhYlaa5PgX4YcnqN5AkN/tk+n4HkuH6rwa+UvT5VRV2/1FJb01f7wMcCPweeB4otEMsIwlJlXNymmrjOeBvIuKpNPvqjWkoq9SxJLl3Ct9tfZqp8xDgrvSz2wH/VeF4ZnVxRWCd5EbgiyR5nnYrWi7gpIh4sHjj9IJaNcdKmgPnWOCVEbFB0h0kFQnAaGzJ0TJG5b+nxRHxkTLLn6l02DLlEnBbRJxSrbxm4+HQkHWSbwOfiYjVJctvAf4+zYSKpDnp8juB96TLXkoSUiq1C7A+rQQOJkkZnbdbgc0VRzpH7d3APEkvSpdNk/TiJpTFuoArAusYEfF4RPxzmVWfJZkScpWke9P3AF8DdkpDQmcBPy/z2R8DU9NtPktyQc7bBcB0SfdKWkkyh+064P3AVWlZ7gYObkJZrAs4+6iZWZfzE4GZWZdzRWBm1uVcEZiZdTlXBGZmXc4VgZlZl3NFYGbW5VwRmJl1uf8PL/ks2swz1EoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(Y_test, Y_pred)\n",
    "plt.xlabel(\"Median Price\")\n",
    "plt.ylabel(\"Predicted Price\")\n",
    "plt.title(\"Median Price vs Predicted Price\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Clean up\n",
    "\n",
    "The default notebook instance on SageMaker doesn't have a lot of excess disk space available. As you continue to complete and execute notebooks you will eventually fill up this disk space, leading to errors which can be difficult to diagnose. Once you are completely finished using a notebook it is a good idea to remove the files that you created along the way. Of course, you can do this from the terminal or from the notebook hub if you would like. The cell below contains some commands to clean up the created files from within the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we will remove all of the files contained in the data_dir directory\n",
    "!rm $data_dir/*\n",
    "\n",
    "# And then we delete the directory itself\n",
    "!rmdir $data_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}