Tutorials/Boston Housing - XGBoost (Hyperparameter Tuning) - High Level.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Boston Housing Prices\n",
    "\n",
    "## Using XGBoost in SageMaker\n",
    "\n",
    "_Deep Learning Nanodegree Program | Deployment_\n",
    "\n",
    "---\n",
    "\n",
    "As an introduction to using SageMaker's High Level Python API we will look at a relatively simple problem. Namely, we will use the [Boston Housing Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) to predict the median value of a home in the area of Boston Mass.\n",
    "\n",
    "The documentation for the high level API can be found on the [ReadTheDocs page](http://sagemaker.readthedocs.io/en/latest/)\n",
    "\n",
    "## General Outline\n",
    "\n",
    "Typically, when using a notebook instance with SageMaker, you will proceed through the following steps. Of course, not every step will need to be done with each project. Also, there is quite a lot of room for variation in many of the steps, as you will see throughout these lessons.\n",
    "\n",
    "1. Download or otherwise retrieve the data.\n",
    "2. Process / Prepare the data.\n",
    "3. Upload the processed data to S3.\n",
    "4. Train a chosen model.\n",
    "5. Test the trained model (typically using a batch transform job).\n",
    "6. Deploy the trained model.\n",
    "7. Use the deployed model.\n",
    "\n",
    "In this notebook we will only be covering steps 1 through 5 as we just want to get a feel for using SageMaker. In later notebooks we will talk about deploying a trained model in much more detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 0: Setting up the notebook\n",
    "\n",
    "We begin by setting up all of the necessary bits required to run our notebook. To start that means loading all of the Python modules we will need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import load_boston\n",
    "import sklearn.model_selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the modules above, we need to import the various bits of SageMaker that we will be using. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "from sagemaker.predictor import csv_serializer\n",
    "\n",
    "# This is an object that represents the SageMaker session that we are currently operating in. This\n",
    "# object contains some useful information that we will need to access later such as our region.\n",
    "session = sagemaker.Session()\n",
    "\n",
    "# This is an object that represents the IAM role that we are currently assigned. When we construct\n",
    "# and launch the training job later we will need to tell it what IAM role it should have. Since our\n",
    "# use case is relatively simple we will simply assign the training job the role we currently have.\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Downloading the data\n",
    "\n",
    "Fortunately, this dataset can be retrieved using sklearn and so this step is relatively straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "boston = load_boston()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Preparing and splitting the data\n",
    "\n",
    "Given that this is clean tabular data, we don't need to do any processing. However, we do need to split the rows in the dataset up into train, test and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we package up the input data and the target variable (the median value) as pandas dataframes. This\n",
    "# will make saving the data to a file a little easier later on.\n",
    "\n",
    "X_bos_pd = pd.DataFrame(boston.data, columns=boston.feature_names)\n",
    "Y_bos_pd = pd.DataFrame(boston.target)\n",
    "\n",
    "# We split the dataset into 2/3 training and 1/3 testing sets.\n",
    "X_train, X_test, Y_train, Y_test = sklearn.model_selection.train_test_split(X_bos_pd, Y_bos_pd, test_size=0.33)\n",
    "\n",
    "# Then we split the training set further into 2/3 training and 1/3 validation sets.\n",
    "X_train, X_val, Y_train, Y_val = sklearn.model_selection.train_test_split(X_train, Y_train, test_size=0.33)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Uploading the data files to S3\n",
    "\n",
    "When a training job is constructed using SageMaker, a container is executed which performs the training operation. This container is given access to data that is stored in S3. This means that we need to upload the data we want to use for training to S3. In addition, when we perform a batch transform job, SageMaker expects the input data to be stored on S3. We can use the SageMaker API to do this and hide some of the details.\n",
    "\n",
    "### Save the data locally\n",
    "\n",
    "First we need to create the test, train and validation csv files which we will then upload to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is our local data directory. We need to make sure that it exists.\n",
    "data_dir = '../data/boston'\n",
    "if not os.path.exists(data_dir):\n",
    "    os.makedirs(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We use pandas to save our test, train and validation data to csv files. Note that we make sure not to include header\n",
    "# information or an index as this is required by the built in algorithms provided by Amazon. Also, for the train and\n",
    "# validation data, it is assumed that the first entry in each row is the target variable.\n",
    "\n",
    "X_test.to_csv(os.path.join(data_dir, 'test.csv'), header=False, index=False)\n",
    "\n",
    "pd.concat([Y_val, X_val], axis=1).to_csv(os.path.join(data_dir, 'validation.csv'), header=False, index=False)\n",
    "pd.concat([Y_train, X_train], axis=1).to_csv(os.path.join(data_dir, 'train.csv'), header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload to S3\n",
    "\n",
    "Since we are currently running inside of a SageMaker session, we can use the object which represents this session to upload our data to the 'default' S3 bucket. Note that it is good practice to provide a custom prefix (essentially an S3 folder) to make sure that you don't accidentally interfere with data uploaded from some other notebook or project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Created S3 bucket: sagemaker-us-east-1-440180731255\n"
     ]
    }
   ],
   "source": [
    "prefix = 'boston-xgboost-HL'\n",
    "\n",
    "test_location = session.upload_data(os.path.join(data_dir, 'test.csv'), key_prefix=prefix)\n",
    "val_location = session.upload_data(os.path.join(data_dir, 'validation.csv'), key_prefix=prefix)\n",
    "train_location = session.upload_data(os.path.join(data_dir, 'train.csv'), key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Train the XGBoost model\n",
    "\n",
    "Now that we have the training and validation data uploaded to S3, we can construct our XGBoost model and train it. Unlike in the previous notebooks, instead of training a single model, we will use SageMakers hyperparameter tuning functionality to train multiple models and use the one that performs the best on the validation set.\n",
    "\n",
    "To begin with, as in the previous approaches, we will need to construct an estimator object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# As stated above, we use this utility method to construct the image name for the training container.\n",
    "container = get_image_uri(session.boto_region_name, 'xgboost')\n",
    "\n",
    "# Now that we know which container to use, we can construct the estimator object.\n",
    "xgb = sagemaker.estimator.Estimator(container, # The name of the training container\n",
    "                                    role,      # The IAM role to use (our current role in this case)\n",
    "                                    train_instance_count=1, # The number of instances to use for training\n",
    "                                    train_instance_type='ml.m4.xlarge', # The type of instance ot use for training\n",
    "                                    output_path='s3://{}/{}/output'.format(session.default_bucket(), prefix),\n",
    "                                                                        # Where to save the output (the model artifacts)\n",
    "                                    sagemaker_session=session) # The current SageMaker session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before beginning the hyperparameter tuning, we should make sure to set any model specific hyperparameters that we wish to have default values. There are quite a few that can be set when using the XGBoost algorithm, below are just a few of them. If you would like to change the hyperparameters below or modify additional ones you can find additional information on the [XGBoost hyperparameter page](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost_hyperparameters.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb.set_hyperparameters(max_depth=5,\n",
    "                        eta=0.2,\n",
    "                        gamma=4,\n",
    "                        min_child_weight=6,\n",
    "                        subsample=0.8,\n",
    "                        objective='reg:linear',\n",
    "                        early_stopping_rounds=10,\n",
    "                        num_round=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our estimator object completely set up, it is time to create the hyperparameter tuner. To do this we need to construct a new object which contains each of the parameters we want to SageMaker to tune. In this case, we wish to find the best values for the `max_depth`, `eta`, `min_child_weight`, `subsample`, and `gamma` parameters. Note that for each parameter that we want SageMaker to tune we need to specify both the *type* of the parameter and the *range* of values that parameter may take on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import IntegerParameter, ContinuousParameter, HyperparameterTuner\n",
    "\n",
    "xgb_hyperparameter_tuner = HyperparameterTuner(estimator = xgb, # The estimator object to use as the basis for the training jobs.\n",
    "                                               objective_metric_name = 'validation:rmse', # The metric used to compare trained models.\n",
    "                                               objective_type = 'Minimize', # Whether we wish to minimize or maximize the metric.\n",
    "                                               max_jobs = 20, # The total number of models to train\n",
    "                                               max_parallel_jobs = 3, # The number of models to train in parallel\n",
    "                                               hyperparameter_ranges = {\n",
    "                                                    'max_depth': IntegerParameter(3, 12),\n",
    "                                                    'eta'      : ContinuousParameter(0.05, 0.5),\n",
    "                                                    'min_child_weight': IntegerParameter(2, 8),\n",
    "                                                    'subsample': ContinuousParameter(0.5, 0.9),\n",
    "                                                    'gamma': ContinuousParameter(0, 10),\n",
    "                                               })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our hyperparameter tuner object completely set up, it is time to train it. To do this we make sure that SageMaker knows our input data is in csv format and then execute the `fit` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating hyperparameter tuning job with name: xgboost-181011-0556\n"
     ]
    }
   ],
   "source": [
    "s3_input_train = sagemaker.s3_input(s3_data=train_location, content_type='csv')\n",
    "s3_input_validation = sagemaker.s3_input(s3_data=val_location, content_type='csv')\n",
    "\n",
    "xgb_hyperparameter_tuner.fit({'train': s3_input_train, 'validation': s3_input_validation})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in many of the examples we have seen so far, the `fit()` method takes care of setting up and fitting a number of different models, each with different hyperparameters. If we wish to wait for this process to finish, we can call the `wait()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "......................................................................................................................................................................................................................................................................!\n"
     ]
    }
   ],
   "source": [
    "xgb_hyperparameter_tuner.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the hyperamater tuner has finished, we can retrieve information about the best performing model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xgboost-181011-0556-015-d5928e01'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb_hyperparameter_tuner.best_training_job()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, since we'd like to set up a batch transform job to test the best model, we can construct a new estimator object from the results of the best training job. The `xgb_attached` object below can now be used as though we constructed an estimator with the best performing hyperparameters and then fit it to our training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-11 06:09:36 Starting - Preparing the instances for training\n",
      "2018-10-11 06:11:28 Downloading - Downloading input data\n",
      "2018-10-11 06:11:50 Training - Training image download completed. Training in progress.\n",
      "2018-10-11 06:12:17 Uploading - Uploading generated training model\n",
      "2018-10-11 06:12:20 Completed - Training job completed\u001b[31mArguments: train\u001b[0m\n",
      "\u001b[31m[2018-10-11:06:12:16:INFO] Running standalone xgboost training.\u001b[0m\n",
      "\u001b[31m[2018-10-11:06:12:16:INFO] Setting up HPO optimized metric to be : rmse\u001b[0m\n",
      "\u001b[31m[2018-10-11:06:12:16:INFO] File size need to be processed in the node: 0.02mb. Available memory size in the node: 8401.49mb\u001b[0m\n",
      "\u001b[31m[2018-10-11:06:12:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[31m[06:12:16] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[31m[06:12:16] 227x13 matrix with 2951 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[31m[2018-10-11:06:12:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[31m[06:12:16] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[31m[06:12:16] 112x13 matrix with 1456 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[31m[0]#011train-rmse:20.7214#011validation-rmse:22.4962\u001b[0m\n",
      "\u001b[31mMultiple eval metrics have been passed: 'validation-rmse' will be used for early stopping.\n",
      "\u001b[0m\n",
      "\u001b[31mWill train until validation-rmse hasn't improved in 10 rounds.\u001b[0m\n",
      "\u001b[31m[1]#011train-rmse:18.7506#011validation-rmse:20.4491\u001b[0m\n",
      "\u001b[31m[2]#011train-rmse:16.9722#011validation-rmse:18.5527\u001b[0m\n",
      "\u001b[31m[3]#011train-rmse:15.3956#011validation-rmse:16.8875\u001b[0m\n",
      "\u001b[31m[4]#011train-rmse:13.929#011validation-rmse:15.4526\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 0 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 6 extra nodes, 0 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 8 extra nodes, 0 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 0 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 0 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[5]#011train-rmse:12.6617#011validation-rmse:14.1778\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 0 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[31m[6]#011train-rmse:11.5116#011validation-rmse:13.0015\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 0 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[31m[7]#011train-rmse:10.4742#011validation-rmse:11.9409\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 0 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[31m[8]#011train-rmse:9.55946#011validation-rmse:10.9985\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 2 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[31m[9]#011train-rmse:8.73849#011validation-rmse:10.2545\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 2 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[31m[10]#011train-rmse:7.96148#011validation-rmse:9.46405\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 0 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[11]#011train-rmse:7.26513#011validation-rmse:8.84866\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 2 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[12]#011train-rmse:6.64402#011validation-rmse:8.25591\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 2 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[13]#011train-rmse:6.09591#011validation-rmse:7.75116\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 2 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[14]#011train-rmse:5.61668#011validation-rmse:7.34941\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 6 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[15]#011train-rmse:5.17377#011validation-rmse:6.95908\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[16]#011train-rmse:4.78398#011validation-rmse:6.64512\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 4 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[17]#011train-rmse:4.42179#011validation-rmse:6.36535\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 12 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[18]#011train-rmse:4.08944#011validation-rmse:6.13031\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 2 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[19]#011train-rmse:3.78988#011validation-rmse:5.94227\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 6 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[20]#011train-rmse:3.51782#011validation-rmse:5.76643\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 4 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[21]#011train-rmse:3.30187#011validation-rmse:5.61459\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 4 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[22]#011train-rmse:3.09202#011validation-rmse:5.48295\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 12 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[23]#011train-rmse:2.90229#011validation-rmse:5.37821\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 4 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[24]#011train-rmse:2.73894#011validation-rmse:5.29406\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 2 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[25]#011train-rmse:2.58358#011validation-rmse:5.20744\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 0 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[26]#011train-rmse:2.44956#011validation-rmse:5.1761\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 16 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[27]#011train-rmse:2.31241#011validation-rmse:5.12394\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 16 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[28]#011train-rmse:2.20558#011validation-rmse:5.09225\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 10 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[29]#011train-rmse:2.11805#011validation-rmse:5.04666\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 0 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[30]#011train-rmse:2.0256#011validation-rmse:5.02305\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 8 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[31]#011train-rmse:1.93202#011validation-rmse:4.98045\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 4 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[32]#011train-rmse:1.84348#011validation-rmse:4.9736\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 10 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[33]#011train-rmse:1.78114#011validation-rmse:4.93574\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 6 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[34]#011train-rmse:1.72209#011validation-rmse:4.89967\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 12 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[35]#011train-rmse:1.66158#011validation-rmse:4.87545\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 20 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[36]#011train-rmse:1.60858#011validation-rmse:4.84253\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 16 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[37]#011train-rmse:1.55994#011validation-rmse:4.8482\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 20 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[38]#011train-rmse:1.52108#011validation-rmse:4.85287\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[39]#011train-rmse:1.48325#011validation-rmse:4.84564\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 14 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[40]#011train-rmse:1.45023#011validation-rmse:4.83341\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 18 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[41]#011train-rmse:1.41073#011validation-rmse:4.82003\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 20 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[42]#011train-rmse:1.37971#011validation-rmse:4.80769\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 10 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[43]#011train-rmse:1.34136#011validation-rmse:4.8104\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 18 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[44]#011train-rmse:1.30451#011validation-rmse:4.80989\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 20 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[45]#011train-rmse:1.28298#011validation-rmse:4.7941\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 14 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[46]#011train-rmse:1.26121#011validation-rmse:4.78348\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 22 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[47]#011train-rmse:1.24469#011validation-rmse:4.78771\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 14 pruned nodes, max_depth=8\u001b[0m\n",
      "\u001b[31m[48]#011train-rmse:1.22064#011validation-rmse:4.79331\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 24 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[49]#011train-rmse:1.19605#011validation-rmse:4.79599\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 20 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[50]#011train-rmse:1.18205#011validation-rmse:4.79698\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 18 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[51]#011train-rmse:1.16448#011validation-rmse:4.79137\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 22 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[52]#011train-rmse:1.1491#011validation-rmse:4.77641\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 24 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[53]#011train-rmse:1.12153#011validation-rmse:4.77117\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 28 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[31m[54]#011train-rmse:1.10877#011validation-rmse:4.75198\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 14 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[55]#011train-rmse:1.09472#011validation-rmse:4.75035\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 22 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[56]#011train-rmse:1.08111#011validation-rmse:4.75024\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[57]#011train-rmse:1.06185#011validation-rmse:4.76657\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 4 pruned nodes, max_depth=9\u001b[0m\n",
      "\u001b[31m[58]#011train-rmse:1.05015#011validation-rmse:4.77194\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 26 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[59]#011train-rmse:1.04229#011validation-rmse:4.76886\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 24 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[60]#011train-rmse:1.03218#011validation-rmse:4.76635\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 24 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[31m[61]#011train-rmse:1.01945#011validation-rmse:4.77187\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=7\u001b[0m\n",
      "\u001b[31m[62]#011train-rmse:1.00903#011validation-rmse:4.77419\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 16 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[63]#011train-rmse:1.00316#011validation-rmse:4.78459\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 12 pruned nodes, max_depth=6\u001b[0m\n",
      "\u001b[31m[64]#011train-rmse:0.991138#011validation-rmse:4.7847\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[65]#011train-rmse:0.981539#011validation-rmse:4.78161\u001b[0m\n",
      "\u001b[31m[06:12:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 26 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[66]#011train-rmse:0.969849#011validation-rmse:4.78588\u001b[0m\n",
      "\u001b[31mStopping. Best iteration:\u001b[0m\n",
      "\u001b[31m[56]#011train-rmse:1.08111#011validation-rmse:4.75024\n",
      "\u001b[0m\n",
      "Billable seconds: 53\n"
     ]
    }
   ],
   "source": [
    "xgb_attached = sagemaker.estimator.Estimator.attach(xgb_hyperparameter_tuner.best_training_job())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Test the model\n",
    "\n",
    "Now that we have our best performing model, we can test it. To do this we will use the batch transform functionality. To start with, we need to build a transformer object from our fit model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: xgboost-181011-0556-015-d5928e01\n"
     ]
    }
   ],
   "source": [
    "xgb_transformer = xgb_attached.transformer(instance_count = 1, instance_type = 'ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we ask SageMaker to begin a batch transform job using our trained model and applying it to the test data we previous stored in S3. We need to make sure to provide SageMaker with the type of data that we are providing to our model, in our case `text/csv`, so that it knows how to serialize our data. In addition, we need to make sure to let SageMaker know how to split our data up into chunks if the entire data set happens to be too large to send to our model all at once.\n",
    "\n",
    "Note that when we ask SageMaker to do this it will execute the batch transform job in the background. Since we need to wait for the results of this job before we can continue, we use the `wait()` method. An added benefit of this is that we get some output from our batch transform job which lets us know if anything went wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Created S3 bucket: sagemaker-us-east-1-440180731255\n",
      "INFO:sagemaker:Creating transform job with name: xgboost-181011-0556-015-d5928e01-2018-10-11-06-20-13-524\n"
     ]
    }
   ],
   "source": [
    "xgb_transformer.transform(test_location, content_type='text/csv', split_type='Line')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".....................................!\n"
     ]
    }
   ],
   "source": [
    "xgb_transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the batch transform job has finished, the resulting output is stored on S3. Since we wish to analyze the output inside of our notebook we can use a bit of notebook magic to copy the output file from its S3 location and save it locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 2.3 KiB/2.3 KiB (14.7 KiB/s) with 1 file(s) remaining\r",
      "download: s3://sagemaker-us-east-1-440180731255/xgboost-181011-0556-015-d5928e01-2018-10-11-06-20-13-524/test.csv.out to ../data/boston/test.csv.out\r\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $xgb_transformer.output_path $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how well our model works we can create a simple scatter plot between the predicted and actual values. If the model was completely accurate the resulting scatter plot would look like the line $x=y$. As we can see, our model seems to have done okay but there is room for improvement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = pd.read_csv(os.path.join(data_dir, 'test.csv.out'), header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Median Price vs Predicted Price')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X2YXHV99/H3J5sFNoBswGAhEIKKIIgQXcQ21gKiUBGJqFVqvbF6S+0TohYN1goq3ETxUmvv1hYfqU9AeYggVeA2UKwt2oTwIAUuFRDZUEDJIpCVbJLv/cc5k5zMnjNzZnfOzO7M53Vde+3MmTPn/Obszvme83v4/hQRmJlZ/5rT7QKYmVl3ORCYmfU5BwIzsz7nQGBm1uccCMzM+pwDgZlZn3MgsKYkhaTnpo//UdLfzIAyLZL0pKSBbpel0yTdL+nY9PEHJX2hA/s8StKDbdzenZKOatf2bHocCHpIeoLYKOmZdctvTU/mi6e7j4h4V0R8bLrbqZeeaLakJ/cnJN0j6Y8blOOBiNglIja3uyzTJekr6d/hSUmPSbpe0kFV7Csi/k9E/O+SZTq3ijKk2w9JT6WfeVTSpxoF6Yg4JCJurKo81hoHgt5zH3BK7YmkQ4Gh7hWnJesiYhfgGcAHgM9LOrh+JUlzO16y1n0i/Sz7AI8AX8lbaZZ8lrIOSz/zK4A/BN5Zv0KPfd6e4UDQe74K/K/M81OBf86uIGlHSZ+U9ICkh9PqnqHM62dKekjSOklvr3vv1itLSfMlfVvSo5LWp4/3yax7o6SPSfpBepV/Xf3dSp5IrATWAwdLWpxecb5D0gPAqsyyuem+dpf05bTM6yWtzJTjNeld0Zik/5D0wrz9psfhk3XLviXpvenjD6RXu7U7lleU+CwbgG8AL0i3cY6kyyR9TdKvgbdJmiNpuaSfSfqVpEsl7Z4pw1sl/Tx97a/ryneOpK9lnr8s/Yxjkn4h6W2STgPeArw/vWK/Ol13b0mXp3+/+ySdntnOUPq3Xi/pv4Ejmn3WzGe+G/h+5jPfnx6724GnJM2tq94aSKu4fpYe2zWS9k1fOyi9o3osPeZ/ULYcVp4DQe+5GXiGpOent+ZvAr5Wt87HgecBhwPPBRYCHwaQdDzwV8ArgQOAYxvsaw7wZWA/YBEwDvzfunX+EPhjYE9gh3TbDaUnxtcBw8AdmZd+D3g+cFzO274KzAMOSff16XRbLwK+BPwJsAfwT8BVknbM2cY3gDdJUvre+cCrgIslHQj8BXBEROyaluH+Ep9lF5KT8NrM4pOAy9LP93XgdGBZ+vn2JgmAf5++/2Dgc8Bb09f2ILnLyNvXIuA7wN8BC0j+vrdGxIXpfj6RVqedKGkOcDVwG8nf/xXAGZJqx/Zs4Dnpz3EkFxSlpGX+3brPfApwAjAcEZvq3vLe9PVXk9wNvh3YIGln4HqSv8ue6Tr/IOmQsmWxkiLCPz3yQ3JiOhb4EHA+cDzJF2kuEMBiQMBTwHMy7/tt4L708ZeAFZnXnpe+97np868A5xbs/3Bgfeb5jcCHMs//DPhuwXuPArYAY8BjwK3Am9PXFqdleHZm/dqyucBe6Xvn52z3c8DH6pbdA/xezroCHgBenj5/J7AqffxckiqeY4HBJn+HrwC/ST/L/wBX1Y43cA5wU936dwGvyDzfC5hIP9uHgYszr+0MbASOzWzva+njs4ArG5Tp3MzzI4EH6tY5C/hy+vhe4PjMa6cBDzb4zAH8miSI/Qw4F5iT+b98e97/aubvcVLONt8EfL9u2T8BZ3f7u9ZrP66v601fBW4C9qeuWojkSnEesCa98IXkBFhr2NsbWJNZ/+dFO5E0j+TK+3hgfrp4V0kDsa0R938yb9kA7NKg3OsiIvdqN/WLguX7Ao9FxPqc1/YDTpX0l5llO5B8zu1EREi6mOTK8yaSu5mvpa/9VNIZJCfeQyRdC7w3ItYVlOmTEfGhkp9jP+BKSVsyyzYDz0rLuXX9iHhK0q8KtrsvyUm4jP2AvSWNZZYNkFTpUL9fGvwfZLwoIn5a8FrR3w6Ky70fcGRdGeeS/H9bG7lqqAdFxM9JGo1fDVxR9/IvSapwDomI4fRnt0ga+QAeIvli1ixqsKv3AQcCR0bEM4CXp8tV/JZpKUqV+wtgd0nDBa+dl/mswxExLyK+WbCtbwJvkLQfyVXz5Vt3HvGNiHgZyQkqSKrY2vE5fgH8fl0Zd4qIUer+Hmnw3aNgu78gqcopu8/76va5a0S8On29lf+DMhqlOS4q9y+Af6sr4y4R8afTLIvVcSDoXe8AjomIp7ILI2IL8Hng05L2BJC0MFM3fClJA+bB6Unn7Ab72JUkqIyljZuN1q1MRDxEUjf+D2kD9qCkWlD6PPAuSUcqsbOkEyTtWrCttcCjwBeAayNiDEDSgZKOSdsWfkPyudvVdfUfgfPS4IOkBZJOSl+7DHhN2gi8A/BRir+3XweOlfQHaYPsHpIOT197GHh2Zt0fAb9OG3GH0gbbF0iqNQpfCpyVHs99gOwdVbt9AfiYpAPSv9ELJe0BfBt4XtpYPpj+HCHp+RWWpS85EPSoiPhZRKwuePkDwE+Bm5X0XPl/JFf2RMR3gM8Aq9J1VjXYzWdIuqb+kqSR+rvtKf2UvJWkXv1ukrr8MwDSY/BOkkbs9SSf6W1NtvVNkraAb2SW7QisIPms/0PSePnBNpX9b0naEa6T9ATJsTwyLf+dwJ+nZXko/Qy5A7si4gGSu8D3sa2d5bD05S+S9MAak7Qyrbo7kaRd5770c30B2C1d/yMk1UH3AddRbXXMp0gCz3Uk7QxfBIYi4gmSxvo3A+tIjvvHSf4W1kaK8MQ0Zmb9zHcEZmZ9zoHAzKzPORCYmfU5BwIzsz43KwaUPfOZz4zFixd3uxhmZrPKmjVrfhkRC5qtNysCweLFi1m9uqgnpJmZ5ZFUZkR4tYFA0v3AEyQDbzZFxEg68OgSklwx9wN/UJAawMzMOqATbQRHR8ThETGSPl8OfC8iDgC+lz43M7Mu6UZj8UnARenji0jS75qZWZdUHQiCZNj8mnRyDIBnpblhajli9sx7o6TTJK2WtPrRRx+tuJhmZv2r6sbipRGxLk1udr2ku8u+MZLJNC4EGBkZcR4MM7OKVBoIarnaI+IRSVcCLwEelrRXRDwkaS+SBGFmZpZauXaUC669h3Vj4+w9PMSZxx3IsiULK9tfZVVDabrfXWuPSbII/pgky2Jt2rtTgW9VVQYzs9lm5dpRzrriDkbHxglgdGycs664g5VrRyvbZ5VtBM8C/l3SbSS5z6+JiO+SpPJ9paSfkMyLu6LCMpiZzSoXXHsP4xPbT3UxPrGZC669p7J9VlY1FBH3si0Xenb5r0gmyjYzszrrxsZbWt4OzjVkZjaDDM8bbGl5OzgQmJnNIEVzhVU5h5gDgZnZDDI2PtHS8nZwIDAzm0EGpJaWt4MDgZnZDLK5oA6oaHk7OBCYmc0gC4eHWlreDg4EZmYzyJnHHcjQ4MB2y4YGBzjzuAMr2+esmJjGzKxf1FJJdDLFhAOBmdkMs2zJwkpP/PVcNWRm1uccCMzM+pwDgZlZn3MgMDPrcw4EZmZ9zoHAzKzPORCYmfU5BwIzsz7nQGBm1uccCMzM+pwDgZlZn3MgMDPrc046Z2ZWsZVrRzuaTbRVDgRmZhVauXaUs664g/GJzQCMjo1z1hV3AMyYYOCqITOzCl1w7T1bg0DN+MRmLrj2ni6VaDIHAjOzCq0bG29peTc4EJiZVWjvgrmGi5Z3gwOBmVmFujEHcavcWGxmVqFuzEHcKgcCM7OKdXoO4la5asjMrM85EJiZ9TkHAjOzPudAYGbW5xwIzMz6nAOBmVmfcyAwM+tzDgRmZn3OgcDMrM85EJiZ9bnKU0xIGgBWA6MR8RpJ+wMXA7sDtwBvjYiNVZfDzGy26PSMZp24I3g3cFfm+ceBT0fEAcB64B0dKIOZ2axQm9FsdGycYNuMZivXjla2z0oDgaR9gBOAL6TPBRwDXJauchGwrMoymJnNJt2Y0azqO4LPAO8HtqTP9wDGImJT+vxBIPd+R9JpklZLWv3oo49WXEwzs5mhGzOaVRYIJL0GeCQi1mQX56waee+PiAsjYiQiRhYsWFBJGc3MZppuzGhWZWPxUuC1kl4N7AQ8g+QOYVjS3PSuYB9gXYVlMLNp6nTD5XTMprIWOfO4Aznriju2qx6qekazyu4IIuKsiNgnIhYDbwZWRcRbgBuAN6SrnQp8q6oymNn0dKPhcqpmU1kbWbZkIeeffCgLh4cQsHB4iPNPPrTSgNaNGco+AFws6VxgLfDFLpTBzEpo1HA50660Z1NZm+n0jGYdCQQRcSNwY/r4XuAlndivmU1PNxoup2o2lXWm8chiMyvUjYbLqZpNZZ1pHAjMZpmVa0dZumIV+y+/hqUrVlVaB37mcQcyNDiw3bKqGy7LyDsGM7Wss4EDgdks0ukG0W40XDZTdAyAGVfW2UIRud34Z5SRkZFYvXp1t4th1nVLV6xiNKfOe+HwED9Yfsyk5b3QnbJeq8egn0laExEjzdbrRq8hM5uiVhpEa1fOtZ402Svn2RwM3Cjcfq4aMptFWmkQ7UbOmk5wo3D7ORCYzSKtNIj26pWzG4Xbz1VDZrNIrUqnTL3/3sNDuXXp2SvnbBvCbkODSDC2YWJGtye0cgysHDcWm/Wo+jYCSK6caz1p8l7Pyq5rs1PZxmJXDZn1qGZdP/PaELJ6oT3BynHVkFkPa5SzpkxbwWxvT7ByfEdg1qfK9LJxT5z+4EBg1qfyet9kuSdO/3DVkFmfqu99M1t6DVn7NQ0E6YTzbwGeHREflbQI+K2I+FHlpTOzhopSSLSy3GkZrGn3UUmfI5l8/piIeL6k+cB1EXFEJwoI7j5qlqeoe+jrX7yQy9eMll5+/smHAu6X34vKdh8tEwhuiYgXSVobEUvSZbdFxGFtKmtTDgRmkxUlXxuQ2JzzvS5aPn/eIL+Z2FI43sBmr3aOI5iQNABEuuEFJHcIZtZFRV078072jZav3zDRkzmJrLwyjcWfBa4E9pR0HsnE8x+qtFRm1lRRColW7wiKjI6Ns3TFKlcX9YGmgSAivi5pDfAKQMCyiLir8pKZ2ST1uYEGB8TE5m0n96m0Eew4dw5j4xOT9iXYGmh6JYW15SvTa+ilwJ0R8ffp810lHRkRP6y8dGa2VX3j8Nj4BINzxPx5g5O6fI7st3tu42/ecmBSo7NI64IzatVFDgTV6/SEQmWqhj4HvCjz/KmcZWZWsbzcQBNbgnk7zGXth1+13fKi1BKNUk5kTzx5VU7glBOd0I0JhcoEAkWma1FEbJHkgWhmbdDKlV+V8wvUB4iiHklOOVG9RhMKVRUIyvQaulfS6ZIG0593A/dWUhqzPtLqRPSdnJnLk790TzcmFCoTCN4F/A4wCjwIHAmcVlmJzPpEq1NJdvLk3CyFtVWnG1Nxluk19Ajw5spKYNZHslVBRR05i678Oj0zV6P2BKvOmccdmDtivMq7scJAIOn9EfEJSX/H5A4ERMTplZXKrAc1mxGsptGVn0/Ova8bU3E2uiOojRVwbgezNmg2Ixi4Ht4SnQ74hYEgIq5OU0u8ICLO7FiJzGahMr1/GjX2CTx617qmYWNxRGwGXtyhspjNSmV7/xRV+QxIHSilWbEyvYbWSrpK0lslnVz7qbxkZrNE2d4/RTOCbY4o1X3UrCplBobtDvwKyM5eEcAVlZTIbJYp2++7vhFwTk4SuEYDhzqddsD6R5lAcGZE/LLykpjNUM1OwEUpGfKqgrKNgPsvvyZ3f3mBpRtpB6x/FFYNSTpR0qPA7ZIelPQ7HSyX2YxQpv5/qgO9Whk41OrgM7NWNGojOA/43YjYG3g9cH5nimQ2c5Q5AU91FG4rAaQbaQesfzSqGtoUEXcDRMQPJe3aoTKZzRit1P+3WkXTysChVqqfzFrVKBDsKem9Rc8j4lPVFctsZqj6BNwogGTbJobnDTI4R0xs2X4SGg8+s3ZoVDX0eWDXzE/9c7Oe160snPVtE+s3TGwXBIaHBp0Eztqm0cjij3SyIGYzUbvzvpTtAtosHcXTm7ZMaf9meSqbYEbSTsBNwI7pfi6LiLMl7Q9cTDI+4RbgrRGxsapymGVNpS9+u/K+tNIFtFkjsKeNtHYqM7J4qp4GjomIw4DDgePT+Y8/Dnw6Ig4A1gPvqLAMZlu1OhFMu7XSBbRMG4R7DFm7VBYIIvFk+nQw/QmSEcqXpcsvApZVVQazrG70xV+5dpSlK1ax//JrWpoHuCgdRZZ7DFm7NJqP4L1Fr0G5XkNp9tI1wHOBvwd+BoxFxKZ0lQeB3HtbSaeRzoS2aNGiZrsya6rTffGnM/9Atm1idGwcsf2kIO4xZO3UqI2g1jPoQOAI4Kr0+Ykkdf9NpdlLD5c0DFwJPD9vtYL3XghcCDAyMlI0mZNZaZ3oi59tg8jLJVSv0Qk92zbhPENWpaa9hiRdB7woIp5In58D/EsrO4mIMUk3Ai8FhiXNTe8K9gHWTa3oZq2pegrA+juARkGg1fkHPDOZValMr6FFQLZXz0ZgcbM3SVoATKRBYAg4lqSh+AbgDSQ9h04FvtVimc2mpGxX0KlefZeZgQySFBQ/WH5M0/XMOqVMIPgq8CNJV5JU47wO+OcS79sLuChtJ5gDXBoR35b038DFks4F1gJfnFrRzVrX7Mp6Olk+y7Q1uG7fZqKmgSAizpP0HeB300V/HBFrS7zvdmBJzvJ7gZe0WlCzTmjUs6hZIChqgxiQ2BLhun2bscoOKJsH/DoivixpgaT9I+K+KgtmVoVm1T7T6VlU1AbhVBA20zUNBJLOBkZIeg99mWQ8wNeApdUWzay9ylT7TKdnUbvTUZh1Spk7gteRVPHcAhAR65yS2majMtU+0+1Z5N49NhuVCQQbIyIkBYCknSsuk1klylT7+Kre+lGZQHCppH8i6f//TuDtwBeqLZZZ+5Wt9vFVvfWbprmGIuKTJLmBLidpJ/hwRHy26oKZtVu35hYwm+nKNBZ/PCI+AFyfs8xs1nC1j1m+MtlHX5mz7PfbXRAzM+uORtlH/xT4M+A5km7PvLQr8B9VF8ysXWpjB+qzeLYyatiskzqdZLBR1dA3gO8A5wPLM8ufiIjHKiuRWZ3pfCnqxw7Up4ErO2rY2T+tU6aT5mSqFE3S5Kazit2ZyT66K3BwRPywkhLlGBkZidWrV3dqdzaD5OX0HxwQO+8wl8fHJ5qelJeuWFU4IUyNgPtWnNBSGRqNGHbQsOko+p+dSrJCSWsiYqTZemXaCD4HPJl5/lS6zKxyeYPAJjYHY+MTpaabLJMaotmo4VZmNuv2dJg2+3V6AiUoFwgUmduGiNhChZPeW3/ITuG4dMWqaZ3IG0032ewkX6b7aCtfzG5Mh2m9peh/tsqpScuc0O+VdDrb7gL+DLi3shJZT8pWl+w2NMhTGzcxsTm5vmhUB1o0CKxe0ck6L2VErcF4Yclqm7ID0VauHW1pXmKzPFVPoJSnzB3Bu4DfAUZJ5hg+knQuYbMy6qtLxsYntgaBmqKr5jKTuEPx1dKyJQs5/+RDWTg8hEhO/p9+0+Hcv+IEfrD8mFJ192UGotU+Y6vlM6uX9z9bdQbbpo3FM4Ebi2e3Mg22UNxou3LtKGdccmvh+wbniF12msvYhuaNx1PVrAG40Wd0KmrrlrKNxY3GEbw/Ij4h6e/ImWA+Ik6fZhmtT5StFqldNdefdI8+aMF2/f+zJECwfsMEUF1Xu2b5hxp9RgcBm+katRHclf72pbhNS9l6/qMPWpDbh/rrNz+QHwSA3XYaZGx8YrvlZccGtFPRZ1w4POQgYDNeYSCIiKvT3xd1rjjWi/Iav/LccPej3HD3o5PWK6q8DODxuiBQ0+nG2W408Jm1S6Oqoasp/g4SEa+tpETWc+qTvRX9U5W5a8hamFYlTXVGsXZyQjubzRpVDX0y/X0y8Fsk01MCnALcX2GZrAdl69jLNh43Mjggnnp6E2PjE5PaD7p1Je55DGy2alQ19G8Akj4WES/PvHS1pJsqL5n1rLJVRUXmzxvkyd9s2to2EDQfG5DX6wd8BW8G5QaULZD07Ii4F0DS/sCCaotlvSxbjTKVO4N5O8zd2kuophYE8nKx5DVAn3nZbRAwsaX5oDazXldmQNl7gBsl3SjpRuAG4IxKS2UzTtmUEGUtW7KQHyw/Zms9f1kDUsu5WIryFdWCQI1TQVi/KjNV5XeBA4B3pz8HRsS1VRfMZo4qE6nljdodnKPC9TdHsNvQYO5rRctb6UHkVBDWj8pMVTkPeC+wX0S8U9IBkg6MiG9XXzybCRolUmu1GqU+55CUbGtAYnPE1jr+j1x956TqH0iqfzZs3JS7bRXEj7LjGGrrmvWbMlVDXwY2Ar+dPn8QOLeyEtmMkK0KalcitbycQ7WT/eaIrb19li1ZyNknHjLpTkEkg87GcgIEULg8965jQJPuPNzv3/pVmcbi50TEmySdAhAR41LRtZf1gryJWPK0evWcd2eRNT6xmXOuunPrHcO8HbY/eQdw+ZpR5u0wwFMbJ29neF5+1VBRH/+8ZW4otn5UJhBslDRE2lVb0nOApystlXVVsxM2TO3qucwdxNj4xNZuoXkn+/GJzRRdhTTKn1jUx98nfrNygeBs4LvAvpK+DiwF3lZloay7Gp2wBVO6el65dpTCzHEtKtpEUboJM2usYSBIq4DuJhld/FKSr/K7I+KXHSibdUj9YKvheYOFDbW1fvq1NoQy1Sq1qqZ2ZTyvNSzXc0Ov2dQ0bCxOp6hcGRG/iohrIuLbDgK9Ja9r6JO/2cTgQHFDat573nPJrSwuGGNQpqqprKHBAU45ct+mE8WYWXllqoZulnRERPxX5aWxjssdbLUlGB4aZOcd5+Ze8ee9p3Z9njdCdzp98wcHxM47zOXx8e0nnRnZb3c39FrPajYRUruVCQRHA++SdD/wFGlNb0S8sLJSWccUnaQfH5/g1rNf1dJ7aurHGLTSj7/em47Yl3OXHTppuRO8Wa/KS4lSdfqTMuMIfh94NnAMcCLwmvS3VazdaR3yFNWrN6pvL1MXnw0WZecdznPN7Q9N6X1ms1WjAZxVKQwEknaSdAZwJnA8MBoRP6/9VFYiA6pN65BVZmL2vPc0u4KoDxY7zs1/R7MBKes3TFQSAM1mqlZzabVDo6qhi4AJ4PskdwUHk+Qasg4ok9ahHfWIZSZUqd/P4j2G2NJgm4NztDWQfGjlHZOmmqz1Iq31/mnWqzQ7yMztAdbriqpSq+wVpyjo0yfpjog4NH08F/hRRLyospI0MDIyEqtX99fUyfsvv6Zwnt77VpyQO/p3aHCg7ROllx1lnDV/3iBrP/wqVq4d5T2X3Fr4OabTm3R4aJBzXnuIA4L1nHZ+tyWtiYiRZus1usPf2pE8IvKzfDUuwL6SbpB0l6Q7Jb07Xb67pOsl/ST9Pb/VbfeDZnX3napHnErXz/UbJth/+TW879LbGs43PB1j4xOVVJWZdduyJQs5/+RDWTg8hEjG77T7Aq9eo6qhwyT9On0sYCh9Xus19Iwm294EvC8ibpG0K7BG0vUko5K/FxErJC0HlgMfmNan6EHNJkPvVD3iVLcXkDvoq52mmgHVbKbrdK+4RlNVTq2bx7b3PwQ8lD5+QtJdwELgJOCodLWLgBtxIJikWd39VOoRG7UpFL02na6f01VLOZ03yrnG8weYTV+ZcQTTJmkxsAT4IfCsNEgQEQ9J2rPgPacBpwEsWrSoE8WccRpdFTS7Y6jXqG8ykPva6p8/xlNPt1wr2Da1z9KojcJpJcymr7CxuG07kHYB/g04LyKukDQWEcOZ19dHRMN2gn5sLC6jlV5DS1esyr2yL8rbMxPUchutXDuaO1FNFY3jZr2kbGNxpXcEkgaBy4GvR8QV6eKHJe2V3g3sBTxSZRl6WSv1iEVVKDM1CMC2Mtc+Z6eG3Xd6eL9Zt1UWCNLMpV8E7oqIT2Veugo4FViR/v5WVWXoN/VXztkullXV9c+fN8i8HbblJDr6oAV87eYH2rLt+mqfTjSgdWN4v1m3VVY1JOllJIPR7oCt448+SNJOcCmwCHgAeGNEPNZoW64aam7l2lHOvOw2JjZv//ecA+yWppVu03QAWw3OERe88bBJJ8jFy69peVv1ZetWtU9RFVo2BbfZbNH1qqGI+HeKMwi8oqr99qsLrr1nUhCAJALX7hCCyaN6pyXnrzvVfv1BcrLtdnVMN4b3m3VbR3oNWfXKnqhqJ9wzjzuQM//lNia2TD0YTGyOSSkvsj2RWjFTrri7MbzfrNvKZB+1WWC3ofyJ2/PUTnS77DT964BsAJrqBDQzaVKZqSThM5vtfEfQIyY2N0oDN1mZ/EFDgwNN18leKZe9K5mjJHCNbdg22QxQeurLKpVJwmfWaxwIesDKtaM8tbG1K/Hxic0N2wnmzxvk7BMP4X2X3la4joCjD1qw9XnZnkkDEmefeMikKqWZ0lPHk95Yv3HVUA+YaqK5zRGT5iaueXzDBB+5+s6GDcoBXL5mdGsDcV61St7WJ7bEdmXuxkQcZraNA0EXtHvmsan2aFk4PMQFbzgM5Zyts72NGsmesPOyJhaFkWyZi8o/OjZe6exsZpZw1VCHVVENMpXBYrUG0GVLFvKeS26d0n5rsify+mqVon752baFRuXPzs5W276ZtZfvCDqs1WqQMncPZecEVvpTn998ul0jG72/TC+cMuV3VZFZdXxH0GGtDFhq5e5hx7lztq638w4DjG/cPGk6ySBJO1GffnrDxullGM02GNcr0wunfp0y1Ulm1j4OBB3WyoClZvMWF2Xl3BKw0+AcNkxM7lJam9mrJq8b6dDgHDZtie1GKjdKT3H5mlFG9tu9sNqmTC+c7DplqpPMrH1cNdRhedUgg3PE2IaNLF5+DYuXX8PhH7mOlWtHG9491O4W8hp0xyc25waB7OsXXHtP4QCw3XfekQvecNjWRt/58wYb5ihqd7WNB3WZdVbl8xG0Q68lncumOd5taJAnnt7E5pxUDzvOncPTmyaf0GsZP6eTTbTWUahoYvn7VpywtaxlBp9l39MOTgVtNn1lk84jJQeuAAAKrUlEQVQ5EEzTdE9YRdUgjQwPDfL4+MS0MokuTKtZiiar2RLB3iWmisxubybkCjKzbcoGAlcNTUPtank0beCsNea20ud9Kg2gj49PTKu+XCTVL0W9dTZHbP08ZYKAq23MZjcHgmlox4jYqZzQa3ceZbqM5gm2Nc5mB4AN5I0sKzAg5XZFNbPZx72GpqHV3PX1bQNSudG7WdmBYLCty+XwvEGe/M2m7dJKF/X0WZgJPtneOvuXnFDGcwWb9RYHgmlopStofaPr2Hi5ALDzDgMMz9shtw2ivltmfXvF0Qct4PI1o9vdtYikymfpilWT2jOKPs/Q4Bx233lHzxVs1qMcCKbhzOMOnNSjpqi+fKq5+jds3MydHy3XCJvXX39kv9254Np7GB0b3+4OIW9wWtFkNZu2ROUn5ZmWgdSsn7iNYBrykqwVVZlMtavndAdRLVuykB8sPyY3AVx9e8ayJQtzJ6upzURWJWcgNese3xFMU9nc9VOZOL6dvXHKtmeMFbRZVJ3ewXMFm3WP7wg6pJUgUEVvnKI7i/rlZddrt27t18wcCCoxnfkGWunC2YqyaRu6ld7BaSXMusdVQ21W1Oi58w4DpaaTrM0I1u7G0rJz8XZrzl7PFWzWPU4x0QbZbo9zCuYBHh4aLN1lNMupG8xsqpxiokPq00wUzfH7+PgEf/TSRS1v342lZlY1Vw2V0GigU9nxAXsPD3HuskO58pbRUlVE2feZmVXJgaCJZgOdylyxi2QWr5VrR9mYk1a6iBtLzawTXDXURLOBTmWu2INkFq+PXH3npFG79Wp9hpzMzcw6xXcETTQb6JSXZiJv8Nj4xOaGVUgC95Qxs65wIGiiWWK5vG6PraaTcM8gM+smB4ImyiSWq08zUTTr2PDQIE9v2lIqSZ2ZWae4jaCJVhLL1RSNkj3ntYe0vC0zs6p5QFlFnFvfzLqt7IAyVw1VpGxWUjOzbnMgmAJf7ZtZL3EgaJFn0jKzXuPG4hZ5Ji0z6zUOBC3yTFpm1mscCFrkmbTMrNdUFggkfUnSI5J+nFm2u6TrJf0k/T2/qv1nTWfGsHqeScvMek2VdwRfAY6vW7Yc+F5EHAB8L31emZVrR1ny0es445Jbt84XUGvcnWowmMoAMzOzmazSAWWSFgPfjogXpM/vAY6KiIck7QXcGBFNL6WnMqCsvndPPef3MbNeN1MHlD0rIh4CSIPBnkUrSjoNOA1g0aLWZ/ZqNmHMurFxjwcwM2MGNxZHxIURMRIRIwsWLGj5/c168ew2NLjdFJPTrTIyM5utOh0IHk6rhEh/P1LVjhr14hkaHEDC4wHMzOh8ILgKODV9fCrwrap2lNe7B5JU0OeffChjGyZy3+fxAGbWbyprI5D0TeAo4JmSHgTOBlYAl0p6B/AA8Maq9p83YUz9pPONJpwxM+sXlQWCiDil4KVXVLXPeo0ygJaZcMbMrB/0bdK5ZncMZmb9om8DAXjOADMzmMHdR83MrDN69o7Ag8XMzMrpyUDgyWPMzMrryaohTx5jZlZeTwYCTx5jZlZeTwYCTx5jZlZeTwYCTx5jZlZeTzYWe7CYmVl5PRkIwIPFzMzK6smqITMzK8+BwMyszzkQmJn1OQcCM7M+50BgZtbnFBHdLkNTkh4Fft7BXT4T+GUH9zfb+Pg052PUnI9Rc9M9RvtFxIJmK82KQNBpklZHxEi3yzFT+fg052PUnI9Rc506Rq4aMjPrcw4EZmZ9zoEg34XdLsAM5+PTnI9Rcz5GzXXkGLmNwMysz/mOwMyszzkQmJn1ub4OBJK+JOkRST/OLNtd0vWSfpL+nt/NMnabpH0l3SDpLkl3Snp3utzHKSVpJ0k/knRbeow+ki7fX9IP02N0iaQdul3WbpM0IGmtpG+nz32MMiTdL+kOSbdKWp0uq/y71teBAPgKcHzdsuXA9yLiAOB76fN+tgl4X0Q8H3gp8OeSDsbHKetp4JiIOAw4HDhe0kuBjwOfTo/ReuAdXSzjTPFu4K7Mcx+jyY6OiMMz4wcq/671dSCIiJuAx+oWnwRclD6+CFjW0ULNMBHxUETckj5+guRLvBAfp60i8WT6dDD9CeAY4LJ0eV8fIwBJ+wAnAF9InwsfozIq/671dSAo8KyIeAiSkyCwZ5fLM2NIWgwsAX6Ij9N20iqPW4FHgOuBnwFjEbEpXeVBkgDazz4DvB/Ykj7fAx+jegFcJ2mNpNPSZZV/13p2hjJrL0m7AJcDZ0TEr5OLOauJiM3A4ZKGgSuB5+et1tlSzRySXgM8EhFrJB1VW5yzat8eo9TSiFgnaU/gekl3d2KnviOY7GFJewGkvx/pcnm6TtIgSRD4ekRckS72ccoREWPAjSTtKcOSahdb+wDrulWuGWAp8FpJ9wMXk1QJfQYfo+1ExLr09yMkFxQvoQPfNQeCya4CTk0fnwp8q4tl6bq0HveLwF0R8anMSz5OKUkL0jsBJA0Bx5K0pdwAvCFdra+PUUScFRH7RMRi4M3Aqoh4Cz5GW0naWdKutcfAq4Af04HvWl+PLJb0TeAoklSvDwNnAyuBS4FFwAPAGyOivkG5b0h6GfB94A621e1+kKSdwMcJkPRCkka8AZKLq0sj4qOSnk1y9bs7sBb4o4h4unslnRnSqqG/iojX+Bhtkx6LK9Onc4FvRMR5kvag4u9aXwcCMzNz1ZCZWd9zIDAz63MOBGZmfc6BwMyszzkQmJn1OQcCm/UkhaSvZp7PlfRoLcNlC9u5UdJI+vhfa2MDplm2t6VluVXSf0t6Z8F6I5I+O939mU2FU0xYL3gKeIGkoYgYB14JjE5ngxHx6raULHFJRPxFmjbgTklXRcTDtRclzY2I1cDqNu7TrDTfEViv+A5JZkuAU4Bv1l5IR2x+SdJ/pbnwT0qXD0m6WNLtki4BhjLvuV/SM9PHK9MkYHdmEoEh6UlJ56XzENws6VmNCpimDfgZsJ+kcyRdKOk64J8lHZXJ0b+LpC+neelvl/T6dPmrJP2npFsk/Uua/8ls2hwIrFdcDLxZ0k7AC0lGPtf8NUlKgyOAo4EL0iH8fwpsiIgXAucBLy7Y9tsj4sXACHB6OtITYGfg5nQegpuA3GqfmnTk6LOBn6aLXgycFBF/WLfq3wCPR8ShadlWpUHpQ8CxEfEikruH9zban1lZrhqynhARt6dpsk8B/rXu5VeRJDz7q/T5TiTD9V8OfDbz/tsLNn+6pNelj/cFDgB+BWwEau0Qa0iqpPK8KU3V8TTwJxHxWJq99aq0KqvesST5eGqfbX2avfNg4Afpe3cA/rNgf2YtcSCwXnIV8EmS/FF7ZJYLeH1E3JNdOT2hNsyxkubFORb47YjYIOlGkkACMBHbcrRspvj7dElE/EXO8qeKdptTLgHXR8QpjcprNhWuGrJe8iXgoxFxR93ya4G/TDOpImlJuvwm4C3psheQVCnV2w1YnwaBg0jSS1ftOmBr4EjnqL0ZWCrpuemyeZKe14GyWB9wILCeEREPRsTf5rz0MZLpI2+X9OP0OcDngF3SKqH3Az/Kee93gbnpOh8jOSFX7VxgvqQfS7qNZA7bR4G3Ad9My3IzcFAHymJ9wNlHzcz6nO8IzMz6nAOBmVmfcyAwM+tzDgRmZn3OgcDMrM85EJiZ9TkHAjOzPvf/ATg0PHOM/4//AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(Y_test, Y_pred)\n",
    "plt.xlabel(\"Median Price\")\n",
    "plt.ylabel(\"Predicted Price\")\n",
    "plt.title(\"Median Price vs Predicted Price\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Clean up\n",
    "\n",
    "The default notebook instance on SageMaker doesn't have a lot of excess disk space available. As you continue to complete and execute notebooks you will eventually fill up this disk space, leading to errors which can be difficult to diagnose. Once you are completely finished using a notebook it is a good idea to remove the files that you created along the way. Of course, you can do this from the terminal or from the notebook hub if you would like. The cell below contains some commands to clean up the created files from within the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we will remove all of the files contained in the data_dir directory\n",
    "!rm $data_dir/*\n",
    "\n",
    "# And then we delete the directory itself\n",
    "!rmdir $data_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}