hw11.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/rzl-ds/gu511_hw/blob/master/hw11.ipynb\" target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises due by EOD 2020.12.04"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## goal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in this homework assignment we will work with the `tensorflow` and `keras` deep learning frameworks and will look at the acceleration capable when using `gpu`s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## method of delivery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*as mentioned in our first lecture, the method of delivery may change from assignment to assignment. we will include this section in every assignment to provide an overview of how we expect homework results to be submitted, and to provide background notes or explanations for \"new\" delivery concepts or methods.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this week you will be submitting the results of your homework via upload to your `s3` submission bucket\n",
    "\n",
    "summary:\n",
    "\n",
    "| exercise | deliverable | method of delivery | points |\n",
    "|----------|-------------|--------------------|--------|\n",
    "| 1 | a file `load_train_positions.py` | uploaded to your `s3` homework bucket | 10 |\n",
    "| 2 | a file `boston_keras.py` | uploaded to your `s3` homework bucket | 15 |\n",
    "| 3 | a commit which resolves a `github` issue and a `merge`d pull request | will be seen on `github` | 10 |\n",
    "\n",
    "there is also a completely optional, ungraded exercise #4 for anyone interested in using a walkthrough of spinning up and utilizing GPU `ec2` instance \n",
    "\n",
    "total points: 35"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border: 1px solid lightgrey;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exercise 1: load a `csv` as a `tensorflow` `dataset`\n",
    "\n",
    "let's use the `tensorflow` `dataset` `api` to process a large `csv` file as a tensor and do some simple calculations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1: acquire the `csv`\n",
    "\n",
    "we will use the `1GB` `train_positions.csv` file I have made publically available on `s3`. download it to your `/tmp` directory on your `ubuntu` `ec2` server with the command\n",
    "\n",
    "```sh\n",
    "# the -P /tmp will save the resulting file in the /tmp directory\n",
    "wget https://s3.amazonaws.com/shared.rzl.gu511.com/train_positions/train_positions.csv -P /tmp\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.1: out of disk space?\n",
    "\n",
    "if in the process of downloading this file you run out of disk space, try running the following and then downloading the file again:\n",
    "\n",
    "```sh\n",
    "sudo apt-get clean\n",
    "sudo apt autoremove -y\n",
    "conda clean -y --all\n",
    "```\n",
    "\n",
    "if that *still* doesn't work, you will need to increase the size of your `ec2`'s hard disk (it's `ebs` volume) through the web console. on the `ec2` dashboard, click on the `ec2` instance with the hard drive you wish to expand, and in the bottom panel find the root device link. click on that link and a popup will show the `ebs` id link, click that link\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1mW1APVujBcS_C31Vd_kWR1Q7Ox1g1pFo\" width=\"1000px\"></div>\n",
    "\n",
    "that link will have dropped you on the `ebs` id page. right click the volume row in the top panel and choose to modify the volume. modify the disk size by adding at least 1 GB."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2: create a `CsvDataset` object\n",
    "\n",
    "in addition to the core `tensorflow` routines and `api`s, the `tensorflow` developers have a rigorous process for allowing developers to contribute new or experimental features. these new features are often saved in the `tf.contrib` namespace, but for datasets there is a special place for experimental (soon-to-be standard?) methods and classes: `tf.data.experimental`.\n",
    "\n",
    "one of the classes defined in that namespace is `tf.data.experimental.CsvDataset`. look at [the docstring](https://www.tensorflow.org/api_docs/python/tf/data/experimental/CsvDataset)\n",
    "\n",
    "```python\n",
    "help(tf.data.experimental.CsvDataset)\n",
    "```\n",
    "\n",
    "why, in 2020, is `tensorflow`'s `csv` reader function experimental? beats me!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.1: initialization arguments\n",
    "\n",
    "a quick review of [the *initialization* function documentation](https://www.tensorflow.org/api_docs/python/tf/data/experimental/CsvDataset#__init__) for this class (the one that is called to build our `CsvDataset` object)\n",
    "\n",
    "```python\n",
    "help(tf.data.experimental.CsvDataset.__init__)\n",
    "```\n",
    "\n",
    "shows us what arguments we have and gives us an idea of what we have to do to build this object.\n",
    "\n",
    "```python\n",
    "__init__(filenames,\n",
    "         record_defaults,\n",
    "         compression_type=None,\n",
    "         buffer_size=None,\n",
    "         header=False,\n",
    "         field_delim=',',\n",
    "         use_quote_delim=True,\n",
    "         na_value='',\n",
    "         select_cols=None)\n",
    "```\n",
    "\n",
    "+ `filenames`: is a single filename string, a list of filenames strings, or a `tensor` of filenames as strings\n",
    "+ `record_default`: a list of default values for incoming records\n",
    "    + each feature is represented by either a default value (e.g. '') if it *is not* required, or a `tensorflow` `dtype` if it *is* required (e.g. `tf.int32`)\n",
    "+ `header`: a `bool` indicating whether or not the file has a `header` row\n",
    "\n",
    "for example, if we had a `csv` file named `data.csv`, with columns\n",
    "\n",
    "| column name | contains `null` values | data type | suggested default value |\n",
    "|-|-|-|-|\n",
    "| `column_a` | no | an integer | `tf.int32` |\n",
    "| `column_b` | no | a string | `''` |\n",
    "\n",
    "that had no header, we would want to run\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "\n",
    "filenames = 'data.csv'\n",
    "record_defaults = [tf.int32, '']\n",
    "header = False\n",
    "\n",
    "data = tf.data.experimental.CsvDataset(\n",
    "    filenames=filenames,\n",
    "    record_defaults=record_defaults,\n",
    "    header=header)\n",
    "```\n",
    "\n",
    "we will need to specify values for those three arguments for our real `trainpositions.csv` dataset; the rest of the arguments can be left as defaults."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.1.1: `record_defaults`\n",
    "\n",
    "check the first few records of the `csv` file with\n",
    "\n",
    "```sh\n",
    "head -n20 /tmp/train_positions.csv\n",
    "```\n",
    "\n",
    "the following table summarizes the columns, whether or not they contain null values, and the suggested default value we should use in the `record_default` parameter. use this table to construct the `record_default` list\n",
    "\n",
    "| column name | contains `null` values | suggested default value |\n",
    "|-|-|-|\n",
    "| `carcount` | no | `tf.int32` |\n",
    "| `circuitid` | no | `tf.int32` |\n",
    "| `destinationstationcode` | no | `''` |\n",
    "| `directionnum` | no | `tf.int32` |\n",
    "| `linecode` | no | `''` |\n",
    "| `secondsatlocation` | no | `tf.int32` |\n",
    "| `servicetype` | no | `tf.string` |\n",
    "| `trainid` | no | `tf.string` |\n",
    "| `timestamp` | no | `tf.string` |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.2: invoking `CsvDataset`\n",
    "\n",
    "fill in the code below to create your dataset\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "\n",
    "filenames = #----------------#\n",
    "            # FILL THIS IN!! #\n",
    "            #----------------#\n",
    "record_defaults = #----------------#\n",
    "                  # FILL THIS IN!! #\n",
    "                  #----------------#\n",
    "header = #----------------#\n",
    "         # FILL THIS IN!! #\n",
    "         #----------------#\n",
    "        \n",
    "train_positions_dataset = tf.data.experimental.CsvDataset(\n",
    "    filenames=filenames,\n",
    "    record_defaults=record_defaults,\n",
    "    header=header)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3: create a `batch`ed `dataset`\n",
    "\n",
    "using your `train_positions_dataset` object's [`.batch`](https://www.tensorflow.org/api_docs/python/tf/data/experimental/CsvDataset#batch) method, create a `dataset` that has a batch size of 3 by filling in the code below\n",
    "\n",
    "```python\n",
    "BATCH_SIZE = 3\n",
    "\n",
    "# make a batched dataset\n",
    "tp_batched = #----------------#\n",
    "             # FILL THIS IN!! #\n",
    "             #----------------#\n",
    "\n",
    "# imported for assertion tests\n",
    "from tensorflow.python.data.ops.dataset_ops import BatchDataset\n",
    "assert isinstance(tp_batched, BatchDataset)\n",
    "```\n",
    "\n",
    "you can verify that this worked by executing\n",
    "\n",
    "\n",
    "```python\n",
    "for elem in tp_batched:\n",
    "    break\n",
    "\n",
    "assert elem[1].numpy().tolist() == [2009, 1912, 1480]\n",
    "assert elem[-2].numpy().tolist() == [b'067', b'175', b'182']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4: put it together\n",
    "\n",
    "fill in the following block of `python` code and save the results as `load_train_positions.py`\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "\n",
    "# imported for assertion tests\n",
    "from tensorflow.python.data.ops.dataset_ops import BatchDataset\n",
    "from tensorflow.python.data.ops.iterator_ops import Iterator\n",
    "\n",
    "BATCH_SIZE = 3\n",
    "\n",
    "def build_train_positions_dataset():\n",
    "    filenames = #----------------#\n",
    "                # FILL THIS IN!! #\n",
    "                #----------------#\n",
    "    record_defaults = #----------------#\n",
    "                      # FILL THIS IN!! #\n",
    "                      #----------------#\n",
    "    header = #----------------#\n",
    "             # FILL THIS IN!! #\n",
    "             #----------------#\n",
    "\n",
    "    train_positions_dataset = tf.data.experimental.CsvDataset(\n",
    "        filenames=filenames,\n",
    "        record_defaults=record_defaults,\n",
    "        header=header)\n",
    "\n",
    "    # make a batched dataset\n",
    "    tp_batched = #----------------#\n",
    "                 # FILL THIS IN!! #\n",
    "                 #----------------#\n",
    "\n",
    "    assert isinstance(tp_batched, BatchDataset)\n",
    "\n",
    "    return tp_batched\n",
    "\n",
    "\n",
    "def validate():\n",
    "    tp_batched = build_train_positions_dataset()\n",
    "    \n",
    "    for elem in tp_batched:\n",
    "        break\n",
    "        \n",
    "    assert elem[1].numpy().tolist() == [2009, 1912, 1480]\n",
    "    assert elem[-2].numpy().tolist() == [b'067', b'175', b'182']\n",
    "    \n",
    "    print(\"you're all good!\")\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    validate()\n",
    "```\n",
    "\n",
    "if everything works as expected, you should be able to run (from the `bash` command line)\n",
    "\n",
    "```sh\n",
    "python load_train_positions.py\n",
    "```\n",
    "\n",
    "and the result will be nothing a handful of noisy tensorflow log messages followed by the phrase `you're all good!`, e.g.\n",
    "\n",
    "```\n",
    "2020-11-20 01:20:52.942501: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory\n",
    "2020-11-20 01:20:52.942682: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
    "2020-11-20 01:20:55.932371: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
    "2020-11-20 01:20:55.932620: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)\n",
    "2020-11-20 01:20:55.932778: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ip-172-31-76-204): /proc/driver/nvidia/version does not exist\n",
    "2020-11-20 01:20:55.935850: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
    "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
    "2020-11-20 01:20:55.982617: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2400080000 Hz\n",
    "2020-11-20 01:20:55.982939: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5600e9d8eec0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
    "2020-11-20 01:20:55.983034: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
    "you're all good!\n",
    "```\n",
    "\n",
    "\n",
    "##### upload your filled-in file `load_train_positions.py` to your s3 homework bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border: 1px solid lightgrey;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exercise 2: simple `keras` models\n",
    "\n",
    "let's create a pair of simple `keras` models to predict housing prices.\n",
    "\n",
    "in the cells below, we will build out a `python` program block by block. the end result will be a full program we will save as a file `boston_keras.py`. in a `python` session, execute the code in each part of the exercise before moving on to the next part"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1: load the Boston housing price dataset\n",
    "\n",
    "`keras` provides built-in access to a number of datasets via the [`keras.datasets`](https://keras.io/datasets/#boston-housing-price-regression-dataset) module. we will use that to load train and test data in a format that is immediately consumable in a `keras` model\n",
    "\n",
    "```python\n",
    "from tensorflow import keras\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data()\n",
    "```\n",
    "\n",
    "additionaly, let's normalize the predictor data:\n",
    "\n",
    "```python\n",
    "mean = x_train.mean(axis=0)\n",
    "std = x_train.std(axis=0)\n",
    "x_train = (x_train - mean) / std\n",
    "x_test = (x_test - mean) / std\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2: a linear model\n",
    "\n",
    "\n",
    "### 2.2.1: build the model\n",
    "\n",
    "we can build a linear regression in `keras` quite easily -- a linear regression is simply a\n",
    "\n",
    "+ one-layer `Sequential` model\n",
    "+ in which the one layer\n",
    "    + is `Dense`\n",
    "    + has only one set of weights (i.e. is only one node tall, aka 1 `unit`)\n",
    "    + takes our `x` datasets as `inputs` (this defines `input_dim`)\n",
    "    + has a `linear` `activation` (this is the default `activation` value, so no argument is necessary to `Dense(...)`)\n",
    "\n",
    "fill in the below snippet to create a linear model\n",
    "\n",
    "```python\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "linear_model = #----------------#\n",
    "               # FILL THIS IN!! #\n",
    "               #----------------#\n",
    "```\n",
    "\n",
    "after doing so, you should be able to run\n",
    "\n",
    "```python\n",
    "linear_model.summary()\n",
    "```\n",
    "\n",
    "and see\n",
    "\n",
    "```\n",
    "Model: \"sequential\"\n",
    "_________________________________________________________________\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "dense (Dense)                (None, 1)                 14        \n",
    "=================================================================\n",
    "Total params: 14\n",
    "Trainable params: 14\n",
    "Non-trainable params: 0\n",
    "_________________________________________________________________\n",
    "```\n",
    "\n",
    "*(the layer name may be `dense_N` for integer `n`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.2: `compile`\n",
    "\n",
    "furthermore, we want to `compile` this model to use the `adam` optimizer algorithm to optimize a `mse` `loss` funciton. let's track the `mean_absolute_error` `metric` as well\n",
    "\n",
    "```python\n",
    "linear_model.compile(loss=,  # FILL THIS IN!!\n",
    "                     optimizer=,  # FILL THIS IN!!\n",
    "                     metrics=)  # FILL THIS IN!!\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.3: `fit`\n",
    "\n",
    "finally, let's fit our training dataset. let's use validation within each `epoch` with a `validation_split` of 0.05. also, in order to treat both model types on an equal footing, rather than stop after a fixed number of epochs we wil stop after our best `mse` value. to do this, we will use `EarlyStopping` and `ModelCheckpoint` `callback`s.\n",
    "\n",
    "```python\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "es_callback = EarlyStopping(monitor='val_loss',\n",
    "                            min_delta=0.01,\n",
    "                            patience=100)\n",
    "\n",
    "mc_callback = ModelCheckpoint('linear.hdf5',\n",
    "                              monitor='val_loss',\n",
    "                              save_best_only=True)\n",
    "\n",
    "callbacks = [es_callback, mc_callback]\n",
    "```\n",
    "\n",
    "set the `validation_split` value to 0.05, set the `verbose` value to 0, the number of `epoch`s to be 10,000, and add the `callbacks` to fit on the `x` and `y` train datasets\n",
    "\n",
    "```python\n",
    "linear_model.fit(\n",
    "    # FILL THIS IN!!\n",
    ")\n",
    "```\n",
    "\n",
    "on your `ec2` instance, this could take a minute or two, but should not take many minutes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.4: `evaluate`\n",
    "\n",
    "load the saved best dataset and view the ultimate accuracy of this model on the held-out test data:\n",
    "\n",
    "```python\n",
    "best_linear_model = keras.models.load_model('linear.hdf5')\n",
    "linear_test_mse, linear_test_mae = best_linear_model.evaluate(x_test, y_test)\n",
    "print(f\"linear test mse: {linear_test_mse}\")\n",
    "print(f\"linear test mae: {linear_test_mae}\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3: a deep neural net model\n",
    "\n",
    "let's repeat the above but with a neural network architecture. create a new `Sequential` model with the following:\n",
    "\n",
    "+ several layers\n",
    "    + one 20-node layer with `relu` activation and `input_dim` determined by the shape of `x_test`\n",
    "    + one 10-node layer with `relu` activation\n",
    "    + one 6-node layer with `relu` activation\n",
    "    + one 1-node output layer with the default activation\n",
    "+ compile with\n",
    "    + an `adam` optimizer\n",
    "    + a `mse` loss\n",
    "    + a `mean_absolute_error` metric\n",
    "+ fit with\n",
    "    + 10,000 `epochs`\n",
    "    + a `validation_split` of 0.05\n",
    "\n",
    "```python\n",
    "dnn_model = #----------------#\n",
    "            # FILL THIS IN!! #\n",
    "            #----------------#\n",
    "\n",
    "dnn_model.compile(loss=,  # FILL THIS IN!!\n",
    "                  optimizer=,  # FILL THIS IN!!\n",
    "                  metrics=)  # FILL THIS IN!!\n",
    "\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "es_callback = EarlyStopping(monitor='val_loss',\n",
    "                            min_delta=0.01,\n",
    "                            patience=100)\n",
    "\n",
    "mc_callback = ModelCheckpoint('dnn.hdf5',\n",
    "                              monitor='val_loss',\n",
    "                              save_best_only=True)\n",
    "\n",
    "callbacks = [es_callback, mc_callback]\n",
    "\n",
    "dnn_model.fit(\n",
    "    # FILL THIS IN!!\n",
    ")\n",
    "\n",
    "best_dnn_model = keras.models.load_model('dnn.hdf5')\n",
    "dnn_test_mse, dnn_test_mae = best_dnn_model.evaluate(x_test, y_test)\n",
    "print(f\"dnn test mse: {dnn_test_mse}\")\n",
    "print(f\"dnn test mae: {dnn_test_mae}\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4: bring it all together\n",
    "\n",
    "fill in all of the above in one file named `boston_keras.py`\n",
    "\n",
    "```python\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "def main():\n",
    "    # load boston data\n",
    "    (x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data()\n",
    "    \n",
    "    # standardize\n",
    "    mean = x_train.mean(axis=0)\n",
    "    std = x_train.std(axis=0)\n",
    "    x_train = (x_train - mean) / std\n",
    "    x_test = (x_test - mean) / std\n",
    "    \n",
    "    # linear model ------------------------------------------------------------\n",
    "    print('{:-<80}'.format('linear model '))\n",
    "    \n",
    "    # init\n",
    "    linear_model = #----------------#\n",
    "                   # FILL THIS IN!! #\n",
    "                   #----------------#\n",
    "\n",
    "    linear_model.summary()\n",
    "    \n",
    "    # compile\n",
    "    linear_model.compile(loss=,  # FILL THIS IN!!\n",
    "                         optimizer=,  # FILL THIS IN!!\n",
    "                         metrics=)  # FILL THIS IN!!\n",
    "    \n",
    "    # linear callbacks\n",
    "    es_callback = EarlyStopping(monitor='val_loss',\n",
    "                                min_delta=0.01,\n",
    "                                patience=1000)\n",
    "\n",
    "    mc_callback = ModelCheckpoint('linear.hdf5',\n",
    "                                  monitor='val_loss',\n",
    "                                  save_best_only=True)\n",
    "\n",
    "    callbacks = [es_callback, mc_callback]\n",
    "\n",
    "    # fit\n",
    "    print('starting training...')\n",
    "    linear_model.fit(\n",
    "        # FILL THIS IN!!\n",
    "    )\n",
    "    print('finished training...')\n",
    "    \n",
    "    # evaluate\n",
    "    best_linear_model = keras.models.load_model('linear.hdf5')\n",
    "    linear_test_mse, linear_test_mae = best_linear_model.evaluate(x_test, y_test)\n",
    "    print(f\"linear test mse: {linear_test_mse}\")\n",
    "    print(f\"linear test mae: {linear_test_mae}\")\n",
    "    \n",
    "    # dnn model ---------------------------------------------------------------\n",
    "    print('{:-<80}'.format('dnn model '))\n",
    "\n",
    "    # init\n",
    "    dnn_model = #----------------#\n",
    "                # FILL THIS IN!! #\n",
    "                #----------------#\n",
    "\n",
    "    dnn_model.summary()\n",
    "            \n",
    "    # compile\n",
    "    dnn_model.compile(loss=,  # FILL THIS IN!!\n",
    "                      optimizer=,  # FILL THIS IN!!\n",
    "                      metrics=)  # FILL THIS IN!!\n",
    "    \n",
    "    # dnn callbacks\n",
    "    es_callback = EarlyStopping(monitor='val_loss',\n",
    "                                min_delta=0.01,\n",
    "                                patience=1000)\n",
    "\n",
    "    mc_callback = ModelCheckpoint('dnn.hdf5',\n",
    "                                  monitor='val_loss',\n",
    "                                  save_best_only=True)\n",
    "\n",
    "    callbacks = [es_callback, mc_callback]\n",
    "\n",
    "    # fit\n",
    "    print('starting training...')\n",
    "    dnn_model.fit(\n",
    "        # FILL THIS IN!!\n",
    "    )\n",
    "    print('finished training...')\n",
    "\n",
    "    # evaluate\n",
    "    best_dnn_model = keras.models.load_model('dnn.hdf5')\n",
    "    dnn_test_mse, dnn_test_mae = best_dnn_model.evaluate(x_test, y_test)\n",
    "    print(f\"dnn test mse: {dnn_test_mse}\")\n",
    "    print(f\"dnn test mae: {dnn_test_mae}\")\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()\n",
    "```\n",
    "\n",
    "\n",
    "##### upload your filled-in `boston_keras.py` to your `s3` homework submission bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border: 1px solid lightgrey;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exercise 3: resolving an issue with a `commit` to `master`\n",
    "\n",
    "`github` -- *not* `git` itself -- has a concept of \"issues\". issues are a way to record \"issues\" you have with the code as it is, including feature requests, bugs, and improvements. you can view these from the \"issues\" tab on the main repository page:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1SYqdKMaNsFvWxEsyAFARir4lZkLZIQ_U\"></div>\n",
    "\n",
    "these provide a great way for people to communicate and discuss their development efforts. you should use them!\n",
    "\n",
    "`github` also has other integrations with `issues`, including (importantly) the ability to *close* issues by referencing them in commit messages. that's what we're going to do.\n",
    "\n",
    "I have already added an issue to your repositories requesting a simple change be made. the issue's title is **pin version numbers in requirements file**, and the goal is to hard-code the version numbers of the packages we want users to install and use to run the `dspipeline.py` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1: viewing and assigning the issue\n",
    "\n",
    "log in to `github` and click on the \"issues\" tab, and open the issue I created for you. in particular, I want you to **assign** it to yourself -- click on the \"assign yourself\" link on the issues page\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1AHYHy4b_A5k4CFC6IUj6Fyyx5_ICnNOz\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2: editing the `requirements.txt` file\n",
    "\n",
    "locally, edit the `requirements.txt` file to read\n",
    "\n",
    "```\n",
    "numpy==1.15.2\n",
    "pandas==0.23.4\n",
    "plotly==3.3.0\n",
    "scikit-learn==0.20.0\n",
    "```\n",
    "\n",
    "hold off on committing for just a moment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3: background on resolving `github` issues with `commit` messages\n",
    "\n",
    "`github` [allows users to resolve and close issues with `commit` message](https://help.github.com/articles/closing-issues-using-keywords/). read the documentation on that page!\n",
    "\n",
    "if you make a `commit` message to any branch (including `master`) that uses keywords like \"fixes\" or \"resolves\" and references the issue by number, `github` will link that commit and the named issue. if that commit is to the `master` branch, `github` will automatically close the issue with a reference to the `commit` `sha`:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1vn8HX4wkn2HhjOBGu842wMuVQSWxHtIu\" width=\"700px\"></div>\n",
    "\n",
    "if the `commit` is made to a non-`master` branch, it will allow users to create something called a \"pull request\" -- more on that in a future exercise!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4: actually resolving `github` issues with `commit` messages\n",
    "\n",
    "`add` and `commit` your update to `requirements.txt` to the `master` branch with commit message\n",
    "\n",
    "```\n",
    "requirements.txt: pin versions, fixes #YOUR_ISSUE_NUMBER\n",
    "```\n",
    "\n",
    "where you replace `YOUR_ISSUE_NUMBER` with the number of your issue as seen in `github`:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1otKcV-eudPkQm-EbJ3alevItgcHhIl_O\" width=\"500px\"></div>\n",
    "\n",
    "your issue number is *probably* 1, but double-check! for example, my commit message was\n",
    "\n",
    "```\n",
    "requirements.txt: pin versions, fixes #1\n",
    "```\n",
    "\n",
    "after you've `commit`ed, `push` to `origin` `master`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5: verify that the issue in your `github` repo is closed\n",
    "\n",
    "check your issue page in `github` and verify that it appears closed and references the comit message you made:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1vn8HX4wkn2HhjOBGu842wMuVQSWxHtIu\" width=\"700px\"></div>\n",
    "\n",
    "\n",
    "##### submission will be verified via `github`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border: 1px solid lightgrey;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:red;font-weight:bold\">OPTIONAL</span> exercise 4: benchmark differences in performance using `gpu`s <span style=\"color:red;font-weight:bold\">OPTIONAL</span>\n",
    "\n",
    "<span style=\"color:red;font-weight:bold\">this exercise is optional and ungraded; it is included for anyone interested in the details of acquiring and using a `gpu`</span>\n",
    "\n",
    "let's spin up a `gpu` `ec2` instance and do a simple benchmark to see the performance improvements available via `gpu`s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1: `gpu`s are expensive!\n",
    "\n",
    "go check [the per-hour price](https://aws.amazon.com/ec2/pricing/on-demand/) of `gpu` compute for a `p3.2xlarge` instance in the US East (Virginia) region. as of writing, it is 3.06 USD per hour.\n",
    "\n",
    "we don't want to leave that on for long, so let's make this quick!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2: spin up a `p3.2xlarge` instance\n",
    "\n",
    "`aws` has already created a deep learning `ami` for us, so let's use it and save time (and money) on downloads.\n",
    "\n",
    "+ open the `ec2` web console and create a new instance\n",
    "+ `ami`: scroll down to \"Deep Learning AMI (Ubuntu 16.04) Version 36.0\" and select that record\n",
    "    + note this is ***NOT*** the Ubuntu 18.04 version, we are using 16.04 (I have not tested this benchmarking script on 18.04)\n",
    "    + if the version is larger (e.g. 36.1), select and continue\n",
    "+ instance type: `p3.2xlarge`\n",
    "    + click \"review and launch\"\n",
    "+ launch the instance\n",
    "    + make sure you have your `ssh` key saved somewhere easy!\n",
    "    \n",
    "**you may receive the following error:**\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=16VCyePLZeDUBI6mymcS_0Zwv4dgtKcvJ\"></div>\n",
    "\n",
    "if you do, you are limited to 0 `p3.2xlarge` instances (verify [here](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Limits:)), and the only path forward is to request a limit increase for your account. if you would like to do so, head to https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=service-code-ec2-instances to request an increase in the `ec2` instance type limit up to a limit value of 1. I am not sure how long this will take!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3: log in\n",
    "\n",
    "after your `ec2` instance is up and running, log in to it using username `ubuntu` and providing the path to the private key `.pem` file you either downloaded just now or when you created that key pair for a previous `ec2` instance\n",
    "\n",
    "if you don't know where this key file is, *terminate the instance* (right click > instance state > terminate) and start over."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4: download a benchmark\n",
    "\n",
    "log in to your new `ec2` instance and then download [this public `gist`](https://gist.github.com/RZachLamberty/fe8e05060b809e90fd2722feeb80fcda):\n",
    "\n",
    "```sh\n",
    "wget https://gist.githubusercontent.com/RZachLamberty/fe8e05060b809e90fd2722feeb80fcda/raw/8009c87f1b7e49c96181b011f226a5097ab89be3/cpu_gpu_benchmark.py\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5: activate an environment\n",
    "\n",
    "the good folks at `aws` have pre-configured this `ami` with a ton of different deep-learning-capable environments, and the commands for entering any one of them are printed out when you log in to the `ami`. one in particular is for us right now:\n",
    "\n",
    "```\n",
    "for TensorFlow(+Keras2) with Python3 (CUDA 9.0 and Intel MKL-DNN) ___________________________ source activate tensorflow_p36\n",
    "```\n",
    "\n",
    "run that command to activate that environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.6: run the benchmark\n",
    "\n",
    "now that we have everything we need already installed (thanks, `aws` `ami`!), go ahead and run the benchmark script:\n",
    "\n",
    "```sh\n",
    "python cpu_gpu_benchmark.py\n",
    "```\n",
    "\n",
    "> **note**: the first time you run on this machine, the process of initializing `tensorflow` for the first time may require enough overhead to cause an error in the benchmarking script. you will see a `ValueError: Empty data passed with indices specified.` or `ValueError: could not broadcast input array from shape (0) into shape (1)` error. if you see this, just run the benchmark script again. if you see it more than three times, terminate your instance and send me an email.\n",
    "\n",
    "the final output will be a dataframe which lists the amount of time it took to create random matrices of increasing sizes and multiply them, as well as the ratio of the speeds for the different devices for each operation.\n",
    "\n",
    "it will also output a `csv` named `results.csv`. download that file (use `scp` to copy it from your `ec2` to your laptop, or just open it, highlight, and copy-paste to a local file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.7: TERMINATE YOUR `gpu` INSTANCE!!\n",
    "\n",
    "don't forget to go back into the `ec2` web console and terminate your instance (right click > instance state > terminate)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border: 1px solid lightgrey;\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}