014_deep_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/rzl-ds/gu511/blob/master/014_deep_learning.ipynb\" target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## intro\n",
    "\n",
    "it's hard to understate the pervasiveness and success of deep learning methods in recent years. knowledge of deep learning techniques is a must for modern data scientists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it's so important, in fact, that GU offers an entire class on it: [Math 514: intro to neural networks](https://myaccess.georgetown.edu/pls/bninbp/bwckctlg.p_display_courses?term_in=201910&one_subj=MATH&sel_crse_strt=514&sel_crse_end=514&sel_subj=&sel_levl=&sel_schd=&sel_coll=&sel_divs=&sel_dept=&sel_attr=#_ga=2.146187319.2080322458.1542654672-115011657.1531320772). this is not that course! you should consider taking it.\n",
    "\n",
    "what follows is merely an incredibly hand-wavy introduction to deep neural nets. I hope to give you enough understanding and context that you feel comfortable executing simple deep learning code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### deep learning vs. deep neural nets\n",
    "\n",
    "for starters, a bit of nomenclature: the lecture is called \"deep learning\" but I will often be talking about \"deep neural nets\" instead. they are related:\n",
    "\n",
    "+ **deep learning** is a family of statistical modelling approaches that attempt to \"learn\" the underlying structure or most convenient representation of data in order to make a specific sort of prediction\n",
    "    + \"learning\" happens through exposure to subsequent examples. a model we have trained should become a better model if exposed to a new example\n",
    "    + the predictions made in deep learning are typically supervised (real-world targets), but not necessarily so (autoencoders)\n",
    "+ **deep neural nets** are a sub-family of *deep learning* models that are specifically constructed out of inter-connected \"neurons\", computation steps that perform a linear transformation and then a subsequent nonlinear transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it's a minor distinction, but there are things that are **deep learning** that are not **deep neural nets**; we're not going to talk about them!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### introduction to deep neural nets\n",
    "\n",
    "let's talk about what a deep neural net is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### TL;DR\n",
    "\n",
    "a neural net is a highly flexible architecture for efficiently learning an optimal representation of input feature for making specific predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "what follows builds up a neural net from the smallest components to the final net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### one neuron / node\n",
    "the fundamental element of a neural net is the neuron. this is so named due to long-standing analogies to the way neurons work in a brain.\n",
    "\n",
    "I think this analogy is more confusing than it is worth. *just watch me* call them nodes instead of neurons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the ~~neuron~~ node is a two-step operation: you do one *linear* transformation with a vector of weights and a bias value, then you do one *nonlinear* transformation with some function (called the **activation function**).\n",
    "\n",
    "what weights? what bias value? what function? to be determined!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "suppose we have a record of data with two features $x_1$ and $x_2$. a neuron that can act on that record would have two weight values ($w_1$ and $w_2$, one for each feature), a bias value $b$, and an activation function $f$\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://ujwlkarn.files.wordpress.com/2016/08/screen-shot-2016-08-09-at-3-42-21-am.png?w=568&h=303\" width=\"800px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the first step is the *linear* transformation. symbolically, this is:\n",
    "\n",
    "$$\n",
    "W \\cdot x + b = \\sum_i W_i x_i + b\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "geometrically, this is a measurement of how large the vector $x$ is when projected along the weight $W$\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1DYpFUfRxuuAhn66402TJOMbyXCzf4Zcw\" width=\"800px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "basically, we are *linearizing* the input by converting every incoming record in whatever space to one single number measuring the amount of that vector pointing in some specific direction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "after we have linearized the input, we add a non-linearity using an **activation function**. this activation function takes one input value (the linearized input value) and outputs something that is specifically non-linear.\n",
    "\n",
    "there are [a lot of these functions](https://en.wikipedia.org/wiki/Activation_function#Comparison_of_activation_functions), but the most common are"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the **sigmoid**, $\\sigma(x) = \\dfrac{1}{1 + \\exp(-x)}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np, plotly.graph_objs as go\n",
    "x = np.linspace(-10, 10, 100)\n",
    "y = 1 / (1 + np.exp(-x))\n",
    "data = [go.Scatter(x=x, y=y)]\n",
    "go.Figure(data).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the **ReLU** (**Re**ctified **L**inear **U**nit),\n",
    "\n",
    "$$\n",
    "\\operatorname{relu}(x) = \\left\\{\\begin{array}{ll}\n",
    "0 & x \\leq 0 \\\\\n",
    "x & x > 0\n",
    "\\end{array}\n",
    "\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "y = np.where(x <= 0, 0, x)\n",
    "data = [go.Scatter(x=x, y=y)]\n",
    "go.Figure(data).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the **leaky ReLU**,\n",
    "\n",
    "$$\n",
    "\\operatorname{relu}(x) = \\left\\{\\begin{array}{ll}\n",
    "0.01 x & x \\leq 0 \\\\\n",
    "x & x > 0\n",
    "\\end{array}\n",
    "\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "y = np.where(x <= 0, 0.01 * x, x)\n",
    "data = [go.Scatter(x=x, y=y)]\n",
    "go.Figure(data).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### a stack of neurons\n",
    "\n",
    "once we understand what one neuron is doing, we could take a whole stack of $N$ of them. each could have different $W$ and $b$ values. they could have different activation functions (but typically don't). and all together they could take any one input record and create $N$ output values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this is often visualized as a \"net\", where the neurons (nodes) are drawn as circles, and the \"weights\" are represented as edges (that is, edge from a node to $x_i$ represents that nodes' $w_i$ value)\n",
    "\n",
    "<br><img src=\"https://draftin.com/images/34466?token=YFsmpDuQfD3DDylinRD8F4sLOgjCFm4Aow1gIWoCY5KED3bnQKs17RaTja95OIQQWdr25dqS2fxq_6mDwwdcs9Y\" width=\"500px\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so with a stack of $N$ nodes we can convert an input record $x$ into an $N$-dimensional output record $z$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### a stack of stack of neurons\n",
    "\n",
    "the output of one stack of neurons is a new record. it's in some crazy $N$-dimensional space which is determined by the weights and biases of the previous layer, but it's basically now just a new record.\n",
    "\n",
    "so we could do the same thing with *that* record that we did with our $x$ records, and feed it into a *new* stack of nodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this is the \"deep\" neural net -- it's a neural net with hidden layers, so it's become \"deep\"\n",
    "\n",
    "<br><img src=\"http://cs231n.github.io/assets/nn1/neural_net2.jpeg\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### finally, an output\n",
    "\n",
    "remember, we started down this path because our model we are constructing should be able to *predict* something. so we need a final layer that will take... whatever it is that we've created -- whatever that representation is -- and predict a value.\n",
    "\n",
    "in practice, this is usually a logistic function (for binary predictions), a softmax (for categorical predictions), a linearization-only node (for regression), or a collection of logistics (for multi-category predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### summary\n",
    "\n",
    "so a neural net is: a series of **layers**, where each **layer** is a stack of some number of **neurons**, and each **neuron** is a linearization (defined by a weight $w$ and a bias $b$) followed by an **activation** function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### why it works\n",
    "\n",
    "if I just gave you a neural net with random number of layers, with layers of random node size, and with random weights and biases throughout, it would be *terrible* at making predictions. so it's not the *structure* that is making good predictions.\n",
    "\n",
    "rather, this particular way of arranging things has some special properties that make it easy to figure out how to tweak weights to incrementally improve those predictions. the process whereby we tweak weights is called *backpropagation* and is, at its heart, just the chain rule applied to millions of variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "slowly but surely, and with enough input data, we can update the weights in our deep neural net to **learn** the ways of representing our data (the elements that come out of each layer of nodes) that are **optimal** for making our predictions.\n",
    "\n",
    "in a way, it's almost like cheating -- we know we want to make predictions, and we have a clever way of mashing together our input features such that what comes out is some $N$ dimensional vector that we can pass to a logistic regression and get amazing results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### why we care (in this class)\n",
    "\n",
    "so why go through the hassle of covering this in \"advanced math and statistical computing\" when it's the topic of an entire different course?\n",
    "\n",
    "because there's so much computing action focused on deep learning!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we spent the bulk of last lecture talking about how anything that can be parallelized is a good candidate for `gpu` analytics and acceleration, and in particular linear algebra.\n",
    "\n",
    "well, as you saw above, deep neural nets are a giant pile of linear algebra. recent advancements in `gpu` availability (price and number) as well as speed have caused an explosion of `gpu` deep learning application development. including ours, in this course"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## higher-level deep learning `api`s\n",
    "\n",
    "let's figure out how we can code deep neural network models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "recall the deep learning stack picture in the `gpu` lecture:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1M3LZQRI8nfCscnyL_h7xjKi4i9e8lo1t\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this is called a stack because each level is providing a new interface and possibly new functionality \"on top of\" the level below it. the bottom three green levels are all very low level, and represent the particular stack made available by `nvidia`.\n",
    "\n",
    "the very lowest level of that diagram (`gpu`) is the hardware level. you could have several types of `gpu`, but in this class we will focus on `nvidia` brand `gpu`s."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the level above that (`cuda`) is a set of `c++` libraries which allow programmers to write code that can be executed on the `gpu`. for `nvidia` `gpu`s, developers at `nvidia` have done the heavy lifting here, creating the bridge from the hardware (extremely low-level instructions!) to `c++` (a full OOP language).\n",
    "\n",
    "they have also used that set of `c++` libraries to create a second set of libraries that use the lower-level `c++` `cuda` code base to implement functionality that is specifically meant to be used in deep learning applications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "at this point, if you are an extremely `l33t h4x0r` `c++` programmer, feel free to hop into your `ide` and bang out some deep neural nets.\n",
    "\n",
    "for the rest of us mere mortals, we will focus on even-higher-level apis in `python`. fortunately, a few exist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "perhaps that begs the question: how can I use this existing `c++` libraries from with `python`?\n",
    "\n",
    "you look for a `python` `api` that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the orange (`tensorflow`) and red (`keras`) boxes represent two levels of abstraction available to `python` coders for interacting with `gpu`s.\n",
    "\n",
    "+ `tensorflow` is a numerical computation framework that defines complicated computations as a directed graph of smaller computations\n",
    "    + we define a \"graph\" of operations (nodes) that we connect by their inputs and ouputs (edges)\n",
    "    + the graph defines how to get from one first node (e.g. loading the ultimate input) to any step downstream in the graph\n",
    "    + this is *not* deep learning specific (we could create another crappy alarm clock script with `tensorflow`, e.g.), but it was created with deep learning in mind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ `keras` is a deep-learning-focused framework\n",
    "    + this is a high-level way of describing deep neural networks and training methods in simple `python` code\n",
    "    + it has *backends*, internal libraries which are used to *implement* the higher-level `keras` framework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### deep learning `api`s\n",
    "\n",
    "each of these libraries is an *interface* to *something* beneath it. they provide developers a set of functions in some runtime (e.g. `python`) that hide the difficult, messy internal implementation details, so that someone who wants to use `tensorflow` to do *whatever* it is that `tensorflow` does won't need to know or care about what's happening a level below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "like any good interface, each of them **can** be an interface to the layer below\n",
    "\n",
    "+ `tensorflow` is a `python` interface to `c++` libraries `cuda` and `c++` (really, it \"goes through\" the `tensorflow` `c++` libraries, for an extra layer of interface-y goodness)\n",
    "+ `keras` is a `python` interface to `tensorflow`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "but they also **are not required** to be using that particular lower-level piece\n",
    "\n",
    "+ `tensorflow` can use non-`nvidia` or non-`gpu` lower-level libraries (that is, it can work on different `gpu`s, or `cpu`s, or google's proprietary `tpu`s, or `android` phones)\n",
    "    + while this is true in theory, in practice you are strongly incentivized to use `nvidia` `gpu`s if you want to use `gpu`s at all. this is less to do with `tensorflow`'s ability to support other types of `gpu`s, and more to do with the lack of replacements for `cuda` and especially `cudnn` for other `gpu` manufacturers. to put it another way: `amd` is way behind in developing code to enable deep learning, even if the architecture could do it in theory.\n",
    "+ `keras` can use other `python` deep learning libraries (e.g. `theano`, `cntk`, or apache `mxnet`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you've used at least one subject-matter-specific `api` library before in this class: the `scikit-learn` library is a framework for creating machine learning models. you are used to relying on the same sort of `api` for `sklearn` models:\n",
    "\n",
    "```python\n",
    "model = sklearn.somemodeltype.MySpecialClassifier(param1, param2)\n",
    "model.fit(X_train, y_train)\n",
    "model.score(X_test, y_test)\n",
    "model.predict(X_test)\n",
    "```\n",
    "\n",
    "all that changes from model to model is the actual `model` object you create, but there are standard ways of creating those models. then you assume they all have the same methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "finally, if you weren't confused enough yet, the `keras` library is available as a standalone `python` package but *also* as a module within the `tensorflow` package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import keras\n",
    "help(keras)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "help(tf.keras)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "note: these are not necessarily the same version!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'keras version: {keras.__version__}')\n",
    "print(f'tf.keras version: {tf.keras.__version__}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's take a step back and re-focus on what we want to do, to help illuminate what these `api`s (`tensorflow` and `keras`) are doing for us.\n",
    "\n",
    "we want to create a deep neural network models, and we'd like to be able to use `gpu`s to accelerate our computation if we would like"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`tensorflow` can help us do this\n",
    "\n",
    "+ **if** we can define our model as a directed graph of computation nodes and input / output edges, **then** `tensorflow` will handle the implementation on different lower-level hardware types (`gpu`, `cpu`, `tpu`, `android`) for us\n",
    "+ **if** we have a novel or experimental neural network architecture we want to try, **then** `tensorflow` provides us with all of the necessary infrastructure to create and train that model in the same way we would train any other model. we should be able to build pretty much *whatever* deep learning model we want\n",
    "+ **if** we want to train a fairly straightforward model type (`dnn`, `cnn`, `rnn`, `lstm`), **then** we may have to work a little bit harder than we'd like to define that model (see `keras`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "also, `keras` can help us do this\n",
    "\n",
    "+ **if** we have a backend which implements neural net computation methods (e.g. `tensorflow` or `theano`), **then** `keras` gives us a *much* simpler interface for writing that code\n",
    "+ **if** we want to use a different backend (`tensorflow` on one computer and `mxnet` on another), **then** `keras` will handle the implementation details for each without requiring code changes\n",
    "\n",
    "if you are just starting out and want to do some simple neural network development, I **strongly** encourage you to start with `keras`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "in particular, the author of the `keras` library (François Chollet) is a prolific author and blogger. his [`keras` blog](https://blog.keras.io/author/francois-chollet.html) is one of the best resources out there for tutorials on how to write deep learning models in `keras`. additionally, he wrote a great book: https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### alternatives\n",
    "\n",
    "the *thing* `keras` gives us is a high-level backend-agnostic interface for creating most types of deep neural net architectures. alternative options include apache `mxnet`, which is a high-level framework with implementations in multiple different languages (and, coincidentally, one of the *backends* to `keras` to boot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the *thing* `tensorflow` gives us is a computation environment with all the basic building blocks of deep neural net models (e.g. activation functions, loss functions, gradient descent algorithms) and supporting implementation on various different hardware types. the main alternative to `tensorflow` for this at this time is `pytorch`.\n",
    "\n",
    "many people prefer `pytorch` to `tensorflow`, so this is by no means a settled dispute. that being said, one of the people that prefers `tensorflow` is `google`, so I feel pretty confident that project will keep advancing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## hands-on\n",
    "\n",
    "enough yaking, more key clacking. let's build some models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">exercise: install `tensorflow` and `keras`</div>**\n",
    "\n",
    "on some machine where you have `conda` and some disk space, let's run\n",
    "\n",
    "```sh\n",
    "conda install -y tensorflow keras\n",
    "```\n",
    "\n",
    "verify it work by running (in a `python` or `ipython` session)\n",
    "\n",
    "```python\n",
    "import keras\n",
    "import tensorflow as tf\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "note: we also could have used `docker` to create a `container` with `tensorflow` (and therefore `keras`, via `tf.keras`) pre-installed. look at https://hub.docker.com/r/tensorflow/tensorflow/ for details, but the basic commands are\n",
    "\n",
    "```sh\n",
    "# pull (if you haven't) and run the latest py v3 tensorflow\n",
    "# container\n",
    "docker run --rm -it -p 8888:8888 tensorflow/tensorflow:latest-py3\n",
    "\n",
    "# open a jupyter notebook at localhost:8888\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## using `tensorflow`\n",
    "\n",
    "the [`tensorflow` documentation](https://www.tensorflow.org/tutorials/) is the definitive source for information on how to write `tensorflow` code, and this is no replacement. I simply want to cover the high-level concepts of working with `tensorflow`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### important context: `tf1` vs. `tf2`\n",
    "\n",
    "a bit more than one year ago (Sep 30, 2019), `tensorflow` officially graduated from major version 1 to major version 2\n",
    "\n",
    "this was a **big** change, and it has particular implications for how you as a data scientist should be using `tensorflow`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "to make a long story short, there were some *features* of `tensorflow` that the -- let's say, less technical (i.e. not software engineers at `google`) -- community didn't like. chief among these were\n",
    "\n",
    "+ an over-abundance of \"official\" `api`s: a *lot* of ways to build models\n",
    "    + this included `tf.estimator`, `tf.keras`, and `tf.slim` `api`s\n",
    "    + these were not necessarily interchangeable and not all things you might want to do in one were available in others\n",
    "+ an insistence on graph execution over eager execution\n",
    "    + **graph execution**: create the calculation you want to perform but *DON'T PERFORM IT UNTIL I SAY SO*\n",
    "    + **eager execution**: calculate things when I press `shift + enter` in my `jupyter notebook`\n",
    "+ confusing locations of \"side projects\", including an enormous dumping ground called `tf.contrib` that held a mix of half-baked and extremely important things"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`google` had many things it wanted to change from v1 to v2 in `tensorflow`, but some of the biggest changes come from concessions on these points above, namely\n",
    "\n",
    "+ while other `api`s are still accessible, `tf.keras` is **the** `api` for `tensorflow`\n",
    "+ eager execution (not graph execution) is now the default and recommended mode)\n",
    "    + useful side effect: much of the `tf` boilerplate (e.g. `tf.Session`, `tf.Graph`, `tf.Placeholder`, and `tf.Session.run` calls) are deprecated\n",
    "+ important contributed code will live in \"the right place\" in the `tf` module, and non-essential things will live in other packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there is [a long walkthrough covering the changes you should implement to migrate from `v1` to `v2`](https://www.tensorflow.org/guide/migrate), but in practice the real implication is this:\n",
    "\n",
    "> when using a pre-trained model or starting your own model from scratch, always use `tf2` if at all possible. if this is not immediately possible (e.g. a pre-trained model that is only available in `tf1`, consider two things\n",
    "> 1. it may be better for you to look for a different pre-trained model (e.g. a `fork` done in `tf2`, or a `pytorch` version)\n",
    "> 1. if you feel confident you can do it, creat that `fork` and migrate it yourself\n",
    "> 1. if neither of the above are possible or efficient, be aware that you are going to do some extra work that will soon be irrelevant. sometimes this is the price you have to pay!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `tensor`s and `operation`s\n",
    "\n",
    "there are many fundamental `python` objects in the `tensorflow` library, but the most important are `tensor`s and `operation`s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `tensor`\n",
    "\n",
    "`tensor`s are arrays of objects (usually numbers) with arbitrary numbers of dimensions. these are the data of a `tensorflow` computation\n",
    "\n",
    "you have plenty of experience with 0, 1, 2 dimensional tensors (generally called scalars, vectors, matrices)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the number of dimensions an object has -- or in `python` code, the number of brackets `[...]` you need to describe them -- is called the **rank**\n",
    "\n",
    "+ `3`, a rank 0 tensor (aka scalar)\n",
    "+ `[1, 2, 3]`, a rank 1 tensor (aka vector)\n",
    "+ `[[1, 2], [3, 4]]`, a rank 2 tensor (aka matrix)\n",
    "+ the below, a rank 3 tensor\n",
    "\n",
    "```python\n",
    "[[[1, 2], [3, 4]],\n",
    " [[5, 6], [7, 8]]]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`tensor`s also have **shape**, a list representing the number of elements in each dimension.\n",
    "\n",
    "+ `3` has shape `[]`\n",
    "+ `[1, 2, 3]` has shape `[3]`\n",
    "+ `[[1, 2, 3], [4, 5, 6]]` has shape `[2, 3]` (2 \"first level\" elements with 3 elements inside each)\n",
    "+ the below has shape `[2, 2, 2]` (2 \"first level\", each with 2 sub-elements, each of those with 2 elements\n",
    "\n",
    "```python\n",
    "[[[1, 2], [3, 4]],\n",
    " [[5, 6], [7, 8]]]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are a few types of `tensor`, but the most important to know about are:\n",
    "\n",
    "+ `tf.Variable`: a tensor that holds a value and can be passed from one \"session\" (defined below) to the next and can change value\n",
    "    + you can think of this like a variable in any program you've written; you instantiate it once but are free to add to it / update it over time.\n",
    "    + for example, the weights and biases might be variables. you will pass them from one training run to the next and you will update them as part of your training\n",
    "+ `tf.constant`: a constant, immutable value\n",
    "+ `tf.placeholder`: a constant, immutable value *that isn't necessarily known yet, but will be*\n",
    "    + note: these are mostly frowned on in v2.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "t_scalar = tf.constant(3)\n",
    "print(f't_scalar = {t_scalar}')\n",
    "print(f't_scalar.shape = {t_scalar.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "t_vector = tf.constant([1, 2, 3])\n",
    "print(f't_vector = {t_vector}')\n",
    "print(f't_vector.shape = {t_vector.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "t_matrix = tf.constant([[1, 2, 3],\n",
    "                        [4, 5, 6]])\n",
    "print(f't_matrix = {t_matrix}')\n",
    "print(f't_matrix.shape = {t_matrix.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "t_rank3 = tf.constant([[[1, 2], [3, 4]],\n",
    "                       [[5, 6], [7, 8]], ])\n",
    "print(f't_rank3 = {t_rank3}')\n",
    "print(f't_rank3.shape = {t_rank3.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `operation`\n",
    "\n",
    "an `operation` is something I *do* to a `tensor` -- think any ol' math expression, `add`, `subtract`, `multiply`, `divide`, etc.\n",
    "\n",
    "```python\n",
    "tf.add(1, 1)\n",
    "tf.multiply(x, y)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "importantly, though: in `python`-speak, **these are not functions**, they are **objects** that represent a computation.\n",
    "\n",
    "as such, they don't return a computed value that is the result of that operation, but rather they return a new tensor which acts as a placeholder for the output of the function. it is not computed in real time (by default), but rather assumed to be eventually computed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "*note: this is slightly abusive terminology because base `python` functions themselves are in fact objects; what I really mean to say here is that `tensorflow` operations are not base `python` functions*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "mysum_op = tf.add(1, 1)\n",
    "mysum_op"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "mymult_op = tf.multiply(t_scalar, t_matrix)\n",
    "mymult_op"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "mycombo_op = mymult_op / mysum_op\n",
    "mycombo_op"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### the execution graph\n",
    "\n",
    "the fundamental abstraction in `tensorflow` is the **execution graph**: a directed graph connecting *computation* nodes (`operation` objects like `add`, `subtract`, `multiply`, etc) with edges that symbolize inputs and outputs (`tensor`s of data that the `operation`s consume or produce)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a graph is **directed** in that the `tensor` outputs of some `operation`s are subsequently consumed by later `operations`. the logic of the computation is still sequential (\"do this, then do that, then do that...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for example, when we wrote `tf.add(1, 1)` above, we were updating the execution graph by\n",
    "\n",
    "1. creating a new `add` operation node\n",
    "1. assigning its `x` and `y` inputs to be two constant scalar `tensor`s with values of 1\n",
    "    1. this implicitly created special operations which \"load\" the scalar value 1 into the graph\n",
    "1. creating a new output tensor implicitly creating two `tf.constant`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "please disregard the mess here, this cell is just to get a `graph` we can actually view"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "!rm -r /tmp/tensorboard-gu511-dl-lecture/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext tensorboard\n",
    "\n",
    "logdir = '/tmp/tensorboard-gu511-dl-lecture'\n",
    "writer = tf.summary.create_file_writer(logdir)\n",
    "\n",
    "t_scalar = tf.constant(3)\n",
    "t_matrix = tf.constant([[1, 2, 3],\n",
    "                        [4, 5, 6]])\n",
    "\n",
    "# The function to be traced.\n",
    "@tf.function\n",
    "def my_func(x, y):\n",
    "    return tf.divide(tf.multiply(x, y), tf.add(1, 1))\n",
    "\n",
    "\n",
    "# tf.summary.trace_on() and tf.summary.trace_export().\n",
    "tf.summary.trace_on(graph=True, profiler=True)\n",
    "# Call only one tf.function when tracing.\n",
    "z = my_func(t_scalar, t_matrix)\n",
    "with writer.as_default():\n",
    "    tf.summary.trace_export(name=\"my_func_trace\",\n",
    "                            step=0,\n",
    "                            profiler_outdir=logdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "%tensorboard --port 6007 --logdir /tmp/tensorboard-gu511-dl-lecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "what the diagram above shows are **`operation` nodes** (e.g. `Add`, `Mul`) connected by **`tensor` edges**. together these make up the **execution graph**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "as we said above, it used to be the case that you would write `tensorflow` code to create the `graph`, but that nothing would be *executed* until you explicitly called for that to happen.\n",
    "\n",
    "for example, even though the program we wrote above is pretty trivial, the default behavior of `tensorflow` in previous versions was to define the execution graph and wait to calculate until you asked for it. it treated the graph like a recipe -- if you provide some materials (input `tensor`s `x` and `y`), `tensorflow` would bake you a `tensor` cake (the result of the function).\n",
    "\n",
    "this mode is gone in `tf2` -- now it calculates `x` when you define `x`, `y` when you define `y`, the sum when you call `tf.add(x, y)`, ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### high-level modelling `api`s: `dataset` and `estimator`\n",
    "\n",
    "the most common complaint about `tensorflow` is that it is over-engineered and not easy or intuitive to write. `tf2` helps with that, but in general I think that complaint is still true.\n",
    "\n",
    "even so, you will *definitely* run into these `api`s if you are doing deep learning, and it is useful to know what these two major `api`s *are* and at a high level how to use them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "google generally recommends that you\n",
    "\n",
    "1. use the `dataset` data `api` to define your datasets you will use for train, validation, dev, and test\n",
    "1. use the `tf.keras` model `api` to define your models\n",
    "1. use the `estimator` model `api` to define your models when `keras` won't do\n",
    "\n",
    "because the `dataset` and `estimator` class are unique to `tensorflow`, but `keras` is useful across different deep learning frameworks, we will talk about those two here and save `keras` for a later section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `tf.data` `api` basics\n",
    "\n",
    "the `tensorflow` developers have created a class of objects `tf.data.Dataset` to handle the most common functionality of loading and manipulating data for deep learning modelling. if you load your data using this object and its `api`, you will be given free access to many pre-built methods for manipulating your data, and other `tensorflow` objects will know exactly what to do with your dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the basic workflow is this:\n",
    "\n",
    "+ identify your pre-`tensorflow` data location on disk (file paths) or in memory (a non-`tensorflow` object)\n",
    "+ create a *source* `tf.data.Dataset` object which takes that data location and knows how to ingest it\n",
    "+ use the *transformation* methods that all `tf.data.Dataset` objects have to transform that data (e.g. shuffling, batching, zero-padding)\n",
    "+ create a `tf.data.Iterator` object which knows how to take a `tf.data.Dataset` collection of records and generate records for a training or evaluation routine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### an example\n",
    "\n",
    "the data that you load into a `tensorflow` dataset is assumed to be a collection of features that have the same first dimension shape but don't necessarily have much to do with each other in terms of their internal makeup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "taking [an example straight from the docs](https://www.tensorflow.org/guide/data#dataset_structure):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a tensor which is 4 x 10 random uniform numbers\n",
    "dataset1 = (tf.data.Dataset\n",
    "            .from_tensor_slices(tf.random.uniform([4, 10])))\n",
    "dataset1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "these `dataset` objects can be iterated over just like any other `python` collection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "for record in dataset1:\n",
    "    print(record)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and we can extract the values inside the individual `tensor` elements as `numpy` arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "record.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the `dataset`s we make can be composed of several `tensor`s of different shape (they must share the first dimension (number of records), but after that, go wild)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# create one 4-element random number vector, and one 4 x 100 element random\n",
    "# number matrix, and combine them into one dataset\n",
    "dataset2 = (tf.data.Dataset\n",
    "            .from_tensor_slices((tf.random.uniform([4]),\n",
    "                                 tf.random.uniform([4, 100],\n",
    "                                                   maxval=100,\n",
    "                                                   dtype=tf.int32))))\n",
    "dataset2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and there are composition functions (e.g. `zip`) to tie multiple `dataset`s together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "dataset3 = tf.data.Dataset.zip((dataset1, dataset2))\n",
    "\n",
    "dataset3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in the above, we are creating a `tf.data.Dataset` which can be iterated to produce four records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for (t1, (t2a, t2b)) in dataset3:\n",
    "    print(f'shapes: {t1.shape}, < {t2a.shape}, {t2b.shape} >')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so elements in `dataset3` have two top-level pieces (from `dataset1` and `dataset2`, respectively); and the second (from `dataset2`) has two sub-pieces, each of which is a `tensor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "print(f\"\"\"\n",
    "t1 = {t1}\n",
    "t2a = {t2a}\n",
    "t2b = {t2b}\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for each of the `dataset` objects above we could have created them with a `dict` or features (with column names as keys) instead of just a tuple of `tensor`s -- this would result in datasets where the components of the records are named, and might help us unpack things a little better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset1 = (tf.data.Dataset\n",
    "            .from_tensor_slices({'t1': tf.random.uniform([4, 10])}))\n",
    "dataset2 = (tf.data.Dataset\n",
    "            .from_tensor_slices({'t2a': tf.random.uniform([4]),\n",
    "                                 't2b': tf.random.uniform([4, 100],\n",
    "                                                          maxval=100,\n",
    "                                                          dtype=tf.int32)}))\n",
    "dataset3 = (tf.data.Dataset.zip({'t3a': dataset1,\n",
    "                                 't3b': dataset2}))\n",
    "dataset3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for record in dataset3:\n",
    "    break\n",
    "record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "record['t3a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "record['t3b']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "record['t3b']['t2b'].numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### loading data with `numpy` or `pandas`\n",
    "\n",
    "in practice, though, you will spend much more time loading datasets from `numpy` arrays or `pandas` dataframes, or flat files, so why not spend some time figuring that out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np, pandas as pd, sklearn.datasets\n",
    "\n",
    "iris = sklearn.datasets.load_iris()\n",
    "\n",
    "x = iris.data\n",
    "y = iris.target\n",
    "df = pd.DataFrame(x, columns=iris.feature_names)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the basic type of `tensor`s in `tensorflow` is (effectively) just a `numpy` array, so you can pass the `numpy` array directly to a `from_tensor_slices` call (as if it were a `tensor` slice) and this will construct your `tensor`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(x))\n",
    "iris_tensor = tf.data.Dataset.from_tensor_slices(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for record in iris_tensor:\n",
    "    print(record)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for a `pd.DataFrame` object, we have a few options. first, we could just use the `df.values` attribute, which is itself a `numpy` array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_tensor = tf.data.Dataset.from_tensor_slices(df.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for record in iris_tensor:\n",
    "    print(record)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "alternatively, we could convert our dataframe into a dictionary with feature names as keys and columns as values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "iris_tensor = tf.data.Dataset.from_tensor_slices(df.to_dict(orient='series'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for record in iris_tensor:\n",
    "    print(record)\n",
    "    print()\n",
    "    print(record['petal width (cm)'].numpy())\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "note that the above two approaches are doing fundamentally different things, and your following code will need to know / care about that:\n",
    "\n",
    "1. passing as an array gives you an iterable which yields one **4-element list of numbers**\n",
    "1. passing as a dictionary gives you an iterable which yields one **dictionary** where each key corresponds to a **one-element scalar tensor**\n",
    "\n",
    "in practice the former (pass as array) is more common"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### loading data from file\n",
    "\n",
    "the above were simple ways to take datasets that fit **in memory** and load them directly into `tensor`s using the `tf.data.Dataset.from_tensor_slices` method.\n",
    "\n",
    "this is great when the data you are working with fits in memory, but in practice this is rarely the case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "instead, it usually the case you are working with datasets that far surpass the memory constraints of your machine (data volume is one of the main reasons deep learning works!). in these cases, you will usually read from files instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "while there is built-in support for `csv`s (`tf.data.experimental.make_csv_dataset`) and `tfrecord`, a proprietary google binary file format for \"records\" (`tf.data.TFRecordDataset`), the general approach is to define a `tf.data.TextLineDataset` dataset object and to apply transformations to it that convert a single string per row into a `tensor`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### `dataset` manipulation methods\n",
    "\n",
    "in addition to loading data, we also have options for making common manipulations of that data before iterating through it. for example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`dataset.batch()` will combine records into a `batch` (or chunk) of a fixed size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_tensor = (tf.data.Dataset\n",
    "               .from_tensor_slices(df.values)\n",
    "               # here we batch\n",
    "               .batch(3))\n",
    "for record in iris_tensor:\n",
    "    print(record)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`dataset.filter()` will apply a filter to every record and only return those for which the predicate (filter function) is `True`. for example, the zeroeth record in our `iris` dataset has a zeroeth element of 5.1, so let's look at a function which checks for first elements `<= 5.0`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def five_filter(x):\n",
    "    return x[0] <= 5.0\n",
    "\n",
    "iris_tensor = (tf.data.Dataset\n",
    "               .from_tensor_slices(df.values)\n",
    "               # here we filter with a lambda function\n",
    "               .filter(five_filter))\n",
    "for record in iris_tensor:\n",
    "    print(record)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`dataset.map()` will apply a single function to every record in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def double(x):\n",
    "    return 2 * x\n",
    "\n",
    "iris_tensor = (tf.data.Dataset\n",
    "               .from_tensor_slices(df.values)\n",
    "               # here we apply the function with map\n",
    "               .map(double))\n",
    "for record in iris_tensor:\n",
    "    print(record)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`dataset.shuffle()` will shuffle the records before iterating through them. in order to do this, `tensorflow` must know how many elements forward it should look for shuffling -- this doesn't have to be the entire dataset, but that's the number we'll use. we'll look at the first few elements and compare with the dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_tensor = (tf.data.Dataset\n",
    "               .from_tensor_slices(df.values)\n",
    "               .shuffle(buffer_size=x.shape[0]))\n",
    "for (i, record) in enumerate(iris_tensor):\n",
    "    print(record)\n",
    "    if i == 2:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for more information on `datasets`, please [refer to the documentation](https://www.tensorflow.org/guide/datasets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `tf.estimator` `api` basics\n",
    "\n",
    "so you've managed to get your data loaded as a `dataset`. good for you!\n",
    "\n",
    "now, let's build a model!\n",
    "\n",
    "but first, we have a decision to make: which `api` to use?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "by default you should be thinking `keras`. *however*, there are certain situations in which you might want to use a different high-level `api` for modelling: the `estimator` `api`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for motivation, here are the reasons the `tensorflow` docs themselves give for using `estimator`s:\n",
    "\n",
    "> Estimators provide the following benefits:\n",
    ">\n",
    "> + You can run Estimator-based models on a local host or on a distributed multi-server environment without changing your model. Furthermore, you can run Estimator-based models on CPUs, GPUs, or TPUs without recoding your model.\n",
    "> + Estimators provide a safe distributed training loop that controls how and when to:\n",
    ">   + load data\n",
    ">   + handle exceptions\n",
    ">   + create checkpoint files and recover from failures\n",
    ">   + save summaries for TensorBoard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "like most frameworks, in using the `estimator` framework for model building / training we are **inverting control**: rather than `tensorflow` providing us with a handful of functions and us writing the over-arching control flow and logic for using those functions, here `tensorflow` is asking us to define a few simple functions or objects and it will then use them to do any number of more complicated tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "those functions or objects we must provide to our `estimator` are:\n",
    "\n",
    "1. `input_fn`: a function which returns a `dict` of `column_name: feature_tensor` key-value pairs and a `tensor` of `label`s\n",
    "    + you may need to define different functions for different modes (e.g. training, eval)\n",
    "1. a sort of schema for the columns in the loaded dataset in the form of a list of `tf.feature_column` objects\n",
    "    + `tf.feature_column` objects which indicate the name, data type, and required pre-processing for each feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "once that has been done, you need to\n",
    "\n",
    "1. build the estimator using one of the `tf.estimator` classes, and\n",
    "1. run one of the `train`, `evaluate`, or `predict` methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### an example\n",
    "\n",
    "let's do the simplest thing we can: a logistic regression.\n",
    "\n",
    "+ are we going to be dealing with big data? no!\n",
    "+ are we going to do distributed training? no!\n",
    "+ are we going to deploy this to production using TFX? no!\n",
    "\n",
    "should we be using the `estimator` `api`? probably not!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "but I ~~re~~digress. start by splitting our `iris` data into a train and test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y,\n",
    "                                                    random_state=1337,\n",
    "                                                    stratify=y,\n",
    "                                                    test_size=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we can create an `input_fn` in a straight-forward way (we have to clean up the feature names first):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = [fn.replace(' ', '_').replace('(', '').replace(')', '')\n",
    "                 for fn in iris.feature_names]\n",
    "\n",
    "def input_fn_train():\n",
    "    feature_dict = {feature_name: x_train[:, i]\n",
    "                    for (i, feature_name) in enumerate(feature_names)}\n",
    "    return feature_dict, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def input_fn_test():\n",
    "    feature_dict = {feature_name: x_test[:, i]\n",
    "                    for (i, feature_name) in enumerate(feature_names)}\n",
    "    return feature_dict, y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "as our columns are each numeric, creating our \"schema\" is also easy. if our data was already normalized it would be trivial, but as it isn't we will add some normalization functions into this step -- note that this means we are assuming data comes in un-normalized going forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_columns = []\n",
    "\n",
    "for (i, feature_name) in enumerate(feature_names):\n",
    "    # build the zscore normalization function here\n",
    "    x_i_mean = x_train[:, i].mean()\n",
    "    x_i_std = x_train[:, i].std()\n",
    "    zscore = lambda x_i: (x_i - x_i_mean) / x_i_std\n",
    "\n",
    "    # add it to the list\n",
    "    feature_columns.append(\n",
    "        tf.feature_column.numeric_column(feature_name,\n",
    "                                         normalizer_fn=zscore))\n",
    "\n",
    "feature_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "now we can initialize our estimator. one quick peak at the docs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.estimator.LinearClassifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# make sure the logging level is at the default\n",
    "tf.get_logger().setLevel('INFO')\n",
    "estimator = (tf.estimator\n",
    "             .LinearClassifier(feature_columns=feature_columns,\n",
    "                               # the number of classes, len(iris.target_names)\n",
    "                               n_classes=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and, finally, we use this model to `train`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator.train?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "estimator.train(input_fn=input_fn_train,\n",
    "                steps=10000)\n",
    "\n",
    "# turning down the logging level now\n",
    "tf.get_logger().setLevel('ERROR')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "our results on training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "result = estimator.evaluate(input_fn_train, steps=1)\n",
    "\n",
    "for key, value in sorted(result.items()):\n",
    "    print(f'{key}: {value:0.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and on test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "result = estimator.evaluate(input_fn_test, steps=1)\n",
    "\n",
    "for key, value in sorted(result.items()):\n",
    "    print(f'{key}: {value:0.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "for more information on `estimator`s, please [refer to the documentation](https://www.tensorflow.org/guide/estimators)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# turn the logger back on...\n",
    "tf.get_logger().setLevel('INFO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `gpu` and `tpu` acceleration\n",
    "\n",
    "we have the ability to accelerate our computations with different hardware -- specifically, `gpu`s and `tpu`s. the way we do this in `tensorflow` is different for each type (though I expect in the long run they will converge. currently, executing computations on `gpu`s is trivial, whereas executing computations on a `tpu` requires a bit more effort"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `gpu`s\n",
    "\n",
    "`tensorflow` has an ingrained understanding of the \"device\" you intend to use to do your computation. currently the supported devices are `cpu`s and `gpu`s, and any model built to be executed on one can seamlessly be executed on the other with minimal code alteration. the same is not true of `tpu`s (more on this in the following)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### `gpu`-capable installation\n",
    "\n",
    "the *ability* to put computations on a `gpu` is something that your particular installation of `tensorflow` either has or doesn't -- it comes down to how you installed `tensorflow`.\n",
    "\n",
    "if you look at [the main `tensorflow` installation page](https://www.tensorflow.org/install/), there are two installation methods: `pip` and `docker`. coincidentally, `tensorflow` is also available via `conda`, but if you are aiming for `gpu` use you may want to consider the \"official\" installation methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "as you can read on [the `pip` install page](https://www.tensorflow.org/install/pip), the main difference between `gpu`-capable and non-`gpu`-capable `tensorflow` installations is the `python` package installed. the base package is called `tensorflow` and the `gpu`-capable package is called (shocker!) `tensorflow-gpu`.\n",
    "\n",
    "if you are using `conda` to manage your environment, you can ignore the discussion about creating virtual environments using `virtualenv` -- just make sure you are in your desired `conda` environment before `pip` installing, and verify that `which pip` returns the path to your `conda` environment's `pip` executable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "meanwhile, [the `docker` install](https://www.tensorflow.org/install/docker) is perhaps even more trivial. for this method of installation we have a number of options to choose from, and once we have made our decisions we use our selected values to build our `docker` image's tag:\n",
    "\n",
    "+ the library name is `tensorflow/tensorflow`\n",
    "+ the base of the tag depends on the particular snapshot of the code base\n",
    "    + `latest` for the latest stable available\n",
    "    + `nightly` for whatever was in the codebase last night (not stable)\n",
    "    + `X.Y.Z` for a specific version of the codebase\n",
    "    + `devel` for a version that includes the source code\n",
    "    + `custom-op` for something crazy you probably don't need\n",
    "+ do we need `gpu` support? add a `-gpu` if so\n",
    "+ do we want to use `jupyter`? add a `-jupyter` if so"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so to get the `docker` image with `gpu` support and `jupyter` notebooks\n",
    "\n",
    "+ `tensorflow/tensorflow`\n",
    "+ the `latest` tag\n",
    "+ add a `-gpu` for `gpu` support\n",
    "+ add a `-jupyter` for `jupyter` to be installed in our container\n",
    "\n",
    "and our tag will be `tensorflow/tensorflow:latest-gpu-jupyter`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it is assumed that you will be working with `nvidia` `gpu`s. in order to verify whether or not that is the case, run (`*nix` only)\n",
    "\n",
    "```sh\n",
    "lspci | grep -i nvidia\n",
    "```\n",
    "\n",
    "machines with no `gpu`s will produce nothing, whereas machines with `gpu`s will produce something like\n",
    "\n",
    "```\n",
    "05:00.0 VGA compatible controller: NVIDIA Corporation Device 1d81 (rev a1)\n",
    "05:00.1 Audio device: NVIDIA Corporation Device 10f2 (rev a1)\n",
    "06:00.0 VGA compatible controller: NVIDIA Corporation Device 1d81 (rev a1)\n",
    "06:00.1 Audio device: NVIDIA Corporation Device 10f2 (rev a1)\n",
    "09:00.0 VGA compatible controller: NVIDIA Corporation Device 1d81 (rev a1)\n",
    "09:00.1 Audio device: NVIDIA Corporation Device 10f2 (rev a1)\n",
    "0a:00.0 VGA compatible controller: NVIDIA Corporation Device 1d81 (rev a1)\n",
    "0a:00.1 Audio device: NVIDIA Corporation Device 10f2 (rev a1)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "finally, to launch a container with `gpu` support, run\n",
    "\n",
    "```sh\n",
    "# --runtime nvidia: using the nvidia runtime takes care of\n",
    "#     much of the complexity of mapping your local host's gpu\n",
    "#     devices to the container (i.e. getting your container to\n",
    "#     \"see\" your local machine's gpus)\n",
    "# -it: make it *i*nteractive and create a *t*ty terminal\n",
    "# --rm: remove the container after we're done\n",
    "# -p 8888:8888 : map the container's internal port 8888 to the\n",
    "#     local host's port 8888 (so you can connect to jupyter)\n",
    "docker run --runtime nvidia -it --rm -p 8888:8888 tensorflow/tensorflow:latest-gpu-jupyter\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "with this done, you will be able to log in to a `jupyter` notebook at `localhost:8888` and execute `tensorflow` code that will run on `gpu`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### placing `operation`s on devices\n",
    "\n",
    "among the many `tensorflow` `operation`s, *some* have implementations on `gpu`s and some do not. left to its own devices (pun intended), `tensorflow` will always attempt to execute a `gpu`-capable operation on a `gpu` before a `cpu`. you don't have to do anything"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "to see this in action on a machine with a `gpu` and in an environment that supports `gpu` execution, run\n",
    "\n",
    "```python\n",
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "# Create some tensors\n",
    "a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "c = tf.matmul(a, b)\n",
    "\n",
    "print(c)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "even though I said nothing in the above code about `gpu`s, on a `gpu`-capable machine I see that my `matmul` operation occurred on my `gpu`:\n",
    "\n",
    "```\n",
    "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n",
    "tf.Tensor(\n",
    "[[22. 28.]\n",
    " [49. 64.]], shape=(2, 2), dtype=float32)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "if you want a bit more control than that, you simply tell `tensorflow` where you want the code to execute. `tensorflow` has names for these devices:\n",
    "\n",
    "+ `cpu` is `/cpu:0` (as in the first (index 0) `cpu`)\n",
    "+ the first `gpu` is `/device:GPU:0`, or `/GPU:0` for short\n",
    "+ the second `gpu` is `/GPU:1`\n",
    "+ etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you can choose to deliberately locate any `tensor` or `operation` on a specific single device with a `context` manager:\n",
    "\n",
    "```python\n",
    "with tf.device('/device:GPU:0'):\n",
    "    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')\n",
    "    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])\n",
    "    c.append(tf.matmul(a, b))\n",
    "```\n",
    "\n",
    "the above code will place the `a`, `b`, and `c` `tensor`s on the first `gpu`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are many more details, for your particular edge case, and your best bet is to check [the `tensorflow` documentation](https://www.tensorflow.org/guide/using_gpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `tpu`s\n",
    "\n",
    "the actual details of implementing code on a `tpu` are beyond the scope of this lecture -- it suffices to say it is not as simple as `with tf.device('/devices:TPU:0')`, though I expect that will eventually be the case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the basic idea is that you replace some of the basic elements of `tensorflow` with classes that duplicate their `api` but for `tpu` use\n",
    "\n",
    "+ `tf.estimator.Estimator` becomes `tf.contrib.tpu.TPUEstimator`\n",
    "    + you must add a parameter `use_tpu=True`\n",
    "    + you must add a `config` (previously this has always been the default run config) with value `tf.contrib.tpu.RunConfig()`\n",
    "+ your chosen training optimizer (above, for the logistic regression, we used `tf.train.FtrlOptimizer`) must be wrapped inside a call to `tf.contrib.tpu.CrossShardOptimizer`\n",
    "+ replace your `tf.estimator.EstimatorSpec` (previously left as default) with a `tf.contrib.tpu.TPUEstimatorSpec`\n",
    "    + this has implications for how you define you metrics that you must resolve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the actual implementation of these code changes is not trivial and requires quite a bit of reference to existing `google` code and [the current somewhat under-complete documentation](https://www.tensorflow.org/guide/using_tpu).\n",
    "\n",
    "for a faster start, consider doing one of the pre-canned tutorials, three of which are listed at the top of the above documentation page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### executing on `google` `colab`\n",
    "\n",
    "recently `google` added both `gpu` and `tpu` support to `google` `colab`. to use an accelerator device, select \"Runtime > Change runtime type > Hardware accelerator\"\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1CcA73qR3-FF6zvvuylD8Wg0zmSKHaBAa\" width=\"1000px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "whew. that was a lot of `tensorflow`... feeling ready to easily crank out deep neural net models?\n",
    "\n",
    "me neither.\n",
    "\n",
    "if only there were an easier way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `keras`\n",
    "\n",
    "`keras` is a backend-agnostic `api` for creating high-level neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it is **backend-agnostic** in that it is *not* implementing neural network computations directly, but rather utilizing *other* libraries that have implemented those computations and providing them to you, the `keras` user, under a unified single `api`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "to put it another way: they've figure out how you would create a simple `CNN` layer in multiple implementation libraries (e.g. `tensorflow`, `cntk`, and `theano`) and given you *one* set of functions that will (under the hood) call out to whichever of those libraries you've installed.\n",
    "\n",
    "`keras` will allow you to write simpler, higher-level, portable neural net code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `keras` vs. `tf.keras`\n",
    "\n",
    "`keras` is itself [an independently developed `python` library](https://github.com/keras-team/keras) which you could install via `pip` or `conda`, and use without any `tensorflow` installation at all. it is completely open source and is developed by a team of OSS developers, and is free to contribute to at any time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in a sense, `keras` is also an `api` specification -- a statement about what *interface* a deep neural net developer could use to create backend-agnostic neural net models. if I write code using the `keras` `api`, someone has created an under-the-hood implementation of that code that leverages `tensorflow`, `cntk`, or `theano` for me, and I can use any of them as I see fit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`google`, in an effort to make their library more user-friendly, incorporated the `keras` **`api`** into the `tensorflow` package as a submodule: `tensorflow.keras`\n",
    "\n",
    "this library contains `google`'s own custom implementation of the `keras` `api` that is completely integrated with (and only with) `tensorflow` core code and writen by `tensorflow` core developers. the author of `keras` discusses this in a [blog post from 2017](https://blog.keras.io/introducing-keras-2.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in summary,\n",
    "\n",
    "+ the `keras` package is *open source* and implements the `keras` `api` in *several* backends\n",
    "+ the `tf.keras` module is *developer by google* and implements the `keras` `api` for `tensorflow` *only*\n",
    "\n",
    "*note*: additionally, the most recent version of `tf.keras` and `keras` may not be the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "practically speaking, while there *may* be some performance differences (with the `tf.keras` implementation being faster for `tensorflow`), the resulting *code* should be immediately transferable from one library to the other because they are implementing a shared `api`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "given `keras`-`api` code, you simply swap which way you `import` the `keras` package:\n",
    "\n",
    "```python\n",
    "import keras\n",
    "```\n",
    "\n",
    "or\n",
    "\n",
    "```python\n",
    "from tensorflow import keras\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "just to make things easier, since tensorflow is already installed and running, let's use the `tf.keras` version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### general `keras` workflow\n",
    "\n",
    "`keras` is a unified deep neural net `api`, so there is an assumed way of creating models:\n",
    "\n",
    "+ create a `keras.models` object to collect your neural net layers\n",
    "+ add layers to your model\n",
    "+ `compile` your model to configure the learning process\n",
    "    + here we specify things like [the `loss` function](https://keras.io/losses/) we wish to optimize, [the `optimizer`](https://keras.io/optimizers/) (optimization algorithm), [`metrics`](https://keras.io/metrics/) we wish to track along the way, etc\n",
    "+ `fit` the model to labelled training data\n",
    "+ `evaluate` the model on test records\n",
    "+ use the model to `predict` the outcome of input predictors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### a simple example\n",
    "\n",
    "the simple example found on [the main `keras` landing page](https://keras.io/) is a good walkthrough of the above workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### create a `keras.models` object\n",
    "\n",
    "we start off by creating a `keras.models` object using the sequential `api` (*\"as opposed to what?\" you ask -- more on this later*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "model = Sequential()\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### add layers\n",
    "\n",
    "this `model` object is a container into which we put our neural net layers.\n",
    "\n",
    "a simple neural net is defined by the number of layers it has (depth), the number of nodes it has in each of those layers, the activation functions used in each of those layers, and perhaps some other information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for the very first layer, we must provide an extra piece of information. recall the neural network picture from before:\n",
    "\n",
    "<br><img src=\"http://cs231n.github.io/assets/nn1/neural_net2.jpeg\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for the hidden layers and the output layer, we already know the dimension of the individual records coming *into* each layer -- it's the number of nodes in the previous layer.\n",
    "\n",
    "for the very first layer, that is not known, and is defined by our input data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's work with the `iris` dataset again, where our input records have four elements in them (4). we need to reload it and resplit it after killing our kernel up above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn.datasets, sklearn.model_selection\n",
    "\n",
    "iris = sklearn.datasets.load_iris()\n",
    "x = iris.data\n",
    "y = iris.target\n",
    "\n",
    "x_train, x_test, y_train, y_test = sklearn.model_selection.train_test_split(\n",
    "    x, y, random_state=1337, stratify=y, test_size=0.3)\n",
    "\n",
    "input_dim = x.shape[1]\n",
    "input_dim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "furthermore, let's arbitrarily choose our neural net architecture (number of layers, nodes per layer, activation, etc) to be\n",
    "\n",
    "+ a first layer that is a dense (fully connected) layer of 64 neurons with a `relu` activation\n",
    "+ a second that is a dense (fully connected) layer of 10 neurons with a `relu` activation\n",
    "+ a final that is a dense (fully connected) layer with a 3-category softmax activation\n",
    "    + the softmax activation is making our prediction among the three possible `iris` categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we specify all layers with calls to a `keras.layers` object such as the onese created by the `keras.layers.Dense` class, and we add them to our `model` (a thing which collects `layer` objects) via `model.add`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "model.add(Dense(units=64, activation='relu', input_dim=input_dim))\n",
    "model.add(Dense(units=10, activation='relu'))  # input_dim inferred from previous layer\n",
    "model.add(Dense(units=3, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "at any time we may access a convenient summary of our `keras` model with the (surprise!) `.summary()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `compile`\n",
    "\n",
    "we specify the way (`optimizer`) that our `model` will be optimized, as well as the goal of that optimization (the `loss` function we wish to minimize) in the `model.compile` function. we can also list the extra `metrics` we care to track over our iterations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `fit`\n",
    "\n",
    "having completely defined our deep learning model, we can now `fit` it to our training data. the only complication is that our output is currently assumed to be a 3-element vector (a one-hot encoding) of the known category. we can build this encoding with the `tf.keras.utils.to_categorical` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_onehot = keras.utils.to_categorical(y_train)\n",
    "y_train_onehot[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# this should take a minute or two. set verbose = 1 if you want progress info\n",
    "model.fit(x_train,\n",
    "          y_train_onehot,\n",
    "          epochs=1000,  # number of times we will iterate over the dataset\n",
    "          verbose=0,  # 0 is silent, 1 is a progress bar, and 2 is a line per epoch\n",
    "          validation_split=0.1)  # hold out some validation data and evaluate at every step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### diversion: `model.history`\n",
    "\n",
    "after fitting, each model acquires a useful attribute `model.history`, which itself has an attribute `model.history.history`. this object is a dictionary which contains the after-each-epoch values of whatever `loss`es and `metric`s were recorded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for example, we just recorded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.history.history.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you could easily plot these values over time (though you may choose to use `tensorboard` for that, more later):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import plotly.graph_objs as go\n",
    "\n",
    "h = model.history.history\n",
    "x = list(range(len(model.history.history['loss'])))\n",
    "data = [go.Scatter(x=x, y=h['loss'], name='training loss'),\n",
    "        go.Scatter(x=x, y=h['val_loss'], name='validation loss')]\n",
    "go.Figure(data).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "data = [go.Scatter(x=x, y=h['accuracy'], name='training accuracy'),\n",
    "        go.Scatter(x=x, y=h['val_accuracy'], name='validation accuracy')]\n",
    "go.Figure(data).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `evaluate`\n",
    "\n",
    "our trained model can now be used to evaluate on our held-out test data -- remember, we have to one-hot encode that too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test_onehot = keras.utils.to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_loss, test_acc = model.evaluate(x_test, y_test_onehot)\n",
    "print(f\"test loss: {test_loss}\")\n",
    "print(f\"test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `predict`\n",
    "\n",
    "and for our held out records we also get softmax predictions, so a probability of every category for a given record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "probas = model.predict(x_test)\n",
    "probas[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "probas[:3].argmax(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict_classes(x_test)\n",
    "predictions[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# the level of certainty we had in our greatest predictions\n",
    "predicted_probas = probas[np.arange(probas.shape[0]), predictions]\n",
    "predicted_probas[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [go.Histogram(x=predicted_probas)]\n",
    "go.Figure(data).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### summary\n",
    "\n",
    "let's bring the above example together into one code block:\n",
    "\n",
    "```python\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "input_dim = x.shape[1]\n",
    "# note: you can replace multiple model.add with a list passed to Sequential\n",
    "model = Sequential([Dense(units=64, activation='relu', input_dim=input_dim),\n",
    "                    Dense(units=10, activation='relu'),\n",
    "                    Dense(units=3, activation='softmax')])\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "y_train_onehot = keras.utils.to_categorical(y_train)\n",
    "model.fit(x_train, y_train_onehot, epochs=1000, verbose=0, validation_split=0.1)\n",
    "y_test_onehot = keras.utils.to_categorical(y_test)\n",
    "test_loss, test_acc = model.evaluate(x_test, y_test_onehot)\n",
    "print(f\"test loss: {test_loss}\")\n",
    "print(f\"test accuracy: {test_acc}\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "hopefully you recognize at this point the value of the `keras` `api` in terms of normalizing and simplifying our neural net model development!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### sequential vs. functional `api`s\n",
    "\n",
    "the way we constructed our model up above used what `keras` calls its [\"sequential\" `api`](https://keras.io/getting-started/sequential-model-guide), in which you create a `Sequential` model object and use that to add on layers of the `tf.keras.layers` object type one layer at a time.\n",
    "\n",
    "if you models you are attempting to build are *sequential* and are composed out of the simplest provided `layer` types, you should default to the sequential `api`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "if, on the other hand, your model architecture is not strictly linear -- perhaps it include branches that split off and join together again, as in a ladder network or a variational auto-encoder; or it include multiple copies of one portion being used in tandem, as in a siamese network -- you may be better off using [the functional `api`](https://keras.io/getting-started/functional-api-guide)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "while in the sequential `api` we treat a model as a list of consecutive layers all concatenated together, the functional `api` hearkens back to the `tensor` graph of base `tensorflow`: every layer (e.g. `Dense(64)`) is an *object*, and each *object* can be *called* as if it were a function. when you *call* the layer, you are effectively attaching some tensor as an *input* to that layer (in effect, building the execution graph)\n",
    "\n",
    "this means, among other things, that you can attach layers in non-sequential ways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it is straight-forward to replicate our work above. we start by defining the different layers (now we are thinking of these as `tensor`s). the first is an input layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Input\n",
    "\n",
    "# input_dim = 4\n",
    "inputs = Input(shape=(4,))\n",
    "inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we construct the graph of `tensor`s one layer-call at a time by declaring what a given layer's input is and capturing the output as a variable. schematically this looks like:\n",
    "\n",
    "```python\n",
    "layer_output = NewLayer(...)(layer_input)\n",
    "```\n",
    "\n",
    "this is really just a very compact way of writing\n",
    "\n",
    "```python\n",
    "new_layer = NewLayer(...)\n",
    "layer_output = new_layer(layer_input)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in practice the intermediate `layer_output` variable is often repeatably called `x` (because we don't need it after we've defined the graph). thus, our layers collectively are built like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = Input(shape=(4,))\n",
    "x = Dense(units=64, activation='relu', autocast=False)(inputs)\n",
    "x = Dense(units=10, activation='relu', autocast=False)(x)\n",
    "predictions = Dense(units=3, activation='softmax', autocast=False)(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "after creating the execution graph this way, just as with `tensorflow` we must define our `input` and our `output` `tensor`s in order to tell the training computation environment which graph components we need to execute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Model\n",
    "\n",
    "model = Model(inputs=inputs, outputs=predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "at this point, everything will proceed the same way it did before. we `compile` the model to define the training algorithm, and then we `fit` the model to training data and `predict` on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "y_train_onehot = keras.utils.to_categorical(y_train)\n",
    "model.fit(x_train, y_train_onehot, epochs=1000, verbose=0, validation_split=0.1)\n",
    "y_test_onehot = keras.utils.to_categorical(y_test)\n",
    "test_loss, test_acc = model.evaluate(x_test, y_test_onehot)\n",
    "print(f\"test loss: {test_loss}\")\n",
    "print(f\"test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "additionally, the **`model` itself** is also callable, returning the predicted values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model(x_test)\n",
    "predictions[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "remember, the motivation for having this `api` is to make more complicated network architectures, especially ones in which we re-use components. when in doubt, **try the sequential `api` first**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### callbacks\n",
    "\n",
    "one last item on `keras` before we call it a day: `keras` has a notion of `callbacks` -- lists of functions that are meant to be called every time you pass some milestone in training. for example, the way we get our progress info printed to screen is via a `callback` function that is activated after every `epoch`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`keras` defines [a handful of useful native `callback` objects](https://keras.io/metrics/), and also provides users with the ability to define their own via inheriting from the `keras.callbacks.Callback` class.\n",
    "\n",
    "`callback`s can be registered to the beginning and end of\n",
    "\n",
    "+ the entire training run\n",
    "+ any single epoch within a training run\n",
    "+ any single batch within an epoch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "among the various callbacks, there are a few worth special mention\n",
    "\n",
    "+ `ModelCheckpoint`: this callback will trigger after every epoch and save a checkpoint version of your current model to some `filepath` value\n",
    "    + importantly, you can set the `save_best_only` parameter to be `True`, in which case the checkpoint will only be written if the last epoch resulted in the best-ever value for your optimization loss (that is, your final checkpoint value will be whatever your best epoch was per your loss function)\n",
    "+ `EarlyStopping`: this callback will fire after every epoch and will terminate the training early if some monitored quantity (e.g. your validation set loss) has not improved in some number of steps\n",
    "    + this requires you defining \"improved\" and \"some number\", but is straight-forward\n",
    "+ `TensorBoard`: this callback will log history in a file consumable by the `tensorboard` application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### callback example\n",
    "\n",
    "the following runs our exact same model as before, but this time when we `fit` the model we provide a `ModelCheckpoint` `callback` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import ModelCheckpoint\n",
    "ckpt_callback = ModelCheckpoint(filepath='/tmp/weights.hdf5',\n",
    "                                      verbose=1,\n",
    "                                      save_best_only=True)\n",
    "model.fit(x_train, y_train_onehot, epochs=100, verbose=0, validation_split=0.1,\n",
    "          callbacks=[ckpt_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the result of this is a checkpointed model file, saved *in the `keras` way* as an `hdf5` file of weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -alh /tmp/weights*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we can load a model from a checkpoint at any time via"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_model = keras.models.load_model('/tmp/weights.hdf5')\n",
    "loaded_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<strong><em><div align=\"center\">it's cold outside, wear layers</div></em></strong>\n",
    "<div align=\"center\"><img src=\"https://cdn-images-1.medium.com/max/1600/1*U_mJ4Yq7pUctpFYwlx1u0g.jpeg\" width=\"500px\"></div>\n",
    "\n",
    "# END OF LECTURE\n",
    "\n",
    "next lecture: [`hadoop`](015_hadoop.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "# appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "## tensorflow `operation`s, `tensor`s, and the execution graph (v1)\n",
    "\n",
    "***NOTE***: the following cells pertain to `tensorflow 1.x` **only**. they are left here as an indication of some of the differences between `tf1` and `tf2`, because `tf1` is still very common"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "what the diagram above (c.f. [the execution graph](#the-execution-graph) section) shows is **`operation` nodes** connected by **`tensor` edges**, i.e. the graph.\n",
    "\n",
    "the ultimate output values are not calculated yet, but if we asked `tensorflow` nicely, saying \"pretty pretty please\" and all that, it will `flow` from the ultimate input `tf.constant` `tensor`s down to whatever output `tensor` we desire."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "if you want to get any information out of any `tensor` edge or `operation` node in this graph, you need to *evaluate* it. `tensorflow` will then perform all the calculations required to get from an ultimate input source node to the requested edge or node.\n",
    "\n",
    "you must ask for these values from within a ***session***: you create a context in which `tensorflow` knows what inputs it should expect (you define them when you `run` the session!) and which outputs to return (you are declaring this in real time)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "there are a few ways to do this. first, for any `tensor` object you may call the `eval` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    mysum_value = mysum_op.eval()\n",
    "mysum_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "alternatively, we could have asked the `tf.Session` object to `run`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "sess.run?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "and request any number of `tensor`s or `operation`s we wanted evaluated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    x = sess.run([mymult_op, mycombo_op])\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "*note: any request for an `operation` to be evaluated within a `tf.Session.run` call will return `None` if successful and will raise an error otherwise*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "this paradigm of creating a `tf.Session` and then evaluating `tensor`s and `operation`s from within that session is often referred to in the `tensorflow` docs as \"graph execution\", as juxtaposed with eager execution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "## graph execution vs. eager execution\n",
    "\n",
    "to make a long story short, `google` tried to make everyone use `tensorflow` in their preferred more rigorous way and everyone hated it, so `google` relented and now let's everything do things the way that you hoped it would work all along\n",
    "\n",
    "when developing code, this extremely structured way of doing things (build a computation graph, then run it in a session) can be... pretty annoying.\n",
    "\n",
    "the google developers created an \"eager execution\" functionality to address exactly this problem. you can\n",
    "\n",
    "1. develop your code in \"eager execution\" mode -- get the results of your operation immediately\n",
    "1. remove one line of code from the beginning of your developed file and put everything inside a `tf.Session` to get the \"production\" behavior"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "note: choosing to work with graph execution or eager execution is a one-time-only decision. if you've started creating graphs with `with tf.Session()` or started performing eager execution (see below), you can't change to the other. you must kill your `python` session and restart to switch. so here, we will restart our `jupyter` kernel and start anew"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "# NOTE:\n",
    "# you must restart your kernel if you want to do this!!!\n",
    "import tensorflow as tf\n",
    "try:\n",
    "    tf.enable_eager_execution()\n",
    "except ValueError:\n",
    "    print(\"restart your kernel!!!!\")\n",
    "    raise\n",
    "\n",
    "mysum_op = tf.add(1, 1)\n",
    "mysum_op"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "mysum_op.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "note: this was calculated for us, no need to create a `tf.Session`. yep, that's the entire point!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `dataset` digression: making iterators\n",
    "\n",
    "in `tf1`, each `Dataset` object is a thing which can *create* an iterator (it is not, itself, a thing which can be iterated). there are a number of methods for creating an `iterator` (an object we can iterate over to yield our individual row-records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "these methods are\n",
    "\n",
    "+ `dataset.make_one_shot_iterator()`: will create a \"one shot\" iterator which we can iterate over exactly once\n",
    "+ `dataset.make_initializable_iterator()`: will give us an iterator we must initialize with some `tf.placeholder` value\n",
    "    + depending on exactly how we create the `dataset` that we then use to create an initializable iterator, this could be something we have to initialize\n",
    "        + once and only once (*initializable*)\n",
    "        + multiple times (*reinitializable*)\n",
    "        + conditionally (*feedable*)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}