Part 2 - Neural Networks in PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural networks with PyTorch\n",
    "\n",
    "Next I'll show you how to build a neural network with PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import things like usual\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "import helper\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from torchvision import datasets, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First up, we need to get our dataset. This is provided through the `torchvision` package. The code below will download the MNIST dataset, then create training and test datasets for us. Don't worry too much about the details here, you'll learn more about this later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a transform to normalize the data\n",
    "transform = transforms.Compose([transforms.ToTensor(),\n",
    "                              transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n",
    "                             ])\n",
    "# Download and load the training data\n",
    "trainset = datasets.MNIST('MNIST_data/', download=True, train=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)\n",
    "\n",
    "# Download and load the test data\n",
    "testset = datasets.MNIST('MNIST_data/', download=True, train=False, transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataiter = iter(trainloader)\n",
    "images, labels = dataiter.next()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have the training data loaded into `trainloader` and we make that an iterator with `iter(trainloader)`. We'd use this to loop through the dataset for training, but here I'm just grabbing the first batch so we can check out the data. We can see below that `images` is just a tensor with size (64, 1, 28, 28). So, 64 images per batch, 1 color channel, and 28x28 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfoAAAH0CAYAAADVH+85AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAHRRJREFUeJzt3X2sbWV9J/DvT297GUgBJW3tTUcRplcSWmHAgkIGAVMHRS0UsPzRio2aypBRfJm0sdq5UCexcSK+X2qpvanGoS2kGqcUnPAiUHBMLwEkCmgBGYIUAQURQV6e+WOvW2+P59yXvfa9+5znfD7Jzjp7rfWs53eXS7772Xu9VGstAECfnjXvAgCAXUfQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DH1sy7gF2hqu5MsneSu+ZcCgBMa/8kj7TWXjhmI10GfSYh/9zhBQCr1ly/uq+qX66qT1fVvVX1RFXdVVUfrqrnjNz0XbOoDwDm7K6xG5jbiL6qDkxyXZJfSPKFJLcmOSLJ25OcUFVHt9YenFd9ANCDeY7oP5lJyL+ttXZSa+0PW2vHJzkvyYuS/I851gYAXajW2u7vtOqAJP+cyVcSB7bWntlq2c8l+U6SSvILrbUfTrH9zUkOm021ADA3N7TWDh+zgXmN6I8fpl/aOuSTpLX2gyT/mGTPJC/d3YUBQE/m9Rv9i4bp7Uss/2aSVyZZn+TypTYyjNwXc9D0pQFAP+Y1ot9nmD68xPIt8/fdDbUAQLeW63X0NUy3eQLBUr9b+I0eACbmNaLfMmLfZ4nley9YDwCYwryC/rZhun6J5b8yTJf6DR8A2AHzCvorh+krq+rf1DBcXnd0kh8l+cruLgwAejKXoG+t/XOSL2Vyw/6zFiw+J8leSf5qmmvoAYCfmOfJeP8lk1vgfrSqXpHkG0mOTHJcJl/Z/9EcawOALsztFrjDqP4lSTZlEvDvSnJgko8meZn73APAeHO9vK619v+S/N48awCAns31MbUAwK4l6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY2vmXQCsduvWrZu67Yc+9KFRfb/+9a+fuu33vve9UX2ffPLJo9pfffXVo9rDamFEDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAd8zx6GOnEE08c1f5Tn/rU1G1/6Zd+aVTfY+y5556j2l9++eWj2n/yk5+cuu3b3/72UX3DSjK3EX1V3VVVbYnXffOqCwB6Mu8R/cNJPrzI/Ed3dyEA0KN5B/33W2sb5lwDAHTLyXgA0LF5j+jXVtXvJHl+kh8muTnJ1a21p+dbFgD0Yd5B/7wkn1kw786q+r3W2pe317iqNi+x6KDRlQFAB+b51f1fJnlFJmG/V5JfS/JnSfZP8g9Vdcj8SgOAPsxtRN9aO2fBrFuSvLWqHk3yriQbkpy8nW0cvtj8YaR/2AzKBIAVbTmejHf+MD1mrlUAQAeWY9DfP0z3mmsVANCB5Rj0Lxumd8y1CgDowFyCvqoOrqrnLjL/BUk+Prz97O6tCgD6M6+T8U5L8odVdWWSO5P8IMmBSU5MskeSS5L8zznVBgDdmFfQX5nkRUn+YyZf1e+V5PtJrs3kuvrPtNbanGoDgG7MJeiHm+Fs94Y4sBK8/OUvH9V+zKNmx34ePu+886Zu++lPf3pU31/72tdGtT/zzDOnbnvrrbeO6nvjxo2j2sPutBxPxgMAZkTQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdKzGPs96OaqqzUkOm3cdrA733HPPqPbr1q2buu2dd945qu8DDzxwVPsx3vjGN45q/6lPfWrqts961rgxzmte85qp21566aWj+mbVuaG1dviYDRjRA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdGzNvAuA5eCII46Yuu2+++47w0p2ztq1a0e132OPPaZu+/jjj4/qe9OmTaPaj3nU7MaNG0f1feaZZ07d9sYbbxzV93333TeqPauPET0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdKxaa/OuYeaqanOSw+ZdByvH5z73uanb/vZv//aovp988smp2/74xz8e1ff69eunbruSn4t+6623jmo/Zr9deumlo/p+9atfPao9K84NrbXDx2zAiB4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBja+ZdAKx0119//aj273//+6duu2nTplF9P//5z5+67Up+TO1pp502qv1NN900ddvjjjtuVN+nnHLK1G0vvvjiUX2zMs1kRF9Vp1bVx6rqmqp6pKpaVX12O22OqqpLquqhqnqsqm6uqrOr6tmzqAkAmN2I/r1JDknyaJJ7khy0rZWr6jeTXJzk8SR/neShJK9Ncl6So5OM+7gNACSZ3W/070iyPsneSc7c1opVtXeSP0/ydJJjW2tvaq39tySHJrk+yalVdfqM6gKAVW0mQd9au7K19s3WWtuB1U9N8vNJLmyt/dNW23g8k28Gku18WAAAdsw8zro/fpheusiyq5M8luSoqlq7+0oCgD7NI+hfNExvX7igtfZUkjszOXfggN1ZFAD0aB6X1+0zTB9eYvmW+ftub0NVtXmJRds8GRAAVovleMOcGqY78ns/ALAN8xjRbxmx77PE8r0XrLek1trhi80fRvqH7XxpANCXeYzobxum6xcuqKo1SV6Y5Kkkd+zOogCgR/MI+iuG6QmLLDsmyZ5JrmutPbH7SgKAPs0j6C9K8kCS06vqJVtmVtUeSbbc9HvjHOoCgO7M5Df6qjopyUnD2+cN05dV1abh7wdaa+9OktbaI1X1lkwC/6qqujCTW+C+LpNL7y7K5La4AMBIszoZ79AkZyyYd0B+ci38t5O8e8uC1trnq+rlSf4oySlJ9kjyrSTvTPLRHbzDHgCwHTMJ+tbahiQbdrLNPyZ59Sz6BwAW53n0kOTII4+cuu1XvvKVUX1feulid4PeMZ/73OdG9f2CF7xg6rZf/epXR/U9T7fccsuo9mP+7UccccSovs8666yp23oe/eq0HG+YAwDMiKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI55TC1d2HPPPUe1X7du3Ywq2b3e+c53zruEFam1Nqr9e97znqnbfvGLXxzV99FHHz1126OOOmpU39ddd92o9syHET0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdMzz6OnCqaeeOqr92rVrp2572223jeqbleeKK66Yuu0XvvCFUX2ffvrpU7f94Ac/OKrvo48+elR75sOIHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGMeU0sX1q1bN7e+77333rn1zcrziU98YlT7MY+pZXUyogeAjgl6AOiYoAeAjgl6AOiYoAeAjgl6AOiYoAeAjgl6AOiYoAeAjgl6AOiYoAeAjgl6AOiYoAeAjgl6AOiYoAeAjnkePV147WtfO+8SYIfceeedo9o/+uijM6qE1WImI/qqOrWqPlZV11TVI1XVquqzS6y7/7B8qdeFs6gJAJjdiP69SQ5J8miSe5IctANtbkry+UXm3zKjmgBg1ZtV0L8jk4D/VpKXJ7lyB9rc2FrbMKP+AYBFzCToW2v/GuxVNYtNAgAzMM+T8dZV1e8n2S/Jg0mub63dPMd6AKA78wz63xhe/6qqrkpyRmvt7h3ZQFVtXmLRjpwjAADdm8d19I8l+ZMkhyd5zvDa8rv+sUkur6q95lAXAHRnt4/oW2v3J/njBbOvrqpXJrk2yZFJ3pzkIzuwrcMXmz+M9A8bWSoArHjL5s54rbWnklwwvD1mnrUAQC+WTdAPvjtMfXUPADOw3IL+pcP0jrlWAQCd2O1BX1VHVtXPLjL/+ExuvJMki94+FwDYOTM5Ga+qTkpy0vD2ecP0ZVW1afj7gdbau4e//zTJwcOldPcM816c5Pjh7/e11q6bRV0AsNrN6qz7Q5OcsWDeAcMrSb6dZEvQfybJyUl+PcmrkvxMkn9J8jdJPt5au2ZGNQHAqjerW+BuSLJhB9f9iyR/MYt+AYBt8zx6GOnrX//6vEtgBbn33ntHtX/kkUdmVAmrxXI76x4AmCFBDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAdE/QA0DFBDwAd85haGOmEE04Y1f66666bUSWsBK95zWtGtd9vv/2mbvvtb397VN+sTEb0ANAxQQ8AHRP0ANAxQQ8AHRP0ANAxQQ8AHRP0ANAxQQ8AHRP0ANAxQQ8AHRP0ANAxQQ8AHRP0ANAxQQ8AHRP0ANAxz6OnCw899NDc+j7wwAPn1jcrz9lnnz2q/dq1a6du++CDD47qm5XJiB4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjHlNLF/72b/92VPsTTzxx6rYHH3zwqL7Z/apqVPtzzz136rbHHnvsqL6feOKJqduec845o/pmZTKiB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COeR49Xbj77rtHtX/66aenbjv2efRvfetbp257/vnnj+p7JXvDG94wdds3vvGNo/oe+0z5MT74wQ9O3Xbz5s0zrISVYvSIvqr2q6o3V9XfVdW3qupHVfVwVV1bVW+qqkX7qKqjquqSqnqoqh6rqpur6uyqevbYmgCAiVmM6E9LsjHJd5JcmeTuJL+Y5LeSXJDkVVV1WmutbWlQVb+Z5OIkjyf56yQPJXltkvOSHD1sEwAYaRZBf3uS1yX5+9baM1tmVtV7knw1ySmZhP7Fw/y9k/x5kqeTHNta+6dh/vuSXJHk1Ko6vbV24QxqA4BVbfRX9621K1prX9w65If59yXZ8gPisVstOjXJzye5cEvID+s/nuS9w9szx9YFAOz6s+6fHKZPbTXv+GF66SLrX53ksSRHVdXaXVkYAKwGu+ys+6pak2TLabFbh/qLhuntC9u01p6qqjuTHJzkgCTf2E4fS51CetDOVQsAfdqVI/oPJPnVJJe01i7bav4+w/ThJdptmb/vrioMAFaLXTKir6q3JXlXkluT/O7ONh+mbZtrJWmtHb5E/5uTHLaT/QJAd2Y+oq+qs5J8JMnXkxzXWntowSpbRuz7ZHF7L1gPAJjSTIO+qs5O8vEkt2QS8vctstptw3T9Iu3XJHlhJifv3THL2gBgNZpZ0FfVH2Ryw5sbMwn5+5dY9YphesIiy45JsmeS61prT8yqNgBYrWYS9MPNbj6QZHOSV7TWHtjG6hcleSDJ6VX1kq22sUeS9w9vN86iLgBY7UafjFdVZyQ5N5M73V2T5G1VtXC1u1prm5KktfZIVb0lk8C/qqouzOQWuK/L5NK7izK5LS4AMNIszrp/4TB9dpKzl1jny0k2bXnTWvt8Vb08yR9lcovcPZJ8K8k7k3x06/viAwDTqx4z1eV17Kwbbrhh6raHHnroqL7H/H/wsssu2/5K2/DUU09tf6UlHHDAAaP6Xr/+p87H3Slr1qzMp2xfcMEFo9qPeazxM888s/2VWG5uWOpS8h21q2+BCwDMkaAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomOfRQ5JDDjlk6rbXXnvtqL732muvUe1Xqqoa1f7pp5+euu1tt902qu9zzz136rYXXXTRqL7H/LtZkTyPHgBYmqAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI6tmXcBsBzcdNNNU7fdsGHDqL7f8pa3TN12/fr1o/p+8sknp257zTXXjOp77GNqN27cOHXbsY+KhZXEiB4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOlattXnXMHNVtTnJYfOuAwBGuqG1dviYDRjRA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHRgd9Ve1XVW+uqr+rqm9V1Y+q6uGquraq3lRVz1qw/v5V1bbxunBsTQDAxJoZbOO0JBuTfCfJlUnuTvKLSX4ryQVJXlVVp7XW2oJ2NyX5/CLbu2UGNQEAmU3Q357kdUn+vrX2zJaZVfWeJF9NckomoX/xgnY3ttY2zKB/AGAJo7+6b61d0Vr74tYhP8y/L8n5w9tjx/YDAOy8WYzot+XJYfrUIsvWVdXvJ9kvyYNJrm+t3byL6wGAVWWXBX1VrUnyhuHtpYus8hvDa+s2VyU5o7V2966qCwBWk105ov9Akl9Ncklr7bKt5j+W5E8yORHvjmHei5NsSHJcksur6tDW2g+310FVbV5i0UHTFg0APamfPhl+BhuteluSjyS5NcnRrbWHdqDNmiTXJjkyydmttY/sQJttBf2eO14xACxLN7TWDh+zgZmP6KvqrExC/utJXrEjIZ8krbWnquqCTIL+mGEb22uz6D9++ABw2A4XDQCdmumd8arq7CQfz+Ra+OOGM+93xneH6V6zrAsAVquZBX1V/UGS85LcmEnI3z/FZl46TO/Y5loAwA6ZSdBX1fsyOflucyZf1z+wjXWPrKqfXWT+8UneMbz97CzqAoDVbvRv9FV1RpJzkzyd5Jokb6uqhavd1VrbNPz9p0kOHi6lu2eY9+Ikxw9/v6+1dt3YugCA2ZyM98Jh+uwkZy+xzpeTbBr+/kySk5P8epJXJfmZJP+S5G+SfLy1ds0MagIAsosur5s3Z90D0InRl9d5Hj0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHeg36/eddAADMwP5jN7BmBkUsR48M07uWWH7QML1115fSDftsOvbbdOy3nWefTWc577f985M8m1q11saXssJU1eYkaa0dPu9aVgr7bDr223Tst51nn01nNey3Xr+6BwAi6AGga4IeADom6AGgY4IeADq2Ks+6B4DVwogeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADq2qoK+qn65qj5dVfdW1RNVdVdVfbiqnjPv2parYR+1JV73zbu+eamqU6vqY1V1TVU9MuyPz26nzVFVdUlVPVRVj1XVzVV1dlU9e3fVPW87s9+qav9tHHutqi7c3fXPQ1XtV1Vvrqq/q6pvVdWPqurhqrq2qt5UVYv+d3y1H287u996Pt56fR79T6mqA5Ncl+QXknwhk2cPH5Hk7UlOqKqjW2sPzrHE5ezhJB9eZP6ju7uQZeS9SQ7JZB/ck58803pRVfWbSS5O8niSv07yUJLXJjkvydFJTtuVxS4jO7XfBjcl+fwi82+ZYV3L2WlJNib5TpIrk9yd5BeT/FaSC5K8qqpOa1vd/czxlmSK/Tbo73hrra2KV5LLkrQk/3XB/A8N88+fd43L8ZXkriR3zbuO5fZKclySX0lSSY4djqHPLrHu3knuT/JEkpdsNX+PTD58tiSnz/vftAz32/7D8k3zrnvO++z4TEL6WQvmPy+T8GpJTtlqvuNtuv3W7fG2Kr66r6oDkrwyk9D6xILF/z3JD5P8blXttZtLY4VqrV3ZWvtmG/4LsR2nJvn5JBe21v5pq208nskIN0nO3AVlLjs7ud9I0lq7orX2xdbaMwvm35fk/OHtsVstcrxlqv3WrdXy1f3xw/RLi/yP/oOq+sdMPgi8NMnlu7u4FWBtVf1Okudn8qHo5iRXt9aenm9ZK8aW4+/SRZZdneSxJEdV1drW2hO7r6wVY11V/X6S/ZI8mOT61trNc65puXhymD611TzH2/Yttt+26O54Wy1B/6JhevsSy7+ZSdCvj6BfzPOSfGbBvDur6vdaa1+eR0ErzJLHX2vtqaq6M8nBSQ5I8o3dWdgK8RvD619V1VVJzmit3T2XipaBqlqT5A3D261D3fG2DdvYb1t0d7ytiq/uk+wzTB9eYvmW+fvuhlpWmr9M8opMwn6vJL+W5M8y+T3rH6rqkPmVtmI4/qbzWJI/SXJ4kucMr5dncmLVsUkuX+U/t30gya8muaS1dtlW8x1v27bUfuv2eFstQb89NUz9brhAa+2c4beuf2mtPdZau6W19tZMTmL8d0k2zLfCLjj+FtFau7+19settRtaa98fXldn8u3b/03yH5K8eb5VzkdVvS3JuzK5euh3d7b5MF11x9u29lvPx9tqCfotn2D3WWL53gvWY/u2nMxyzFyrWBkcfzPUWnsqk8ujklV4/FXVWUk+kuTrSY5rrT20YBXH2yJ2YL8tqofjbbUE/W3DdP0Sy39lmC71Gz4/7f5huiK/ytrNljz+ht8LX5jJSUF37M6iVrjvDtNVdfxV1dlJPp7JNd3HDWeQL+R4W2AH99u2rOjjbbUE/ZXD9JWL3A3p5zK5gcSPknxldxe2gr1smK6a/1iMcMUwPWGRZcck2TPJdav4DOhpvHSYrprjr6r+IJMb3tyYSVjdv8Sqjret7MR+25YVfbytiqBvrf1zki9lcgLZWQsWn5PJp7S/aq39cDeXtqxV1cFV9dxF5r8gk0/HSbLN276SJLkoyQNJTq+ql2yZWVV7JHn/8HbjPApbzqrqyKr62UXmH5/kHcPbVXH8VdX7MjmJbHOSV7TWHtjG6o63wc7st56Pt1ot961Y5Ba430hyZCZ36ro9yVHNLXD/jarakOQPM/lG5M4kP0hyYJITM7nL1iVJTm6t/XheNc5LVZ2U5KTh7fOS/OdMPu1fM8x7oLX27gXrX5TJLUkvzOSWpK/L5FKoi5K8fjXcRGZn9ttwSdPBSa7K5Ha5SfLi/OQ68fe11rYEV7eq6owkm5I8neRjWfy39btaa5u2arPqj7ed3W9dH2/zvjXf7nwl+feZXC72nSQ/TvLtTE7OeO68a1uOr0wuLflfmZyh+v1MbjLx3ST/J5PrUGveNc5x32zI5KzlpV53LdLm6Ew+HH0vk5+KvpbJSOHZ8/73LMf9luRNSf53Jne0fDSTW7rencm92//TvP8ty2iftSRXOd7G7beej7dVM6IHgNVoVfxGDwCrlaAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDo2P8H3X3jPMTtNWMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11b12eba8>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 253
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(images[1].numpy().squeeze(), cmap='Greys_r');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building networks with PyTorch\n",
    "\n",
    "Here I'll use PyTorch to build a simple feedfoward network to classify the MNIST images. That is, the network will receive a digit image as input and predict the digit in the image.\n",
    "\n",
    "<img src=\"assets/mlp_mnist.png\" width=600px>\n",
    "\n",
    "To build a neural network with PyTorch, you use the `torch.nn` module. The network itself is a class inheriting from `torch.nn.Module`. You define each of the operations separately, like `nn.Linear(784, 128)` for a fully connected linear layer with 784 inputs and 128 units.\n",
    "\n",
    "The class needs to include a `forward` method that implements the forward pass through the network. In this method, you pass some input tensor `x` through each of the operations you defined earlier. The `torch.nn` module also has functional equivalents for things like ReLUs in `torch.nn.functional`. This module is usually imported as `F`. Then to use a ReLU activation on some layer (which is just a tensor), you'd do `F.relu(x)`. Below are a few different commonly used activation functions.\n",
    "\n",
    "<img src=\"assets/activation.png\" width=700px>\n",
    "\n",
    "So, for this network, I'll build it with three fully connected layers, then a softmax output for predicting classes. The softmax function is similar to the sigmoid in that it squashes inputs between 0 and 1, but it's also normalized so that all the values sum to one like a proper probability distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "class Network(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # Defining the layers, 128, 64, 10 units each\n",
    "        self.fc1 = nn.Linear(784, 128)\n",
    "        self.fc2 = nn.Linear(128, 64)\n",
    "        # Output layer, 10 units - one for each digit\n",
    "        self.fc3 = nn.Linear(64, 10)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        ''' Forward pass through the network, returns the output logits '''\n",
    "        \n",
    "        x = self.fc1(x)\n",
    "        x = F.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = F.relu(x)\n",
    "        x = self.fc3(x)\n",
    "        x = F.softmax(x, dim=1)\n",
    "        \n",
    "        return x\n",
    "\n",
    "model = Network()\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing weights and biases\n",
    "\n",
    "The weights and such are automatically initialized for you, but it's possible to customize how they are initialized. The weights and biases are tensors attached to the layer you defined, you can get them with `model.fc1.weight` for instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.fc1.weight)\n",
    "print(model.fc1.bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For custom initialization, we want to modify these tensors in place. These are actually autograd *Variables*, so we need to get back the actual tensors with `model.fc1.weight.data`. Once we have the tensors, we can fill them with zeros (for biases) or random normal values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set biases to all zeros\n",
    "model.fc1.bias.data.fill_(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sample from random normal with standard dev = 0.01\n",
    "model.fc1.weight.data.normal_(std=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forward pass\n",
    "\n",
    "Now that we have a network, let's see what happens when we pass in an image. This is called the forward pass. We're going to convert the image data into a tensor, then pass it through the operations defined by the network architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA04AAAHACAYAAACVhTgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xu8bWVdL/7PFxBFhI1giOJli0cFg0LI+wXRLI1U8lL9TPOSWmpZpueEl05a2sFf5b2Omhe8lRfKLDEzEy8Jpm20RBG8bQW8IKBcFBHYz/ljjCXL5Vp77LH2XGvOtef7/XrN19hrjPEdz3eMNffe8zufZzyjWmsBAABgZbtNOwEAAIBZp3ACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYoHACAHY5VdX61+Zp5zIvpnXNd6bdqjqpj33ujh63qh7Tr//g6jJmo1I4AQAzq6quX1VPqqp/qqqvVtX3quq7VfXlqjq5qh5ZVXtNO8/1UlVbF32gX3hdU1UXVdVHquppVXX9aec5r/qi6rlVdeS0c2Hy9ph2AgAAy6mqByZ5dZKDFq3+bpJtSTb3r4cmeWFVPaq19oH1znGKvpvk8v7PeybZP8k9+tfjq+rY1toF00puA/l6krOTXDgi5pI+5qvLbHtMkmOSbE3yqZ3MjRmjxwkAmDlV9Zgk/5CuaDo7yaOS3Ki1doPW2r5J9kvysCQfTHLTJPeaTqZT8+ettYP61/5JbpTkBUlaktunKzgZ0Fp7Zmvt0NbaK0bEvLOP+fW1zI3Zo3ACAGZKVf1Uklem+5zyniR3aK29ubV20cI+rbVLWmt/11o7NsmvJLlsOtnOhtbaRa215yR5fb/qwVV102nmBLsahRMAMGtekOS6Sc5P8ojW2hXb27m19vYkL9qRA1fV7lV1bFW9tKq2VNU3q+oHVfW1qnpnVd1nO7G79fewnNrfU3RVVX2rqj5TVa+rqvsvE3Orqvq/VXVOVV3R36P1lar6YFU9s6putCN5j/C3i/581KI8fjgJQlVdt6qeXVX/XVWX9ev3W5L3sVX191X1jf76fGPo+iyJP7yq3trHfb+qPldVf1hV111h/xtU1cOr6i1VdWZVfae/Xl+oqldX1W3WqN0VJ4fYThs/NjnEwrp0w/SS5PVL7kPb2u/3uv7nkwfaeF6/32k7mhdrzz1OAMDMqKqDkxzX//iy1tolOxLXWms72MRhSRbfC3Vlkh8kuUmS45McX1XPbq396TKxb0ryiEU/X5Jk33TD5G7fv967sLGqjko3lHCfftVV6e5NukX/OibJJxfHTMD5i/687zLbr5fkw0nu1OfzvaU7VNXzkzy7/7GlO88Dc+31ObG19szt5HC3dEMF905yaZJKcrskf5zkF6rqfq21y5fEPCbJyxf9fFm6L/hv3b8eUVXHt9beP+F2J+WKJN9Md6/Zdfr2Fxf83+qXr0ny2CQPrKoDFveiLqiqSvLo/sfXrVG+rIIeJwBgltw73QfeJPnHNTj+D5K8I8kD090/tVdr7QZJbpzkD5Nck+T5VXXnxUFVda90RdO2JE9Lsm9rbb90hchN033w//clbf15uqLpP5Ic1Vrbs7V2w3Qf7O+Y5CXpipJJusWiP39nme1PSXLbJL+a5Ab9OWxOV9Clqn411xZNr0hyYJ/zT+TawuaEqnrkdnL4qySfTfJTrbVN6a7BY9MVEnfJ8r2DF/XHv1uS/fr72K6XrtB9S7pr9jdVtfeE252I1trbWmsHJVnoIfrdRfegHdRau2O/32l9jnsm+bUVDnffJLdM9zt521rlzHgKJwBglhzWL69MNynERLXWzmmt/XJr7d2ttW8u9FS11i5orT0/yfPSFW6/tST0Lv3yfa21l7TWLuvjWmvt6621N7TWnrFCzO+21j65KIfvtdb+s7X2tNba6RM+xScsNJPkE8tsv0GSX+k/6P+gz+crrbWr+p6OP+n3e2tr7Xdaaxf2+1zUWntqrh0K+PyqWulz5JVJ7t9a+3Qf+4PW2klJntxv/42quuXigNba37bWntpaO32hl7G/tp9LNzHI+9MVbw/bzrmPbndKXtMvH7vC9sf1y5MX3mfMBoUTADBLDuiX3x4x/G6S/qlf3n3J+kv75YHbKRiWWoi5yU5ntR1VtWdV3b6qXpNuevakK3y+tczu/91ae98Khzoyyf/o//z8FfZ5Xr+8Zbrhfst5ZWvt4mXWvzHJeek+f/7SCrE/pn8fnNL/uPT3smbtrqE3puv5PLKq7rB4Q1VtyrU5GqY3YxROAMBcqaq9+gfFfrCqLugneWj9zf0LPUNLZ6R7f7oPu0cl+WB1D94dmrXuPf3yjVV1YlXdpaquM6HT+KNFOV+Z5DNJfqPf9rFc28uy1PZ6uBYmk/hWa+0zy+3QWjs7195HddRy+6S7r2u52G1JPrJSbFXdrKpe2E/a8Z3qHuy7cI4v7nfb3jVfVbvrrb+v6R/6H5f2Oj0i3RDFz7fWPryuiTFI4QQAzJKFm+Vv2A8dm6iqukm6B5O+KN3kDD+RrvD4Vrqb+xcehPoj99K01r6Q5Enp7pe5Z7qJIs6vqi/3s+b9SM9B73+mu+dlnyR/kK5oubSqPlBVT6qqvXbiVL7b5/vNJF9LclaSv083rO2erbXl7m9Krp2kYDk/0S/P384+Sdd7s3j/pbYXv7DtR2Kr6ph05/C/0hU3m9JNELFwjgu9d9u7x2l0u1O0MFzvEVW156L1C8P0Xh9mjsIJAJglZ/XL66abEW3SXpJucoQvpRvWtn//UN0D+5v777JSYGvtdUluleT3krwrXZG3Od39UFuq6llL9r8oyT2S3C/Jy9L1Zu2Z5Nh0ExmcWVU3W+V5LH4A7sGttdu31h7aP+/q6u3EXbMDx1526u4J+bFiuO+Fe3O6+6/en+5hxnu11vZbOMckv79S/GrbnbL3J/lyuqGpD0qSqvrJJD+T7nf0humlxkoUTgDALPlQuokNkv4D5aT03+w/uP/x11prf99a+/aS3W68vWP0E0q8tLV2fLreizsleWe6D+Z/Ut3Dexfv31pr72+t/W5r7ah0U5f/ZpKLkxySa4egzYKF3qhbbHevZKHYW6n3anvD6Rbu91oce9f+mBcneXBr7SOtte8vidvu72WV7U5Nf9/Wwj1MC8P1FoZa/ktr7WvrnxVDFE4AwMxorZ2Xa+8N+p2qWu5ZRD9mB4f13SjX9qZ8coV9fnZH2kt+WBR9IsnDc+3kA/cYiPl2a+3VSRZ6p47Z3v7r7Ix+uXdVLTvxQ1XdNsnBS/Zfatlz6n9H91wmdqEQO6e19mPPlertyO9lbLtrYdtCszuw7+vT9S79fD/b38IU7yaFmFEKJwBg1jwn3X1HN0v37J7rbW/nqvrlXDuUa3suzbW9WUcsc5ybJPmdFdrYc7n1SdJauybdw2STvjCrqt2qao/t5HLF4v1nxKeSfKH/87NW2Oe5/XJrko+vsM+Tqmq/ZdY/MsnN0xUXf79o/cKzrG6z3O+6qn4u3fDGIWPbXQsL92Itl8ePaK2dn+Sfk+ye7llVP5GuR2wtnl/GBCicAICZ0lr7VLoHtbYkxyX5ZD+L3f4L+1TVpqp6SFWdmu4hofvswHEvTzfjXJK8rqqO7I+1W1XdN90wwZV6Cv60qk6uquOX5HHjqnpZunufWpJ/7Tftm+QLVfXsqjqiqnZf0tYL+v3+ZfiKrI9++Nhz+h8fXFUvr6oDkqSqDujP8//rtz+nn61uOddL8t6qOryPvU5VPTrJK/vtr22tfXXR/h9N8r109/u8sS9gF2Y/fFySv8u1k4Zsz9h218LCbIQP6acWH7IwScTCNOtvbq1dtdLOTNf2vgkBAJiK1tprq+qiJK9Kcmi6WexSVZenK1AWF0pfSfKBHTz005Kcmq7H6ZNV9d10XyTvle4em8fl2qmiF9sj3WQSD+3zuDRdkbU4j+e01s5c9PMt0z0P6flJrqqqy9LNFrd7v/1L2bGesnXTWntbVR2R5NlJfjvJk6vqknR5L3zhfmJr7S3bOcyTk/x1kk/3sXulmxQj6QrXHznn1tp3quqZSV6abtjjw/u4vdNd90+lG772soH0R7W7Rt6U5BnphmxeWFUXpOuNPK+1ttwwzlOSfD3X3oNlmN4M0+MEAMyk1to/pJtA4Snp7ns6L90H6T3SDRU7Od1zb263o8+8aa39R7rJCP4hybeTXCfJBekKtCOT/NcKoS9O8tR0s+mdk65oum6Sc9P1eN2rtfani/a/NMkvppvF7+PphmDtk24a8U+kK0yO7O/pmimtteckuW+6c70w3Wx3F6UbQvazrbVnDhzitCR3TvL2dEMuW5Kzk/zvJPfue/6WtvmyJA/Jtb1PeyT5XJI/SnK3dFOTDxnd7qS11j6XbhbF96YbgnhQugJ62dkT+xkQFx66/IklhTczpqbzUG4AAKCqzklymyRPaq29cmh/pkfhBAAAU9Df7/b+dD2RN22tXToQwhQZqgcAAOusqm6U5M/6H1+naJp9epwAAGCdVNWfJ/nldPc/XSfdfWQ/2Vq7YKqJMUiPEwAArJ8bpXuu1BVJ3pfkPoqmjUGPEwAAwAA9TgAAAAMUTgAAAAP2mHYCa+V+uz3cGESAGfSv295R084BAMbS4wQAADBA4QQAADBglx2qBwDrqaq+nGTfJFunnAoA19qc5NLW2q129kAKJwCYjH332muv/Q877LD9p50IAJ2zzjorV1xxxUSOpXACgMnYethhh+2/ZcuWaecBQO/oo4/OGWecsXUSx3KPEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwIA9pp0AAOwqzjz/kmw+4ZRppzFo64nHTTsFgA1HjxMAAMAAhRMAAMAAhRMAAMAAhRMAAMAAhRMAAMAAhRMAAMAAhRMAc6E6j6uqj1XVZVX1var6ZFU9tap2n3Z+AMw2hRMA8+INSV6b5FZJ3pbkr5PsmeSlSd5WVTXF3ACYcR6AC8Aur6qOT/KoJF9OcqfW2oX9+uskeXuShyZ5dJKTppUjALNNjxMA8+Ah/fIvFoqmJGmtXZXkD/sff2fdswJgw1A4ATAPDuqXX1pm28K6o6pqv3XKB4ANxlA9AObBQi/TrZbZdsiiPx+a5GPbO1BVbVlh06GryAuADUKPEwDz4N398verav+FlVW1R5LnLdrvhuuaFQAbhh4nAObBW5M8MskDkny2qv4xyfeS/GySWyf5fJLbJLlm6ECttaOXW9/3RB01qYQBmC16nADY5bXWtiV5UJJnJPlGuhn2HpfkvCT3SHJRv+sFU0kQgJmnxwmAudBauzrJX/SvH6qqvZIcmeSKJJ+ZQmoAbAB6nACYd49Kcr0kb++nJweAH6NwAmAuVNW+y6y7Y5ITk1ye5I/XPSkANgxD9WCd7XHzm42Oedy/fXh0zM2vc9HwTks8+g2/OzrmkNd+ddT+V5973ug2YEL+taquSHJmksuS/GSSX0hyZZKHtNaWe8YTACRROAEwP05O8qvpZtfbK8nXkrwmyYmtta1TzAuADUDhBMBcaK39WZI/m3YeAGxM7nECAAAYoHACAAAYoHACAAAYoHACAAAYYHIIAJiQww/elC0nHjftNABYA3qcAAAABiicAAAABiicAAAABiicAAAABpgcAtbZuS/fZ3TMHa/3tdExN9l9r9Exn3rCS0fHPPzYB47a/+pjRjcBADB1CicAmJAzz78km084ZWrtbzWjH8CaMVQPAABggMIJAABggMIJAABggMIJAABggMIJAABggMIJAABggMIJAABggMIJgLlRVcdV1fuq6ryquqKqvlRV76iqu047NwBmm8IJgLlQVS9M8u4kRyV5b5KXJjkjyYOTfLSqHjnF9ACYcXtMOwEAWGtVdVCSZyT5ZpKfaq1dsGjbsUk+kOSPk7x5OhkCMOv0OAEwD26Z7v+8/1hcNCVJa+3UJJcl+YlpJAbAxqDHCXq777dpdMylbz1gdMwnj3jL6Jht2Wt0zG6p0TGr+S7lr2518qj9H3/HJ41uo33i06NjYInPJ/lBkjtV1Y1aaxcubKiqeyXZJ8k/7MiBqmrLCpsO3eksAZhZCicAdnmttYur6g+SvCjJZ6vqH5JclOTWSR6U5F+T/OYUUwRgximcAJgLrbWXVNXWJK9L8oRFm76Q5KSlQ/i2c5yjl1vf90QdtbN5AjCb3OMEwFyoqv+V5OQkJ6Xrado7ydFJvpTkLVX1/08vOwBmncIJgF1eVd07yQuT/GNr7fdba19qrX2vtXZGkl9Kcn6Sp1fVIdPME4DZpXACYB78Yr88demG1tr3knw83f+Jd1jPpADYOBROAMyD6/bLlaYcX1j/g3XIBYANSOEEwDz4SL98YlUdvHhDVT0gyd2TfD/JaeudGAAbg1n1AJgHJyd5f5KfTXJWVb0zyTeSHJZuGF8lOaG1dtH0UgRglimcANjltda2VdUvJHlKkl9NNyHE9ZNcnOQ9SV7WWnvfFFMEYMYpnACYC621q5K8pH8BwCjucQIAABigcAIAABhgqB70Ln3rAaNj/u2It42O2baK7yu2ZdvomNV8L7Kadv7qoruN2r994tOj2wAAmDY9TgAAAAP0OAHAhBx+8KZsOfG4aacBwBrQ4wQAADBA4QQAADBA4QQAADBA4QQAADBA4QQAADDArHoAMCFnnn9JNp9wylTa3mo2P4A1pccJAABggMIJAABggMIJAABggMIJAABggMkh2BB232/T6JhL33rAqP0/dMTJo9vYtorvHnZLjY5ZzXcc69XO+15x91H7H5DTR7cBADBtepwAAAAGKJwAmAtV9ZiqagOva6adJwCzyVA9AObFp5I8b4Vt90xynyT/vH7pALCRKJwAmAuttU+lK55+TFUt3Hz36vXLCICNxFA9AOZaVR2e5C5Jzk9yypTTAWBGKZwAmHe/2S9f21pzjxMAyzJUD4C5VVV7JXlkkm1JXrODMVtW2HTopPICYPbocQJgnv1ykv2S/HNr7dxpJwPA7NLjBMA8e2K/fNWOBrTWjl5ufd8TddQkkgJg9uhxAmAuVdXtk9wtyXlJ3jPldACYcQonAOaVSSEA2GEKJwDmTlVdL8mj0k0K8doppwPABuAeJzaEz/3J+MmqzjriFaP237aK7xG2ZdvomNV8X7Fe7fzld249OuaA154+vBPMnocnuWGSd5sUAoAdoccJgHm0MCnEq6eaBQAbhsIJgLlSVYcluUdMCgHACIbqATBXWmtnJalp5wHAxqLHCQAAYIDCCQAAYIDCCQAAYIDCCQAAYIDJIQBgQg4/eFO2nHjctNMAYA3ocQIAABigcAIAABigcAIAABigcAIAABhgcgjW3R43v9nomKff9z2jY3Yb+b3AbqnRbazmu4f1aueb11wxOuYtf/GA0TH75/TRMQAAG40eJwAAgAF6nABgQs48/5JsPuGUdW1zq+nPAdaFHicAAIABCicAAIABCicAAIABCicAAIABCicAAIABCicAAIABCicA5k5V3bOq/q6qvl5VV/bL91XVL0w7NwBmk+c4ATBXquo5Sf4kyYVJ3p3k60lulOQOSe6d5D1TSw6AmaVwAmBuVNXD0xVN70/ykNbaZUu2X2cqiQEw8wzVA2AuVNVuSV6Y5HtJHrG0aEqS1tpV654YABuCHifW3VcecYvRMU/c9K7RMduybWTE+O8Rxrexfu08YMsTR8fc9HWnj46BDeRuSW6V5OQk366q45IcnuT7ST7eWvMXAIAVKZwAmBd37JffTHJGkiMWb6yqDyd5WGvtW9s7SFVtWWHToTudIQAzy1A9AObFgf3yt5LsleRnk+yTrtfpX5LcK8k7ppMaALNOjxMA82L3flnpepb+q//5M1X1S0nOSXJMVd11e8P2WmtHL7e+74k6apIJAzA79DgBMC++3S+/tKhoSpK01q5I1+uUJHda16wA2BAUTgDMi7P75XdW2L5QWO21DrkAsMEonACYFx9OcnWS21TVnstsP7xfbl23jADYMBROAMyF1tqFSd6WZFOS/714W1XdL8nPJ7kkyXvXPzsAZp3JIQCYJ7+f5M5Jnl1V90ry8SS3TPJLSa5J8oTW2kpD+QCYYwonAOZGa+2CqrpzkuekK5bukuSyJKck+T+ttY9NMz8AZpfCCYC50lq7OF3P0+9POxcANg73OAEAAAzQ48RO2X2/TaNjfvPXTxkds1tqdMzY7wXWo43VtvPEc+8zOuamv/TZ0TEAACxPjxMAAMAAhRMAAMAAQ/UAYEIOP3hTtpx43LTTAGAN6HECAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYYFY9AJiQM8+/JJtPGP+Q70nYajY/gDWlxwkAAGCAwgkAAGCAwgkAAGCAe5zYKQ/46NbRMU/c7wujY7atosbflm0jI9ajjeTen/6V0TGbnrb76JjkslXEAACwHD1OAAAAAxROAAAAAxROAAAAAxROAMyNqtpaVW2F1zemnR8As8vkEADMm0uSvGSZ9ZevdyIAbBwKJwDmzXdaa8+ddhIAbCyG6gEAAAzQ4wTAvLluVT0yyS2SfDfJfyf5cGvtmummBcAsUzgBMG8OSvKmJeu+XFWPba19aCi4qrassOnQnc4MgJllqB4A8+T1Se6brnjaO8kRSV6VZHOSf66qn55eagDMMj1OAMyN1trzlqw6M8lvVdXlSZ6e5LlJfmngGEcvt77viTpqAmkCMIP0OAFA8sp+ea+pZgHAzNLjxA9d8eA7jY558n6vHN5piW2rqNd3S42OGfu9wGra2HLl+HO5wf2/NDrmmjsdMTrmi39z5OiYN9/ltaNj/vqCY0bt/6Ev/Y/Rbdzg9OuPjrnpG84cHXPNpZeOjmGXcUG/3HuqWQAws/Q4AUBy1345/psNAOaCwgmAuVBVP1lV+y+z/pZJXtH/+Ob1zQqAjcJQPQDmxcOTnFBVpyb5cpLLktw6yXFJrpfkPUn+fHrpATDLFE4AzItTk9wuyR3SDc3bO8l3kvx7uuc6vam11qaXHgCzTOEEwFzoH247+IBbAFiOe5wAAAAGKJwAAAAGKJwAAAAGKJwAAAAGmBwCACbk8IM3ZcuJx007DQDWgB4nAACAAQonAACAAYbq8UMXHzr+7bAt458VuS3bRsespsYf3874Nh55+uNHx9w6nxwdc/Sr/mt0zLsPHN/Oan43r7z5uMfibLv5qaPb2O2Y8b+bv/ytW4+O+efH3nN0TD7+6fExAMCGo8cJAABggMIJAABggKF6ADAhZ55/STafcMq6trnVLH4A60KPEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwBzq6oeVVWtfz1+2vkAMLsUTgDMpaq6eZKXJ7l82rkAMPsUTgDMnaqqJK9PclGSV045HQA2gD2mnQAb226pVUXNYjura2O8q+9z9OiY5x346tExs/u7WZ/f/1P2++LomLu97fOjY575G781OmaPD2wZHcPEPTXJfZLcu18CwHbpcQJgrlTVYUlOTPLS1tqHp50PABuDHicA5kZV7ZHkTUm+muRZqzzGSl2Gh642LwBmn8IJgHnyv5PcIck9WmtXTDsZADYOhRMAc6Gq7pSul+kvWmunr/Y4rbVlb1Tse6KOWu1xAZht7nECYJe3aIjeOUn+cMrpALABKZwAmAc3SHLbJIcl+f6ih962JH/U7/PX/bqXTC1LAGaWoXoAzIMrk7x2hW1Hpbvv6d+TnJ1k1cP4ANh1KZwA2OX1E0E8frltVfXcdIXTG1prr1nPvADYOAzVAwAAGKBwAgAAGKBwAmCutdae21orw/QA2B6FEwAAwACTQ/BDt3zLV0bH/NWv32p0zBP3+8LomNXU+Nuybc3beMNdV5qka2V/cMDDRseMP5dkfa7Z+HZm+VzusOf4dm7xp+eMjvnaXUaHAABTpscJAABggMIJAABggKF6ADAhhx+8KVtOPG7aaQCwBvQ4AQAADFA4AQAADFA4AQAADFA4AQAADFA4AQAADFA4AQAADDAdOQBMyJnnX5LNJ5yypm1sNd05wFTocQIAABigcAIAABhgqB4/dPV554+Oeel/HTs65reO+dLomN1So2PGfi+wmjbudN02OuZDR5w8OmbbKr7jWI9rtrp2dqVzSV598w+Ojjn+5g8eHXP1ueeNjgEAJkePEwAAwACFEwAAwACFEwAAwACFEwBzo6peWFX/VlXnVtUVVXVxVX2yqv6oqg6Ydn4AzC6FEwDz5GlJ9k7yr0lemuQtSa5O8twk/11VN59eagDMMrPqATBP9m2tfX/pyqp6QZJnJXlmkieve1YAzDw9TgDMjeWKpt7b++Vt1isXADYWhRMAJA/sl/891SwAmFmG6gEwd6rqGUlukGRTkp9Jco90RdOJOxC7ZYVNh04sQQBmjsIJgHn0jCQ3XvTze5M8prX2rSnlA8CMUzgBMHdaawclSVXdOMnd0vU0fbKqfrG1dsZA7NHLre97oo6adK4AzAaFEzvlmEO+MDpmW7atoqXxt+ONb2c92tDOrnQuq23n8iNvOjrmeueeNzqGYa21byZ5Z1WdkeScJG9Mcvh0swJgFpkcAoC511r7SpLPJvnJqrrRtPMBYPYonACgs9AVeM1UswBgJimcAJgLVXVoVR20zPrd+gfgHpjktNbat9c/OwBmnXucAJgX90/yZ1X14SRfTHJRupn1jklySJJvJHnC9NIDYJYpnACYF+9P8uokd0/y00n2S/LddJNCvCnJy1prF08vPQBmmcIJgLnQWjszyVOmnQcAG5N7nAAAAAYonAAAAAYonAAAAAYonAAAAAaYHAIAJuTwgzdly4nHTTsNANaAHicAAIABepzYKec99VajY7558ntHxxy8+/VHx4z9XmC31Jq3oZ31amN17Vyndh8dc1UbHZLvHji+neuNbwYAmCA9TgAAAAMUTgAAAAMUTgAAAAPc4wQAE3Lm+Zdk8wmnTDuNH7PVTH8AO02PEwDQGeeCAAAP6UlEQVQAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwAAwACFEwBzoaoOqKrHV9U7q+oLVXVFVV1SVf9eVb9RVf5PBGBFHoDLzvn4p0eH3Pdv/+fomM8+8hWjY7Zl28iI8Z+ZxrehnVk+l6va6JDc679/eXTMAa89fXxDTMLDk/zfJF9PcmqSrya5cZKHJHlNkgdU1cNba6t4JwCwq1M4ATAvzknyoCSntNZ+WFlX1bOSfDzJQ9MVUX83nfQAmGWGJQAwF1prH2it/dPioqlf/40kr+x/vPe6JwbAhqBwAoDkqn559VSzAGBmGaoHwFyrqj2S/Hr/43t3YP8tK2w6dGJJATBz9DgBMO9OTHJ4kve01v5l2skAMJv0OAEwt6rqqUmenuRzSR61IzGttaNXONaWJEdNLjsAZokeJwDmUlU9JclLk3w2ybGttYunnBIAM0zhBMDcqarfS/KKJGemK5q+MeWUAJhxCicA5kpV/UGSFyf5VLqi6YIppwTABqBwAmBuVNUfppsMYkuS+7bWLpxySgBsECaHAGAuVNWjk/xxkmuSfCTJU6tq6W5bW2snrXNqAGwACicA5sWt+uXuSX5vhX0+lOSkdckGgA1F4cS6O+QPTh8dc9/TnjQ65twHbhu1/zPuNv7xLU/ctHV0zG75sW+4dyhq12lnds/llO9tGh1z9TsOHB2TfHEVMeys1tpzkzx3ymkAsEG5xwkAAGCAwgkAAGCAwgkAAGCAwgkAAGCAySEAYEIOP3hTtpx43LTTAGAN6HECAAAYoHACAAAYoHACAAAYoHACAAAYoHACAAAYYFY9AJiQM8+/JJtPOGVq7W81ox/AmtHjBAAAMECPExvCXu/6+OiY275r3P6n7Hur0W288+j7jY753o33HB1zwQOvHB3zu0d+YHTMNanRMbunjdr/ift9YXQbTzz3PqNjzn7xT46OueFHzx0ds/95p4+OAQA2Hj1OAAAAAxROAAAAAxROAAAAAxROAAAAAxROAMyFqnpYVb28qj5SVZdWVauqN087LwA2BrPqATAvnpPkp5NcnuS8JIdONx0ANhI9TgDMi6cluW2SfZM8acq5ALDB6HECYC601k5d+HPV+GeWATDf9DgBAAAM0OMEACNU1ZYVNrlnCmAXpscJAABggB4nABihtXb0cuv7nqij1jkdANaJwgl611x66eiY3U89Y3TMPqMjkn3eOj7m3bnhKlpae+/OHVcRddnoiH3ysdExV4+OAADmhaF6AAAAAxROAAAAAxROAAAAA9zjBMBcqKrjkxzf/3hQv7xrVZ3U//nC1toz1j0xADYEhRMA8+LIJI9esu6Q/pUkX0micAJgWYbqATAXWmvPba3Vdl6bp50jALNL4QQAADBA4QQAADBA4QQAADBA4QQAADDArHoAMCGHH7wpW048btppALAG9DgBAAAMUDgBAAAMUDgBAAAMUDgBAAAMUDgBAAAMUDgBAAAMMB05AEzImedfks0nnDLtNJIkW02LDjBRepwAAAAGKJwAAAAGKJwAAAAGKJwAAAAGKJwAAAAGKJwAAAAGKJwAmBtVdbOqel1Vfa2qrqyqrVX1kqq64bRzA2C2eY4TAHOhqm6d5LQkByZ5V5LPJblTkt9Ncv+quntr7aIppgjADNPjBMC8+Kt0RdNTW2vHt9ZOaK3dJ8mLk9wuyQummh0AM03hBMAur6oOSfJzSbYm+cslm/8oyXeTPKqq9l7n1ADYIBROAMyD+/TL97XWti3e0Fq7LMlHk1w/yV3WOzEANgb3OAEwD27XL89ZYfvn0/VI3TbJv23vQFW1ZYVNh64uNQA2Aj1OAMyDTf3ykhW2L6zfbx1yAWAD0uMEAEn1yza0Y2vt6GUP0PVEHTXJpACYHXqcAJgHCz1Km1bYvu+S/QDgRyicAJgHZ/fL266w/Tb9cqV7oACYcwonAObBqf3y56rqR/7vq6p9ktw9yRVJPrbeiQGwMSicANjltda+mOR9STYnecqSzc9LsneSN7bWvrvOqQGwQZgcAoB58eQkpyV5WVXdN8lZSe6c5Nh0Q/SePcXcAJhxepwAmAt9r9PPJDkpXcH09CS3TvKyJHdtrV00vewAmHV6nACYG621c5M8dtp5ALDx6HECAAAYoHACAAAYoHACAAAYoHACAAAYYHIIAJiQww/elC0nHjftNABYA3qcAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABiicAAAABuwx7QQAYBex+ayzzsrRRx897TwA6J111llJsnkSx1I4AcBk3OCKK6645owzzvivaSeywR3aLz831Sw2NtdwMlzHyZj2ddyc5NJJHEjhBACTcWaStNZ0Oe2EqtqSuI47wzWcDNdxMnal6+geJwAAgAEKJwAAgAG77FC9f932jpp2DgAAwK5BjxMAAMAAhRMAAMCAaq1NOwcAAICZpscJAABggMIJAABggMIJAABggMIJAABggMIJAABggMIJAABggMIJAABggMIJAABggMIJgLlWVTerqtdV1deq6sqq2lpVL6mqG448zv593Nb+OF/rj3uztW57FuzsuVTV3lX1a1X1N1X1uar6blVdVlX/WVVPr6o9V4hr23l9bLJnubYm8X6oqg8OXJPrrRB3+6p6e1VdUFXfr6qzq+p5VbXX5M5wfUzgvXjvgWu48Lr5krhd4r1YVQ+rqpdX1Ueq6tI+/zev8lijfxez/F6s1tq0cwCAqaiqWyc5LcmBSd6V5HNJ7pTk2CRnJ7l7a+2iHTjOAf1xbpvkA0k+keTQJA9OckGSu7bWvrQWbc+CSZxLVd0/yT8nuTjJqUm+kGT/JA9MclB//Pu21r6/JK4l+UqSk5Y57Hmttdes+sTW0QTfix9MckyS562wy/Nba1cviblzuvftdZKcnOTcJPdJ8jNJPpruul85/qzW34Tei5uTPGaFzUckeUiSz7TWDl8St6u8Fz+V5KeTXJ7kvHT/lr2ltfbIkccZ/buY+fdia83Ly8vLy2suX0n+JUlL8jtL1r+oX//KHTzOq/r9X7Rk/VP79e9dq7Zn4TWJc0lyZJJfS7LnkvX7JNnSH+fpy8S1JB+c9jWYhWvY7//B7uPdDre7e5LP9m08aNH63dJ9cG1JTpj29Vnv67id4/9tf5ynLrNtV3kvHpvkNkkqyb3783rzWv8uNsJ7UY8TAHOpqg5J8sUkW5PcurW2bdG2fZJ8Pd0HhwNba9/dznH2TvKtJNuS3KS1dtmibbv1bWzu2/jSJNueBetxLlX1iCRvSfLu1toDl2xrST7UWrv3qk5gBkzyGi70OLXWagfbvk+Sf0vy4dbaMSvk9ZUkt2oz/qFxrd+Lfc/y+en+rh/cWvv2ku0b/r24VFXdO10P8Kgep9X8LjbCe9E9TgDMq/v0y/ct/k89Sfri56NJrp/kLgPHuWuSvZJ8dHHR1B9nW5L39T8euwZtz4L1OJer+uXVK2zfr6oeV1XPqqqnVNVGuG6LTfwaVtWvVNUJVfX7VfWAqrruQNvvXbqhL/TPSXLLJIfsaNtTtNbvxcckuW6SdywtmhbZ6O/FSVnN72Lm34sKJwDm1e365TkrbP98v7ztGhxnUm3PgvU4l8f1yx/7QNX76SSvTfKCJK9IcnpVfaqqjtiJNtfTWlzDtyb5P0n+Isl7kny1qh62Tm1Py1qfy+P75au2s89Gfy9Oyi7576LCCYB5talfXrLC9oX1+63BcSbV9ixY03Opqt9Ocv8kn0ryumV2eVGSuyf5iXT3Q90x3f0QP53kA1V18GraXWeTvIbvSjehxs3S9YQemq6A2i/J26rqAWvY9rSt2blU1THpruVnWmunrbDbrvBenJRd8t9FhRMALG/hHpGdHUu/muNMqu1ZsOpzqaqHJHlJkm8keWhr7aql+7TWnt5aO621dmFr7fLW2n+21h6e5O+S3CjJM3Yi91mxw9ewtfbi1tq7W2vnt9a+31o7u7X2rCRPT/e570/Xqu0NYGfO5Yn9csXepjl5L07Khvx3UeEEwLxa+PZy0wrb912y3ySPM6m2Z8GanEtVHZ9uuNkFSe7dlkznvgNe2S/vNTJuGtbj/fCadPeIHdnfnL+eba+XtXov7p/koUmuSPKmVeS1kd6Lk7JL/ruocAJgXp3dL1caL3+bfrnSePudOc6k2p4FEz+Xqnp4knck+Wa6GeLOHghZzrf65d6riF1va/5+aN3zrxYmL1l8TbwXhz063aQQb2+tfWcVeW2k9+Kk7JL/LiqcAJhXp/bLn+unDf+h/hv5u6f7hvljA8f5WL/f3Zd8k78wHfnPLWlvkm3PgomeSz/1+N8m+Vq6ounzAyErWZita2xP1TSs+fuhqm6X5IbpiqcLF236QL+8/zIxh6T7EPuVzPd1fEK/fPUq89pI78VJWc3vYubfiwonAOZSa+2L6aYK35zkKUs2Py/dt8NvXPy8l6o6tKoOXXKcy9MN39k7yXOXHOe3++P/y+KhZqtpe1ZN6jr26x+d7lp+Ncm9hobnVdVR/XO0lq7/qXSzmiXJm3f8bKZjUtewqg5ZbgKCqrpRktf3P761tbZ4WvcPJTkryb2q6kGLYnZL8sL+x1fO+jOcksm+Fxdtv2eSw5KcuZ1JIXaZ9+JYVXWd/hreevH6Vf4bN/PvRQ/ABWBu9f/Zn5bkwHSzkZ2V5M7pnrl0TpK7tdYuWrR/S5KlDxftH4x5WrpvRD+Q5OPpPmw9ON09OnfrP0isuu1ZNonrWFXHJnl/ui91X5fk3GWa+k5r7SWLYk5K8pB01/zcJFemm/ns/kl2T/LXSX5zI3zon9A1fEy6e5k+lO5hoRcnuUWSX0h338h/Jrnf0uFmVXXndNfwOulmgftqkvsm+Zl0z9u5b2vtykmf81qY1N/pRdvflOSRSZ7aWnv5dto9KbvOe/H4JMf3Px6U5OfT9fJ8pF93YWvtGf2+m5N8OclXWmublxxn9L9xM/9ebK15eXl5eXnN7SvJzdN9G//1JD9INxTkpUn2X2bf1v3Xuexx9u/jvtIf5+vpCoCbTaLtWX/t7HVM93DRNvDauiTm+CR/n+QLSS5ddN3/KcmDpn1NpnANj0hyUpJPJ7ko3YODL073gfd3kuy5nbZvn+6+sgvTfeg/J13PwF7Tvi7rfR0XbbthuuFk30uy30Cbu8x7MV3P+Q79PUzXo/RjfzdX87vYCO9FPU4AAAAD3OMEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAwQOEEAAAw4P8BikKdxDz1KasAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1193a0fd0>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 224,
       "width": 423
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Grab some data \n",
    "dataiter = iter(trainloader)\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "# Resize images into a 1D vector, new shape is (batch size, color channels, image pixels) \n",
    "images.resize_(64, 1, 784)\n",
    "# or images.resize_(images.shape[0], 1, 784) to not automatically get batch size\n",
    "\n",
    "# Forward pass through the network\n",
    "img_idx = 0\n",
    "ps = model.forward(images[img_idx,:])\n",
    "\n",
    "img = images[img_idx]\n",
    "helper.view_classify(img.view(1, 28, 28), ps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, our network has basically no idea what this digit is. It's because we haven't trained it yet, all the weights are random!\n",
    "\n",
    "PyTorch provides a convenient way to build networks like this where a tensor is passed sequentially through operations, `nn.Sequential` ([documentation](https://pytorch.org/docs/master/nn.html#torch.nn.Sequential)). Using this to build the equivalent network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Linear(in_features=784, out_features=128, bias=True)\n",
      "  (1): ReLU()\n",
      "  (2): Linear(in_features=128, out_features=64, bias=True)\n",
      "  (3): ReLU()\n",
      "  (4): Linear(in_features=64, out_features=10, bias=True)\n",
      "  (5): Softmax()\n",
      ")\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA04AAAHACAYAAACVhTgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XmcJWV9L/7Pl1VEARGRiIYRfyoYjAhxVxaNhkhURMli9LrE3ESNJi6/G1ySgNFcvEkUNDdXDSJxuXFLNIkYRCIuEY1mkBgUwW0QEEUW2QRE5rl/VLU0bffUnOZ0n9Nz3u/X67xqTlU9Vd9TXTNzPv1UPVWttQAAALC0rSZdAAAAwLQTnAAAAAYITgAAAAMEJwAAgAGCEwAAwADBCQAAYIDgBAAAMEBwAgAAGCA4AQAADBCcAAAABghOAAAAAwQnAACAAYITAADAAMEJANjiVFXrX+smXcusmNQxvy37raqT+7bHbO52q+pZ/fxPLK9i1irBCQCYWlV1+6p6XlX9c1V9u6p+WFXXVdW3quoDVfX0qtph0nWulqraMO8L/dzr5qq6vKo+XVUvrqrbT7rOWdWHqmOqav9J18L4bTPpAgAAFlNVT0jy1iR7zJt9XZKNSdb1r6ckeV1VPaO19vHVrnGCrktybf/n7ZLsmuSR/eu5VXVoa+3SSRW3hlyS5Lwkl43Q5qq+zbcXWfasJAcn2ZDk7NtYG1NGjxMAMHWq6llJPpQuNJ2X5BlJdmut3aG1tlOSXZI8NcknktwtyUGTqXRi/qK1tkf/2jXJbklem6QluV+6wMmA1trLW2v7tNb+aoQ2H+zb/LeVrI3pIzgBAFOlqn4+yZvTfU/5SJIHttbe1Vq7fG6d1tpVrbW/b60dmuTXklwzmWqnQ2vt8tbaq5K8vZ/1pKq62yRrgi2N4AQATJvXJtk+ycVJntZau35TK7fW3pfk9Zuz4arauqoOraoTqmp9VX2vqn5UVd+pqg9W1aM30Xar/h6WM/p7im6qqu9X1Zer6qSqOmyRNvesqv9TVedX1fX9PVoXVNUnqurlVbXb5tQ9gr+b9+cD5tXxk0EQqmr7qnplVX2pqq7p5++yoO5Dq+ofquq7/fH57tDxWdB+v6p6T9/uhqr6alX9UVVtv8T6d6iqo6rq3VV1TlX9oD9eX6+qt1bVvVdov0sODrGJffzU4BBz89Jdppckb19wH9qGfr2T+vcfGNjHsf16Z25uXaw89zgBAFOjqvZMcnj/9o2ttas2p11rrW3mLvZNMv9eqBuT/CjJzyQ5IskRVfXK1tqfLdL2nUmeNu/9VUl2SneZ3P3616lzC6vqgHSXEt6xn3VTunuTfrZ/HZzki/PbjMHF8/680yLLb5fkU0ke3Nfzw4UrVNVrkryyf9vSfc7dc8vxOa619vJN1PDwdJcK7pjk6iSV5L5JXp3k8VX12NbatQvaPCvJm+a9vybdL/jv1b+eVlVHtNZOH/N+x+X6JN9Ld6/Ztv3+5wf+7/fTE5M8O8kTqurO83tR51RVJXlm//akFaqXZdDjBABMk0PSfeFNkn9age3/KMn7kzwh3f1TO7TW7pDkrkn+KMnNSV5TVQ+Z36iqDkoXmjYmeXGSnVpru6QLIndL98X/3xbs6y/ShaZ/T3JAa2271tqd0n2xf1CS49OFknH62Xl//sEiy1+Q5D5Jfj3JHfrPsC5doEtV/XpuCU1/lWT3vua75JZgc3RVPX0TNfx1kq8k+fnW2s7pjsGz0wWJh2bx3sHL++0/PMku/X1st0sXdN+d7pj936raccz7HYvW2ntba3skmesh+v1596Dt0Vp7UL/emX2N2yX5zSU295gke6X7mbx3pWpmdIITADBN9u2nN6YbFGKsWmvnt9Z+tbX24dba9+Z6qlprl7bWXpPk2HTB7XcXNH1oPz2ttXZ8a+2avl1rrV3SWvvb1trLlmjz+621L86r4Yettf9orb24tfbZMX/E357bTZIvLLL8Dkl+rf+i/6O+ngtaazf1PR1/2q/3ntbaC1trl/XrXN5ae1FuuRTwNVW11PfIG5Mc1lr7r77tj1prJyd5fr/8t6pqr/kNWmt/11p7UWvts3O9jP2x/Wq6gUFOTxfenrqJzz7yfifkxH767CWWP6effmDuPGM6CE4AwDS5cz+9coTL78bpn/vpIxbMv7qf7r6JwLDQXJufuc1VbUJVbVdV96uqE9MNz550wef7i6z+pdbaaUtsav8k/1//59cssc6x/XSvdJf7LebNrbUrFpn/jiQXpfv++eQl2v6U/jw4pX+78OeyYvtdQe9I1/O5f1U9cP6Cqto5t9ToMr0pIzgBADOlqnboHxT7iaq6tB/kofU398/1DC0cke70dF92D0jyieoevDs0at1H+uk7quq4qnpoVW07po/xJ/NqvjHJl5P8Vr/sc7mll2WhTfVwzQ0m8f3W2pcXW6G1dl5uuY/qgMXWSXdf12JtNyb59FJtq+ruVfW6ftCOH1T3YN+5z/iGfrVNHfNl7Xe19fc1fah/u7DX6WnpLlH8WmvtU6taGIMEJwBgmszdLH+n/tKxsaqqn0n3YNLXpxuc4S7pgsf3093cP/cg1FvdS9Na+3qS56W7X+ZR6QaKuLiqvtWPmnernoPe/5/unpc7JvnDdKHl6qr6eFU9r6p2uA0f5bq+3u8l+U6Sc5P8Q7rL2h7VWlvs/qbklkEKFnOXfnrxJtZJut6b+esvtKn2c8tu1baqDk73Gf5HunCzc7oBIuY+41zv3abucRp5vxM0d7ne06pqu3nz5y7Te3uYOoITADBNzu2n26cbEW3cjk83OMI3013Wtmv/UN3d+5v7H7pUw9baSUnumeQPkvxjupC3Lt39UOur6hUL1r88ySOTPDbJG9P1Zm2X5NB0AxmcU1V3X+bnmP8A3D1ba/drrT2lf97VjzfR7ubN2PaiQ3ePyU+F4b4X7l3p7r86Pd3DjHdore0y9xmTvGSp9svd74SdnuRb6S5NfWKSVNXPJfmFdD+jv51caSxFcAIApskn0w1skPRfKMel/83+k/q3v9la+4fW2pULVrvrprbRDyhxQmvtiHS9Fw9O8sF0X8z/tLqH985fv7XWTm+t/X5r7YB0Q5f/TpIrkuydWy5BmwZzvVE/u8m1krmwt1Tv1aYup5u732t+24f127wiyZNaa59urd2woN0mfy7L3O/E9Pdtzd3DNHe53tyllh9trX1n9atiiOAEAEyN1tpFueXeoBdW1WLPIvopm3lZ3265pTfli0us84ubs7/kJ6HoC0mOyi2DDzxyoM2VrbW3JpnrnTp4U+uvsrP66Y5VtejAD1V1nyR7Llh/oUU/U/8zetQibeeC2PmttZ96rlRvc34uo+53JWyc2+1mrPv2dL1Lv9SP9jc3xLtBIaaU4AQATJtXpbvv6O7pnt1zu02tXFW/mlsu5dqUq3NLb9b9F9nOzyR54RL72G6x+UnSWrs53cNkkz6YVdVWVbXNJmq5fv76U+LsJF/v//yKJdY5pp9uSPL5JdZ5XlXtssj8pye5R7pw8Q/z5s89y+rei/2sq+px6S5vHDLqflfC3L1Yi9VxK621i5P8S5Kt0z2r6i7pesRW4vlljIHgBABMldba2eke1NqSHJ7ki/0odrvOrVNVO1fVkVV1RrqHhN5xM7Z7bboR55LkpKrav9/WVlX1mHSXCS7VU/BnVfWBqjpiQR13rao3prv3qSX5WL9opyRfr6pXVtX9q2rrBft6bb/eR4ePyOroLx97Vf/2SVX1pqq6c5JU1Z37z/kb/fJX9aPVLeZ2SU6tqv36tttW1TOTvLlf/rbW2rfnrf+ZJD9Md7/PO/oAOzf64XOS/H1uGTRkU0bd70qYG43wyH5o8SFzg0TMDbP+rtbaTUutzGRt6jchAAAT0Vp7W1VdnuQtSfZJN4pdquradAFlflC6IMnHN3PTL05yRroepy9W1XXpfpG8Q7p7bJ6TW4aKnm+bdINJPKWv4+p0IWt+Ha9qrZ0z7/1e6Z6H9JokN1XVNelGi9u6X/7NbF5P2apprb23qu6f5JVJfi/J86vqqnR1z/3C/bjW2rs3sZnnJ/mbJP/Vt90h3aAYSRdcb/WZW2s/qKqXJzkh3WWPR/Xtdkx33M9Od/naGwfKH2m/K+SdSV6W7pLNy6rq0nS9kRe11ha7jPOUJJfklnuwXKY3xfQ4AQBTqbX2oXQDKLwg3X1PF6X7Ir1NukvFPpDuuTf33dxn3rTW/j3dYAQfSnJlkm2TXJouoO2f5D+XaPqGJC9KN5re+elC0/ZJLkzX43VQa+3P5q1/dZJfSTeK3+fTXYJ1x3TDiH8hXTDZv7+na6q01l6V5DHpPutl6Ua7uzzdJWS/2Fp7+cAmzkzykCTvS3fJZUtyXpI/TnJI3/O3cJ9vTHJkbul92ibJV5P8SZKHpxuafMjI+x231tpX042ieGq6SxD3SBegFx09sR8Bce6hy19YELyZMjWZh3IDAABVdX6Seyd5XmvtzUPrMzmCEwAATEB/v9vp6Xoi79Zau3qgCRPkUj0AAFhlVbVbkj/v354kNE0/PU4AALBKquovkvxquvuftk13H9nPtdYunWhhDNLjBAAAq2e3dM+Vuj7JaUkeLTStDXqcAAAABuhxAgAAGCA4AQAADNhm0gWslMdudZRrEAGm0Mc2vr8mXQMAjEqPEwAAwADBCQAAYMAWe6keAKymqvpWkp2SbJhwKQDcYl2Sq1tr97ytGxKcAGA8dtphhx123XfffXeddCEAdM4999xcf/31Y9mW4AQA47Fh33333XX9+vWTrgOA3oEHHpizzjprwzi25R4nAACAAYITAADAAMEJAABggOAEAAAwQHACAAAYIDgBAAAMEJwAAAAGCE4AAAADBCcAAIABghMAAMAAwQkAAGCA4AQAADBAcAIAABggOAEAAAzYZtIFAMCW4pyLr8q6o0+ZdBk/seG4wyddAsAWQ48TAADAAMEJAABggOAEAAAwQHACAAAYIDgBAAAMEJwAAAAGCE4AzITqPKeqPldV11TVD6vqi1X1oqraetL1ATDdBCcAZsXfJnlbknsmeW+Sv0myXZITkry3qmqCtQEw5TwAF4AtXlUdkeQZSb6V5MGttcv6+dsmeV+SpyR5ZpKTJ1UjANNNjxMAs+DIfvqXc6EpSVprNyX5o/7tC1e9KgDWDMEJgFmwRz/95iLL5uYdUFW7rFI9AKwxLtUDYBbM9TLdc5Fle8/78z5JPrepDVXV+iUW7bOMugBYI/Q4ATALPtxPX1JVu87NrKptkhw7b707rWpVAKwZepwAmAXvSfL0JL+c5CtV9U9JfpjkF5PcK8nXktw7yc1DG2qtHbjY/L4n6oBxFQzAdNHjBMAWr7W2MckTk7wsyXfTjbD3nCQXJXlkksv7VS+dSIEATD09TgDMhNbaj5P8Zf/6iaraIcn+Sa5P8uUJlAbAGqDHCYBZ94wkt0vyvn54cgD4KYITADOhqnZaZN6DkhyX5Nokr171ogBYM1yqB8Cs+FhVXZ/knCTXJPm5JI9PcmOSI1triz3jCQCSCE4AzI4PJPn1dKPr7ZDkO0lOTHJca23DBOsCYA0QnACYCa21P0/y55OuA4C1yT1OAAAAAwQnAACAAYITAADAAMEJAABggMEhAGBM9ttz56w/7vBJlwHACtDjBAAAMEBwAgAAGCA4AQAADBCcAAAABghOAAAAA4yqBwBjcs7FV2Xd0aes+n43GMkPYMXpcQIAABggOAEAAAwQnAAAAAYITgAAAAMEJwAAgAGCEwAAwADBCQAAYIDgBMDMqKrDq+q0qrqoqq6vqm9W1fur6mGTrg2A6SY4ATATqup1ST6c5IAkpyY5IclZSZ6U5DNV9fQJlgfAlNtm0gUAwEqrqj2SvCzJ95L8fGvt0nnLDk3y8SSvTvKuyVQIwLTT4wTALNgr3f95/z4/NCVJa+2MJNckucskCgNgbdDjBMAs+FqSHyV5cFXt1lq7bG5BVR2U5I5JPrQ5G6qq9Uss2uc2VwnA1BKcANjitdauqKo/TPL6JF+pqg8luTzJvZI8McnHkvzOBEsEYMoJTgDMhNba8VW1IclJSX573qKvJzl54SV8m9jOgYvN73uiDritdQIwndzjBMBMqKr/keQDSU5O19O0Y5IDk3wzybur6n9NrjoApp3gBMAWr6oOSfK6JP/UWntJa+2brbUfttbOSvLkJBcneWlV7T3JOgGYXoITALPgV/rpGQsXtNZ+mOTz6f5PfOBqFgXA2iE4ATALtu+nSw05Pjf/R6tQCwBrkOAEwCz4dD/971W15/wFVfXLSR6R5IYkZ652YQCsDUbVA2AWfCDJ6Ul+Mcm5VfXBJN9Nsm+6y/gqydGttcsnVyIA00xwAmCL11rbWFWPT/KCJL+ebkCI2ye5IslHkryxtXbaBEsEYMoJTgDMhNbaTUmO718AMBL3OAEAAAwQnAAAAAYITgAAAAMEJwAAgAEGhwCAMdlvz52z/rjDJ10GACtAjxMAAMAAwQkAAGCA4AQAADBAcAIAABggOAEAAAwwqh4AjMk5F1+VdUefMpF9bzCaH8CK0uMEAAAwQHACAAAYIDgBAAAMEJwAAAAGGByCVbfVjjuO3GbDyXuP3OZLDz955DbTatvaeuQ2h375SSO32fDtu4zcZtfPbzvS+rt/9sqR95FvXDhyk43XXTf6fgAAlqDHCQAAYIDgBMBMqKpnVVUbeN086ToBmE4u1QNgVpyd5Ngllj0qyaOT/MvqlQPAWiI4ATATWmtnpwtPP6WqPtv/8a2rVxEAa4lL9QCYaVW1X5KHJrk4ySkTLgeAKSU4ATDrfqefvq215h4nABblUj0AZlZV7ZDk6Uk2JjlxM9usX2LRPuOqC4Dpo8cJgFn2q0l2SfIvrbXRHxgGwMzQ4wTALPvv/fQtm9ugtXbgYvP7nqgDxlEUANNHjxMAM6mq7pfk4UkuSvKRCZcDwJQTnACYVQaFAGCzCU4AzJyqul2SZ6QbFOJtEy4HgDXAPU6sukt/8+dHbnP2w984cpuNI7dYHR++7s4jt3nijleO3ObU+31g5Da53+hNctgy2ozo/p967sht7vXs80Zus/GGG0Zuw5p1VJI7JfmwQSEA2Bx6nACYRXODQrx1olUAsGYITgDMlKraN8kjY1AIAEbgUj0AZkpr7dwkNek6AFhb9DgBAAAMEJwAAAAGCE4AAAADBCcAAIABBocAgDHZb8+ds/64wyddBgArQI8TAADAAMEJAABggOAEAAAwQHACAAAYYHAIbpNt9rzbyG3e/oo3LGNPW4/c4s8u23+k9T/81weNvI87/9cPR26z7XevGrnNW/bYeeQ233ri7Udu855fO2HkNvttVyO3GdV/HXTiyG2OvMuTR26z8cKLRm4DAMwGPU4AAAAD9DgBwJicc/FVWXf0KSu6jw2GOweYCD1OAAAAAwQnAACAAYITAADAAMEJAABggOAEAAAwQHACAAAYIDgBMHOq6lFV9fdVdUlV3dhPT6uqx0+6NgCmk+c4ATBTqupVSf40yWVJPpzkkiS7JXlgkkOSfGRixQEwtQQnAGZGVR2VLjSdnuTI1to1C5ZvO5HCAJh6LtUDYCZU1VZJXpfkh0metjA0JUlr7aZVLwyANUGPE7fNVqNn77ts9eNl7GjrkVt89H89aqT1d3v3Z0fex3Is59PXN0dvs/eZo7d51WsfM3Kbr/75vqOt/4T/PfI+YEwenuSeST6Q5MqqOjzJfkluSPL51trq/CMAwJokOAEwKx7UT7+X5Kwk95+/sKo+leSprbXvb2ojVbV+iUX73OYKAZhaLtUDYFbs3k9/N8kOSX4xyR3T9Tp9NMlBSd4/mdIAmHZ6nACYFXPX/Fa6nqX/7N9/uaqenOT8JAdX1cM2ddlea+3Axeb3PVEHjLNgAKaHHicAZsWV/fSb80JTkqS1dn26XqckefCqVgXAmiA4ATArzuunP1hi+Vyw2mEVagFgjRGcAJgVn0o3sOW9q2q7RZbv1083rFpFAKwZghMAM6G1dlmS9ybZOckfz19WVY9N8ktJrkpy6upXB8C0MzgEALPkJUkekuSVVXVQks8n2SvJk5PcnOS3W2tLXcoHwAwTnACYGa21S6vqIUlelS4sPTTJNUlOSfI/W2ufm2R9AEwvwQmAmdJauyJdz9NLJl0LAGuHe5wAAAAG6HHiNvnxhReN3OY3fvfFI7e5eq/RT9W7fuhLI62/ceQ9bHk2XnPNyG3qplqBSm7t9y46ZOQ2G6+4cnglAIDNpMcJAABggOAEAAAwwKV6ADAm++25c9Yfd/ikywBgBehxAgAAGCA4AQAADBCcAAAABghOAAAAAwQnAACAAUbVA4AxOefiq7Lu6FMmsu8NRvMDWFF6nAAAAAYITgAAAAMEJwAAgAHucWLVbf+RL4zc5i7L2M/GZbSZdVs9YN+R2xz3S+9ZgUpu7Yx/u//Ibe513edWoBIAYFbpcQIAABggOAEAAAwQnAAAAAYITgDMjKraUFVtidd3J10fANPL4BAAzJqrkhy/yPxrV7sQANYOwQmAWfOD1toxky4CgLXFpXoAAAAD9DgBMGu2r6qnJ/nZJNcl+VKST7XWbp5sWQBMM8EJgFmzR5J3Lpj3rap6dmvtk0ONq2r9Eov2uc2VATC1XKoHwCx5e5LHpAtPOya5f5K3JFmX5F+q6gGTKw2AaabHCYCZ0Vo7dsGsc5L8blVdm+SlSY5J8uSBbRy42Py+J+qAMZQJwBTS4wQAyZv76UETrQKAqaXHCfiJr/3h9iO3edKOl420/k3LuP9+r4/cNHIbGNGl/XTHiVYBwNTS4wQAycP66TcnWgUAU0twAmAmVNXPVdWui8zfK8lf9W/ftbpVAbBWuFQPgFlxVJKjq+qMJN9Kck2SeyU5PMntknwkyV9MrjwAppngBMCsOCPJfZM8MN2leTsm+UGSf0v3XKd3ttba5MoDYJoJTgDMhP7htoMPuAWAxbjHCQAAYIDgBAAAMEBwAgAAGCA4AQAADDA4BACMyX577pz1xx0+6TIAWAF6nAAAAAYITgAAAANcqgdbqK3vdKeR27z8gaeuQCW3dvQlB4/cZpt/Xb8ClQAAbD49TgAAAAMEJwAAgAEu1QOAMTnn4quy7uhTJl1GNhjZD2Ds9DgBAAAMEJwAAAAGCE4AAAADBCcAAIABghMAAMAAwQkAAGCA4ATAzKqqZ1RV61/PnXQ9AEwvwQmAmVRV90jypiTXTroWAKaf4ATAzKmqSvL2JJcnefOEywFgDdhm0gUAK+PC5+47cpun73TayG2u2vijkdb/zNsPHHkfu+fMkdvAgBcleXSSQ/opAGySHicAZkpV7ZvkuCQntNY+Nel6AFgb9DgBMDOqapsk70zy7SSvWOY21i+xaJ/l1gXA9BOcAJglf5zkgUke2Vq7ftLFALB2CE4AzISqenC6Xqa/bK19drnbaa0teqNe3xN1wHK3C8B0c48TAFu8eZfonZ/kjyZcDgBrkOAEwCy4Q5L7JNk3yQ3zHnrbkvxJv87f9POOn1iVAEwtl+oBMAtuTPK2JZYdkO6+p39Lcl6SZV/GB8CWS3ACYIvXDwTx3MWWVdUx6YLT37bWTlzNugBYO1yqBwAAMEBwAgAAGCA4ATDTWmvHtNbKZXoAbIrgBAAAMMDgELCF2u1xF6/Kfr7yozuOtP7uf33mClUCALBy9DgBAAAMEJwAAAAGuFQPAMZkvz13zvrjDp90GQCsAD1OAAAAAwQnAACAAYITAADAAMEJAABggOAEAAAwQHACAAAYYDhyABiTcy6+KuuOPmXSZfzEBkOjA4yNHicAAIABghMAAMAAl+rBGrDx4AeO3Oa0+524jD2N/ruU55/0uyOtf4+cOfI+AAAmTY8TAADAAMEJAABggOAEAAAwQHACYGZU1euq6l+r6sKqur6qrqiqL1bVn1TVnSddHwDTS3ACYJa8OMmOST6W5IQk707y4yTHJPlSVd1jcqUBMM2MqgfALNmptXbDwplV9dokr0jy8iTPX/WqAJh6epwAmBmLhabe+/rpvVerFgDWFsEJAJIn9NMvTbQKAKaWS/UAmDlV9bIkd0iyc5JfSPLIdKHpuM1ou36JRfuMrUAApo7gBMAselmSu857f2qSZ7XWvj+hegCYcoITADOntbZHklTVXZM8PF1P0xer6ldaa2cNtD1wsfl9T9QB464VgOkgOMEa8Ky3/tPIbTZm48htTr/+jiO32ev4/xxp/dGrgpXTWvtekg9W1VlJzk/yjiT7TbYqAKaRwSEAmHmttQuSfCXJz1XVbpOuB4DpIzgBQOdu/fTmiVYBwFQSnACYCVW1T1Xtscj8rfoH4O6e5MzW2pWrXx0A0849TgDMisOS/HlVfSrJN5Jcnm5kvYOT7J3ku0l+e3LlATDNBCcAZsXpSd6a5BFJHpBklyTXpRsU4p1J3thau2Jy5QEwzQQnAGZCa+2cJC+YdB0ArE3ucQIAABggOAEAAAwQnAAAAAYITgAAAAMMDgEAY7Lfnjtn/XGHT7oMAFaAHicAAIABepxgDfjVO1w6cpuNy9jPH3z+10dus/d1Zy9jTwAAa4seJwAAgAGCEwAAwADBCQAAYIB7nABgTM65+KqsO/qUSZexSRuM+gewLHqcAAAABghOAAAAAwQnAACAAYITAADAAMEJAABggOAEAAAwQHACAAAYIDgBMBOq6s5V9dyq+mBVfb2qrq+qq6rq36rqt6rK/4kALMkDcGGVXfjKhy+j1fqx17GYH1/vnwS2aEcl+T9JLklyRpJvJ7lrkiOTnJjkl6vqqNZam1yJAEwr35IAmBXnJ3liklNaaxvnZlbVK5J8PslT0oWov59MeQBMM5clADATWmsfb6398/zQ1M//bpI3928PWfXCAFgTBCcASG7qpz+eaBUATC2X6gEw06pqmyT/rX976masv9RNh/uMrSgApo4eJwBm3XFJ9kvykdbaRyddDADTSY8TADOrql6U5KVJvprkGZvTprV24BLbWp/kgPFVB8A00eMEwEyqqhckOSHJV5Ic2lq7YsIlATDFBCcAZk5V/UGSv0pyTrrQ9N0JlwTAlBOcAJgpVfWHSd7MO8iqAAAOSElEQVSQ5Ox0oenSCZcEwBogOAEwM6rqj9INBrE+yWNaa5dNuCQA1giDQwAwE6rqmUleneTmJJ9O8qKqWrjahtbayatcGgBrgOAEwKy4Zz/dOskfLLHOJ5OcvCrVALCmCE6wyvZ+7LdWZT9fv+nHI7fZ54RrR26zceQWMBmttWOSHDPhMgBYo9zjBAAAMEBwAgAAGCA4AQAADBCcAAAABhgcAgDGZL89d8764w6fdBkArAA9TgAAAAMEJwAAgAGCEwAAwADBCQAAYIDgBAAAMMCoegAwJudcfFXWHX3Kiu5jg1H7ACZCjxMAAMAAPU5wG9zwhAeP3ObkvV+/jD3tMHKLJ3zyBSO3ufeXzhq5DQDALNDjBAAAMEBwAgAAGCA4AQAADBCcAAAABghOAMyEqnpqVb2pqj5dVVdXVauqd026LgDWBqPqATArXpXkAUmuTXJRkn0mWw4Aa4keJwBmxYuT3CfJTkmeN+FaAFhj9DgBMBNaa2fM/bmqJlkKAGuQHicAAIABepwAYARVtX6JRe6ZAtiC6XECAAAYoMcJAEbQWjtwsfl9T9QBq1wOAKtEcII5W209cpMLf2n0G8x33Xr7kdssx13+dXX2AwAwC1yqBwAAMEBwAgAAGCA4AQAADHCPEwAzoaqOSHJE/3aPfvqwqjq5//NlrbWXrXphAKwJghMAs2L/JM9cMG/v/pUkFyQRnABYlEv1AJgJrbVjWmu1ide6SdcIwPQSnAAAAAYITgAAAAMEJwAAgAGCEwAAwACj6gHAmOy3585Zf9zhky4DgBWgxwkAAGCAHifoffuPHzJym3OffMIKVPLTrt1448httr/65hWoBABgNulxAgAAGCA4AQAADBCcAAAABghOAAAAAwwOAQBjcs7FV2Xd0adMuoxb2WB4dICx0OMEAAAwQHACAAAYIDgBAAAMEJwAAAAGCE4AAAADBCcAAIABhiOH3s33vW7SJSzpdy54wshtdvjQ51egEljbquruSV6d5LAkd05ySZIPJTm2tXblJGsDYLoJTgDMhKq6V5Izk+ye5B+TfDXJg5P8fpLDquoRrbXLJ1giAFPMpXoAzIq/TheaXtRaO6K1dnRr7dFJ3pDkvkleO9HqAJhqghMAW7yq2jvJ45JsSPK/Fyz+kyTXJXlGVe24yqUBsEYITgDMgkf309NaaxvnL2itXZPkM0lun+Shq10YAGuDe5wAmAX37afnL7H8a+l6pO6T5F83taGqWr/Eon2WVxoAa4EeJwBmwc799Kolls/N32UVagFgDdLjBABJ9dM2tGJr7cBFN9D1RB0wzqIAmB56nACYBXM9SjsvsXynBesBwK0ITgDMgvP66X2WWH7vfrrUPVAAzDjBCYBZcEY/fVxV3er/vqq6Y5JHJLk+yedWuzAA1gbBCYAtXmvtG0lOS7IuyQsWLD42yY5J3tFau26VSwNgjTA4BACz4vlJzkzyxqp6TJJzkzwkyaHpLtF75QRrA2DKCU6wBlx7xKQrgLWvtfaNqvqFJK9OcliSxye5JMkbkxzbWrtikvUBMN0EJwBmRmvtwiTPnnQdAKw97nECAAAYIDgBAAAMEJwAAAAGCE4AAAADDA4BAGOy3547Z/1xh0+6DABWgB4nAACAAYITAADAAMEJAABggOAEAAAwQHACAAAYIDgBAAAMMBw59O75G/85cpsn5kErUMliLl+l/QAAsBg9TgAAAAMEJwAAgAGCEwAAwADBCQAAYIDgBAAAMEBwAgAAGCA4AQAADPAcJwAYj3XnnntuDjzwwEnXAUDv3HPPTZJ149iW4AQA43GH66+//uazzjpr9KdpM98+/fSrE61ibXMMx8NxHI9JH8d1Sa4ex4YEJwAYj3OSpLWmy+k2qKr1ieN4WziG4+E4jseWdBzd4wQAADBAcAIAABiwxV6q97GN769J1wAAAGwZ9DgBAAAMEJwAAAAGVGtt0jUAAABMNT1OAAAAAwQnAACAAYITAADAAMEJAABggOAEAAAwQHACAAAYIDgBAAAMEJwAAAAGCE4AzLSquntVnVRV36mqG6tqQ1UdX1V3GnE7u/btNvTb+U6/3buv9L6nwW39LFW1Y1X9ZlX936r6alVdV1XXVNV/VNVLq2q7Jdq1Tbw+N95PubLGcT5U1ScGjsntlmh3v6p6X1VdWlU3VNV5VXVsVe0wvk+4OsZwLh4ycAznXvdY0G6LOBer6qlV9aaq+nRVXd3X/65lbmvkn8U0n4vVWpt0DQAwEVV1ryRnJtk9yT8m+WqSByc5NMl5SR7RWrt8M7Zz534790ny8SRfSLJPkicluTTJw1pr31yJfU+DcXyWqjosyb8kuSLJGUm+nmTXJE9Iske//ce01m5Y0K4luSDJyYts9qLW2onL/mCraIzn4ieSHJzk2CVWeU1r7ccL2jwk3Xm7bZIPJLkwyaOT/EKSz6Q77jeO/qlW35jOxXVJnrXE4vsnOTLJl1tr+y1ot6Wci2cneUCSa5NclO7fsne31p4+4nZG/llM/bnYWvPy8vLy8prJV5KPJmlJXrhg/uv7+W/ezO28pV//9Qvmv6iff+pK7XsaXuP4LEn2T/KbSbZbMP+OSdb323npIu1akk9M+hhMwzHs1/9E9/Vus/e7dZKv9Pt44rz5W6X74tqSHD3p47Pax3ET2/+7fjsvWmTZlnIuHprk3kkqySH953rXSv8s1sK5qMcJgJlUVXsn+UaSDUnu1VrbOG/ZHZNcku6Lw+6ttes2sZ0dk3w/ycYkP9Nau2besq36fazr9/HNce57GqzGZ6mqpyV5d5IPt9aesGBZS/LJ1tohy/oAU2Ccx3Cux6m1Vpu570cn+dckn2qtHbxEXRckuWeb8i+NK30u9j3LF6f7u75na+3KBcvX/Lm4UFUdkq4HeKQep+X8LNbCuegeJwBm1aP76Wnz/1NPkj78fCbJ7ZM8dGA7D0uyQ5LPzA9N/XY2Jjmtf3voCux7GqzGZ7mpn/54ieW7VNVzquoVVfWCqloLx22+sR/Dqvq1qjq6ql5SVb9cVdsP7PvUhQv6oH9+kr2S7L25+56glT4Xn5Vk+yTvXxia5lnr5+K4LOdnMfXnouAEwKy6bz89f4nlX+un91mB7Yxr39NgNT7Lc/rpT32h6j0gyduSvDbJXyX5bFWdXVX3vw37XE0rcQzfk+R/JvnLJB9J8u2qeuoq7XtSVvqzPLefvmUT66z1c3Fctsh/FwUnAGbVzv30qiWWz83fZQW2M659T4MV/SxV9XtJDktydpKTFlnl9UkekeQu6e6HelC6+yEekOTjVbXncva7ysZ5DP8x3YAad0/XE7pPugC1S5L3VtUvr+C+J23FPktVHZzuWH65tXbmEqttCefiuGyR/y4KTgCwuLl7RG7rtfTL2c649j0Nlv1ZqurIJMcn+W6Sp7TWblq4Tmvtpa21M1trl7XWrm2t/Udr7agkf59ktyQvuw21T4vNPoattTe01j7cWru4tXZDa+281torkrw03fe+P1upfa8Bt+Wz/Pd+umRv04yci+OyJv9dFJwAmFVzv73ceYnlOy1Yb5zbGde+p8GKfJaqOiLd5WaXJjmkLRjOfTO8uZ8eNGK7SViN8+HEdPeI7d/fnL+a+14tK3Uu7prkKUmuT/LOZdS1ls7Fcdki/10UnACYVef106Wul793P13qevvbsp1x7XsajP2zVNVRSd6f5HvpRog7b6DJYr7fT3dcRtvVtuLnQ+uefzU3eMn8Y+JcHPbMdINCvK+19oNl1LWWzsVx2SL/XRScAJhVZ/TTx/XDhv9E/xv5R6T7DfPnBrbzuX69Ryz4Tf7ccOSPW7C/ce57Goz1s/RDj/9dku+kC01fG2iylLnRukbtqZqEFT8fquq+Se6ULjxdNm/Rx/vpYYu02Tvdl9gLMtvH8bf76VuXWddaOhfHZTk/i6k/FwUnAGZSa+0b6YYKX5fkBQsWH5vut8PvmP+8l6rap6r2WbCda9NdvrNjkmMWbOf3+u1/dP6lZsvZ97Qa13Hs5z8z3bH8dpKDhi7Pq6oD+udoLZz/8+lGNUuSd23+p5mMcR3Dqtp7sQEIqmq3JG/v376ntTZ/WPdPJjk3yUFV9cR5bbZK8rr+7Zun/RlOyXjPxXnLH5Vk3yTnbGJQiC3mXBxVVW3bH8N7zZ+/zH/jpv5c9ABcAGZW/5/9mUl2Tzca2blJHpLumUvnJ3l4a+3yeeu3JFn4cNH+wZhnpvuN6MeTfD7dl60npbtH5+H9F4ll73uajeM4VtWhSU5P90vdk5JcuMiuftBaO35em5OTHJnumF+Y5MZ0I58dlmTrJH+T5HfWwpf+MR3DZ6W7l+mT6R4WekWSn03y+HT3jfxHkscuvNysqh6S7hhum24UuG8neUySX0j3vJ3HtNZuHPdnXgnj+js9b/k7kzw9yYtaa2/axH5PzpZzLh6R5Ij+7R5JfildL8+n+3mXtdZe1q+7Lsm3klzQWlu3YDsj/xs39edia83Ly8vLy2tmX0nuke638Zck+VG6S0FOSLLrIuu27r/ORbeza9/ugn47l6QLAHcfx76n/XVbj2O6h4u2gdeGBW2OSPIPSb6e5Op5x/2fkzxx0sdkAsfw/klOTvJfSS5P9+DgK9J94X1hku02se/7pbuv7LJ0X/rPT9czsMOkj8tqH8d5y+6U7nKyHybZZWCfW8y5mK7nfLP+HqbrUfqpv5vL+VmshXNRjxMAAMAA9zgBAAAMEJwAAAAGCE4AAAADBCcAAIABghMAAMAAwQkAAGCA4AQAADBAcAIAABggOAEAAAwQnAAAAAYITgAAAAMEJwAAgAGCEwAAwADBCQAAYIDgBAAAMEBwAgAAGCA4AQAADBCcAAAABghOAAAAAwQnAACAAf8P3gY7GEHvin0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1193aa630>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 224,
       "width": 423
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Hyperparameters for our network\n",
    "input_size = 784\n",
    "hidden_sizes = [128, 64]\n",
    "output_size = 10\n",
    "\n",
    "# Build a feed-forward network\n",
    "model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),\n",
    "                      nn.ReLU(),\n",
    "                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),\n",
    "                      nn.ReLU(),\n",
    "                      nn.Linear(hidden_sizes[1], output_size),\n",
    "                      nn.Softmax(dim=1))\n",
    "print(model)\n",
    "\n",
    "# Forward pass through the network and display output\n",
    "images, labels = next(iter(trainloader))\n",
    "images.resize_(images.shape[0], 1, 784)\n",
    "ps = model.forward(images[0,:])\n",
    "helper.view_classify(images[0].view(1, 28, 28), ps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also pass in an `OrderedDict` to name the individual layers and operations. Note that a dictionary keys must be unique, so _each operation must have a different name_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (fc1): Linear(in_features=784, out_features=128, bias=True)\n",
       "  (relu1): ReLU()\n",
       "  (fc2): Linear(in_features=128, out_features=64, bias=True)\n",
       "  (relu2): ReLU()\n",
       "  (output): Linear(in_features=64, out_features=10, bias=True)\n",
       "  (softmax): Softmax()\n",
       ")"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "model = nn.Sequential(OrderedDict([\n",
    "                      ('fc1', nn.Linear(input_size, hidden_sizes[0])),\n",
    "                      ('relu1', nn.ReLU()),\n",
    "                      ('fc2', nn.Linear(hidden_sizes[0], hidden_sizes[1])),\n",
    "                      ('relu2', nn.ReLU()),\n",
    "                      ('output', nn.Linear(hidden_sizes[1], output_size)),\n",
    "                      ('softmax', nn.Softmax(dim=1))]))\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's your turn to build a simple network, use any method I've covered so far. In the next notebook, you'll learn how to train a network so it can make good predictions.\n",
    "\n",
    ">**Exercise:** Build a network to classify the MNIST images with _three_ hidden layers. Use 400 units in the first hidden layer, 200 units in the second layer, and 100 units in the third layer. Each hidden layer should have a ReLU activation function, and use softmax on the output layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: Your network here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Run this cell with your model to make sure it works ##\n",
    "# Forward pass through the network and display output\n",
    "images, labels = next(iter(trainloader))\n",
    "images.resize_(images.shape[0], 1, 784)\n",
    "ps = model.forward(images[0,:])\n",
    "helper.view_classify(images[0].view(1, 28, 28), ps)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}