Lab4_ConvolutionalNeuralNetwork - 講解 - Pooling - Predicate.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "29b9bd1d-766f-4422-ad96-de0accc1ce58"
    }
   },
   "source": [
    "# Lab 4 - Convolutional Neural Network with MNIST\n",
    "\n",
    "This lab corresponds to Module 4 of the \"Deep Learning Explained\" course. We assume that you have successfully Lab 1 to download the MNIST data.\n",
    "\n",
    "We will train a Convolutional Neural Network (CNN) on MNIST data. \n",
    "\n",
    "## Introduction\n",
    "\n",
    "A [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN, or ConvNet) is a type of [feed-forward](https://en.wikipedia.org/wiki/Feedforward_neural_network) artificial neural network made up of neurons that have learnable weights and biases, very similar to ordinary multi-layer perceptron (MLP) networks introduced in Module 3. The CNNs take advantage of the spatial nature of the data. \n",
    "\n",
    "In nature, we perceive different objects by their shapes, size and colors. For example, objects in a natural scene are typically composed of edges, corners/vertices (defined by two of more edges), color patches etc. These primitives are often identified using different detectors (e.g., edge detection, color detector) or combination of detectors interacting to facilitate image interpretation (object classification, region of interest detection, scene description etc.) in real world vision related tasks. These detectors are also known as filters. \n",
    "\n",
    "Convolution is a mathematical operator that takes an image and a filter as input and produces a filtered output (representing say egdges, corners, colors etc in the input image).  Historically, these filters are a set of weights that were often hand crafted or modeled with mathematical functions (e.g., [Gaussian](https://en.wikipedia.org/wiki/Gaussian_filter) / [Laplacian](http://homepages.inf.ed.ac.uk/rbf/HIPR2/log.htm) / [Canny](https://en.wikipedia.org/wiki/Canny_edge_detector) filter).  The filter outputs are mapped through non-linear activation functions mimicking human brain cells called [neurons](https://en.wikipedia.org/wiki/Neuron).\n",
    "\n",
    "Convolutional networks provide a machinery to learn these filters from the data directly instead of explicit mathematical models and have been found to be superior (in real world tasks) compared to historically crafted filters.  With convolutional networks, the focus is on learning the filter weights instead of learning individually fully connected pair-wise (between inputs and outputs) weights. In this way, the number of weights to learn is reduced when compared with the traditional MLP networks from the previous tutorials.  In a convolutional network, one learns several filters ranging in number from single digits to thousands depending on the network complexity.\n",
    "\n",
    "Many of the CNN primitives have been shown to have a conceptually parallel components in brain's [visual cortex](https://en.wikipedia.org/wiki/Visual_cortex). A neuron in the visual cortex will emit responses when a certain region of its input cells are stimulated. This region is known as the receptive field (RF) of the neuron. \n",
    "\n",
    "Equivalently, in a convolution layer, the input region corresponding to the filter dimensions at certain locations in the input layer can be considered as the receptive field of the nodes in the convolutional layer. Popular deep CNNs or ConvNets (such as [AlexNet](https://en.wikipedia.org/wiki/AlexNet), [VGG](https://arxiv.org/abs/1409.1556), [Inception](http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Szegedy_Going_Deeper_With_2015_CVPR_paper.pdf), [ResNet](https://arxiv.org/pdf/1512.03385v1.pdf)) that are used for various [computer vision](https://en.wikipedia.org/wiki/Computer_vision) tasks have many of these architectural primitives (inspired from biology).  \n",
    "\n",
    "We will introduce the convolution operation and gain familiarity with the different parameters in CNNs.\n",
    "\n",
    "**Problem**:\n",
    "We will continue to work on the same problem of recognizing digits in MNIST data. The MNIST data comprises of hand-written digits with little background noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"http://3.bp.blogspot.com/_UpN7DfJA0j4/TJtUBWPk0SI/AAAAAAAAABY/oWPMtmqJn3k/s1600/mnist_originals.png\" width=\"200\" height=\"200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 1\n",
    "Image(url= \"http://3.bp.blogspot.com/_UpN7DfJA0j4/TJtUBWPk0SI/AAAAAAAAABY/oWPMtmqJn3k/s1600/mnist_originals.png\", width=200, height=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Goal**:\n",
    "Our goal is to train a classifier that will identify the digits in the MNIST dataset. \n",
    "\n",
    "**Approach**:\n",
    "\n",
    "The same 5 stages we have used in the previous labs are applicable: Data reading, Data preprocessing, Creating a model, Learning the model parameters and Evaluating (a.k.a. testing/prediction) the model. \n",
    "\n",
    "We will experiment with two models with different architechtural components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbpresent": {
     "id": "138d1a78-02e2-4bd6-a20e-07b83f303563"
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "\n",
    "import cntk as C\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Select the right target device when this notebook is being tested:\n",
    "# if 'TEST_DEVICE' in os.environ:\n",
    "#     if os.environ['TEST_DEVICE'] == 'cpu':\n",
    "#         C.device.try_set_default_device(C.device.cpu())\n",
    "#     else:\n",
    "#         C.device.try_set_default_device(C.device.gpu(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test for CNTK version\n",
    "# if not C.__version__ == \"2.0\":\n",
    "#     raise Exception(\"this notebook is designed to work with 2.0. Current Version: \" + C.__version__) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data reading\n",
    "In this section, we will read the data generated in Lab1_MNIST_DataLoader.\n",
    "\n",
    "We are using the MNIST data that you have downloaded using the Lab1_MNIST_DataLoader notebook. The dataset has 60,000 training images and 10,000 test images with each image being 28 x 28 pixels. Thus the number of features is equal to 784 (= 28 x 28 pixels), 1 per pixel. The variable `num_output_classes` is set to 10 corresponding to the number of digits (0-9) in the dataset.\n",
    "\n",
    "In previous labs, as shown below, we have always flattened the input image into a vector.  With convoultional networks, we do **not** flatten the image - we preserve its 2D shape..\n",
    "\n",
    "**Input Dimensions**:  \n",
    "\n",
    "In convolutional networks for images, the input data is often shaped as a 3D matrix (number of channels, image width, image height), which preserves the spatial relationship between the pixels. In the MNIST data, the image is a single channel (grayscale) data, so the input dimension is specified as a (1, image width, image height) tuple. \n",
    "\n",
    "![input-rgb](https://www.cntk.ai/jup/cntk103d_rgb.png)\n",
    "\n",
    "Natural scene color images are often presented as Red-Green-Blue (RGB) color channels. The input dimension of such images are specified as a (3, image width, image height) tuple. If one has RGB input data as a volumetric scan with volume width, volume height and volume depth representing the 3 axes, the input data format would be specified by a tuple of 4 values (3, volume width, volume height, volume depth). In this way CNTK enables specification of input images in arbitrary higher-dimensional space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure we always get the same amount of randomness\n",
    "np.random.seed(0)\n",
    "C.cntk_py.set_fixed_random_seed(1)\n",
    "C.cntk_py.force_deterministic_algorithms()\n",
    "\n",
    "# Define the data dimensions\n",
    "input_dim_model = (1, 28, 28)    # images are 28 x 28 with 1 channel of color (gray)\n",
    "input_dim = 28*28                # used by readers to treat input data as a vector\n",
    "num_output_classes = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data reading\n",
    "\n",
    "There are different ways one can read data into CNTK. The easiest way is to load the data in memory using NumPy / SciPy / Pandas readers. However, this can be done only for small data sets. Since deep learning requires large amount of data we have chosen in this course to show how to leverage built-in distributed readers that can scale to terrabytes of data with little extra effort. \n",
    "\n",
    "We are using the MNIST data you have downloaded using Lab 1 DataLoader notebook. The dataset has 60,000 training images and 10,000 test images with each image being 28 x 28 pixels. Thus the number of features is equal to 784 (= 28 x 28 pixels), 1 per pixel. The variable `num_output_classes` is set to 10 corresponding to the number of digits (0-9) in the dataset.\n",
    "\n",
    "In Lab 1, the data was downloaded and written to 2 CTF (CNTK Text Format) files, 1 for training, and 1 for testing. Each line of these text files takes the form:\n",
    "\n",
    "    |labels 0 0 0 1 0 0 0 0 0 0 |features 0 0 0 0 ... \n",
    "                                                  (784 integers each representing a pixel)\n",
    "    \n",
    "We are going to use the image pixels corresponding the integer stream named \"features\". We define a `create_reader` function to read the training and test data using the [CTF deserializer](https://cntk.ai/pythondocs/cntk.io.html?highlight=ctfdeserializer#cntk.io.CTFDeserializer). The labels are [1-hot encoded](https://en.wikipedia.org/wiki/One-hot). Refer to Lab 1 for data format visualizations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read a CTF formatted text (as mentioned above) using the CTF deserializer from a file\n",
    "def create_reader(path, is_training, input_dim, num_label_classes):\n",
    "    \n",
    "    ctf = C.io.CTFDeserializer(path, C.io.StreamDefs(\n",
    "          labels=C.io.StreamDef(field='labels', shape=num_label_classes, is_sparse=False),\n",
    "          features=C.io.StreamDef(field='features', shape=input_dim, is_sparse=False)))\n",
    "                          \n",
    "    return C.io.MinibatchSource(ctf,\n",
    "        randomize = is_training, max_sweeps = C.io.INFINITELY_REPEAT if is_training else 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data directory is data\\MNIST\n"
     ]
    }
   ],
   "source": [
    "# Ensure the training and test data is available\n",
    "# We search in two locations in the toolkit for the cached MNIST data set.\n",
    "\n",
    "data_found=False # A flag to indicate if train/test data found in local cache\n",
    "for data_dir in [os.path.join(\"..\", \"Examples\", \"Image\", \"DataSets\", \"MNIST\"),\n",
    "                 os.path.join(\"data\", \"MNIST\")]:\n",
    "    \n",
    "    train_file=os.path.join(data_dir, \"Train-28x28_cntk_text.txt\")\n",
    "    test_file=os.path.join(data_dir, \"Test-28x28_cntk_text.txt\")\n",
    "    \n",
    "    if os.path.isfile(train_file) and os.path.isfile(test_file):\n",
    "        data_found=True\n",
    "        break\n",
    "        \n",
    "if not data_found:\n",
    "    raise ValueError(\"Please generate the data by completing Lab1_MNIST_DataLoader\")\n",
    "    \n",
    "print(\"Data directory is {0}\".format(data_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='#Model Creation'></a>\n",
    "## CNN Model Creation\n",
    "\n",
    "CNN is a feedforward network made up of bunch of layers in such a way that the output of one layer becomes the input to the next layer (similar to MLP). In MLP, all possible pairs of input pixels are connected to the output nodes with each pair having a weight, thus leading to a combinatorial explosion of parameters to be learnt and also increasing the possibility of overfitting ([details](http://cs231n.github.io/neural-networks-1/)). Convolution layers take advantage of the spatial arrangement of the pixels and learn multiple filters that significantly reduce the amount of parameters in the network ([details](http://cs231n.github.io/convolutional-networks/)). The size of the filter is a parameter of the convolution layer.  \n",
    "\n",
    "In this section, we introduce the basics of convolution operations. We show the illustrations in the context of RGB images (3 channels), eventhough the MNIST data we are using a grayscale image (single channel).\n",
    "\n",
    "![input-rgb](https://www.cntk.ai/jup/cntk103d_rgb.png)\n",
    "\n",
    "### Convolution Layer\n",
    "\n",
    "A convolution layer is a set of filters. Each filter is defined by a weight (**W**) matrix, and  bias ($b$).\n",
    "\n",
    "![input-filter](https://www.cntk.ai/jup/cntk103d_filterset_v2.png)\n",
    "\n",
    "These filters are scanned across the image performing the dot product between the weights and corresponding input value (${x}$). The bias value is added to the output of the dot product and the resulting sum is optionally mapped through an activation function. This process is illustrated in the following animation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/cntk103d_conv2d_final.gif\" width=\"300\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url=\"https://www.cntk.ai/jup/cntk103d_conv2d_final.gif\", width= 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convolution layers incorporate the following key features:\n",
    "\n",
    "   - Instead of being fully-connected to all input nodes , each convolution node is **locally-connected** to a subset of input nodes localized to a smaller input region, also referred to as receptive field (RF). The figure above illustrates a small 3 x 3 regions in the image as the RF region. In the case of an RGB, image there would be 3 such 3 x 3 regions, one for each of the 3 color channels. \n",
    "   \n",
    "   \n",
    "   - Instead of having a single set of weights (as in a Dense layer), convolutional layers have multiple sets (shown in figure with multiple colors), called **filters**. Each filter detects features within each possible RF in the input image.  The output of the convolution is a set of `n` sub-layers (shown in the animation below) where `n` is the number of filters (refer to the above figure).  \n",
    "   \n",
    "     \n",
    "   - Within a sublayer, instead of each node having its own set of weights, a single set of **shared weights** are used by all nodes in that sublayer. This reduces the number of parameters to be learnt and help reduce the risk of overfitting. This also opens the door for several aspects of deep learning which has enabled very practical solutions to be built:\n",
    "    - Handling larger images (say 512 x 512)\n",
    "    - Trying larger filter sizes (corresponding to a larger RF) say 11 x 11\n",
    "    - Learning more filters (say 128)\n",
    "    - Explore deeper architectures (100+ layers)\n",
    "    - Achieve translation invariance (the ability to recognize a feature independent of where they appear in the image). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strides and Pad parameters\n",
    "\n",
    "**How are filters positioned?** In general, the filters are arranged in overlapping tiles, from left to right, and top to bottom.  Each convolution layer has a parameter to specify the `filter_shape`, specifying the width and height of the filter.  There is a parameter (`strides`) that controls the how far to step to right when moving the filters through multiple RF's in a row, and how far to step down when moving to the next row.  The boolean parameter `pad` controls if the input should be padded around the edges to allow a complete tiling of the RF's near the borders. \n",
    "\n",
    "The animation above shows the results with a `filter_shape` = (3, 3), `strides` = (2, 2) and `pad` = False. The two animations below show the results when `pad` is set to True. First, with a stride of 2 and second having a stride of 1.\n",
    "Note: the shape of the output (the teal layer) is different between the two stride settings. In many problems, the stride and pad values are chosen to control the size of output layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With stride = 2\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/cntk103d_padding_strides.gif\" width=\"200\" height=\"200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With stride = 1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/cntk103d_same_padding_no_strides.gif\" width=\"200\" height=\"200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot images with strides of 2 and 1 with padding turned on\n",
    "images = [(\"https://www.cntk.ai/jup/cntk103d_padding_strides.gif\" , 'With stride = 2'),\n",
    "          (\"https://www.cntk.ai/jup/cntk103d_same_padding_no_strides.gif\", 'With stride = 1')]\n",
    "\n",
    "for im in images:\n",
    "    print(im[1])\n",
    "    display(Image(url=im[0], width=200, height=200))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building our CNN models\n",
    "\n",
    "We define two containers. One for the input MNIST image and the second one being the labels corresponding to the 10 digits. When reading the data, the reader automatically maps the 784 pixels per image to a shape defined by the `input_dim_model` tuple (in this example it is set to (1, 28, 28))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = C.input_variable(input_dim_model) # (channel, width, height) = (1, 28, 28)\n",
    "y = C.input_variable(num_output_classes) # 0123456789 classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first model we build is a simple convolution only network. Here we have two convolutional layers. Since, our task is to detect the 10 digits in the MNIST database, the output of the network should be a vector of length 10, 1 element corresponding to each output class. This is achieved by projecting the output of the last convolutional layer using a dense layer with the output being `num_output_classes`. We have seen this before with Logistic Regression and MLP where features were mapped to the number of classes in the final layer. Also, note that since we will be using the `softmax` operation that is combined with the `cross entropy` loss function during training (see a few cells below), the final dense layer has no activation function associated with it.\n",
    "\n",
    "The following figure illustrates the model we are going to build. Note the parameters in the model below are to be experimented with. These are often called network hyperparameters. Increasing the filter shape leads to an increase in the number of model parameters, increases the compute time and helps the model better fit to the data. However, one runs the risk of [overfitting](https://en.wikipedia.org/wiki/Overfitting). Typically, the number of filters in the deeper layers are more than the number of filters in the layers before them. We have chosen 8, 16 for the first and second layers, respectively. These hyperparameters should be experimented with during model building.\n",
    "\n",
    "![conv-only](https://www.cntk.ai/jup/cntk103d_convonly2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to build model\n",
    "# Sigmoid, Tanh, ReLu, LeakyRelu\n",
    "def create_model(features):\n",
    "    with C.layers.default_options(init=C.glorot_uniform(), activation=C.relu):\n",
    "            h = features\n",
    "            h = C.layers.Convolution2D(filter_shape=(5,5), \n",
    "                                       num_filters=8, \n",
    "                                       strides=(2,2), \n",
    "                                       pad=True, name='first_conv')(h)\n",
    "            h = C.layers.Convolution2D(filter_shape=(5,5), \n",
    "                                       num_filters=16, \n",
    "                                       strides=(2,2), \n",
    "                                       pad=True, name='second_conv')(h)\n",
    "            r = C.layers.Dense(num_output_classes, activation=None, name='classify')(h)\n",
    "            return r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create an instance of the model and inspect the different components of the model. `z` will be used to represent the output of a network. In this model, we use the `relu` activation function. Note: using the `C.layers.default_options` is an elegant and concise way to build models. This is key to minimizing modeling errors, saving precious debugging time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output Shape of the first convolution layer: (8, 14, 14)\n",
      "Output Shape of the second convolution layer: (16, 7, 7)\n",
      "Bias value of the last dense layer: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "# Create the model\n",
    "z = create_model(x)\n",
    "\n",
    "# Print the output shapes / parameters of different components\n",
    "print(\"Output Shape of the first convolution layer:\", z.first_conv.shape)\n",
    "print(\"Output Shape of the second convolution layer:\", z.second_conv.shape)\n",
    "print(\"Bias value of the last dense layer:\", z.classify.b.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding the number of learnable parameters in a model is key to deep learning since there is a  dependency between the number of parameters and the amount of data one needs to have to train the model. \n",
    "\n",
    "You need more data for a model that has a larger number of parameters to prevent overfitting. In other words, with a fixed amount of data, one has to constrain the number of parameters. There is no golden rule between the amount of data one needs for a model. However, there are ways one can boost performance of model training with [data augmentation](https://deeplearningmania.quora.com/The-Power-of-Data-Augmentation-2). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 11274 parameters in 6 parameter tensors.\n"
     ]
    }
   ],
   "source": [
    "# Number of parameters in the network\n",
    "C.logging.log_number_of_parameters(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Understanding Parameters**:\n",
    "\n",
    "\n",
    "Our model has 2 convolution layers each having a weight and bias. This adds up to 4 parameter tensors. Additionally the dense layer has weight and bias tensors. Thus, the 6 parameter tensors.\n",
    "\n",
    "Remember that in a convolutional layer, the number of parameters is not dependent on the number of nodes, only on the shared weights and bias of each filter.\n",
    "\n",
    "Let us now count the number of parameters:\n",
    "- *First convolution layer*: There are 8 filters each of size (1 x 5 x 5) where 1 is the number of channels in the input image. This adds up to 200 values in the weight matrix and 8 bias values.   \n",
    "\n",
    "\n",
    "- *Second convolution layer*: There are 16 filters each of size (8 x 5 x 5) where 8 is the number of channels in the input to the second layer (= output of the first layer). This adds up to 3200 values in the weight matrix and 16 bias values.\n",
    "\n",
    "\n",
    "- *Last dense layer*: There are 16 x 7 x 7 input values and it produces 10 output values corresponding to the 10 digits in the MNIST dataset. This corresponds to (16 x 7 x 7) x 10 weight values and 10 bias values.\n",
    "\n",
    "Adding these up gives the 11274 parameters in the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Knowledge Check**: Does the dense layer shape align with the task (MNIST digit classification)?\n",
    "\n",
    "** Suggested Explorations **\n",
    "- Try printing shapes and parameters of different network layers,\n",
    "- Record the training error you get with `relu` as the activation function,\n",
    "- Now change to `sigmoid` as the activation function and see if you can improve your training error.\n",
    "- Different supported activation functions can be [found here][]. Which activation function gives the least training error?\n",
    "\n",
    "[found here]: https://github.com/Microsoft/CNTK/wiki/Activation-Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning model parameters\n",
    "\n",
    "We use the `softmax` function to map the accumulated evidences or activations to a probability distribution over the classes (Details of the [softmax function][] and other [activation][] functions).\n",
    "\n",
    "[softmax function]: http://cntk.ai/pythondocs/cntk.ops.html#cntk.ops.softmax\n",
    "\n",
    "[activation]: https://github.com/Microsoft/CNTK/wiki/Activation-Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "We minimize the cross-entropy between the label and predicted probability by the network. Since we are going to build more than one model, we will create a few helper functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_criterion_function(model, labels):\n",
    "    loss = C.cross_entropy_with_softmax(model, labels)\n",
    "    errs = C.classification_error(model, labels)\n",
    "    return loss, errs # (model, labels) -> (loss, error metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will need a helper function to perform the model training. First let us create additional helper functions that will be needed to visualize different functions associated with training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a utility function to compute the moving average sum.\n",
    "# A more efficient implementation is possible with np.cumsum() function\n",
    "def moving_average(a, w=5):\n",
    "    if len(a) < w:\n",
    "        return a[:]    # Need to send a copy of the array\n",
    "    return [val if idx < w else sum(a[(idx-w):idx])/w for idx, val in enumerate(a)]\n",
    "\n",
    "\n",
    "# Defines a utility that prints the training progress\n",
    "def print_training_progress(trainer, mb, frequency, verbose=1):\n",
    "    training_loss = \"NA\"\n",
    "    eval_error = \"NA\"\n",
    "\n",
    "    if mb%frequency == 0:\n",
    "        training_loss = trainer.previous_minibatch_loss_average\n",
    "        eval_error = trainer.previous_minibatch_evaluation_average\n",
    "        if verbose: \n",
    "            print (\"Minibatch: {0}, Loss: {1:.4f}, Error: {2:.2f}%\".format(mb, training_loss, eval_error*100))\n",
    "        \n",
    "    return mb, training_loss, eval_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure training\n",
    "\n",
    "Previously we have described the concepts of `loss` function, the optimizers or [learners](https://cntk.ai/pythondocs/cntk.learners.html) and the associated machinery needed to train a model. Please refer to earlier labs for gaining familiarility with these concepts. Here we combine model training and testing in a helper function below. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_test(train_reader, test_reader, model_func, num_sweeps_to_train_with=10):\n",
    "    \n",
    "    # Instantiate the model function; x is the input (feature) variable \n",
    "    # We will scale the input image pixels within 0-1 range by dividing all input value by 255.\n",
    "    model = model_func(x/255)\n",
    "    \n",
    "    # Instantiate the loss and error function\n",
    "    loss, label_error = create_criterion_function(model, y)\n",
    "    \n",
    "    # Instantiate the trainer object to drive the model training\n",
    "    learning_rate = 0.2\n",
    "    lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)\n",
    "    learner = C.sgd(z.parameters, lr_schedule)\n",
    "    trainer = C.Trainer(z, (loss, label_error), [learner])\n",
    "    \n",
    "    # Initialize the parameters for the trainer\n",
    "    minibatch_size = 64\n",
    "    num_samples_per_sweep = 60000\n",
    "    num_minibatches_to_train = (num_samples_per_sweep * num_sweeps_to_train_with) / minibatch_size\n",
    "    \n",
    "    # Map the data streams to the input and labels.\n",
    "    input_map={\n",
    "        y  : train_reader.streams.labels,\n",
    "        x  : train_reader.streams.features\n",
    "    } \n",
    "    \n",
    "    # Uncomment below for more detailed logging\n",
    "    training_progress_output_freq = 500\n",
    "     \n",
    "    # Start a timer\n",
    "    start = time.time()\n",
    "\n",
    "    for i in range(0, int(num_minibatches_to_train)):\n",
    "        # Read a mini batch from the training data file\n",
    "        data=train_reader.next_minibatch(minibatch_size, input_map=input_map) \n",
    "        trainer.train_minibatch(data)\n",
    "        print_training_progress(trainer, i, training_progress_output_freq, verbose=1)\n",
    "     \n",
    "    # Print training time\n",
    "    print(\"Training took {:.1f} sec\".format(time.time() - start))\n",
    "    \n",
    "    # Test the model\n",
    "    test_input_map = {\n",
    "        y  : test_reader.streams.labels,\n",
    "        x  : test_reader.streams.features\n",
    "    }\n",
    "\n",
    "    # Test data for trained model\n",
    "    test_minibatch_size = 512\n",
    "    num_samples = 10000\n",
    "    num_minibatches_to_test = num_samples // test_minibatch_size\n",
    "\n",
    "    test_result = 0.0   \n",
    "\n",
    "    for i in range(num_minibatches_to_test):\n",
    "    \n",
    "        # We are loading test data in batches specified by test_minibatch_size\n",
    "        # Each data point in the minibatch is a MNIST digit image of 784 dimensions \n",
    "        # with one pixel per dimension that we will encode / decode with the \n",
    "        # trained model.\n",
    "        data = test_reader.next_minibatch(test_minibatch_size, input_map=test_input_map)\n",
    "        eval_error = trainer.test_minibatch(data)\n",
    "        test_result = test_result + eval_error\n",
    "\n",
    "    # Average of evaluation errors of all test minibatches\n",
    "    print(\"Average test error: {0:.2f}%\".format(test_result*100 / num_minibatches_to_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='#Run the trainer'></a>\n",
    "### Run the trainer and test model\n",
    "\n",
    "We are now ready to train our convolutional neural net. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch: 0, Loss: 2.3093, Error: 89.06%\n",
      "Minibatch: 500, Loss: 0.2157, Error: 9.38%\n",
      "Minibatch: 1000, Loss: 0.1370, Error: 4.69%\n",
      "Minibatch: 1500, Loss: 0.1173, Error: 3.12%\n",
      "Minibatch: 2000, Loss: 0.0235, Error: 0.00%\n",
      "Minibatch: 2500, Loss: 0.0032, Error: 0.00%\n",
      "Minibatch: 3000, Loss: 0.0101, Error: 0.00%\n",
      "Minibatch: 3500, Loss: 0.0499, Error: 3.12%\n",
      "Minibatch: 4000, Loss: 0.0252, Error: 1.56%\n",
      "Minibatch: 4500, Loss: 0.0378, Error: 1.56%\n",
      "Minibatch: 5000, Loss: 0.0157, Error: 1.56%\n",
      "Minibatch: 5500, Loss: 0.0023, Error: 0.00%\n",
      "Minibatch: 6000, Loss: 0.0097, Error: 0.00%\n",
      "Minibatch: 6500, Loss: 0.0177, Error: 0.00%\n",
      "Minibatch: 7000, Loss: 0.0152, Error: 0.00%\n",
      "Minibatch: 7500, Loss: 0.0027, Error: 0.00%\n",
      "Minibatch: 8000, Loss: 0.0033, Error: 0.00%\n",
      "Minibatch: 8500, Loss: 0.0157, Error: 0.00%\n",
      "Minibatch: 9000, Loss: 0.0402, Error: 1.56%\n",
      "Training took 43.7 sec\n",
      "Average test error: 1.37%\n"
     ]
    }
   ],
   "source": [
    "def do_train_test():\n",
    "    global z\n",
    "    z = create_model(x)\n",
    "    reader_train = create_reader(train_file, True, input_dim, num_output_classes)\n",
    "    reader_test = create_reader(test_file, False, input_dim, num_output_classes)\n",
    "    train_test(reader_train, reader_test, z)\n",
    "    \n",
    "do_train_test()\n",
    "# Sigmoid 2.69, Tanh 1.40, ReLu 1.37, LeakyRelu 1.31"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, the average test error is very comparable to our training error indicating that our model has good \"out of sample\" error a.k.a. [generalization error](https://en.wikipedia.org/wiki/Generalization_error). This implies that our model can very effectively deal with previously unseen observations (during the training process). This is key to avoid [overfitting](https://en.wikipedia.org/wiki/Overfitting).\n",
    "\n",
    "Let us check what is the value of some of the network parameters. We will check the bias value of the output dense layer. Previously, it was all 0. Now you see there are non-zero values, indicating that a model parameters were updated during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bias value of the last dense layer: [-0.01862428 -0.05158406  0.07529527 -0.08441418 -0.02108228 -0.06890385\n",
      " -0.00570317 -0.13155235  0.26497364  0.04138236]\n"
     ]
    }
   ],
   "source": [
    "print(\"Bias value of the last dense layer:\", z.classify.b.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation / Prediction\n",
    "We have so far been dealing with aggregate measures of error. Let us now get the probabilities associated with individual data points. For each observation, the `eval` function returns the probability distribution across all the classes. The classifier is trained to recognize digits, hence has 10 classes. First let us route the network output through a `softmax` function. This maps the aggregated activations across the network to probabilities across the 10 classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "out = C.softmax(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us test a small minibatch sample from the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the data for evaluation\n",
    "reader_eval=create_reader(test_file, False, input_dim, num_output_classes)\n",
    "\n",
    "eval_minibatch_size = 25\n",
    "eval_input_map = {x: reader_eval.streams.features, y:reader_eval.streams.labels} \n",
    "\n",
    "data = reader_eval.next_minibatch(eval_minibatch_size, input_map=eval_input_map)\n",
    "\n",
    "img_label = data[y].asarray()\n",
    "img_data = data[x].asarray()\n",
    "\n",
    "# reshape img_data to: M x 1 x 28 x 28 to be compatible with model\n",
    "img_data = np.reshape(img_data, (eval_minibatch_size, 1, 28, 28))\n",
    "\n",
    "predicted_label_prob = [out.eval(img_data[i]) for i in range(len(img_data))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the index with the maximum value for both predicted as well as the ground truth\n",
    "pred = [np.argmax(predicted_label_prob[i]) for i in range(len(predicted_label_prob))]\n",
    "gtlabel = [np.argmax(img_label[i]) for i in range(len(img_label))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label    : [7, 2, 1, 0, 4, 1, 4, 9, 5, 9, 0, 6, 9, 0, 1, 5, 9, 7, 3, 4, 9, 6, 6, 5, 4]\n",
      "Predicted: [7, 2, 1, 0, 4, 1, 4, 9, 5, 9, 0, 6, 9, 0, 1, 5, 9, 7, 3, 4, 9, 6, 6, 5, 4]\n"
     ]
    }
   ],
   "source": [
    "print(\"Label    :\", gtlabel[:25])\n",
    "print(\"Predicted:\", pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us visualize some of the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image Label:  1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAAptJREFUeJztmzFP6lAYhh9ADE44sIKTGwmJiS4mbqwwO7iSwOSPUP8AP4PExclRY7owGjcHBxgJm4lx4A6mXit1uvY9eu77TLQdzseTt6dfT9vSarXCFEs5dAH/A5YswJIFWLIASxZgyQIsWYAlC9gQjxfznU/pqwNOsgBLFmDJAixZgCULsGQBlizAkgVYsgD1HV8wrq6uAOj1egCMx2MAhsMhAJVKpbCxnWQBJfGDVPnaxWKxAKDT6QAwn88zx5+fnwHY2tr616G8dhGS6Ofk29tbYD3Bx8fHANRqtcJrcJIFRJvkl5cXAM7OznKPn5ycAFAqfTmVfhtOsoBou4vpdArAwcFBZv/GxtvJ+/r6+t1DursISbRz8uXlZe7+brcrrsRJlhBtkm9ubjLbm5ubAFxcXMhrcZIFRNldJEnC4eFhZt/29jYAy+WyqGHdXYQkyjk57ZE/kq4bhyB6yek0MRqNQpXj6UJBVBe+u7s7AI6Ojkj/187ODgBPT09FDg2+8IUlqjk5fdT08ewMcRv9GSdZQFRJnkwm77/TrmIwGIQq5x0nWUAU3cVsNgOg1Wq9DbJa0W63Abi/vy9iyDzcXYQkijk5SRIg21X0+/1Q5azhJAuIIslpf5zSaDQ4PT0NVM06TrKAKJJ8fX2d2W42m9Tr9UDVrOMkC/jVSU5fUHl8fMzsr9VqVKvVECXl4iQL+NVJLpffMrK/vw/Aw8MDALu7u8FqysNJFvCrk5x+THN+fg78fQ12b28vWE15OMkColiF+yF4FS4klizAkgVYsgBLFqDuk4v/nusH4iQLsGQBlizAkgVYsgBLFmDJAixZgCULsGQBlizAkgVYsgBLFmDJAixZgCULsGQBlizAkgVYsgBLFmDJAv4ABcGGMBb9blwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 72x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot a random image\n",
    "sample_number = 5\n",
    "plt.figure(figsize=(1,1))\n",
    "plt.imshow(img_data[sample_number].reshape(28,28), cmap=\"gray_r\")\n",
    "plt.axis('off')\n",
    "\n",
    "img_gt, img_pred = gtlabel[sample_number], pred[sample_number]\n",
    "print(\"Image Label: \", img_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pooling Layer\n",
    "\n",
    "Often times, one needs to control the number of parameters especially when having deep networks. For every layer of the convolution layer output (each layer, corresponds to the output of a filter), one can have a pooling layer. Pooling layers are typically introduced to:\n",
    "- Reduce the shape of the current layer (speeding up the network),\n",
    "- Make the model more tolerant to changes in object location in the image. For example, even when a digit is shifted to one side of the image instead of being in the middle, the classifer would perform the classification task well.\n",
    "\n",
    "The calculation on a pooling node is much simpler than a normal feedforward node.  It has no weight, bias, or activation function.  It uses a simple aggregation function (like max or average) to compute its output.  The most commonly used function is \"max\" - a max pooling node simply outputs the maximum of the input values corresponding to the filter position of the input. The figure below shows the input values in a 4 x 4 region. The max pooling window size is 2 x 2 and starts from the top left corner, and uses a stride of 2x2.  The maximum value within the window becomes the output of the region. Every time the model is shifted by the amount specified by the stride parameter (as shown in the figure below) and the maximum pooling operation is repeated. \n",
    "![maxppool](https://cntk.ai/jup/201/MaxPooling.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another alternative is average pooling, which emits that average value instead of the maximum value. The two different pooling opearations are summarized in the animation below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max pooling\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/c103d_max_pooling.gif\" width=\"200\" height=\"200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average pooling\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/c103d_average_pooling.gif\" width=\"200\" height=\"200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot images with strides of 2 and 1 with padding turned on\n",
    "images = [(\"https://www.cntk.ai/jup/c103d_max_pooling.gif\" , 'Max pooling'),\n",
    "          (\"https://www.cntk.ai/jup/c103d_average_pooling.gif\", 'Average pooling')]\n",
    "\n",
    "for im in images:\n",
    "    print(im[1])\n",
    "    display(Image(url=im[0], width=200, height=200))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Typical convolution network\n",
    "\n",
    "![mnist-conv-mp](http://www.cntk.ai/jup/conv103d_mnist-conv-mp.png)\n",
    "\n",
    "A typical CNN contains a set of alternating convolution and pooling layers followed by a dense output layer for classification. You will find variants of this structure in many classical deep networks (VGG, AlexNet etc).  This is in contrast to the MLP network we used in Lab 3, which consisted of 2 dense layers followed by a dense output layer.  \n",
    "\n",
    "The illustrations are presented in the context of 2-dimensional (2D) images, but the concept and the CNTK components can operate on any dimensional data. The above schematic shows 2 convolution layer and 2 pooling layers. A typical strategy is to increase the number of filters in the deeper layers while reducing the spatial size of each intermediate layers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task: Create a network with Average Pooling\n",
    "\n",
    "Typical convolutional networks have interlacing convolution and pooling layers. The previous model had only convolution layer. In this section, you will create a model with the following architecture.\n",
    "\n",
    "![conv-only](https://courses.edx.org/asset-v1:Microsoft+DAT236x+2T2017+type@asset+block@conv_pool2.png)\n",
    "\n",
    "You will use the CNTK [Average Pooling](https://cntk.ai/pythondocs/cntk.layers.layers.html#cntk.layers.layers.AveragePooling) function to achieve this task. You will edit the `create_model` function below and add the Average Pooling operation. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify this model 2019.11.24 挑戰\n",
    "# relu       1.10%\n",
    "# leaky_relu 1.16%\n",
    "def create_model(features):\n",
    "    with C.layers.default_options(init = C.glorot_uniform(), activation = C.tanh):\n",
    "            h = features\n",
    "            \n",
    "            h = C.layers.Convolution2D(filter_shape=(5,5), \n",
    "                                       num_filters=8, \n",
    "                                       strides=(1,1), \n",
    "                                       pad=True, name='first_conv')(h)\n",
    "            h = C.layers.AveragePooling((5,5), strides=(2,2), name='first_pooling' )(h)\n",
    "            h = C.layers.Convolution2D(filter_shape=(5,5), \n",
    "                                       num_filters=16, \n",
    "                                       strides=(1,1), \n",
    "                                       pad=True, name='second_conv')(h)\n",
    "            h = C.layers.AveragePooling((5,5), strides=(2,2), name='second_pooling' )(h)\n",
    "            r = C.layers.Dense(num_output_classes, activation = None, name='classify')(h)\n",
    "            return r\n",
    "\n",
    "# relu \n",
    "# sigmoid = 3.89 %\n",
    "# tanh = 1.47%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch: 0, Loss: 2.2833, Error: 89.06%\n",
      "Minibatch: 500, Loss: 0.4262, Error: 15.62%\n",
      "Minibatch: 1000, Loss: 0.2001, Error: 7.81%\n",
      "Minibatch: 1500, Loss: 0.2382, Error: 7.81%\n",
      "Minibatch: 2000, Loss: 0.0813, Error: 3.12%\n",
      "Minibatch: 2500, Loss: 0.0414, Error: 1.56%\n",
      "Minibatch: 3000, Loss: 0.0291, Error: 0.00%\n",
      "Minibatch: 3500, Loss: 0.2426, Error: 12.50%\n",
      "Minibatch: 4000, Loss: 0.0891, Error: 1.56%\n",
      "Minibatch: 4500, Loss: 0.0463, Error: 1.56%\n",
      "Minibatch: 5000, Loss: 0.0497, Error: 1.56%\n",
      "Minibatch: 5500, Loss: 0.0327, Error: 1.56%\n",
      "Minibatch: 6000, Loss: 0.0474, Error: 1.56%\n",
      "Minibatch: 6500, Loss: 0.0799, Error: 1.56%\n",
      "Minibatch: 7000, Loss: 0.0312, Error: 1.56%\n",
      "Minibatch: 7500, Loss: 0.0582, Error: 1.56%\n",
      "Minibatch: 8000, Loss: 0.0266, Error: 0.00%\n",
      "Minibatch: 8500, Loss: 0.0507, Error: 1.56%\n",
      "Minibatch: 9000, Loss: 0.0139, Error: 0.00%\n",
      "Training took 113.1 sec\n",
      "Average test error: 1.47%\n"
     ]
    }
   ],
   "source": [
    "do_train_test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output Shape of the first convolution layer: (8, 28, 28)\n",
      "Output Shape of the first pooling layer: (8, 12, 12)\n",
      "Output Shape of the second convolution layer: (16, 12, 12)\n",
      "Output Shape of the second pooling layer: (16, 4, 4)\n",
      "Training 5994 parameters in 6 parameter tensors.\n"
     ]
    }
   ],
   "source": [
    "# Print the output shapes / parameters of different components\n",
    "print(\"Output Shape of the first convolution layer:\", z.first_conv.shape)\n",
    "print(\"Output Shape of the first pooling layer:\", z.first_pooling.shape)\n",
    "print(\"Output Shape of the second convolution layer:\", z.second_conv.shape)\n",
    "print(\"Output Shape of the second pooling layer:\", z.second_pooling.shape)\n",
    "#print(\"Bias value of the last dense layer:\", z.classify.b.value)\n",
    "C.logging.log_number_of_parameters(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Knowledge Check**: How many parameters do we have in this second model (as shown in the figure)? \n",
    "\n",
    "\n",
    "**Suggested Explorations**\n",
    "- Add average pooling layer after each of the two convolution layer. Use the parameters as shown in the figure. \n",
    "- Does use of LeakyRelu help improve the error rate?\n",
    "- What percentage of the parameter does the last dense layer contribute w.r.t. the overall number of parameters for (a) purely two convolutional layer and (b) alternating 2 convolutional and average pooling layers   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 28, 28)\n",
      "[[2.9713529e-10 4.1965654e-15 9.9579966e-01 1.1525006e-05 1.0036472e-19\n",
      "  3.5585375e-14 1.1761815e-13 1.2263200e-16 4.1888221e-03 7.6530240e-09]]\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "# 預測\n",
    "\n",
    "# 請讀入BMP檔案進行PREDICT HW\n",
    "\n",
    "#!dir\n",
    "# https://www.datasciencecapstone.org/competitions/13/identifying-malaria-in-blood/page/41/\n",
    "# channel = RGB + A\n",
    "# exam =  plt.imread('MysteryNumberD.bmp')\n",
    "# exam.shape\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from skimage import io\n",
    "exam = io.imread('MysteryNumberD.bmp', as_gray=True)\n",
    "exam = exam.astype(np.float32).reshape(1, 28, 28)/255.0\n",
    "\n",
    "# plt.imshow(exam, cmap='gray')\n",
    "# plt.show()\n",
    "\n",
    "# predict\n",
    "print(exam.shape)\n",
    "print(out.eval(exam))\n",
    "print(np.argmax(out.eval(exam)))\n",
    "\n",
    "# ValueError: only integer, bool, float32, float64 and float16 are supported, you gave uint8\n",
    "# ValueError: The trailing dimensions of the Value shape '[28 x 28]' do not match the Variable 'Input('Input3', [#], [784])' shape '[784]'.\n",
    "# ValueError: Value's rank (2) should be >= the Variable's rank (3); Variable = 'Input('Input5', [#], [1 x 28 x 28])', Value shape = '[1 x 784]'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}