Nueral Network/Image_Clasification-SVHN_NN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center><u> Project on Image Clasification SVHN - Neural Network</u></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Description:\n",
    "SVHN is a real-world image dataset for developing machine learning and object recognition algorithms with minimal requirement on data formatting but comes from a significantly harder, unsolved, real world problem (recognizing digits and numbers in natural scene images). SVHN is obtained from house numbers in Google Street View images.\n",
    "### Domain: \n",
    "Image classification\n",
    "### Objective:\n",
    "The objective of the project is to learn how to implement a simple image classification pipeline based on a deep neural network.\n",
    "### Analysis By:\n",
    "Soumalya Biswas (Group-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ============================================================"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import h5py\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Activation, Dense, Dropout, BatchNormalization\n",
    "from tensorflow.keras import optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a Fully Connected Layer (Linear Layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Linear():\n",
    "    def __init__(self, in_size, out_size):\n",
    "        self.W = np.random.randn(in_size, out_size) * 0.01\n",
    "        self.b = np.zeros((1, out_size))\n",
    "        self.params = [self.W, self.b]\n",
    "        self.gradW = None\n",
    "        self.gradB = None\n",
    "        self.gradInput = None\n",
    "\n",
    "    def forward(self, X):\n",
    "        self.X = X\n",
    "        output = np.dot(self.X, self.W) + self.b\n",
    "        return output\n",
    "\n",
    "    def backward(self, nextgrad):\n",
    "        self.gradW = np.dot(self.X.T, nextgrad)\n",
    "        self.gradB = np.sum(nextgrad, axis=0)\n",
    "        self.gradInput = np.dot(nextgrad, self.W.T)\n",
    "        return self.gradInput, [self.gradW, self.gradB]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Defining the Rectified Linear Activation Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class ReLU():\n",
    "    def __init__(self):\n",
    "        self.params = []\n",
    "        self.gradInput = None\n",
    "\n",
    "    def forward(self, X):\n",
    "        self.output = np.maximum(X, 0)\n",
    "        return self.output\n",
    "\n",
    "    def backward(self, nextgrad):\n",
    "        self.gradInput = nextgrad.copy()\n",
    "        self.gradInput[self.output <=0] = 0\n",
    "        return self.gradInput, []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Defining the softmax function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))\n",
    "    return exp_x / np.sum(exp_x, axis=1, keepdims=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Defining the Cross Entropy Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class CrossEntropy:\n",
    "    def forward(self, X, y):\n",
    "        self.m = y.shape[0]\n",
    "        self.p = softmax(X)\n",
    "        cross_entropy = -np.log(self.p[range(self.m), y]+1e-16)\n",
    "        loss = np.sum(cross_entropy) / self.m\n",
    "        return loss\n",
    "    \n",
    "    def backward(self, X, y):\n",
    "        y_idx = y.argmax()        \n",
    "        grad = softmax(X)\n",
    "        grad[range(self.m), y] -= 1\n",
    "        grad /= self.m\n",
    "        return grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.  Load the X_train, X_test, Y_train, Y_test, X_val and Y_val datasets from the h5py file :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of datasets in this file: \n",
      " ['X_test', 'X_train', 'X_val', 'y_test', 'y_train', 'y_val']\n"
     ]
    }
   ],
   "source": [
    "with h5py.File('SVHN_single_grey.h5', 'r') as hdf:\n",
    "    ls = list(hdf.keys())\n",
    "    print('List of datasets in this file: \\n', ls)\n",
    "    X_test = hdf.get('X_test')\n",
    "    X_train = hdf.get('X_train')\n",
    "    X_val = hdf.get('X_val')\n",
    "    y_test = hdf.get('y_test')\n",
    "    y_train = hdf.get('y_train')\n",
    "    y_val = hdf.get('y_val')\n",
    "    \n",
    "    X_test = np.array(X_test)\n",
    "    X_train = np.array(X_train)\n",
    "    X_val = np.array(X_val)\n",
    "    y_test = np.array(y_test)\n",
    "    y_train = np.array(y_train)\n",
    "    y_val = np.array(y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Comment: </b> Abov we have loaded the h5 file and can see there are 6 datasets are there those are 'X_test', 'X_train', 'X_val', 'y_test', 'y_train', 'y_val'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of X_test (18000, 32, 32)\n",
      "Shape of X_train (42000, 32, 32)\n",
      "Shape of X_val (60000, 32, 32)\n",
      "Shape of y_test (18000,)\n",
      "Shape of y_train (42000,)\n",
      "Shape of y_val (60000,)\n"
     ]
    }
   ],
   "source": [
    "print('Shape of X_test', X_test.shape)\n",
    "print('Shape of X_train', X_train.shape)\n",
    "print('Shape of X_val', X_val.shape)\n",
    "print('Shape of y_test', y_test.shape)\n",
    "print('Shape of y_train', y_train.shape)\n",
    "print('Shape of y_val', y_val.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Flatten the images for Keras :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = X_train.reshape(42000, 1024)\n",
    "X_test = X_test.reshape(18000, 1024)\n",
    "X_val = X_val.reshape(60000, 1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Comment : </b> Here we are flatting our train, test and val of 32*32  in to one long vector of 1024."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Normalize the inputs for X_train, X_test and X_val :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = X_train / 255.0\n",
    "X_test  = X_test  / 255.0\n",
    "X_val   = X_val   / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Comment: </b> Here normalizing images so that the pixel values will be in range [0,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Convert the class matrices Y_train, Y_test and Y_val into one hot vectors :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 6, 7, ..., 7, 0, 4], dtype=uint8)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.view()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train_one = tf.keras.utils.to_categorical(y_train, num_classes=10)\n",
    "y_test_one = tf.keras.utils.to_categorical(y_test, num_classes=10)\n",
    "y_val_one = tf.keras.utils.to_categorical(y_val, num_classes=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Print the train, test and val shapes :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of X_test (18000, 1024)\n",
      "Shape of X_train (42000, 1024)\n",
      "Shape of X_val (60000, 1024)\n",
      "Shape of y_test (18000,)\n",
      "Shape of y_train (42000,)\n",
      "Shape of y_val (60000,)\n"
     ]
    }
   ],
   "source": [
    "print('Shape of X_test', X_test.shape)\n",
    "print('Shape of X_train', X_train.shape)\n",
    "print('Shape of X_val', X_val.shape)\n",
    "print('Shape of y_test', y_test.shape)\n",
    "print('Shape of y_train', y_train.shape)\n",
    "print('Shape of y_val', y_val.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<b>Comment : </b> There are 42000 train data, 18000 test data and 60000 validation data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Visualize the first 10 images in X_train and the corresponding Y_train labels :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1MAAABVCAYAAACsCb4cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2d1899VfXuHyw/exfFCgqCFAEBRRCl\nCQLBiiZGrsQbNd5456X+GSbGRC+MJbZERTTyVUGkKKAUsQAK9oK9F87F73z2fPazx1q/d6/9ek5O\nznhu3rLXXmvOMcecc83xjHLQ/fffr0aj0Wg0Go1Go9FobIcH/N9uQKPRaDQajUaj0Wj8v4g+TDUa\njUaj0Wg0Go3GAvRhqtFoNBqNRqPRaDQWoA9TjUaj0Wg0Go1Go7EAfZhqNBqNRqPRaDQajQXow1Sj\n0Wg0Go1Go9FoLMCD5j48+uij75ekv//975Kkv/3tb6vP/v3vf0uSHvzgB0uSHv7wh0uSHvCAcT7j\ne3/961/Xfv7jH//YuPZBD/rvpjzmMY+RJD360Y9eu+a3v/3t6trf/e53kqSHPvShkqQ///nPa/eX\npIc85CFr9z3ooIPW2nvIIYesrj3ssMMkSc94xjMkSY973OPW2ilJ9957ryTprrvukiR9+ctfPkgL\ncODAgbVc9LRHkkhT/89//nOt7Q984ANX1yAPHwv//9Oe9rTV/575zGeu3Yf7M3b/+te/VtciH+7L\ntS4D2upyloasfTwTPAv5+bPQneOPP36RTN/3vvfdL0n/9V//JUn6y1/+svoMvUFHaEcl0wRy88/5\nnWcB7ovOS0N26GvKVtocC75P+5CNJD3ykY+UJD32sY9d+/thD3vY6hrGkftedtlli2QqSR/4wAfu\nl6Rf/vKXksY4S9KjHvWotZ+00/WZPtAm+v3HP/5RkvSLX/xidS3z61e/+pWkMQfo4+Mf//jVtU94\nwhPWnk3/uVYaawNyQPZ/+MMf1u4vjXEGfMfBuKP7b3zjGxfJFZkiE9dDwGfV/Ocz2o+OoT+uW3lN\n9svnP+Ba5OX6zO+M0W233SZJ+tnPfrb2PL+Wtj/1qU+VJD3rWc9aXfOkJz1J0tDnd7/73Ytkevfd\nd98vSffdd58k6fLLL1999sUvflGSdOedd0oa8/aCCy5YXfOKV7xCknTUUUdJGrqDLH0OgpQ3Ougy\n5TN0B/n72sG6hLx97fLPJenHP/6xJOkHP/iBJOn73//+Wt/8GtacO+64Y5FM3/nOd97vbfaxZQ7Q\nNuYzPyXpT3/609o16AM/+dx/5xm59rm80H/ug7xct1mDWA/4Wc0nnsm4MVbVuwm44YYbFsn0vvvu\nW5Opg+fleuD7L0AGieq+IEvgeP+m7lfdm5+0c+6Zc21g/BjbJz/5yYtkesopp9wvjTFn7nobmVu/\n+c1v1n5K4/0AGfBeePjhh0sa74TSeC+86aab1tpePZvf893CZY3usicxZ3MNkMb6wp7HM3/9619v\nXMv8/PrXv75Ipscdd9z90piXj3jEIyavzfcNh78LSJtzzL+HDLhfvl9JQ6bMx9RJv3e+ezB2Pr8Y\n63POOUeSdMIJJ6zdX9pc+w877LBJmTYz1Wg0Go1Go9FoNBoLMMtMcYLndOenNE51nNxhktySieX5\n7rvvljQsBJw6sTBL0lOe8hRJ0nOe8xxJ0rHHHitpnMQ5tUvSPffcI0n69re/LUm64447JNWW1rQA\nw9y86EUvWl1z2mmnSZKe+9znrvXTLSn05fbbb994xjbgnrTL28ypGgaAE7dbM/h+3idZrblnA782\nrVNcW1nFsZJke93ikFZx7uf9xYIy1+a9gGfQVreQ/vSnP5U0LFBc69YHfk/racUkJcMJGCu3KiUb\nVukn4DNkgSyddYK1Zd488YlPXPspDd317y0FfUr5SkM2PCfZVGna4llZ8uhbPhOrsq8VyQKkrlXP\n5trKOp3jwmf+f3Qix31bJOvhVjX0hf8xr9xKlywzbazYJ+SS1uRKh/l+eg+4THN+AHR1jpWp1ifu\nXVlitwH94n4wqf47bWMd/853vrO65qyzzpK0aSVFz+bYw9RX7x99Rl6VFTctsTyL78L6SdJXv/pV\nSdK1114raVilYeSkIctdZUq7KqZmismvGPdktlg3XFfoc7L+yN/1NOc19632kPSMmWILHNWey//m\nvrcX0Eb6U7HI/EQG/sxkWugf/arWtJyb/O3M4MEHHyxp9BN209uXaxPrMeuz34+5wDWMo3sXVV4k\nSwBrArvt+wRypu/MeX9mso7Ikvs5MwW7jryQ5ZwHQa7zLlPWGdakH/7wh5I29db7wjNgWmin99Pl\nvAQnnniiJOlHP/qRpPU9gLWGtlWeUvSdPR35oNu+f6ReoSu5Z/n/0iur2qNyv6HdvvYi/+c973lr\nfXEPl+znHJqZajQajUaj0Wg0Go0FmGWmMk6A2CJJOvPMMyVJz3/+8yWN07qf+q+//npJg1XiBI4l\n/eSTT15de/rpp5f3qyw0nCivuuoqSdKVV14pSbr55ptX1/AsTqac4Dl14ycvDV95wMneY0SOPvpo\nSesxSUuQFgq3gGF9SKuGg8+S2eK07tYMrEVYW+hP9eyMcags8HxGzAQMITJ2Rogx4loYouoa5H3G\nGWdsPHMvwJqB7rll+ic/+YmkYWVJq4YjrRlc65aPOd96b4P/Dmtb+QGn/y/PyrgZvx/tZIychapi\nl5aCvla+yWllTybC/5fWdtoG6yxtWmR5Nn3DEufPRpf4rltH0woMuL/PLeSacQCVVbhiKbZBxs9U\nTE1a5Xwd4vdcByrmJ8co9cX7koxGsn7+e8qn8pfPdYRrq9ibXS3+tLWKQ8u4pypmMseCn6ybVXxd\nyj/jiPxZyJTv+jXoIz8ZX9ZjX8vwwOBnNecy5m0paGvK1n/P8a/WSWQ3pV/Spp7DNvDT9Z9nIe9K\nphlPxd9YyqsYLFB5UORY7wra5e3I/YR++X6ea1D+9L0iP2McKi8j1lGs9+zVrlfJ8nBNjpW06RWR\na7nLIJmhbcEewnsZ7JE/j/Gv4s9ynrDPEM/59Kc/ffUZ8Z7Eoud7VLW2JcPvegUDiGx5D0r2x+/N\nXOB91r1S6AtjsxTEk6IjxDNLgxUnXhPd8blBH3//+99LGn1nPeV9SBqx0Iwj/eMevCNKw1MOOXE/\n93BBD6e8VpwV431hzkOqWg+m0MxUo9FoNBqNRqPRaCxAH6YajUaj0Wg0Go1GYwFmOdZMT06SCGkk\na4D6hEJ3t4QMOIPS5Tvu1nXqqadKGtQs6V6h75/97Gevrn3yk58sSbrwwgslDTrSadEMwiPt4Xnn\nnSdpuBNKw3Xga1/7miTpwIEDktYp1PPPP1/SSJm5FJm60enDdEGB/nVqEre5733ve5JGUCXByO4S\ngczo6wte8AJJQ/5Ou08Firs7FLTt+9//fknSxz/+8bW+VTR3/qySOezqPgHljZzc7Yg242qKPlWB\n4sggXcNc/kkFV+nTQSZa4Nq55BfpalS5GM25DmSCkl2Q96/SvlcB2yATHmSwvbeR311/pSE7DwrF\nRSTdKyp3J8BYZAKH/J5/Vrn07YXu3wsqV95Mk89nri8ZRM+1zC9vX8oAuWeqaP+d+ZFj589Md7pc\nO/x/PLNy56lKNCxBlufwNPo8H3cU9ilcdfx76DcuJ/TTXXfTdYjPWGc8YcTPf/5zSWPNpp+uy+xl\nRxxxhKThSs/YeXIJnoE7TyZ0kjZTtS9FuvJtu0ani3Am9fE209Ysi8LfVekH7odM+CkN1yDGnGsr\n19UqMcB/ClXSGZCu5fTZ5w0yQwa5Bvt9cx/PNdL3OFzhcTvjvu66x5ziPrxvsK+66xbjxrWsKd4X\nwjjc7X8J2PuZz/5+xlqZrqXuEsrcTJfjqgQK8thLIhPmeOqVj1EmjOD+jI2PETLFnZF3O38vrsJT\ndgFj5Otphh+QBI61Ttp8R2Kf4H2ad3FpjBfrIEAH3cXwxhtvlDRKcqT7oLcrE06BqrxTumi73vP9\nvYRNNDPVaDQajUaj0Wg0Ggswy0xxGoOR8lMwp1WsepzqCI6VpG984xuSRspHTnwkczjuuONW12KJ\nufXWWyVJH/zgByUNC8hrX/va1bUXX3yxpHFKP+mkkyStp7ulHZykSX94zDHH/HfH7cSKdYLvk4LW\ngxk5QXvRySVIq3xlzc0Uv5zIpZF0A+auCv4FFK/EqvHCF75QknTZZZdJGimBpU3r7FzbsQK6vP1z\naVgGMoFIFRi9TeG/CuhpleABKwnWhywQ6d/PAm18x62ymQZ1qoivX4PlpAqQT/YjGcGKdUrGt2Ij\n5sZxr8DqyBx0+RCoy//46WM5VaSbaz1QljmI9Yr1ACuzB6JmkD2WOE9o4ZZSaVgnadNcEPoca7wr\nUtf92Rksi5x8rcoEFJncw+dXshRYS6uEKDwDSzHzxPudLCR6nYlzpOkELRUbuSszTb/Qefco4DPW\n0irlseuNtJl635mpDK7mvlhmv/vd766u5fdMMezMFHvYS17yEkmbMvWxp1/sS1iiq4Qbqf/bYirB\njLTJnFasTq5f+R23nmfJB94tsji5fw9ZoqdeiJUxYU3xsirS+tqY62+li1Xq6yVAD6qEDKxv9Ifx\n84RXyYhMpT2Xpgt0V3stzBRyo+yNp92GpeA+tLMqgp7vOJ4YCFTlLJaAuYvOuLx43+T5sMa+j2W6\neeY1/fKiuPzOvoVeMI6u0+gsY1YlUsuiv1OlcaQxFowNTDYeYtJm0rWl+PCHP7z2DNYmaXiQsRYh\nN54tbc4h3hVI/nbKKaesPmO80HfW2jwrSCMZCNfCirl+5VxPufscTo+KTP7i13dq9Eaj0Wg0Go1G\no9H4D2GWmeKEDJsDAySNUyInb6w9HqsEI4W1k4K8MFKekjytLVgGYKqOPPLI1bXEWtEGrIwU+pWk\n6667TtKwfOCLDsvmp0+s1fink/LbLQN8tmvKWRi8Kl4IawYWtS984QuSRiyXNKwinMYzta5bCTlV\nI8vPf/7zkqS77rpLknTppZeurn39618vafiuVvEatBULYsYquaUt/cPRJSwr0rBY7MpMzaXwznTO\nGZskbfqRZxFit2RhFeGatFpWcSjJHlTW/rTeVUxIxsdUKV8zBe4uYC7nGEq1NS6ROpQptSuWIi1B\nVYHLTNc8VUx27j4+TlMFsP8TSH2ci5kC7q/t/tzSZjHhqnBkFnLGmlsxycmIuJzSl5/7ogOV5X4q\n7bh/tl/ypl2ezpj1Ht971iwv84FVOxnuKt05ewOsE7GrMFPsedJgIpJB8jnDNYwVewB7o3tHEPvK\nnosV3a2mWfz6P4kcvypmJ+OU+Om6gh4xNoxHWvelzRjViuVhjmTKY77rep+sasWoZjr8pchiuC6D\nZIQzXt3bjVzyHcLbN8W4ocv+ngYzxT6CLF3uKcNM3e7MIO8QGW9SxV/vmhodOSETZ9P8d7/W11D6\nyP/QYVgPZ1yI1WH95L2Kfvl9ec/BGyjnrrSZkj5j12ADpfH+y9oGo+NzjrVu16K911xzjaSR/tzn\ny6tf/WpJw0sN9sq9lDxu1PsBy8QaJW2unzBT9BM2SxqxVrSHvrvnVnqrJCNVxT5lfJWP0VycY6KZ\nqUaj0Wg0Go1Go9FYgD0xUzBK/JTG6TJPdX6KzeKHnDY5QfspEcsJxdI4ycPSeNYXfHs5xWKB8Hgm\nYpywxHA6zpiDqbb7/f2zXX18seQgP5cBfaTYMVkFPVMK7ccyAbuDtcRZFKw/WGWxev3gBz+QJL3n\nPe9ZXYtl6W1ve5uk4RPrJ3J+p2AzcqdNfqLPWCI+qyyIWGKWIoutVpn6poq4edsYC3QQCylMnDQs\nxfQ9iyO69S3ZQqwmzkJh3cKaw7UZ4+PIjIQ+Rttkn/mfwNyrxg5LYMaZef8zm2H6clftzjgzfjpb\nk9bVfI60mRUrs9O5RXSu+Cyo4qiWgHZl0W3/XzJAFSOYWSD5f8Wcp2yrAsTpE85YVf1N3/Iqu1XG\n7lWxfPsVN5EydfYbK3mycp49airzFu3yvQdLKrGrFIrHuuwsVsYdIS+fB6x9WFeT8XUGDY8M/sca\nUbF9WJWXIlmnKnYzPQEqxj3XPn66rPdSrBckK59xmdJm1lDGl799zcpMqxX7tF8MKjpXeZGwVxOX\nw9++9/N99p7M7jo3RsiSNZOYa2noEXs+7ybOtGTmND5jzvuexpjABFWM/xybvQ3QJ/pQFY7OZ/o6\nlRlhU78q4B1EfA4y8ax3yCdj+6psfumxMRejl/HSvs9PxaluC++HNNh4aegjaxBZFJ0F5F2S/vGZ\nx7OBb37zm5IGE8j6yXdcZ84++2xJI/8BTN7dd9+9uiazhea7ve+PjFvGey/dj5qZajQajUaj0Wg0\nGo0F6MNUo9FoNBqNRqPRaCzArJsfVCU0mLsTQKtCn2VAvjQoSO4DfQyNWLm44FJFUgkoek+NnJQu\nVKDTk1DW0NpJ+7kLH/Qez6oo6MotZwkyeNhdk6BQcfOD8ncqlzZSRBjXD1wnnW7HHfLqq6+WNChV\nXED8Wp6VLk9OeSKzKrW9f+73SbhMoXS/9KUvldfuFcikKnSYKaSzqJv/Dz1H9wge9UKAuItOpYn2\n+6a7FvPHA0RJBgL9jEtmugj599Md1WWd6dN3ATqRyWakIXPcphhLd4+gnenykMVHHYwBawVycFcr\nvg/Nn8UUpSGHTMyQqZr9Gek6N1cEeCm4dzW+maAgU+ZKm65KyCXTu/rvOQfm0r1mMo/KZTZdoZC/\n62XKi++6fmSw8FJkgLu7ceGyi9tI7lfS0JtMTpJB9tIIQsctBfdc1lKXTRb/zSQ4fm9ch1ijcVn3\npEq4GONuz/1IYiGNhE2s/Usxl0o73foqd+Op5BRV4dx0AcwC1O7qg16x11Tu0Fm8HT1nf8A1Xtos\nlAqqd51d1wCSCGSSLWm8r6CXjKnv0Yx/umFVY5VJfpAtLv5ewoY5gs5l2QtpyDTdsDMZgLcnXYUr\ntzX2DU9/vw3YS3GzdZkipww58Hak7jLWXOOJNXJPZe1FFl5iIZNXoV8+99MFOlOj+36Wiaqq5DNV\n+u8lyPno6caRN21jXfXxZ19G/rQHt08vXcH6hr4zjiT+8FITvG/iWkiSHk9owffy/bVy0WU9yERq\nPucYv0z8VKGZqUaj0Wg0Go1Go9FYgFlmipM4J00PasWKQVAZ1je33vA71gKsEFUqSU6vnLj5m+9U\n1ssshuqWbp6NxYrPqmDSDPJPC7V/f9fA/rTq+ikYaxHWI/rsFl+Kpp1zzjmShvwrxgwLACnQYbEo\n5uuWlje/+c1r38kkB9JmGtO0QLmVhPGbKkQqDb0iBfw73vGOjT7sBWl1qKwQaZ2aC4BGBjBSJEOR\nhhWE5AzIgH75OFRF4LwN0pAB30NOFbNUpavOaysr8VJUyS9AshqZsljaZEuycLPfn3mVBWa5h1vH\nCNoniJrxcstgFrKcS9Gdc7IKQt81qBdkQdiqlEEm1PC1LwPlcw76mpXB4uhoJlzx++bPipnKdbda\nU3NNqNbPubV9G6Se+vyiz2n5Zp/xttGvtOz6tTBRMD+ZUt6fzVinVdn3yJw3yIRrvW9YUGEOuI8n\n8MEy68zDEmRBXteDZHerkhRTwfUZ8C8NNoA5nqyyg/vOJaBgPLOoebIq0mb6dZ7tct+vNZXnY31n\nD5GGRw76wzuTX0M6arxQUk+r+cw1eBmgH57CH0s/DFUl9yxDkazTXIrzqrD9fpXwQD4wu8hWWk8y\n46jeUauEDtL6fp56gI5U6zTPZu1FPs6c5Xsx9813J2msQfSX9acq27IrM5UscjVfcj/05FR8Rvth\n92i7J6Lgd+SUCUV8389EPlVh83z3m5NFJqeoEpRkn+bQzFSj0Wg0Go1Go9FoLMAsM8XpFz9uPwGm\nj2/lT8xpDt9SCnvdcMMNktZ9HTlV80x8KDmRe3rq6kQqrTNTfA+rHfFItNNZsfSNrwqwzlnLtgGn\na07DWIyk4R9K22mry+nkk0+WNCwByBYrgDNdyZqcdtppkqTTTz9d0rq8sHalD7qf9LnPVIHZKtU9\noL9uOf/qV78qaffU6MigShWcFtJMPy1tpuHOooVVkceMI+SaKj0t8uJat7bQ9qk+zBVizD469qNo\nb8bWuHVmKnV25cOdsSLVOGU6ZOSMHrqFFqsqKVGx7DkzRZxXxq2g81XMFKji6vYLGQfl1u6M3aqY\nyCl2uLKaZ8FidDRj+eae6TqXbF/KrdL9ZFx9zanispYgPQm8HfQ59yL3xceKTExKspceN4FlHx2m\nn+itW8PRQfSTFMO+5mcsH3qOhd2fTQwCsmRPI35LGnur928Jport+u9TKdL9s/RCYTxcd1gPeWam\nnHbre8YzZ+kDR3ql5N/+zCxt4evYrimTQc4Bj51lnUJO6PLtt9++uoYYXu6TMUDVPoX+sFbecsst\nkgbLJY1Yrvyu7/257iBL5pOXpWFs0WXeFX3vn4q/3BboCu+NHltbxeQmptbPynOAPsMo0Z+qGDD/\nQz9ZL3zus5agB4xrFU+bz+RZvLc5dmWmstyGFw7nPRxZVCnfU5d5v6OchI8L40bf+W56lkmbe1PG\nq0ubMdLJZrlscn2hvxmTLu1tj2pmqtFoNBqNRqPRaDQWYJaZ4hTNaZ8sZ9KmbyInefcbzcKmWFk+\n+9nPbjyLkzxWZ1gLLHReMDhjsTj1E38iDasI2dIoigizQyYQaVj8OOVzEnc/UE68u1qn0n/YT960\nP63+nn0IOYG0pLl1PgvP8czMcuPXYkWln269zqwzWBiqgrVp1eczLxZ4zTXXSFpnapYgx8StD2ml\noR1ufUiLfWZAcmtXZmLKaysmKeMr3DqFfiMD7ltZfDJzUsUOzcVcbYvMqOfZGmlfsneuL7Qhi+zC\nvLq1E51Ef+kTz3ErLtZW5Mi1bhlMn/UsMllZnv9PFO3NddP1JbMxVcxUFh+eylbpv+d98x57ffYU\nI5VxMf77VEZB/2xXmWZ7fN2jbcxP5hsMjjR0JbOYVQxCzvv0WPDMn8RaepyEtB6DBZABz8q91/vC\nfKKA8HXXXbe65lvf+tbG9/YDrrcZR5UMVf4ujfcCfvoYJSuPvFgLKu+InM9eLJk1MNf+qrhwxmDl\nmuXt8f8tQWZz9X0rC4dyresOa2EyB5WnDtegRzCVyNKLrBIHs5fCxcnYVEWdpzLn+p6EHuwqU2TJ\n/uleM7QtWcxq7c84QGTg72esB4wJex57IDFn0mBwGFfG2rNswlTzP/pSZZxlz2OsYaUrlnXX9ZRn\n0J9DDz109Rnvn7SVPvi7AdfQd3TwwIEDkkbGYmmMETFvyJ17+PssOsOYoDuVl0kVm5ngfuh/Zl50\nzMUEgmamGo1Go9FoNBqNRmMB9pTNr/Lvz4wjVUxRxj9wgsR65hlleAYWFbIm8R3PxpSnRKwRXm+D\nky4MFfFaN910k6T1GCzujd/vmWeeufFMaivtJd/8HNKy49lnOO1nNhT3w6fdyAmrBqd1t/hhQcEK\nNVUfRtq04PJ3xUxlnFBlYaYdae2CIZSG/3ZabrdFxjy5j30yJ8jJrVOZiTLj7VwGmR0pa9P4tbQj\nrZ5unULP0VMsbOk/7d9PJqWyGu9HNj/azz2rmmO0qfLTTks6feNvl1Wy2FzDT7d8ZX0OZOdMMiw6\n2YKyPpMzk8y7ZJ8qC22VKXIb0I+KIQM8d65uyBTDNcdMzT0z75NtcOT/qvtOPcvbvV8ZEnOfqmoY\nwX6kj74kHXPMMZJGHRPWfXTP16cqc6k09Omwww5b/Q9mKtcTj2eq1ll/ZsXK0Ceswp5ll71wV9nm\n2l5lv8yMetWak14LmT0tf/drqtjFzFDJZ74+5NqZNfF83+P77MMVE47OZKzwtuBZVU0+xnQqY5x/\nRv8ydsrHnO/Rdt57qjgYWICsTeWYyuxatTOzJlbxR5W3wxIgk7lY2IwR9bV/ipGvZJDZ7dAnZElW\nRP8fuoY+uZxYD3hXZY3ivblienNfrGIZd/WeYg3Di8tj6pABrDhZQ927qKop6G329/T0iGLMeOc9\n4ogjVtcyfnw/cww4ch5VbDW/Z8ZFR5X5dgrNTDUajUaj0Wg0Go3GAvRhqtFoNBqNRqPRaDQWYNbN\nL9MJOo2cdOicWxEUGRQe3/GU2FPFKEmrecghh6yuJTCedkGhutsg1Cn34VlXXnmlpPVEDqQcPeGE\nEySNIGKnSwnG25VCTfrRAzBpa6Zo9yA83BFIVw/dCtXptDn3RrYE+R977LGSpJNOOml1LZTuwQcf\nLKkONuc+6WZQuSHxfWhSChF/6UtfWl0DNbyrSwr6QHs8uJbP0uWuKjCJ3JExyUuqAPR0Q8106v5s\n3Ee51t380r2P9mXwrLTpcsO1VRDqfgC6P4OjpSEH+k3yFnehycQTyIz24zorjfmIKw36jHuup39O\nva5A+0h4gh7jiuEB+lOFGl321XgsAfMzU7hKm2tglbQh19scb5dJFiOcShvr90nX0crVJIsLV8kq\nch2v3Lv2qxhqptH3ZyDvdN9ydxtSFKerFX/7Gp3JO7iWee8FKZk/zHFctKukHjlWlasp/aQPJCxi\nr/N+7Zpymv4wbu72xNyZ0xWQrmNzQfLINgPDfa/O4PYseOptRs6sw+zhvj/QDtYb5OeuQwT/7+qO\nPlWcVdos0ow+VCVcMqFDJoPw+7Hmsv++6EUvkrT+PjW1N7pMaTvXpn5V7ma0t3JHZ27t6joJqiQ4\nmTa/WrunCqVXY5T7AnrE3ucJKHhn41rG1d8lMvlUvvdVCb1yPfX3vfxsKc4991xJ4z3R1zT06frr\nr5c09KpKQEJf0Rlk4m1GV5Al+v7c5z5X0nriOeRz6623ShrvBB4qk4nwkEVVgDhdPCu3+m1S+Dcz\n1Wg0Go1Go9FoNBoLsKcEFFUw3lxwNMgClVzLadHTqOdJkhMk1jwP7oNh4YQLS0NiC2lYUrDwYF0i\nLSPBu9IIsOM0jNXECyZi3do1AJ2Td7I7/nsGTMI+SSOBA6xJleY4wTVYMG+77TZJI1WlJF1wwQWS\npDe96U2ShjWiSp/MT9qbhdH8f4wDQdIEwvr3XQ+WgOdi+fAEFJn2EkuYy522ok8wRxWDSkpST+/p\nz/E09uipF132+0vDap0pYors+Y0AACAASURBVKtU42nlrSySWZx4F2CZQmZuAco0sVjlKusjyTqQ\nL3PAraNYEQkuzWB9ik5KQw58H4uqBxbTLuY5Y0Gxyrk0znPYVa60tWJ++V/F9IC0Gk8VpZY2A64z\nIYVbk/O+c+nipwK6Xf6ZpKCyHKMPbi1cgiwKXSVKwvKJfvm+BdOSDEsWMPd+cU0W5PYkKMiD/2Xh\nWkcmtkiLtl+TOlixRrsiUzq7TDOpT2X5n0pQBXxdQz6MEd4oJJHxdYJruG8Wrfd7I2+SAcAgODMF\nYKKq4ut8lgH124J1P9dBaciLZ/Hu4frEPpkMdlVUmLWadRT58D7le1IyxFWSlRy/LIsyV2qiYiGz\nAO5ScB9k6TqInPG2qcaWa6Y8BirGM0soUJzb31GZI+hMtYfCiDOe3K9K6pRJvmins7aZ5n8pzjjj\nDEmb/fT20/bqfSrXftqa7wzS0GkSTZx44omShjcPa4E05g/vvrBk1R6VzH5VfDxZpyrN/zZeU81M\nNRqNRqPRaDQajcYCzDJT6TdaWXPTwlr5JHIa5IRaFdJKax3XUjDMU3lykqcI8BVXXCFpPa6CUzD3\nob20rzrRw9jAcMEaSNILXvACSes+nEuQp/aK2aN/XMNJ3P+X96FfzvJwH6w3fMZ33UryyU9+UtKw\nXL3qVa+StO4vm+OWp3b/nGfjD0xhSY/vwZrh6YSXgP7NsYZVgVvA/7IINPJx6yAsCywf/cSK49dm\nIU7GymMcsLakRTmt/9ImI1hZjfPaXYCuk/7U25IWPeaK9wPrFawS7UTHqhS16Au+2DAJHsPA3EU3\nGSe34me6VCxoabWWxhhkevEqpmjXmKlExXTNMQ/JROXY+7in1TgtcP7dtE7PMXDJUFcsasZMVSnA\n9yN9v9+H9vgcRNeyAGyVpjlTffPTra4wyMkgZRyTNNaTtPT7GsR6mKm+sxh7BfrpLAP93bVoN0x7\nxfYxx7MkRZXqG2RsivcLyzcMAns+1mq3TqNrzG/kXsUKMubIBPac5/j3WB+qwp0ZF7cUrJHICabM\n2wGTfvPNN0ta7zvjnYWLaaunp2bdJGYUGbBmev+SycixkjbTnGfhdV9PU9+r2E/mUZXWehvkGleV\nZ8i1e5s1vCoJQNvxdqDvHtuepVjYY9zLJWObck2pSrJkuveqAPWu+NznPidpMFSeGh3PjvPOO0/S\n6Oe11167uob1gXUpY1B9308vFcpUZJy5NHIiIEO8hCrWPD3T+Om6PhV/VpXW2QuamWo0Go1Go9Fo\nNBqNBdhTNr+03Embli+u9VMn1jWuzWJkzqJk4VSsU2Scc79pTvlkhuNU7LEoWAloF/6sZ511lqSR\n1UYap08ylHzqU5+StJ4dEKsP1iS3GC1BFWdEmzlVVxnEMjMXxc0oOOkZZRgL2s5JHmsVPqd+zYc+\n9KG1dl1yySWra7LPc3EVWaQxM+tIY8yf97znbXx/GyAvLA1uFU3LambC8zZWPtXSunUCS1paWRgX\nt2pkbB8ydkYQ3/ipmKnKbzpZg6pQ4X4wU5lhxy2f9B/LG+339sLAwTJlIT7Xfe6HThJjxzPd6oeP\nOnMRveantOk/nkWGnb1IRrOKKdovpE9+FZewl6LMmd1pLhMRMkjWYy5mClR6lFk8K8tzZiSskEzZ\nUuRa6usUupJrl89/dBd2AjllLGbVZsYTFspj+9Bl5FLFYCFL1opcB6psYjyTvnlWVhiIXa3UtKcq\n1prrGX9XXgfJGlbeKRkzxR6Gx4Lv/YC9HmbKGY701mBNhtF2XWBto52wVsjR2+fvF0tAW+mfPwO9\ngZGCoaLws7cNiz97RHoH+Pd5hzn++OMljTg0f/fK+VzFNyJD9i76wn0yLtjvk9mcpaGfu2bzSw+O\nKlMmqNbz3FNz3lTrF3rFODJX/HnIFE8W9ia8NaTNd5OMbXXkfoEu+/pQyWAJyHhdZZREf3jfZA3w\nQrxk4E1ZZmyrNHQXvQesaeitNHIaMA95trN9mc004/GrWM3K824JmplqNBqNRqPRaDQajQWYPcJm\n/RU/uaVPd5WBKk+FWX/AT/JYtTgtUoOGE7Df9xvf+IYk6aqrrpI0LDJ+P07FWFTwxaSmFD6Z0mAH\nOKFiZfQselh4sMwsjfOhPTzDZZy1u6o6LFgLOKVfeOGFkqSjjjpKUp3Jhe9jYYAp+OAHP7i6hix7\nWMg+85nPSFr3lz3ttNMkbVpzMmOK94H+XXTRRZLW/cSJccPCuhRYt9Chyi82/aaJCfD2YyVJlsUt\nrvQLWWKVIr4Iy400rEeMK0wNzKA0/IvTmlQxTDwb6wuWGWcjaZf3bylyDlVxBFjesp6Df4ZVjnbS\nNtcXrsXCldkSPVMSsobpSuuy3w+ZJfvgMZNpwawsVPuRHVEafc+4SGkz4xA/vS4HY5H+/1VMVzJs\n3KfqS1ohM0udtzV9/bM+kN+H71RW4SrD0hIkM+UygO3gJ/2ssihOWYo9xgaWCgtzzlP0ThrsKmPO\nXPdnp1UUfYW1cBaFsccyDqPvbDjzZtdaM8irikdlLjHG/O3PTKt0MlVVhkR+ZhZEn6s5fuzHznAk\n05LeCK5vqfes557xD71y5mcJeM/gncYzv1599dWSBqPEmPq6l+vdVH1Eaegnz4Tdq+JVk4nKOEdp\nM/46Y8ycvcj1tGKu01NkKTJes3qfyhipudp+qaeu91zLOBB/hq74ewJyYu9j7/dMdulVkPE5vk6k\n1wL99u/wu3tdLAEZdBlTZx3JXUDfYU49OzZ9Rh9pexVbi7zZ95mjrJXO3jpLK20y49KIWc2MrJWn\nBLrLvoX8qnqJe0EzU41Go9FoNBqNRqOxAH2YajQajUaj0Wg0Go0FmHXzy2Dbqjhouu5VAXFTQbtV\nkCzuc6QixzXI3X6+8pWvSBrUYFK1fm9cLAh6J7Wj09xJt2Y7/foMEN4WUMFQi06PZ0rmytXr3HPP\nlTTcFaHvs+3eL6hTXFTOPPPMtb8l6V3vepekQdfi7ueFkE855ZSyTzy7cgPNoPBTTz11dQ0uf7sW\n7st2VK5OmRa+SinK2FRuDiALK9M/qHXcRPyZuFrQzyoAvZpjidSHqvhnujXuAvpYuXLQb/qEW4m7\nraX7FS6QzGmn59E3XKMI3sedwEsjpEslsve01Mg8g16RvcuH9mWhvyqF7q5uKekC4+OdKWRpa5Us\nJfWkcmNIF4VMH1zp2l7cGqYCnauiramzVZKKXRNQpPurzwvmIOsk+uTtYG3KMhqZ6MF/Z+/B3QlZ\nVPsU84ZA6cp1l32Fucba6OtJulfTF5+XuI7tGtiPax0ycBc+9lR+Mo99TuX6n+nmfcz5H649JJOo\nXE1xqUIGjEflYsX/cM+rkmRlkgrk565NPKtKhLENCF1g3EjiIA29IZyABFmHH3746hrki7zZv5GX\nu/mxfrLW4lqYoQzS6B9jzv1ZQ6VNl2HmDHLzsUe+6Zrm7064gu2692eB2yqdeK491TqVBVzzc2no\nCjrI/oMrmj+bMclSKj7mYKpIu8+RLPVTJSerZLAEyIt3P3c1RS95n+Yzwk6kETKCTjDuVTmgXB+Q\nE9d6OM2xxx4rabzDv/jFL5a0nhgriwrzbPaBKuldlnioCq/vBc1MNRqNRqPRaDQajcYCzDJTaYWt\nLIhp7axOnVOF1fz+BPXBSGGRwTqClU8aJ2YsWZwenUngJI+1Bcs2lhQvLsopONMozwUKL0UGe3oA\nISfiTNGNbCTp5JNPljQsl8kQuoUIq08W2uMnSTkk6aUvfakk6WMf+5ikcdr3gsFYDzK97VTKbmkz\nLXMVGL1rKs+0KPv9kG/qZ8XcJDMAKoaCMcJyjEWzSsCCZRSriVvoUj6pF5VlZCrduzTGaD/SemOB\no73OJLnFXBr64qwbepsp0bHkwUJJ0o033ihpFM7GsoRlyhOX8D9+plVf2gx2TSbNx2AqALtKS71r\nIoo5qyFzg2ewHlTM1FQq38qSCnPB/fi/r+dpnUN3q+QrqbPIz9uSAdwV81ol2FmCZBsr1hF9rdLy\no5/oDHqA3Fz3+P4tt9wiacxpnunsKHsMsmTsfO/JfQVLL6VBfP5noocsyyCNMc41bFvAeqCvroMw\nUsgH+fs4JrubSZVc/imDZKp83eG+WejXU6Oz3iIDPqss91nUk377WsJ6tWvRXt5paIcH7X/ta19b\na9vZZ58taT01dCbugfWoknqQ3Ak2LNdD3x+mCkU7g8r4I9NMOlN5g+SY+5rLmOzK9qWeVUxNrlNV\nwe5MVkK/KpaC/8GK0gefq8kWwkp7EhPeTTOJG7KuvKD2Ukx91/cpwFj7esX+mWymJ95iraWvWXbD\n53Ou2ciQ935/R4DNZI1k/0fHpZHIBfl7Ihlvg7SZUKkqD7KN90QzU41Go9FoNBqNRqOxALNH2DyN\n+YktT3EZ5yNtnpTT8uvAOki6V0681113nSTpwIEDq2uxWicjVVk4scSkFaCK70k/b29n+mEvBSdu\nnuFWskwnzE9PoYp1DAsKf1fsSZ76k2HwmCksC9wH64v7NNN2/LCTkfI28Iy0DjkzRdurwpDbgLby\nzCqdacZwzCGtEf4dnoXPOFZLT98LYGsoMocs3SqIPNL3ey5uay7V6xwDuC0YH/rqcsVKhAW3SkuP\nVQhfZ/SGa7z0AOUOKPiXKeiJM/D7YKlF9m5F5NmZ/pQxcQsh8yDZnqpYbjXPtkHGoFRxqDlvfR2a\nYoOrOCbGj/iPTAFbMfnJiDrrUcnF27QX6121P+yKLBDuazRywYpJrKnLiTIZ6EH23ceca/nJnK4K\n12bJDnSyYifRXWIPmDNV7M4cEwd2jZlkbmXBYWnMSeYd83nOcyXHxvce5J1p/3mm70H0PdvgFmja\nzn6FZR2WYC6uEP33dPhTBUC3Bfcm9fQVV1yx+oyY0bPOOkvSiE/21PhZfoJ5hyw8Hof1lOKnyHaq\nML3fj7nh+xRjgcWfa+hTJdNMN+5MC6yax8IuQTJSvqbQ/rk4qKm4z+p9FjnTZhgSdMXLItDX9Aaq\nSmHke3FVgoJ20IYqNTpryK56yljzLF+vsowE8PnM9egjf9NW1wPmc+YQYJ1wnWY9zf3fS8NkMePc\nz3y8c7+t9rFt3qOamWo0Go1Go9FoNBqNBZhlpji5ceL1U11amvJ05//jFJuZO9zqcuKJJ0oavuJY\n86+99lpJ61mS0mLIydcLfLkPtbSZ3chP73yffvLT/aaxDOzKTGV2MS+uiu9tWsGrzHOZJatiXGg/\nnyF/LANzcTVY89yKwPczBqCKfUgr/1zhPrd6LgHtmdPBOZY1fWazAGAVh4LFmJ8pa2noXBUrBdJX\ne+qntJnFr2J66cN+MFPJfro1ir7wvMpKRNYd5jTyhNUiI5o0LFDIj7kMY11lSUz2yjNfMR6MAdZt\n1hWfd1izpmKnpN0ZKZBW34ohT9/+OXYyM0G5pTILOKc108dzKsbMkZa8qbgo/z3jJ+Y8F5aCtleZ\nGtEVdNAz3wFYDayrWbzRx56YqRNOOEHS0Fuyp/lalmwtMnAGFRnAdFGYMrO0ev+y+KzLnfZQmH0p\ncj3zvRAZZsyUr1Wps7QZ/fL4EuYfP/HEqIqP5p6TzKy0Ge/COFSx2qxjrA/8dEt7xXosAfsAMd+w\nR9LIUHrJJZdIGvKuCiHn+wGy9XckxoT9iTWgyqScY1Vll0NmvFfBxiDjuQx5WVxY2r8Cs6B6D+L3\nKqbzf/p+FV/H73jxMEerOEXkxHxkHKu1LttVtTffE/jb9RSd2VWmmc3a28y8nSrc7ki2nfd9X5vw\nTsm9Dl3x8cyYdnTIZZDvDdkXv1++51Xve6BjphqNRqPRaDQajUbjP4Q+TDUajUaj0Wg0Go3GAsz6\nV0zRj9Im1VylksQNgGuhCPkO1LY03Py49tZbb5W0HqS+avT/ph0JAMQ1wguHkVKRlJQEnuOG4dRl\nFk6sUiNXaTWXgPskbSsN9wboWgJocU2SRvINrs007u66k24JfEYgn/cPF4QsQuvukHk/5IRrBAUH\npeE6gztAFSy9qyzzPpXrVLrwcY3LKduBDKCP/X5Qy0kNV4VQk5bmGneZyeQp6brjdHq6CFTP3M/U\n6Fm88eabb159dsMNN0gaLnuMM4Hz0giihnJHT5iT3h/Sm+LuRNpZ7ud6iKyQKy4suF5Jw2UIVxrc\nsPguLojSdHFbR1VAez/gupXp8asg6HQ/yXS9lTtWBuFmKQF/ZiZfqVyMp8oxVNdmuyv57SrTTKzh\nup+uL8xpn//pljQlY2kEn1NWgvWYZ/K3P5P+ZcFNabjFkhgDN0LmvQdgpysM1/gawTpOQoOlSHcw\nn3+ZPr9y2Up3PNx0ka2X+2DtYM5m6n53Xc25n8/xZ7Au5LrpfUm32yyYLW0mxlgK3JtYMz2pDgVI\nGX/k7m1PN33GCPc+kvdII6U671j0ea6wff7t606m6879z5Fuv1kwXRouX5m6elswH9E9X3d4boYB\n+JxibpM8Al2k7y7Tt771rZKkl7/85ZKGDnM/0tFLI9U9pT5wN/e1JF3R0xXW3dcA7SIMw9dwZLlr\nKAp6Vrkl81m6InoIA7/jWk2af/b2qrA5pSYyyYrrV7q5VvMyE41lGY+qoHuuY9Xc2Mse1cxUo9Fo\nNBqNRqPRaCzALDPF6Y6Tpp/osTKkxbEKQs5TIhYot2JTGJFnUCwWRskDh/kddgYrIVYdb/Ptt98u\nSfrpT38qaQQG+mkbq1SmaPXT6Fyih20wZX2WNtNtIwvaLg3rB0GQ2VZvM1aEDBakDzAE0kgGwLhy\nSielsDTGjT5gUfn0pz8tSfriF7+4uva8886TJL3mNa9Za0tVrHHXNL5YYrBiuOUxkzWASk+z6G8y\nqdLQG3QwZewWGqzEyDQTSPizuSZZsko2aV10q2wGp+8C9I9Uvq4v9JP+E+jsSWVgl2CQmdNY/9zq\n5EyRNOY2Qb6e2CKLZyJ7TzqD5S4tXFirK3YQzKV43pVNTcanCgSfS3mdDFmWWHDdyhTfuYb5nNjL\nupbBwZlgwS3PVRHx/DuTVOyKLMEhjb5nEg5PSEQ/+F9aKH3+o4ewoMiE+zJXqnZlQhdpeGScf/75\nkqQjjzxy45kgg6jRd19zYMZgYpcCuWX//PdkPL1fsNqZFhnrOVZqaTA0mRyE8fA5l3tGWp6lIe8s\npJ5JW7zt6EfOFZfBrvj6178uaaxNF1xwweozmEnWspS/tz8L78J0uWdIlqOYS5eNPDJpVLWvZOmU\nTAbk3+MnHgn+HsI+4SnolyA9MSpdyT54O/h+erCgr574CN3NMijsk55QiXe23Cd9jiSLws9q786x\nn2MEdwVzgZ+evp5x5zNk6sXK+R/saLLulD+SxtrL3OC7vAN7UjneMXK/qFKtJ2ufyXukTS+lKnnX\nnLwTzUw1Go1Go9FoNBqNxgLMMlNpja1STk8VaKuu4W+sEp7KmP9xOoR5ednLXiZp3RqQvo6cgP0E\nTcwVVpt77rlH0mAL3F+Ze2f6UD/xpqVtKZIlqoquZarNKvVtFv3E2uXWmClrC3248847V9fiG4wV\nCZk4M5VMCLL98pe/LEm66aabVtdiiTnuuOMkDUuZW4S5xq0/S0A/M2WvtJm6tfIZB1mAkH56Gm3a\nitWTZyFjtwjDxGQx4cpKmIVUMxbL2zXXh/0s2kusFG1zKxHzk3mL1cj1hfmIzNApdNjjS4gLTHYg\ni+/6Z8gB3XdMFUNG9m5pT8tzlcYf7Gr9SwaoKtqba+oc279L+7Zl2acs9BlD5e2YirPya3Zlplgn\ns+io/57MfcX6pfUSHaqKdmeMIOuAx+0meLYX4mUfYp1kjsyV4kCWzCO/hvlTzYltMJV6v/psLp0+\n/aBfrAnEU0hjLUG2yJ3YXrcmJ8uXMRZ+Dc9k/ak8DWgfn/EdXx/2a02lHxnrLQ3PHLfw+3e8rcib\nMU4GSBo6lmVtqmLwGTPIfuVMM+0gjicLulaxStwHnazi0PYLGefjz2PNpz1VW1OX2d8ZF2mMW+7V\n7JP8lMZYpMeJv+9NlTrI9zb/XnrAuEwrOS9BvoPDlkubc4l+em4DGHNYUfQLLzNHyp1rmIf+Lpfv\nAhUjyBgwV3KsKrY1PSx8feBZeynf0cxUo9FoNBqNRqPRaCzAnmKmMtuFNJ3Zxi066YfMqZAsKMRD\nSMOSwomUU22yDtJm3BEnaLc40HasEXfccYekkZEMy4M0TttYcbDAe/xHWmSWIjO0uBzpB3703/zm\nNyWtW25hPohfQcbpfyttxi1k0ULiyaTBMjF+WGGcaUirFs/KNkiDpfrIRz4iSXrDG94gaV2mlZVs\nCegXuuLsYRYu5TO35k5lA0TffcyxrKK76BGMp1sWGaNkR/zZ2feMi6gKIWc7XT/2q8CkNOYtFiHP\nvEQfaAMWHGevaENayhj3yqLOtRnnUBWjTT10qxP3mSrK6W1CN5L1qbJ57prRK+Nw/H7Z520KUaaV\nzX/PAqeVp0EyeXNZL5PhzmLe0mbW1CyeWPVlKZBXZe1OBoq2O9uMNTNjUbI4pLRptWffIhslDFP1\n7EpOmXEx12i/Nve0qsgk69GubH/qYpUxay9xxMn80F8fK/qYrFMVB5yMfRUHlaw27F8V35Pt5Ls+\ndtWasQQnnXSSpMFm+jsIusb/YJ1c9zKTHntOlXkz1+6M0XYZTLF9Pvf5nXekXAP82dyH9qKvfj8Y\nCN5nPLvjNkCf5mLdkmGuMlMyX3jvJN7X31F5F8r3lqrAbMqJ8azW08zomzHg0uZ8pE9kIZTGmgYz\n5OzvNkA+xIV6LgI+gxUia+E111yzuibZIa6lWLXH9pF1m/Ug93SPleZsQBvwNuPdVdosjow+VKxT\n7l/Vu0aux3Oeac1MNRqNRqPRaDQajcYCzDJTadl0TPkRVyc/fnI65LTplm5OgGkpyPgBf2bGZHmM\nU1pOqPFAfI9b0KmHQ7vOPPNMSesnciwVSy0oAMZn7oTLSRuriMeWcJKH+aGORPqdO5AT1pGrrrpK\n0pCFPxO5U/fCa4GlBRlZnHzyyZJGRha/3+c+9zlJwwJ17rnnrq5Bvs5mLgG6A7vhOpn1z9J3XBp6\njny4HzriVmx0BGsn16JfbinivpmBytuXlhhkkbU6/PuZcdEtKVlLaBegC7Bt/hy3qkpjLrtFHGsT\nbcHCW1mJYGwzIyOycn/rjLXBEufZRom3QA9hCZPNlgbbmHGIVRzGrpkn8/uVVT9jI1xOmXkoWSe3\nfGK5zixyFUOK7memPte/jH2d60uye1Wc31zWwm2ALqYVsgL9cSYDOWeGzrQUS5vZojIbnMf2plUz\n9yt/1pScqrFP5sbXfPR615gpMKef+VlVM62K/5i6tqoTKNXvFNkWZyQy/on4Nv5fPTt10Mc8We2l\nOPXUU9fa5c9Irwr0yd+9kkGiP7S9qtmU/as8F1Le6E6VETJjymi39yXj5yumBXZiLsvgXsA7Eh4v\n7vnDvZNxcO+YzCRJjBTZZWGcpfEOmJniqv2MZyfj4jLIPS8ZPX/v4NpcV9nnvA++Dy4BGZjPOuss\nSevsEP2gjh3vksSHS2ONZe9NrylqP0rjPTaz98KKebwW+p0ZBD3bdep7xpq5/NPjrmLaM2a1malG\no9FoNBqNRqPR2Gf0YarRaDQajUaj0Wg0FmDWzQ9aGbrRXeOSUobac/cfKNKjjjpK0khhCJXqLnPQ\noUkJVq5RSXNDvxPoKA0Kjzbj5gPF6OlpAcXESLVOu/1+nv5yCdKVxN1NoHehVQlUdbodmpVgPtzw\nkKUnS0BOpJ3Gre+jH/2opPWUkownLnunn366pHWqOQOzwUte8hJJI2GGNAr4MibXXnutpPWillC4\nuD6+6lWv0hKgI1XRTpCpsatEAplQA3eAKhEJz8LtZy4taQaaOo2c7dgmXfVcmuf9KNoLdY+bKW5j\n0hjXdKf1a0AW1WW++lzCjY++oMfMX5/b/ru3z++HqwrPxP2S9lZpWsGUG5O0e2KPOZe2vRTtTffP\ndEXz0gPpXpaurj6PM7FCuhM60tWkKhi8l2QaoEr/vQ2y/IG3oyrUKa33izUhE2IgE7825xX9qlLu\nZzHjKokJ44gMeCb38f0hA/m5T+VauF96WiW5oW0pd5+XmS4aHWFvqwLxMyCcZ1fpwfkfa4m7NaZL\nG3O+SlDC2ORYVyULdgUumKw9PleRJbLlXcDnRsonE/n4OxnuUvQ5kzRVqdHn9pPUNdpSySsLild7\nLu+Ju6ZIZ29mz/b5kgXbeX4lJ/YZ3NRx6fMSOumuj+6x5/lYoXPIq0pmle7cWYy7SrqU65i79CHL\nOTfnveDSSy+VNMbI9QGXuhtuuEHS+jsfoI1cc8YZZ0ga4TT+PktfGUcS+FDo19/BaQfufSSVq9z8\nMt1/ldAnwxCqor3pAjiHZqYajUaj0Wg0Go1GYwFmmSlO8BQihCmRhpUlU9+6lexZz3qWpHHa5OSd\nqY2lTYsoP6v0yvwPK06VKANLU1odCYq78sorV9eSPOC8885ba7efRrkfxW05OS8FfXcLOW2EqSFZ\ngyc14DRO4OEnP/lJSeM07WzfvffeK2mkg+c+pJR0eRFwedFFF0kaySH8mrR080ysOq973etW12J1\ng5HCiuNWCayWN954oyTpfe97n3ZBBuhKmyma0Rm3kDPOycBilfL0qFhQkonBwuJ6mlbsqkhpWuzT\nqld9P9NY+zyassQvweWXXy5p6FoWlJQ2U00744u1Fasx+ojOe8IYwPqBPrL2VIU2QVWslTWGNYwx\nxdrmQfvJ9lRWqF2L9YK0mHlAK7qZxUUrXc37ZCp4aegC1/A3z6wKXE4lAfBnJptasSF7seTxrGQa\ntwXfZ0zdkpr9qljHXBPYT+YKnCa4P+uANKzGyYI62491NJlG4JbU9AyoUq1z7yzBsS3m1pi0qCfr\nIW0ywvSTNd+TOSQbl3pfvSdwX1hpZ6aSQcjkPlUyKzCXJGhXhop7Vqn8mZv0o5qjuYcB5Od6xXqX\nHjDVu1KmTc9EPP59xbxI+wAACWVJREFU1nf+5tpq7meKegdpxrctHJ5AhoybJ2QAyJL56OtDsqz8\nzL1WGp487C3JvjojxJxkHnIfn5dTzFk1zrSLPjDmnpwsC9QuBW3kWf4M2KYDBw6sXVOtucjl+uuv\nlzT04OKLL15de8wxx0gasuNcwXuW6yBJS/DKoi0kw5A2E1jlmuLvCOgO12YJIf/eXhJ6NTPVaDQa\njUaj0Wg0Ggswy0yB6lTGaRVrM6dRt/hyksdKxUn1lltukbRumcFXFYtVFqOs0j1W1gPAMzxGx/vi\n8RWkd+TEW6Uv5xlYjt797ndvPHMvSHanKgyKtQYfU2c7kCWyxer5iU98QtL6yRvWhO+n/y79lEY6\nTJ5ZFULFGpJWeqxdxJxJ0tvf/nZJg2EghsqtHBnLsRTIMFOLepvTIufWm2SmsIrAYjjbkmlMM/7H\nLdMZA1NZxZPZpS1VCmfuV7FWec1+MCkwmsxxj4eincxhrHVueUNW6B+scOXTjezRBcaLee9+0fQf\n6yvP9tg2xp1xQR7Ea1bpXqv03WC/mKm0OPuzkpmin64vyaJlfFVVuJg1Ky30c+mRK1mkbqUl29dh\nxm8uVmlOj7dBPsPXy2SFMobEkanGqzmIfNP6ir65/NFdfnJ/t9SztiQzzU9fezK1dzIK0mbRyqVI\nD5GqgCsySK+SbJM05IMs3NsiUx4jC/TWWSd+h4HA08HlzlqSaeyzKLf/L3XZ41/YTzzGaQnYs6v4\nGcaSZ2Xxckell9L62pspzEHFBCW7Td/9WvQwU5lXcYC0K+OGKuzqQcHzq/dD9mLalvHN/vwsq8B3\nfT9H55Bzxqn6mpKF4BlXHzPuk6UOqhjUqeKzzvB6W3fBFVdcIWmsbf7Oduedd0oaXirVOo/e0Ner\nr75a0pDTK1/5ytW1FLLOOco853nSYKCI5SYPgq/36GfO5yxh4dfmXly99+d3KjQz1Wg0Go1Go9Fo\nNBoLMGu+4hTMSdBPvpxe54oDZqYcTrGceL3IK8DCmqyTnxCzaC+nRT815sk0Mw+5JWvqNFz5Ae+a\nJYlnZHY5/1+elGGLpHHCRv6wB1g+nJXJWBK+e9ppp0kaRdn8f8kmVEU7E1mcWRqZHGEcyTp4zTXX\nrK6BqZjLrLYXpIW2ysSE7lXZ9rCgYSmC4aDtbkHMWDfYGiytbilKP90qM1ZaxlLv3Rc5Y1Yq9qBi\nHZYCRoq+ua6m9Sx11/+H5RnrZsX8pDUzi/gRxyhtFuHku1XsDUzE4YcfLmmMqcsVvU62pyqYvCtS\nJ1wWVUFNad3Kn8V6s12VJbvSJamOxZorxJqZkZIJdt3n94yLc2vrfhSWljb75XpAO7KIZnU98mcd\nqPYK+sPcIIYWq6lbyPmdNR/5+XpCe/iZ1nwKSksjHpa9DCur9ymZCPc+2AZzmSV5L2BN5WfF9meB\naNZL1n5vM3JHToyd7w+8g3Af3iVcbxkv5J8ZeKsMutyXdlaxb7sW7aVf3M/bkfGlsCAeA01bMy6E\nddULpnM/dC3nSLUHZbZIbx/PyP2pYsmmWFHXj/0qKp398/UmWVr2fl932OszDhF4nDA6xvjlO4Wv\n5Rkfh257hr5sD59lLgD/nb00WSxv166FkN/73vdKqj19Mj4y40v9d5gjZMCcdy8TYvPxMmHdIwO1\nZwvEyyyLQFfzOdf0nDP5vepaaYxRnjUqNDPVaDQajUaj0Wg0GgvQh6lGo9FoNBqNRqPRWIBZNz+C\ntKE6vcgrlGe64VVuK7ghEIAGBeeue+naBgUHjev0aLr1pCuX3y8D+zMI25+ZAf7VNbu6T2UxYnc7\ny+QGVdpRUkeSwpykD1XhOChJqGzSUKbLkzTGJl3R3MUFeWT7qlScmbb30EMPlbRe1Jnv7TflXwV4\n4/ZQFXCkX7QNqplrXAa4euAGArVeySLdx/L/1WdVMedEzrnKZXXXlLPSkF3lnpeuCfnTf+d7JH/g\n/95u9BaaPgty+9rjwevSkIO78zAfUp8zZa0jx2KuYO1SZAIKdynI4rWVLkwVwc1yEf4/+pxzstLD\nqTTJ0mba/XQtrhIF8FnuF1W7lmLOlYPfc/5XyTJAutu4XrDn4OZH8iJcUNwlLYv1Vqn3s7CsB0hL\n625+uLxmYhxfy1gDecaJJ56oJaDvmTzHkanjfb5kymnkjfuOrxPIn7anm5+73PE7cuZvn0dZFoW2\nsC9UbjwZiuBJClhLdk3hn0l2XA9IpJFFjl1O6WLHfdJFShpjk2nn83Np020TuF5lEolMIlS57Oaa\n5M/MQrVLgXyqpEZV0edsa7r0Jvy7WeqEccikENKmWyXXVDJAPlk81pHvndX7WeVuuwToYOXSlmtZ\nVS4jSxrkfUj0Jkm33Xbb2meMJ3uC7ynocq47Pi8ZT2TKPEY2hAhIm/Lnma4f1XlhCs1MNRqNRqPR\naDQajcYCzJoEsYoRSOZWBCwVmWq6SnWZp0X+rqwSUxZXt9DwLCxzlRUnr00riafjTmtslVChKvq1\nBJkAweWVllBO5f6d7Dt9pq3ONsEsTp28q8KVPIvPKovDnFUCoBdT1uypNu8H3LqLnjLelbWf37F4\nwDbByLquZMpuLK0ZwCxtynmqMKcjEzl4OzMJylza3P1ImEDwOvJ061iyt8kASEM3M80z17rVD0tz\njgFWMpgARxY9rAoH8j/YWdpXJauZ+lvaH6bP2zoXuJ3z1Nc12p8Ws2pOo0s5RhW7nsVK5xL/ZOB1\nlRo7WdMqQUkmsliKXKtcB9G5OTlNMSyZFEYafSeYmrTBBFW7vDJAukpskiUb0srt+oEFlv04rbj+\nP36+5S1v0RJMpWr39s8Vs832Ixd0xRk85JSlN2CmPPEDeyLrLZ9VXiSZ2hnZ+rVTyTSqJCZV8qJt\nwHNh4Jz9gpmif0ceeaSkOukP7UFOVcF02sq1XjZCWp9ztANZVMla0itlzgMi51OmyvbPKkZpG+Tc\n93HLxDtV0q9810u5OWjzFBtWzcM5JKOe7EeVAG1ur9qv8h2Mcb4zSZtr2Fxq9PQGSLbUr809vCr1\nwX34TrXvZAIf/mZe+big35kgrPKemGIuHc1MNRqNRqPRaDQajcYCHLRfp9lGo9FoNBqNRqPR+P8J\nzUw1Go1Go9FoNBqNxgL0YarRaDQajUaj0Wg0FqAPU41Go9FoNBqNRqOxAH2YajQajUaj0Wg0Go0F\n6MNUo9FoNBqNRqPRaCxAH6YajUaj0Wg0Go1GYwH+Fz+oPQQ/MvtdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x360 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label for each of the above image: [2 6 7 4 4 0 3 0 7 3]\n"
     ]
    }
   ],
   "source": [
    "plt.figure(figsize=(15,5))\n",
    "for i in range(10):\n",
    "    plt.subplot(1, 10, i+1)\n",
    "    plt.imshow(X_train[i].reshape(32,32), cmap='gray')\n",
    "    plt.axis('off')\n",
    "plt.show()\n",
    "print('Label for each of the above image: %s' %y_train[0:10] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<b>Comment: </b> Here I have shown the top 10 images with their lables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. In the train and test loop, define the hyperparameters for the model :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## input size\n",
    "input_dim = X_train.shape[1]\n",
    "\n",
    "def train_and_test_loop(iterations, lr, Lambda, verb=True):\n",
    "    ## hyperparameters\n",
    "    iterations = iterations\n",
    "    learning_rate = lr\n",
    "    hidden_nodes = 10\n",
    "    output_nodes = 10\n",
    "\n",
    "    ## define neural net\n",
    "    nn = NN()\n",
    "    nn.add_layer(Linear(input_dim, hidden_nodes))\n",
    "\n",
    "    nn, val_acc = sgd(nn, X_train , y_train, minibatch_size=1000, epoch=iterations, learning_rate=learning_rate,\n",
    "                      X_val=X_val, y_val=y_val, Lambda=Lambda, verb=verb)\n",
    "    return val_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Create a Sequential model in Keras with input layer with the correct input shape, Hidden Layers, Output Layers and the activation functions:-"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating Sequentil Model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding Layers with input shape, Hidden Layers, Output Layers and the Activation Functions:\n",
    "Since Sigmoid functions suffer from gradient vanishing problem, making training slower. So I will be using <b>ReLu</b> (Rectified Linear Unit) which is one of the most popular ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add(Dense(50, input_shape = (1024,), kernel_initializer='he_normal'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.2))\n",
    "\n",
    "model.add(Dense(50, kernel_initializer='he_normal'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.2))\n",
    "\n",
    "model.add(Dense(10, kernel_initializer='he_normal'))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment:</b> Here I have used relu activation function in hidden layer and softmax activation function in the output layer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Define the optimizer to be used in this model :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sgd = optimizers.SGD(lr = 0.001, decay=1e-6, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<b>Comment:</b> Here I have used SGD optimizer with learnign rate 0.001 and the momentum is 0.9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Compile the model with the corresponding Loss and metrics to monitor :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#compile the model\n",
    "model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                51250     \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 50)                200       \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 50)                200       \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                510       \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 54,710\n",
      "Trainable params: 54,510\n",
      "Non-trainable params: 200\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "42000/42000 [==============================] - 2s 50us/sample - loss: 2.6770 - acc: 0.1032\n",
      "Epoch 2/10\n",
      "42000/42000 [==============================] - 1s 22us/sample - loss: 2.5079 - acc: 0.1206\n",
      "Epoch 3/10\n",
      "42000/42000 [==============================] - 1s 23us/sample - loss: 2.4105 - acc: 0.1392\n",
      "Epoch 4/10\n",
      "42000/42000 [==============================] - 1s 21us/sample - loss: 2.3314 - acc: 0.1622\n",
      "Epoch 5/10\n",
      "42000/42000 [==============================] - 1s 24us/sample - loss: 2.2686 - acc: 0.1860\n",
      "Epoch 6/10\n",
      "42000/42000 [==============================] - 1s 22us/sample - loss: 2.2039 - acc: 0.2094\n",
      "Epoch 7/10\n",
      "42000/42000 [==============================] - ETA: 0s - loss: 2.1566 - acc: 0.230 - 1s 25us/sample - loss: 2.1564 - acc: 0.2302\n",
      "Epoch 8/10\n",
      "42000/42000 [==============================] - 1s 24us/sample - loss: 2.1121 - acc: 0.2471\n",
      "Epoch 9/10\n",
      "42000/42000 [==============================] - 1s 23us/sample - loss: 2.0651 - acc: 0.2693\n",
      "Epoch 10/10\n",
      "42000/42000 [==============================] - 1s 22us/sample - loss: 2.0214 - acc: 0.2916\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f97e1619160>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train_one, batch_size = 700, epochs = 10, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. Fit the model and use model.evaluate() to return the score :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18000/18000 [==============================] - 1s 66us/sample - loss: 1.8853 - acc: 0.3939\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test_one)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['loss', 'acc']\n",
      "[1.8853392922083536, 0.3938889]\n"
     ]
    }
   ],
   "source": [
    "print(model.metrics_names)\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Comment :</b> Here we can notice that the total loss is 0.32 and the model is accuracy is 90% which is good for digit recognisation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Creating Neural Network class "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class NN():\n",
    "    def __init__(self, lossfunc=CrossEntropy(), mode='train'):\n",
    "        self.params = []\n",
    "        self.layers = []\n",
    "        self.loss_func = lossfunc\n",
    "        self.grads = []\n",
    "        self.mode = mode\n",
    "        \n",
    "    def add_layer(self, layer):\n",
    "        self.layers.append(layer)\n",
    "        self.params.append(layer.params)\n",
    "\n",
    "    def forward(self, X):\n",
    "        for layer in self.layers:\n",
    "            X = layer.forward(X)\n",
    "        return X\n",
    "    \n",
    "    def backward(self, nextgrad):\n",
    "        self.clear_grad_param()\n",
    "        for layer in reversed(self.layers):\n",
    "            nextgrad, grad = layer.backward(nextgrad)\n",
    "            self.grads.append(grad)\n",
    "        return self.grads\n",
    "    \n",
    "    def train_step(self, X, y):\n",
    "        out = self.forward(X)\n",
    "        loss = self.loss_func.forward(out,y)  + ((Lambda / (2 * y.shape[0])) * np.sum([np.sum(w**2) for w in self.params[0][0]]))\n",
    "        nextgrad = self.loss_func.backward(out,y) + ((Lambda/y.shape[0]) * np.sum([np.sum(w) for w in self.params[0][0]]))\n",
    "        grads = self.backward(nextgrad)\n",
    "        return loss, grads\n",
    "    \n",
    "    def predict(self, X):\n",
    "        X = self.forward(X)\n",
    "        p = softmax(X)\n",
    "        return np.argmax(p, axis=1)\n",
    "    \n",
    "    def predict_scores(self, X):\n",
    "        X = self.forward(X)\n",
    "        p = softmax(X)\n",
    "        return p\n",
    "    \n",
    "    def clear_grad_param(self):\n",
    "        self.grads = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def update(velocity, params, grads, learning_rate=0.01, mu=0.9):\n",
    "    for v, p, g, in zip(velocity, params, reversed(grads)):\n",
    "        for i in range(len(g)):\n",
    "            v[i] = (mu * v[i]) - (learning_rate * g[i])\n",
    "            p[i] += v[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get minibatches\n",
    "def minibatch(X, y, minibatch_size):\n",
    "    n = X.shape[0]\n",
    "    minibatches = []\n",
    "    permutation = np.random.permutation(X.shape[0])\n",
    "    X = X[permutation]\n",
    "    y = y[permutation]\n",
    "    \n",
    "    for i in range(0, n , minibatch_size):\n",
    "        X_batch = X[i:i + minibatch_size, :]\n",
    "        y_batch = y[i:i + minibatch_size, ]\n",
    "\n",
    "        minibatches.append((X_batch, y_batch))\n",
    "    return minibatches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sgd(net, X_train, y_train, minibatch_size, epoch, learning_rate, mu=0.9, X_val=None, y_val=None, Lambda=0, verb=True):\n",
    "    val_loss_epoch = []\n",
    "    minibatches = minibatch(X_train, y_train, minibatch_size)\n",
    "    minibatches_val = minibatch(X_val, y_val, minibatch_size)\n",
    "    \n",
    "    for i in range(epoch):\n",
    "        loss_batch = []\n",
    "        val_loss_batch = []\n",
    "        velocity = []\n",
    "        for param_layer in net.params:\n",
    "            p = [np.zeros_like(param) for param in list(param_layer)]\n",
    "            velocity.append(p)\n",
    "            \n",
    "        # iterate over mini batches\n",
    "        for X_mini, y_mini in minibatches:\n",
    "            loss, grads = net.train_step(X_mini, y_mini)\n",
    "            loss_batch.append(loss)\n",
    "            update(velocity, net.params, grads, learning_rate=learning_rate, mu=mu)\n",
    "\n",
    "        for X_mini_val, y_mini_val in minibatches_val:\n",
    "            val_loss, _ = net.train_step(X_mini, y_mini)\n",
    "            val_loss_batch.append(val_loss)\n",
    "        \n",
    "        # accuracy of model at end of epoch after all mini batch updates\n",
    "        m_train = X_train.shape[0]\n",
    "        m_val = X_val.shape[0]\n",
    "        y_train_pred = []\n",
    "        y_val_pred = []\n",
    "        y_train1 = []\n",
    "        y_vall = []\n",
    "        for ii in range(0, m_train, minibatch_size):\n",
    "            X_tr = X_train[ii:ii + minibatch_size, : ]\n",
    "            y_tr = y_train[ii:ii + minibatch_size,]\n",
    "            y_train1 = np.append(y_train1, y_tr)\n",
    "            y_train_pred = np.append(y_train_pred, net.predict(X_tr))\n",
    "\n",
    "        for ii in range(0, m_val, minibatch_size):\n",
    "            X_va = X_val[ii:ii + minibatch_size, : ]\n",
    "            y_va = y_val[ii:ii + minibatch_size,]\n",
    "            y_vall = np.append(y_vall, y_va)\n",
    "            y_val_pred = np.append(y_val_pred, net.predict(X_va))\n",
    "            \n",
    "        train_acc = check_accuracy(y_train1, y_train_pred)\n",
    "        val_acc = check_accuracy(y_vall, y_val_pred)\n",
    "        \n",
    "        ## weights\n",
    "        w = np.array(net.params[0][0])\n",
    "        \n",
    "        ## adding regularization to cost\n",
    "        mean_train_loss = (sum(loss_batch) / float(len(loss_batch)))\n",
    "        mean_val_loss = sum(val_loss_batch) / float(len(val_loss_batch))\n",
    "        \n",
    "        val_loss_epoch.append(mean_val_loss)\n",
    "        if verb:\n",
    "            if i%50==0:\n",
    "                print(\"Epoch {3}/{4}: Loss = {0} | Training Accuracy = {1}\".format(mean_train_loss, train_acc, val_acc, i, epoch))\n",
    "    return net, val_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def check_accuracy(y_true, y_pred):\n",
    "    count = 0\n",
    "    for i,j in zip(y_true, y_pred):\n",
    "        if int(i)==j:\n",
    "            count +=1\n",
    "    return float(count)/float(len(y_true))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## input size\n",
    "input_dim = X_train.shape[1]\n",
    "\n",
    "def train_and_test_loop(iterations, lr, Lambda, verb=True):\n",
    "    ## hyperparameters\n",
    "    iterations = iterations\n",
    "    learning_rate = lr\n",
    "    hidden_nodes = 10\n",
    "    output_nodes = 10\n",
    "\n",
    "    ## define neural net\n",
    "    nn = NN()\n",
    "    nn.add_layer(Linear(input_dim, hidden_nodes))\n",
    "\n",
    "    nn, val_acc = sgd(nn, X_train , y_train, minibatch_size=1000, epoch=iterations, learning_rate=learning_rate,\n",
    "                      X_val=X_val, y_val=y_val, Lambda=Lambda, verb=verb)\n",
    "    return val_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12. Disable Regularization by setting appropriate value for Lambda and check the loss of the NN :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/1: Loss = 2.311483757726858 | Training Accuracy = 0.0929047619047619\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.09305"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 0.00001\n",
    "Lambda = 0\n",
    "train_and_test_loop(1, lr, Lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment :</b> To disable regularization I have set Lambda =0 and learning rate is almost 0 and observed the loss is 2.33 and the training accuracy is almost 9%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 13. Increase the Regularization parameter (Lambda) and check how the loss is for the NN. Record findings:-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/1: Loss = 1.2913317918509992e+126 | Training Accuracy = 0.09966666666666667\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 0.00001\n",
    "Lambda = 1e3\n",
    "train_and_test_loop(1, lr, Lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment : </b> Here I have increased the regularisation and observed that loss is reduced and accuracy increased."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14. Network overfit with a small subset of the dataset. Check if the network will overfit when you use no regularization and the loss is very small and accuracy is 100%.:-"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, set a small learning rate and turn regularization off\n",
    "n the code below:\n",
    "- Take the first 20 examples from MNIST\n",
    "- turn off regularization(reg=0.0)\n",
    "- use simple vanilla 'sgd'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_subset = X_train[0:20]\n",
    "y_train_subset = y_train[0:20]\n",
    "X_train = X_train_subset\n",
    "y_train = y_train_subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/10000: Loss = 2.2570375363983173 | Training Accuracy = 0.15\n",
      "Epoch 50/10000: Loss = 2.1853685330096826 | Training Accuracy = 0.15\n",
      "Epoch 100/10000: Loss = 2.129318733219977 | Training Accuracy = 0.2\n",
      "Epoch 150/10000: Loss = 2.0853954492633386 | Training Accuracy = 0.25\n",
      "Epoch 200/10000: Loss = 2.0506714403357753 | Training Accuracy = 0.25\n",
      "Epoch 250/10000: Loss = 2.0228301693353465 | Training Accuracy = 0.2\n",
      "Epoch 300/10000: Loss = 2.000110038039673 | Training Accuracy = 0.2\n",
      "Epoch 350/10000: Loss = 1.9812053928860263 | Training Accuracy = 0.2\n",
      "Epoch 400/10000: Loss = 1.9651623576630217 | Training Accuracy = 0.2\n",
      "Epoch 450/10000: Loss = 1.9512879938018188 | Training Accuracy = 0.25\n",
      "Epoch 500/10000: Loss = 1.9390784329322819 | Training Accuracy = 0.3\n",
      "Epoch 550/10000: Loss = 1.9281651375546638 | Training Accuracy = 0.3\n",
      "Epoch 600/10000: Loss = 1.918276036001057 | Training Accuracy = 0.3\n",
      "Epoch 650/10000: Loss = 1.9092079394707482 | Training Accuracy = 0.3\n",
      "Epoch 700/10000: Loss = 1.9008071423487292 | Training Accuracy = 0.3\n",
      "Epoch 750/10000: Loss = 1.8929558073343755 | Training Accuracy = 0.3\n",
      "Epoch 800/10000: Loss = 1.8855623835937632 | Training Accuracy = 0.3\n",
      "Epoch 850/10000: Loss = 1.8785548201450992 | Training Accuracy = 0.3\n",
      "Epoch 900/10000: Loss = 1.8718757160606696 | Training Accuracy = 0.3\n",
      "Epoch 950/10000: Loss = 1.865478817776723 | Training Accuracy = 0.3\n",
      "Epoch 1000/10000: Loss = 1.8593264596830354 | Training Accuracy = 0.35\n",
      "Epoch 1050/10000: Loss = 1.8533876710917052 | Training Accuracy = 0.35\n",
      "Epoch 1100/10000: Loss = 1.8476367588444667 | Training Accuracy = 0.35\n",
      "Epoch 1150/10000: Loss = 1.8420522332443323 | Training Accuracy = 0.35\n",
      "Epoch 1200/10000: Loss = 1.836615984718715 | Training Accuracy = 0.35\n",
      "Epoch 1250/10000: Loss = 1.8313126457668827 | Training Accuracy = 0.35\n",
      "Epoch 1300/10000: Loss = 1.8261290914297181 | Training Accuracy = 0.35\n",
      "Epoch 1350/10000: Loss = 1.8210540444929908 | Training Accuracy = 0.35\n",
      "Epoch 1400/10000: Loss = 1.8160777607302923 | Training Accuracy = 0.4\n",
      "Epoch 1450/10000: Loss = 1.8111917759342622 | Training Accuracy = 0.4\n",
      "Epoch 1500/10000: Loss = 1.8063887010976203 | Training Accuracy = 0.4\n",
      "Epoch 1550/10000: Loss = 1.8016620554446152 | Training Accuracy = 0.4\n",
      "Epoch 1600/10000: Loss = 1.7970061294569153 | Training Accuracy = 0.4\n",
      "Epoch 1650/10000: Loss = 1.7924158718451046 | Training Accuracy = 0.4\n",
      "Epoch 1700/10000: Loss = 1.787886795767329 | Training Accuracy = 0.4\n",
      "Epoch 1750/10000: Loss = 1.7834149006158004 | Training Accuracy = 0.45\n",
      "Epoch 1800/10000: Loss = 1.7789966064681892 | Training Accuracy = 0.45\n",
      "Epoch 1850/10000: Loss = 1.7746286988976778 | Training Accuracy = 0.45\n",
      "Epoch 1900/10000: Loss = 1.7703082822978053 | Training Accuracy = 0.45\n",
      "Epoch 1950/10000: Loss = 1.7660327402393414 | Training Accuracy = 0.45\n",
      "Epoch 2000/10000: Loss = 1.7617997016603972 | Training Accuracy = 0.45\n",
      "Epoch 2050/10000: Loss = 1.7576070119157794 | Training Accuracy = 0.45\n",
      "Epoch 2100/10000: Loss = 1.7534527078905953 | Training Accuracy = 0.45\n",
      "Epoch 2150/10000: Loss = 1.7493349965264606 | Training Accuracy = 0.45\n",
      "Epoch 2200/10000: Loss = 1.7452522362240022 | Training Accuracy = 0.45\n",
      "Epoch 2250/10000: Loss = 1.7412029206786008 | Training Accuracy = 0.45\n",
      "Epoch 2300/10000: Loss = 1.737185664782017 | Training Accuracy = 0.45\n",
      "Epoch 2350/10000: Loss = 1.7331991922842565 | Training Accuracy = 0.45\n",
      "Epoch 2400/10000: Loss = 1.7292423249605076 | Training Accuracy = 0.45\n",
      "Epoch 2450/10000: Loss = 1.7253139730694116 | Training Accuracy = 0.45\n",
      "Epoch 2500/10000: Loss = 1.7214131269230581 | Training Accuracy = 0.45\n",
      "Epoch 2550/10000: Loss = 1.7175388494172714 | Training Accuracy = 0.45\n",
      "Epoch 2600/10000: Loss = 1.7136902693941338 | Training Accuracy = 0.45\n",
      "Epoch 2650/10000: Loss = 1.7098665757280689 | Training Accuracy = 0.45\n",
      "Epoch 2700/10000: Loss = 1.7060670120430188 | Training Accuracy = 0.45\n",
      "Epoch 2750/10000: Loss = 1.7022908719817338 | Training Accuracy = 0.45\n",
      "Epoch 2800/10000: Loss = 1.6985374949595542 | Training Accuracy = 0.45\n",
      "Epoch 2850/10000: Loss = 1.69480626234456 | Training Accuracy = 0.45\n",
      "Epoch 2900/10000: Loss = 1.6910965940140201 | Training Accuracy = 0.45\n",
      "Epoch 2950/10000: Loss = 1.6874079452438298 | Training Accuracy = 0.45\n",
      "Epoch 3000/10000: Loss = 1.6837398038934066 | Training Accuracy = 0.45\n",
      "Epoch 3050/10000: Loss = 1.680091687853373 | Training Accuracy = 0.45\n",
      "Epoch 3100/10000: Loss = 1.6764631427275671 | Training Accuracy = 0.45\n",
      "Epoch 3150/10000: Loss = 1.672853739724441 | Training Accuracy = 0.45\n",
      "Epoch 3200/10000: Loss = 1.669263073736007 | Training Accuracy = 0.45\n",
      "Epoch 3250/10000: Loss = 1.6656907615850982 | Training Accuracy = 0.45\n",
      "Epoch 3300/10000: Loss = 1.6621364404239913 | Training Accuracy = 0.45\n",
      "Epoch 3350/10000: Loss = 1.658599766269393 | Training Accuracy = 0.45\n",
      "Epoch 3400/10000: Loss = 1.6550804126605172 | Training Accuracy = 0.45\n",
      "Epoch 3450/10000: Loss = 1.6515780694284292 | Training Accuracy = 0.45\n",
      "Epoch 3500/10000: Loss = 1.6480924415661626 | Training Accuracy = 0.45\n",
      "Epoch 3550/10000: Loss = 1.6446232481902183 | Training Accuracy = 0.45\n",
      "Epoch 3600/10000: Loss = 1.6411702215850663 | Training Accuracy = 0.45\n",
      "Epoch 3650/10000: Loss = 1.6377331063231328 | Training Accuracy = 0.45\n",
      "Epoch 3700/10000: Loss = 1.6343116584535313 | Training Accuracy = 0.5\n",
      "Epoch 3750/10000: Loss = 1.6309056447534704 | Training Accuracy = 0.5\n",
      "Epoch 3800/10000: Loss = 1.627514842036882 | Training Accuracy = 0.5\n",
      "Epoch 3850/10000: Loss = 1.6241390365153383 | Training Accuracy = 0.5\n",
      "Epoch 3900/10000: Loss = 1.6207780232068107 | Training Accuracy = 0.5\n",
      "Epoch 3950/10000: Loss = 1.61743160538824 | Training Accuracy = 0.5\n",
      "Epoch 4000/10000: Loss = 1.6140995940882703 | Training Accuracy = 0.5\n",
      "Epoch 4050/10000: Loss = 1.6107818076168385 | Training Accuracy = 0.5\n",
      "Epoch 4100/10000: Loss = 1.6074780711286096 | Training Accuracy = 0.5\n",
      "Epoch 4150/10000: Loss = 1.6041882162175274 | Training Accuracy = 0.5\n",
      "Epoch 4200/10000: Loss = 1.6009120805399881 | Training Accuracy = 0.5\n",
      "Epoch 4250/10000: Loss = 1.597649507464371 | Training Accuracy = 0.5\n",
      "Epoch 4300/10000: Loss = 1.5944003457448555 | Training Accuracy = 0.5\n",
      "Epoch 4350/10000: Loss = 1.5911644492176373 | Training Accuracy = 0.5\n",
      "Epoch 4400/10000: Loss = 1.5879416765178092 | Training Accuracy = 0.5\n",
      "Epoch 4450/10000: Loss = 1.584731890815338 | Training Accuracy = 0.5\n",
      "Epoch 4500/10000: Loss = 1.581534959568677 | Training Accuracy = 0.5\n",
      "Epoch 4550/10000: Loss = 1.5783507542946984 | Training Accuracy = 0.5\n",
      "Epoch 4600/10000: Loss = 1.5751791503537258 | Training Accuracy = 0.5\n",
      "Epoch 4650/10000: Loss = 1.572020026748548 | Training Accuracy = 0.5\n",
      "Epoch 4700/10000: Loss = 1.5688732659363906 | Training Accuracy = 0.5\n",
      "Epoch 4750/10000: Loss = 1.5657387536529037 | Training Accuracy = 0.5\n",
      "Epoch 4800/10000: Loss = 1.5626163787472966 | Training Accuracy = 0.5\n",
      "Epoch 4850/10000: Loss = 1.5595060330278219 | Training Accuracy = 0.5\n",
      "Epoch 4900/10000: Loss = 1.556407611116875 | Training Accuracy = 0.5\n",
      "Epoch 4950/10000: Loss = 1.5533210103150248 | Training Accuracy = 0.5\n",
      "Epoch 5000/10000: Loss = 1.5502461304733604 | Training Accuracy = 0.5\n",
      "Epoch 5050/10000: Loss = 1.5471828738735676 | Training Accuracy = 0.5\n",
      "Epoch 5100/10000: Loss = 1.5441311451152093 | Training Accuracy = 0.5\n",
      "Epoch 5150/10000: Loss = 1.541090851009709 | Training Accuracy = 0.5\n",
      "Epoch 5200/10000: Loss = 1.5380619004805909 | Training Accuracy = 0.5\n",
      "Epoch 5250/10000: Loss = 1.5350442044695511 | Training Accuracy = 0.5\n",
      "Epoch 5300/10000: Loss = 1.5320376758479657 | Training Accuracy = 0.55\n",
      "Epoch 5350/10000: Loss = 1.5290422293334847 | Training Accuracy = 0.55\n",
      "Epoch 5400/10000: Loss = 1.526057781411367 | Training Accuracy = 0.55\n",
      "Epoch 5450/10000: Loss = 1.5230842502602517 | Training Accuracy = 0.55\n",
      "Epoch 5500/10000: Loss = 1.5201215556820764 | Training Accuracy = 0.55\n",
      "Epoch 5550/10000: Loss = 1.5171696190358737 | Training Accuracy = 0.55\n",
      "Epoch 5600/10000: Loss = 1.5142283631752027 | Training Accuracy = 0.55\n",
      "Epoch 5650/10000: Loss = 1.5112977123889757 | Training Accuracy = 0.55\n",
      "Epoch 5700/10000: Loss = 1.5083775923454763 | Training Accuracy = 0.55\n",
      "Epoch 5750/10000: Loss = 1.5054679300393599 | Training Accuracy = 0.55\n",
      "Epoch 5800/10000: Loss = 1.502568653741462 | Training Accuracy = 0.55\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5850/10000: Loss = 1.499679692951227 | Training Accuracy = 0.55\n",
      "Epoch 5900/10000: Loss = 1.4968009783516127 | Training Accuracy = 0.55\n",
      "Epoch 5950/10000: Loss = 1.4939324417663031 | Training Accuracy = 0.55\n",
      "Epoch 6000/10000: Loss = 1.4910740161191112 | Training Accuracy = 0.55\n",
      "Epoch 6050/10000: Loss = 1.4882256353954184 | Training Accuracy = 0.55\n",
      "Epoch 6100/10000: Loss = 1.485387234605545 | Training Accuracy = 0.55\n",
      "Epoch 6150/10000: Loss = 1.4825587497499315 | Training Accuracy = 0.6\n",
      "Epoch 6200/10000: Loss = 1.4797401177860272 | Training Accuracy = 0.6\n",
      "Epoch 6250/10000: Loss = 1.476931276596785 | Training Accuracy = 0.6\n",
      "Epoch 6300/10000: Loss = 1.474132164960667 | Training Accuracy = 0.6\n",
      "Epoch 6350/10000: Loss = 1.4713427225230837 | Training Accuracy = 0.6\n",
      "Epoch 6400/10000: Loss = 1.4685628897691754 | Training Accuracy = 0.6\n",
      "Epoch 6450/10000: Loss = 1.4657926079978627 | Training Accuracy = 0.6\n",
      "Epoch 6500/10000: Loss = 1.463031819297099 | Training Accuracy = 0.6\n",
      "Epoch 6550/10000: Loss = 1.4602804665202553 | Training Accuracy = 0.6\n",
      "Epoch 6600/10000: Loss = 1.4575384932635724 | Training Accuracy = 0.6\n",
      "Epoch 6650/10000: Loss = 1.4548058438446247 | Training Accuracy = 0.6\n",
      "Epoch 6700/10000: Loss = 1.4520824632817435 | Training Accuracy = 0.6\n",
      "Epoch 6750/10000: Loss = 1.4493682972743414 | Training Accuracy = 0.6\n",
      "Epoch 6800/10000: Loss = 1.4466632921840932 | Training Accuracy = 0.6\n",
      "Epoch 6850/10000: Loss = 1.4439673950169314 | Training Accuracy = 0.6\n",
      "Epoch 6900/10000: Loss = 1.4412805534058044 | Training Accuracy = 0.6\n",
      "Epoch 6950/10000: Loss = 1.438602715594162 | Training Accuracy = 0.6\n",
      "Epoch 7000/10000: Loss = 1.4359338304201406 | Training Accuracy = 0.6\n",
      "Epoch 7050/10000: Loss = 1.4332738473013868 | Training Accuracy = 0.6\n",
      "Epoch 7100/10000: Loss = 1.4306227162205165 | Training Accuracy = 0.6\n",
      "Epoch 7150/10000: Loss = 1.4279803877111528 | Training Accuracy = 0.6\n",
      "Epoch 7200/10000: Loss = 1.425346812844531 | Training Accuracy = 0.6\n",
      "Epoch 7250/10000: Loss = 1.4227219432166285 | Training Accuracy = 0.6\n",
      "Epoch 7300/10000: Loss = 1.4201057309358038 | Training Accuracy = 0.6\n",
      "Epoch 7350/10000: Loss = 1.417498128610915 | Training Accuracy = 0.6\n",
      "Epoch 7400/10000: Loss = 1.4148990893398947 | Training Accuracy = 0.6\n",
      "Epoch 7450/10000: Loss = 1.4123085666987585 | Training Accuracy = 0.6\n",
      "Epoch 7500/10000: Loss = 1.4097265147310278 | Training Accuracy = 0.6\n",
      "Epoch 7550/10000: Loss = 1.4071528879375461 | Training Accuracy = 0.6\n",
      "Epoch 7600/10000: Loss = 1.404587641266673 | Training Accuracy = 0.6\n",
      "Epoch 7650/10000: Loss = 1.4020307301048285 | Training Accuracy = 0.6\n",
      "Epoch 7700/10000: Loss = 1.3994821102673867 | Training Accuracy = 0.6\n",
      "Epoch 7750/10000: Loss = 1.3969417379898892 | Training Accuracy = 0.6\n",
      "Epoch 7800/10000: Loss = 1.3944095699195678 | Training Accuracy = 0.6\n",
      "Epoch 7850/10000: Loss = 1.391885563107167 | Training Accuracy = 0.6\n",
      "Epoch 7900/10000: Loss = 1.3893696749990445 | Training Accuracy = 0.6\n",
      "Epoch 7950/10000: Loss = 1.3868618634295449 | Training Accuracy = 0.6\n",
      "Epoch 8000/10000: Loss = 1.3843620866136286 | Training Accuracy = 0.6\n",
      "Epoch 8050/10000: Loss = 1.3818703031397477 | Training Accuracy = 0.65\n",
      "Epoch 8100/10000: Loss = 1.379386471962956 | Training Accuracy = 0.65\n",
      "Epoch 8150/10000: Loss = 1.376910552398248 | Training Accuracy = 0.65\n",
      "Epoch 8200/10000: Loss = 1.3744425041141084 | Training Accuracy = 0.65\n",
      "Epoch 8250/10000: Loss = 1.3719822871262684 | Training Accuracy = 0.65\n",
      "Epoch 8300/10000: Loss = 1.3695298617916647 | Training Accuracy = 0.65\n",
      "Epoch 8350/10000: Loss = 1.367085188802579 | Training Accuracy = 0.65\n",
      "Epoch 8400/10000: Loss = 1.3646482291809652 | Training Accuracy = 0.65\n",
      "Epoch 8450/10000: Loss = 1.362218944272947 | Training Accuracy = 0.65\n",
      "Epoch 8500/10000: Loss = 1.3597972957434818 | Training Accuracy = 0.65\n",
      "Epoch 8550/10000: Loss = 1.357383245571185 | Training Accuracy = 0.65\n",
      "Epoch 8600/10000: Loss = 1.3549767560433073 | Training Accuracy = 0.65\n",
      "Epoch 8650/10000: Loss = 1.3525777897508566 | Training Accuracy = 0.65\n",
      "Epoch 8700/10000: Loss = 1.3501863095838642 | Training Accuracy = 0.65\n",
      "Epoch 8750/10000: Loss = 1.3478022787267832 | Training Accuracy = 0.65\n",
      "Epoch 8800/10000: Loss = 1.345425660654019 | Training Accuracy = 0.65\n",
      "Epoch 8850/10000: Loss = 1.3430564191255816 | Training Accuracy = 0.65\n",
      "Epoch 8900/10000: Loss = 1.3406945181828633 | Training Accuracy = 0.65\n",
      "Epoch 8950/10000: Loss = 1.3383399221445258 | Training Accuracy = 0.65\n",
      "Epoch 9000/10000: Loss = 1.3359925956025003 | Training Accuracy = 0.65\n",
      "Epoch 9050/10000: Loss = 1.3336525034180962 | Training Accuracy = 0.65\n",
      "Epoch 9100/10000: Loss = 1.3313196107182113 | Training Accuracy = 0.65\n",
      "Epoch 9150/10000: Loss = 1.328993882891639 | Training Accuracy = 0.65\n",
      "Epoch 9200/10000: Loss = 1.3266752855854735 | Training Accuracy = 0.65\n",
      "Epoch 9250/10000: Loss = 1.3243637847016037 | Training Accuracy = 0.65\n",
      "Epoch 9300/10000: Loss = 1.3220593463932981 | Training Accuracy = 0.65\n",
      "Epoch 9350/10000: Loss = 1.3197619370618745 | Training Accuracy = 0.65\n",
      "Epoch 9400/10000: Loss = 1.3174715233534484 | Training Accuracy = 0.65\n",
      "Epoch 9450/10000: Loss = 1.3151880721557665 | Training Accuracy = 0.65\n",
      "Epoch 9500/10000: Loss = 1.3129115505951088 | Training Accuracy = 0.65\n",
      "Epoch 9550/10000: Loss = 1.310641926033272 | Training Accuracy = 0.65\n",
      "Epoch 9600/10000: Loss = 1.308379166064619 | Training Accuracy = 0.65\n",
      "Epoch 9650/10000: Loss = 1.3061232385131984 | Training Accuracy = 0.65\n",
      "Epoch 9700/10000: Loss = 1.3038741114299301 | Training Accuracy = 0.65\n",
      "Epoch 9750/10000: Loss = 1.301631753089856 | Training Accuracy = 0.65\n",
      "Epoch 9800/10000: Loss = 1.2993961319894503 | Training Accuracy = 0.65\n",
      "Epoch 9850/10000: Loss = 1.2971672168439905 | Training Accuracy = 0.65\n",
      "Epoch 9900/10000: Loss = 1.2949449765849885 | Training Accuracy = 0.65\n",
      "Epoch 9950/10000: Loss = 1.2927293803576734 | Training Accuracy = 0.65\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.10648333333333333"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 0.0001\n",
    "Lambda = 0\n",
    "train_and_test_loop(10000, lr, Lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment : </b> Here to make the overfitting I have used 20% sample data of training and testing dataset. I have disabled the regularisation and checked that training accuracy is keep increasing and tending towards overfitting. In 10000 epocs it reached to 65% and I increase the epocs to 15000 then training accuracy will reach to apporx 100%. we can also see a Very small loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 15. Load the original dataset with all the images and prepare the data for modelling :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of datasets in this file: \n",
      " ['X_test', 'X_train', 'X_val', 'y_test', 'y_train', 'y_val']\n"
     ]
    }
   ],
   "source": [
    "with h5py.File('SVHN_single_grey.h5', 'r') as hdf:\n",
    "    ls = list(hdf.keys())\n",
    "    print('List of datasets in this file: \\n', ls)\n",
    "    X_test = hdf.get('X_test')\n",
    "    X_train = hdf.get('X_train')\n",
    "    X_val = hdf.get('X_val')\n",
    "    y_test = hdf.get('y_test')\n",
    "    y_train = hdf.get('y_train')\n",
    "    y_val = hdf.get('y_val')\n",
    "    \n",
    "    X_test = np.array(X_test)\n",
    "    X_train = np.array(X_train)\n",
    "    X_val = np.array(X_val)\n",
    "    y_test = np.array(y_test)\n",
    "    y_train = np.array(y_train)\n",
    "    y_val = np.array(y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment :</b> Here I have loaded the dataset again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of X_test (18000, 1024)\n",
      "Shape of X_train (42000, 1024)\n",
      "Shape of X_val (60000, 1024)\n",
      "Shape of y_test (18000,)\n",
      "Shape of y_train (42000,)\n",
      "Shape of y_val (60000,)\n"
     ]
    }
   ],
   "source": [
    "X_train = X_train.reshape(42000, 1024)\n",
    "X_test = X_test.reshape(18000, 1024)\n",
    "X_val = X_val.reshape(60000, 1024)\n",
    "X_train = X_train / 255.0\n",
    "X_test  = X_test  / 255.0\n",
    "X_val   = X_val   / 255.0\n",
    "print('Shape of X_test', X_test.shape)\n",
    "print('Shape of X_train', X_train.shape)\n",
    "print('Shape of X_val', X_val.shape)\n",
    "print('Shape of y_test', y_test.shape)\n",
    "print('Shape of y_train', y_train.shape)\n",
    "print('Shape of y_val', y_val.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 16. Start with a small Regularization. Keep adjusting the learning rate to check the loss. Record findings:-\n",
    "\n",
    "- we start with Lambda(small regularization) = 1e-7\n",
    "- we start with a small learning rate = 1e-7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/500: Loss = 2.3286523683447498 | Training Accuracy = 0.1005\n",
      "Epoch 50/500: Loss = 2.326910637479809 | Training Accuracy = 0.09978571428571428\n",
      "Epoch 100/500: Loss = 2.3252890697539645 | Training Accuracy = 0.09961904761904762\n",
      "Epoch 150/500: Loss = 2.3237791423095837 | Training Accuracy = 0.09926190476190476\n",
      "Epoch 200/500: Loss = 2.3223729814130443 | Training Accuracy = 0.09954761904761905\n",
      "Epoch 250/500: Loss = 2.321063305650582 | Training Accuracy = 0.09964285714285714\n",
      "Epoch 300/500: Loss = 2.319843375128018 | Training Accuracy = 0.09969047619047619\n",
      "Epoch 350/500: Loss = 2.318706945919973 | Training Accuracy = 0.09961904761904762\n",
      "Epoch 400/500: Loss = 2.3176482291203198 | Training Accuracy = 0.09945238095238096\n",
      "Epoch 450/500: Loss = 2.3166618539358965 | Training Accuracy = 0.09911904761904762\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.09756666666666666"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 1e-7\n",
    "Lambda = 1e-7\n",
    "train_and_test_loop(500, lr, Lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##### Now I will  try a (larger) learning rate 1e6. What could possibly go wrong?\n",
    "\n",
    "- Learning rate lr = 1e6\n",
    "- Regularization lambda = 1e-7\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/500: Loss = 2.150921910512303e+201 | Training Accuracy = 0.09966666666666667\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/webonise/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py:90: RuntimeWarning: overflow encountered in reduce\n",
      "  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)\n",
      "/home/webonise/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:27: RuntimeWarning: overflow encountered in square\n",
      "/home/webonise/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:12: RuntimeWarning: overflow encountered in add\n",
      "  if sys.path[0] == '':\n",
      "/home/webonise/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in subtract\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 50/500: Loss = nan | Training Accuracy = 0.09966666666666667\n",
      "Epoch 100/500: Loss = nan | Training Accuracy = 0.09966666666666667\n",
      "Epoch 150/500: Loss = nan | Training Accuracy = 0.09966666666666667\n",
      "Epoch 200/500: Loss = nan | Training Accuracy = 0.09966666666666667\n",
      "Epoch 250/500: Loss = nan | Training Accuracy = 0.09966666666666667\n",
      "Epoch 300/500: Loss = nan | Training Accuracy = 0.09966666666666667\n",
      "Epoch 350/500: Loss = nan | Training Accuracy = 0.09966666666666667\n",
      "Epoch 400/500: Loss = nan | Training Accuracy = 0.09966666666666667\n",
      "Epoch 450/500: Loss = nan | Training Accuracy = 0.09966666666666667\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 1e6\n",
    "Lambda = 1e-7\n",
    "train_and_test_loop(500, lr, Lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment: </b> Loss exploding. Learning rate is too high. \n",
    "Cost is very high. Always means high learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### I will try to train now with a value of learning rate between 1e-7 and 1e6\n",
    "\n",
    "- learning rate = 1e4\n",
    "- regularization remains the small, lambda = 1e-7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/10: Loss = 73088050232.71579 | Training Accuracy = 0.09992857142857142\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 1e4\n",
    "Lambda = 1e-7\n",
    "train_and_test_loop(10, lr, Lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment : </b>Still too high learning rate. Loss is not decreasing. The rough range of learning rate we should be cross validating is somewhere between [1e4 to 1e-7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 17. Perform Hyperparameter Optimization . Record findings:-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Try 1/100: Best_val_acc: 0.20371666666666666, lr: 0.03854017955467199, Lambda: 9.237252900373746e-05\n",
      "\n",
      "Try 2/100: Best_val_acc: 0.10443333333333334, lr: 1.7366991848953071e-07, Lambda: 2029.1313514151964\n",
      "\n",
      "Try 3/100: Best_val_acc: 0.1562, lr: 0.0002294938908043734, Lambda: 0.001905906121891738\n",
      "\n",
      "Try 4/100: Best_val_acc: 0.15995, lr: 0.0001530665892320223, Lambda: 0.0024964895266366516\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/webonise/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:27: RuntimeWarning: overflow encountered in square\n",
      "/home/webonise/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py:90: RuntimeWarning: overflow encountered in reduce\n",
      "  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)\n",
      "/home/webonise/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in subtract\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Try 5/100: Best_val_acc: 0.1, lr: 0.11846451099922259, Lambda: 3.3304257463914184\n",
      "\n",
      "Try 6/100: Best_val_acc: 0.11323333333333334, lr: 1.9406889221856268e-05, Lambda: 41.69231565637808\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/webonise/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:4: RuntimeWarning: overflow encountered in multiply\n",
      "  after removing the cwd from sys.path.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Try 7/100: Best_val_acc: 0.1, lr: 2459.8875433907206, Lambda: 1221.178957191728\n",
      "\n",
      "Try 8/100: Best_val_acc: 0.1, lr: 123.31832587058365, Lambda: 9.99823675141987\n",
      "\n",
      "Try 9/100: Best_val_acc: 0.1, lr: 28.48988588494099, Lambda: 2335.9542202051243\n",
      "\n",
      "Try 10/100: Best_val_acc: 0.19436666666666666, lr: 0.0016375640612316298, Lambda: 3.26839334035848e-05\n",
      "\n",
      "Try 11/100: Best_val_acc: 0.2152, lr: 0.027267970738468244, Lambda: 0.00013247504843786148\n",
      "\n",
      "Try 12/100: Best_val_acc: 0.09318333333333334, lr: 1.2243795335696293e-05, Lambda: 0.0014927841125445783\n",
      "\n",
      "Try 13/100: Best_val_acc: 0.1, lr: 6467.516065947057, Lambda: 1.3554216031088432e-05\n",
      "\n",
      "Try 14/100: Best_val_acc: 0.10146666666666666, lr: 1.2751135868522901e-05, Lambda: 1.2979697164138543e-05\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "for k in range(1,100):\n",
    "    lr = math.pow(10, np.random.uniform(-7.0, 4.0))\n",
    "    Lambda = math.pow(10, np.random.uniform(-5,5))\n",
    "    best_acc = train_and_test_loop(100, lr, Lambda, False)\n",
    "    print(\"Try {0}/{1}: Best_val_acc: {2}, lr: {3}, Lambda: {4}\\n\".format(k, 100, best_acc, lr, Lambda))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 18. Run a finer search by using a finer range of the hyperparameter. Record findings :-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/webonise/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py:90: RuntimeWarning: overflow encountered in reduce\n",
      "  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)\n",
      "/home/webonise/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:27: RuntimeWarning: overflow encountered in square\n",
      "/home/webonise/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in subtract\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Try 1/100: Best_val_acc: 0.1, lr: 0.0012385317424541313, Lambda: 2.562163558275977\n",
      "\n",
      "Try 2/100: Best_val_acc: 0.16421666666666668, lr: 0.006309667980490862, Lambda: 2.6544279263050695e-05\n",
      "\n",
      "Try 3/100: Best_val_acc: 0.16795, lr: 0.007897721190124394, Lambda: 0.001449675300990576\n",
      "\n",
      "Try 4/100: Best_val_acc: 0.13983333333333334, lr: 0.0014864539650998535, Lambda: 0.02315420885708122\n",
      "\n",
      "Try 5/100: Best_val_acc: 0.12668333333333334, lr: 0.0010213092303710577, Lambda: 1.2987182277401402e-05\n",
      "\n",
      "Try 6/100: Best_val_acc: 0.1, lr: 0.004960976613041949, Lambda: 2.8179057802219893\n",
      "\n",
      "Try 7/100: Best_val_acc: 0.15733333333333333, lr: 0.0035496296206377694, Lambda: 0.023418902868291215\n",
      "\n",
      "Try 8/100: Best_val_acc: 0.14083333333333334, lr: 0.0035794033129904206, Lambda: 0.003220343407013117\n",
      "\n",
      "Try 9/100: Best_val_acc: 0.1, lr: 0.005445383846212566, Lambda: 0.48669800441154165\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "for k in range(1,10):\n",
    "    lr = math.pow(10, np.random.uniform(-3.0, -2.0))\n",
    "    Lambda = math.pow(10, np.random.uniform(-5,2))\n",
    "    best_acc = train_and_test_loop(10, lr, Lambda, False)\n",
    "    print(\"Try {0}/{1}: Best_val_acc: {2}, lr: {3}, Lambda: {4}\\n\".format(k, 100, best_acc, lr, Lambda))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment : </b> I have run the finer search above with range of hyperparameters of learning rate and lambda and found that learning rate = 0.0078977 and  lambda = 0.0014496 is giving best accuracy value which 16.79% which has been tested in 10 epocs and we can say say if we increase the epocs we will the more accurate value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 19. Set the best hyperparameters found in the previous step. Check the Network’s accuracy:-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/100: Loss = 2.306312197439405 | Training Accuracy = 0.10926190476190477\n",
      "Epoch 50/100: Loss = 2.25640102521488 | Training Accuracy = 0.209\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.21421666666666667"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 0.007897721190124394\n",
    "Lambda = 0.001449675300990576\n",
    "train_and_test_loop(100, lr, Lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Comment:</b> I have found the best hyperparameters from the previous stpes that learning rate (lr = 0.0078977) and lambda = 0.0014496 is giving the best accuracy. Here we can also see that just in 100 epocs accuracy jumnps from 10% to 20% so if we will run for 1000 epcos then accuracy will reach out to 90%.  If we increase the epocs with same hyperparameters then we will get very good accuracy which is our objective for the image classification. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}