Backpropogation_with_Images.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backprop is very simple. Who made it Complicated?\n",
    "### Learning Outcome: You will be able to build your own Neural Network on a Paper and understand the maths beyond it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Almost 1 year back when I first wanted to try my hands on Neural network, I scratched my head for a long time on how Back-Propagation works. When I talk to peers around my circle, I see a lot of people facing this problem. Most people consider it as a black-box and use libraries like Keras, TensorFlow and PyTorch which provide automatic differentiation. Though it is not necessary to write code on how to compute gradients and backprop errors, having knowledge on it helps you in understanding a few concepts like Vanishing Gradients, Saturation of Neurons and reasons for random initialization of weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![NN](Images/NN.png \"NN\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initalizing the Network\n",
    "\n",
    "\\begin{equation*}\n",
    "Input = \\left[ \\begin{array}{cccc}\n",
    "0.1 & 0.2 & 0.7 \\\\\\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "W_{ij} = \\left[ \\begin{array}{cccc}\n",
    "W_{i1j1} & W_{i1j2} & W_{i1j3} \\\\\n",
    "W_{i2j1} & W_{i2j2} & W_{i2j3} \\\\\n",
    "W_{i3j1} & W_{i3j2} & W_{i3j3} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "0.1 & 0.2 & 0.3 \\\\\n",
    "0.3 & 0.2 & 0.7 \\\\\n",
    "0.4 & 0.3 & 0.9 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "\\begin{equation*}\n",
    "W_{jk} = \\left[ \\begin{array}{cccc}\n",
    "W_{j1k1} & W_{j1k2} & W_{j1k3} \\\\\n",
    "W_{j2k1} & W_{j2k2} & W_{j2k3} \\\\\n",
    "W_{j3k1} & W_{j3k2} & W_{j3k3} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "0.2 & 0.3 & 0.5 \\\\\n",
    "0.3 & 0.5 & 0.7 \\\\\n",
    "0.6 & 0.4 & 0.8 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "\\begin{equation*}\n",
    "W_{kl} = \\left[ \\begin{array}{cccc}\n",
    "W_{k1l1} & W_{k1l2} & W_{k1l3} \\\\\n",
    "W_{k2l1} & W_{k2l2} & W_{k2l3} \\\\\n",
    "W_{k3l1} & W_{k3l2} & W_{k3l3} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "0.1 & 0.4 & 0.8 \\\\\n",
    "0.3 & 0.7 & 0.2 \\\\\n",
    "0.5 & 0.2 & 0.9 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "Output = \\left[ \\begin{array}{cccc}\n",
    "1.0 & 0.0 & 0.0 \\\\\\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Layer1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![forward1](Images/Forword_1.png \"Forword\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Operation:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "i_{1} & i_{2} & i_{3} \\\\\\end{array} \\right] \\times \\left[ \\begin{array}{cccc}\n",
    "W_{i1j1} & W_{i1j2} & W_{i1j3} \\\\\n",
    "W_{i2j1} & W_{i2j2} & W_{i2j3} \\\\\n",
    "W_{i3j1} & W_{i3j2} & W_{i3j3} \\\\ \\end{array} \\right] + \\left[ \\begin{array}{cccc}\n",
    "b_{j1} & b_{j2} & b_{j3} \\\\\\end{array} \\right]= \\left[ \\begin{array}{cccc}\n",
    "h1_{in1} & h1_{in2} & h1_{in3} \\\\\\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "### Relu operation \n",
    "\\begin{equation*}\n",
    "relu = max(0, x) \n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "h1_{out1} & h1_{out2} &  h1_{out3} \\\\\\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    "max(0, h1_{in1}) & max(0, h1_{in2}) & max(0, h1_{in3}) \\\\\\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "## Example:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "0.1 & 0.2 & 0.7 \\\\ \\end{array} \\right] \\times \\left[ \\begin{array}{cccc}\n",
    "0.1 & 0.4 & 0.3 \\\\\n",
    "0.3 & 0.7 & 0.7 \\\\\n",
    "0.5 & 0.2 & 0.9 \\\\ \\end{array} \\right]  + \\left[ \\begin{array}{cccc}\n",
    "1.0 & 1.0 & 1.0 \\\\\\end{array} \\right] =  \\left[\\begin{array}{cccc}\n",
    "1.35 & 1.27 & 1.8 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "h1_{out1} & h1_{out2} &  h1_{out3} \\\\\\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    "1.35 & 1.27 & 1.8 \\\\\\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Layer-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![forward2](Images/Forword_2.png \"Forword2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Operation:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "h1_{out1} & h1_{out2} & h1_{out3} \\\\\\end{array} \\right] \\times \\left[ \\begin{array}{cccc}\n",
    "W_{j1k1} & W_{j1k2} & W_{j1k3} \\\\\n",
    "W_{j2k1} & W_{j2k2} & W_{j2k3} \\\\\n",
    "W_{j3k1} & W_{j3k2} & W_{j3k3} \\\\ \\end{array} \\right] + \\left[ \\begin{array}{cccc}\n",
    "b_{k1} & b_{k2} & b_{k3} \\\\\\end{array} \\right]= \\left[ \\begin{array}{cccc}\n",
    "h2_{in1} & h2_{in1} & h2_{in3} \\\\\\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "### Sigmoid operation \n",
    "\\begin{equation*}\n",
    "Sigmoid = 1/(1+\\mathrm{e}^{-x}) \n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "h2_{out1} & h2_{out2} &  h2_{out3} \\\\\\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    "1/(1+\\mathrm{e}^{-h2_{in1}})  & 1/(1+\\mathrm{e}^{-h2_{in2}})  & 1/(1+\\mathrm{e}^{-h2_{in3}})  \\\\\\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "## Example:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "1.35 & 1.27 & 1.8 \\\\ \\end{array} \\right] \\times \\left[ \\begin{array}{cccc}\n",
    "0.2 & 0.3 & 0.5 \\\\\n",
    "0.3 & 0.5 & 0.7 \\\\\n",
    "0.6 & 0.4 & 0.8 \\\\ \\end{array} \\right]  + \\left[ \\begin{array}{cccc}\n",
    "1.0 & 1.0 & 1.0 \\\\\\end{array} \\right] =  \\left[\\begin{array}{cccc}\n",
    "2.73 & 2.76 & 4.001 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "h2_{out1} & h2_{out2} &  h2_{out3} \\\\\\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    "0.938 & 0.94 & 0.98 \\\\\\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Layer-3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![forward3](Images/Forword_3.png \"Forword3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Operation:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "h2_{out1} & h2_{out2} & h2_{out3} \\\\\\end{array} \\right] \\times \\left[ \\begin{array}{cccc}\n",
    "W_{k1l1} & W_{k1l2} & W_{k1l3} \\\\\n",
    "W_{k2l1} & W_{k2l2} & W_{k2l3} \\\\\n",
    "W_{k3l1} & W_{k3l2} & W_{k3l3} \\\\ \\end{array} \\right] + \\left[ \\begin{array}{cccc}\n",
    "b_{l1} & b_{l2} & b_{l3} \\\\\\end{array} \\right]= \\left[ \\begin{array}{cccc}\n",
    "O_{in1} & O_{in2} & O_{in3} \\\\\\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "### Softmax operation \n",
    "\\begin{equation*}\n",
    "Softmax = \\mathrm{e}^{l_{ina}}/(\\sum_{a=1}^{3}\\mathrm{e}^{O_{ina}}) \n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "O_{out1} & O_{out2} & O_{out3} \\\\\\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    "\\mathrm{e}^{O_{in1}}/(\\sum_{a=1}^{3}\\mathrm{e}^{O_{ina}})   & \\mathrm{e}^{O_{in2}}/(\\sum_{a=1}^{3}\\mathrm{e}^{O_{ina}})   & \\mathrm{e}^{O_{in3}}/(\\sum_{a=1}^{3}\\mathrm{e}^{O_{ina}})   \\\\\\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "## Example:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "0.938 & 0.94 & 0.98 \\\\ \\end{array} \\right] \\times \\left[ \\begin{array}{cccc}\n",
    "0.1 & 0.4 & 0.8 \\\\\n",
    "0.3 & 0.7 & 0.2 \\\\\n",
    "0.5 & 0.2 & 0.9 \\\\ \\end{array} \\right]  + \\left[ \\begin{array}{cccc}\n",
    "1.0 & 1.0 & 1.0 \\\\\\end{array} \\right] =  \\left[\\begin{array}{cccc}\n",
    "1.8658 & 2.2292 & 2.8204 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "O_{out1} & O_{out2} &  O_{out3} \\\\\\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    "0.2698 & 0.3223 & 0.4078 \\\\\\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The Actual Output should be [1.0, 0.0, 0.0] but we got [0.2698, 0.3223, 0.4078]. \n",
    "- To calculate error lets use cross-entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-Entropy Error \n",
    "\\begin{equation*}\n",
    "cross entropy = - (1/n)(\\sum_{i=1}^{3} (y_{i} \\times \\log(O_{outi})) + ((1-y_{i}) \\times \\log((1-O_{outi}))))\n",
    "\\end{equation*}\n",
    "\n",
    "## Example \n",
    "\\begin{equation*}\n",
    "Error = - ((1*\\log(0.2698) + 0 + 0*\\log(0.3223) + 1 * \\log(1- 0.3223)+ 0*\\log(0.4078) + 1 * \\log(1 - 0.4078))\n",
    "\\end{equation*}\n",
    "\\begin{equation*}\n",
    "Error = - \\log(0.2698) - \\log(0.6777) - \\log(0.5922)\n",
    "\\end{equation*}\n",
    "\\begin{equation*}\n",
    "Error = + 0.569858 + 0.16886 + 0.22753 = 0.985\n",
    "\\end{equation*}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important Derivatives \n",
    "\n",
    "### Sigmoid\n",
    "\\begin{equation*}\n",
    "Sigmoid = 1/(1+\\mathrm{e}^{-x}) \n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial (1/(1+\\mathrm{e}^{-x}))}{\\partial x} = 1/(1+\\mathrm{e}^{-x}) \\times (1- 1/(1+\\mathrm{e}^{-x}))\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial Sigmoid}{\\partial x} = Sigmoid \\times (1- Sigmoid)\n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "### Relu\n",
    "\\begin{equation*}\n",
    "relu = max(0, x) \n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\ if x >0 , \\frac{\\partial (relu)}{\\partial x} = 1\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\ Otherwise , \\frac{\\partial (relu)}{\\partial x} = 0\n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "### Softmax operation \n",
    "\\begin{equation*}\n",
    "Softmax = \\mathrm{e}^{x_{a}}/(\\sum_{a=1}^{n}\\mathrm{e}^{x_{a}}) = \\mathrm{e}^{x_{1}}/(\\mathrm{e}^{x_{1}} + \\mathrm{e}^{x_{2}} + \\mathrm{e}^{x_{3}}) \n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial (Softmax)}{\\partial x_{1}} = (\\mathrm{e}^{x_{1}} \\times (\\mathrm{e}^{x_{2}}+\\mathrm{e}^{x_{3}}))/ (\\mathrm{e}^{x_{1}}+\\mathrm{e}^{x_{2}}+\\mathrm{e}^{x_{3}})^2\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BackPropagating the error - (Hidden Layer2 - Output Layer) Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BP_layer3](Images/BP_layer3.png \"BP_layer3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets calculate a few derviates upfront so these become handy and we can reuse them whenever necessary. \n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{total}}{\\partial O_{out1}} = \\frac{\\partial (-1 * ((y_{1} * \\log(O_{out1}) + (1-y_{1}) * \\log((1-O_{out1}))}{\\partial O_{out1}}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Here are we are using only one example (batch_size=1), if there are more examples then just average everything.\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{1}}{\\partial O_{out1}} =  -1 *  ((y_{1} * (1/O_{out1}) + (1-y_{1})*(1/(1-O_{out1}))\n",
    "\\end{equation*}\n",
    "\n",
    "#### by symmetry, We can calculate other derviatives also \n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{1}}{\\partial O_{out1}} \\\\\n",
    "\\frac{\\partial E_{2}}{\\partial O_{out2}}  \\\\\n",
    "\\frac{\\partial E_{3}}{\\partial O_{out3}}\\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "-1  * ((y_{1} * (1/O_{out1}) + (1-y_{1})*(1/(1-O_{out1})) \\\\\n",
    "-1  * ((y_{2} * (1/O_{out2}) + (1-y_{2})*(1/(1-O_{out2})) \\\\\n",
    "-1  * ((y_{3} * (1/O_{out3}) + (1-y_{3})*(1/(1-O_{out3})) \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "### In our Example The values will be\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{1}}{\\partial O_{out1}} \\\\\n",
    "\\frac{\\partial E_{2}}{\\partial O_{out2}}   \\\\\n",
    "\\frac{\\partial E_{3}}{\\partial O_{out3}}  \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " -3.70644\\\\\n",
    " -1.4755\\\\\n",
    " -1.6886\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next let us calculate the derviative of each output with respect to their input.\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial O_{out1}}{\\partial O_{in1}} = \\frac{\\partial (\\mathrm{e}^{O_{in1}}/(\\mathrm{e}^{O_{in1}} + \\mathrm{e}^{O_{in2}} + \\mathrm{e}^{O_{in3}}))}{\\partial O_{in1}}  \n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial O_{out1}}{\\partial O_{in1}} = (\\mathrm{e}^{O_{in1}} \\times (\\mathrm{e}^{O_{in2}}+\\mathrm{e}^{O_{in3}}))/ (\\mathrm{e}^{O_{in1}}+\\mathrm{e}^{O_{in2}}+\\mathrm{e}^{O_{in3}})^2\n",
    "\\end{equation*}\n",
    "\n",
    "#### by symmetry, We can calculate other derviatives also \n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial O_{out1}}{\\partial O_{in1}} \\\\\n",
    "\\frac{\\partial O_{out2}}{\\partial O_{in2}}  \\\\\n",
    "\\frac{\\partial O_{out3}}{\\partial O_{in3}}\\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "(\\mathrm{e}^{O_{in1}} \\times (\\mathrm{e}^{O_{in2}}+\\mathrm{e}^{O_{in3}}))/ (\\mathrm{e}^{O_{in1}}+\\mathrm{e}^{O_{in2}}+\\mathrm{e}^{O_{in3}})^2 \\\\\n",
    "(\\mathrm{e}^{O_{in2}} \\times (\\mathrm{e}^{O_{in1}}+\\mathrm{e}^{O_{in3}}))/ (\\mathrm{e}^{O_{in1}}+\\mathrm{e}^{O_{in2}}+\\mathrm{e}^{O_{in3}})^2 \\\\\n",
    "(\\mathrm{e}^{O_{in3}} \\times (\\mathrm{e}^{O_{in1}}+\\mathrm{e}^{O_{in2}}))/ (\\mathrm{e}^{O_{in1}}+\\mathrm{e}^{O_{in2}}+\\mathrm{e}^{O_{in3}})^2 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "### In our Example The values will be\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial O_{out1}}{\\partial O_{in1}} \\\\\n",
    "\\frac{\\partial O_{out2}}{\\partial O_{in2}}  \\\\\n",
    "\\frac{\\partial O_{out3}}{\\partial O_{in3}}\\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " 0.15911\\\\\n",
    " 0.2040\\\\\n",
    " 0.3685\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For each input to neuron lets calculate the derivative with respect to each weight.\n",
    "\n",
    "##### Now let us look at the final derivative\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial O_{in1}}{\\partial W_{k1l1}} = \\frac{\\partial ((h2_{out1}*W_{j1k1}) + (h2_{out2} * W{j2k1}) + (h2_{out3} * W{j3k1}) + b_{l1})}{\\partial W_{k1l1}}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Now let us look at the final derivative\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial O_{in1}}{\\partial W_{k1l1}} = h2_{out1}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Using similarity we can write:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial O_{in1}}{\\partial W_{k1l1}}  \\\\\n",
    "\\frac{\\partial O_{in1}}{\\partial W_{k2l1}}   \\\\\n",
    "\\frac{\\partial O_{in1}}{\\partial W_{k3l1}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " h2_{out1}\\\\\n",
    " h2_{out2}\\\\\n",
    " h2_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 0.938\\\\\n",
    " 0.94\\\\\n",
    " 0.98\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial O_{in2}}{\\partial W_{k1l2}}  \\\\\n",
    "\\frac{\\partial O_{in2}}{\\partial W_{k2l2}}   \\\\\n",
    "\\frac{\\partial O_{in2}}{\\partial W_{k3l2}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " h2_{out1}\\\\\n",
    " h2_{out2}\\\\\n",
    " h2_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 0.938\\\\\n",
    " 0.94\\\\\n",
    " 0.98\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial O_{in3}}{\\partial W_{k1l3}}  \\\\\n",
    "\\frac{\\partial O_{in3}}{\\partial W_{k2l3}}   \\\\\n",
    "\\frac{\\partial O_{in3}}{\\partial W_{k3l3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " h2_{out1}\\\\\n",
    " h2_{out2}\\\\\n",
    " h2_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 0.938\\\\\n",
    " 0.94\\\\\n",
    " 0.98\\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now we will calulate the change in \n",
    "\\begin{equation*}\n",
    "W_{k1l1}\n",
    "\\end{equation*}\n",
    "\n",
    "#### This will be simply\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{1}}{\\partial W_{k1l1}}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Using chain rule:\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{1}}{\\partial W_{k1l1}} = \\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O\n",
    "_{out1}}{\\partial O_{in1}} * \\frac{\\partial O_{in1}}{\\partial W_{k1l1}}\n",
    "\\end{equation*}\n",
    "\n",
    "### By symmetry\n",
    "\\begin{equation*}\n",
    "\\delta W_{kl} = \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{1}}{\\partial W_{k1l1}} & \\frac{\\partial E_{2}}{\\partial W_{k1l2}} & \\frac{\\partial E_{3}}{\\partial W_{k1l3}} \\\\\n",
    "\\frac{\\partial E_{1}}{\\partial W_{k2l1}} & \\frac{\\partial E_{2}}{\\partial W_{k2l2}} & \\frac{\\partial E_{3}}{\\partial W_{k2l3}} \\\\\n",
    "\\frac{\\partial E_{1}}{\\partial W_{k3l1}} & \\frac{\\partial E_{2}}{\\partial W_{k3l2}} & \\frac{\\partial E_{3}}{\\partial W_{k3l3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O\n",
    "_{out1}}{\\partial O_{in1}} * \\frac{\\partial O_{in1}}{\\partial W_{k1l1}} & \\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O\n",
    "_{out2}}{\\partial O_{in2}} * \\frac{\\partial O_{in2}}{\\partial W_{k1l2}}& \\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O\n",
    "_{out3}}{\\partial O_{in3}} * \\frac{\\partial O_{in3}}{\\partial W_{k1l3}} \\\\\n",
    "\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O\n",
    "_{out1}}{\\partial O_{in1}} * \\frac{\\partial O_{in1}}{\\partial W_{k2l1}}& \\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O\n",
    "_{out2}}{\\partial O_{in2}} * \\frac{\\partial O_{in2}}{\\partial W_{k2l2}} & \\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O\n",
    "_{out3}}{\\partial O_{in3}} * \\frac{\\partial O_{in3}}{\\partial W_{k2l3}} \\\\\n",
    "\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O\n",
    "_{out1}}{\\partial O_{in1}} * \\frac{\\partial O_{in1}}{\\partial W_{k3l1}} & \\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O\n",
    "_{out2}}{\\partial O_{in2}} * \\frac{\\partial O_{in2}}{\\partial W_{k3l2}} & \\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O\n",
    "_{out3}}{\\partial O_{in3}} * \\frac{\\partial O_{in3}}{\\partial W_{k3l3}} \\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### All the above values are calculated above, We just need to substitute these values \n",
    "\\begin{equation*}\n",
    "\\delta W_{kl} = \\left[ \\begin{array}{cccc}\n",
    "\\delta W_{k1l1} & \\delta W_{k1l2} & \\delta W_{k1l3} \\\\\n",
    "\\delta W_{k2l1} & \\delta W_{k2l2} & \\delta W_{k2l3} \\\\\n",
    "\\delta W_{k3l1} & \\delta W_{k3l2} & \\delta W_{k3l3} \\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    "-3.7064 *0.1591 * 0.938&-0.301 * 0.204 * 0.938&-1.6886 * 0.3685 * 0.938 \\\\\n",
    "-3.7064 * 0.1591 * 0.94&-0.301 * 0.204 * 0.94&-1.6886 * 0.3685 * 0.94 \\\\\n",
    "-3.7064 * 0.1591* 0.98&-0.301  * 0.204 * 0.98&-1.6886  * 0.3685 * 0.98 \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "-0.5531 & -0.0576 & -0.5836\\\\\n",
    "-0.554347 & -0.0577 & -0.5849 \\\\\n",
    "-0.577937 & -0.06017 & -0.6098 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Consider a learning rate (lr) of 0.01 We get our final Weight matrix as \n",
    "\\begin{equation*}\n",
    "\\acute{W_{kl}} = \\left[ \\begin{array}{cccc}\n",
    "W_{k1l1} - (lr*\\delta W_{k1l1}) & W_{k1l2} - (lr * \\delta W_{k1l2}) &W_{k1l3} - (lr * \\delta W_{k1l3}) \\\\\n",
    "W_{k2l1} - (lr* \\delta W_{k2l1}) & W_{k2l2} - (lr * \\delta W_{k2l2}) &W_{k2l3} - (lr * \\delta W_{k2l3}) \\\\\n",
    "W_{k3l1} - (lr * \\delta W_{k3l1}) & W_{k3l2} - (lr *\\delta W_{k3l2}) & W_{k3l3} - (lr * \\delta W_{k3l3}) \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\acute{W_{kl}} = \\left[ \\begin{array}{cccc}\n",
    "0.1 - (0.01*-0.5531) & 0.4 - (0.01 * -0.0576) & 0.8 - (0.01 * -0.5836) \\\\\n",
    "0.3 - (0.01* -0.554347) & 0.7 - (0.01* -0.0577) &0.2 - (0.01* -0.5849) \\\\\n",
    "0.5 - (0.01 * -0.577937) & 0.2 - (0.01 -0.06017) & 0.9 - (0.01 * -0.6098) \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\acute{W_{kl}} = \\left[ \\begin{array}{cccc}\n",
    "0.105531 & 0.400576 & 0.805836 \\\\\n",
    "0.30055 & 0.700577 &0.2005849 \\\\\n",
    "0.5005779 & 0.2006017 & 0.9006098 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "#### Finally We made it, Lets jump to the next layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BackPropagating the error - (Hidden Layer1 - Hidden Layer2)  Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BP_layer2](Images/BP_layer2.png \"BP_layer2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets calculate a few handy derviatives before we actually calculate the error derviatives wrt Weights in this layer. \n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial h2_{out1}}{\\partial h2_{in1}} = \\frac{\\partial Sigmoid(h2_{in1})}{\\partial h2_{in1}}\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial h2_{out1}}{\\partial h2_{in1}} = Sigmoid(h2_{in1}) * (1 - Sigmoid(h2_{in1}))\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h2_{out1}}{\\partial h2_{in1}} \\\\\n",
    "\\frac{\\partial h2_{out2}}{\\partial h2_{in2}}   \\\\\n",
    "\\frac{\\partial h2_{out3}}{\\partial h2_{in3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " Sigmoid(h2_{in1}) * (1 - Sigmoid(h2_{in1}))\\\\\n",
    "Sigmoid(h2_{in1}) * (1 - Sigmoid(h2_{in1}))\\\\\n",
    "Sigmoid(h2_{in1}) * (1 - Sigmoid(h2_{in1}))\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "#### In our example , this will be \n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h2_{out1}}{\\partial h2_{in1}} \\\\\n",
    "\\frac{\\partial h2_{out2}}{\\partial h2_{in2}}   \\\\\n",
    "\\frac{\\partial h2_{out3}}{\\partial h2_{in3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " Sigmoid(2.73) * (1 - Sigmoid(2.73))\\\\\n",
    "Sigmoid(2.76) * (1 - Sigmoid(2.76))\\\\\n",
    "Sigmoid(4.001) * (1 - Sigmoid(4.001))\\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " 0.058156\\\\\n",
    "0.0564\\\\\n",
    "0.0196\\\\ \\end{array} \\right] \n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For each input to neuron lets calculate the derivative with respect to each weight.\n",
    "\n",
    "##### Now let us look at the final derivative\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial h2_{in1}}{\\partial W_{j1k1}} = \\frac{\\partial ((h1_{out1}*W_{j1k1}) + (h1_{out2} * W{j2k1}) + (h1_{out3} * W{j3k1}) + b_{k1})}{\\partial W_{j1k1}}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Now let us look at the final derivative\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial h2_{in1}}{\\partial W_{j1k1}} = h1_{out1}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Using similarity we can write:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h2_{in1}}{\\partial W_{j1k1}}  \\\\\n",
    "\\frac{\\partial h2_{in1}}{\\partial W_{j2k1}}   \\\\\n",
    "\\frac{\\partial h2_{in1}}{\\partial W_{j3k1}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " h1_{out1}\\\\\n",
    " h1_{out2}\\\\\n",
    " h1_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 1.35\\\\\n",
    " 1.27\\\\\n",
    " 1.8\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h2_{in2}}{\\partial W_{j1k2}}  \\\\\n",
    "\\frac{\\partial h2_{in2}}{\\partial W_{j2k2}}   \\\\\n",
    "\\frac{\\partial h2_{in2}}{\\partial W_{j3k2}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " h1_{out1}\\\\\n",
    " h1_{out2}\\\\\n",
    " h1_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 1.35\\\\\n",
    " 1.27\\\\\n",
    " 1.8\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h2_{in3}}{\\partial W_{j1k3}}  \\\\\n",
    "\\frac{\\partial h2_{in3}}{\\partial W_{j2k3}}   \\\\\n",
    "\\frac{\\partial h2_{in3}}{\\partial W_{j3k3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " h1_{out1}\\\\\n",
    " h1_{out2}\\\\\n",
    " h1_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 1.35\\\\\n",
    " 1.27\\\\\n",
    " 1.8\\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Now we will calulate the change in \n",
    "\\begin{equation*}\n",
    "W_{j3k1}\n",
    "\\end{equation*} \n",
    "##### and generalize it for all other variables.\n",
    "\n",
    "#### Caution: Make sure that you have understood everything we discussed till here. \n",
    "\n",
    "##### This will be simply\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{total}}{\\partial W_{j3k1}}\n",
    "\\end{equation*}\n",
    "\n",
    "##### Using chain rule:\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{total}}{\\partial W_{j3k1}} = \\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2\n",
    "_{out1}}{\\partial h2_{in1}} * \\frac{\\partial h2_{in1}}{\\partial W_{j3k1}}\n",
    "\\end{equation*}\n",
    "\n",
    "##### Now we will see each and every equation individually.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets look at the matrix\n",
    "### By symmetry\n",
    "\\begin{equation*}\n",
    "\\delta W_{jk} = \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial W_{j1k1}} & \\frac{\\partial E_{total}}{\\partial W_{j1k2}} & \\frac{\\partial E_{total}}{\\partial W_{j1k3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial W_{j2k1}} & \\frac{\\partial E_{total}}{\\partial W_{j2k2}} & \\frac{\\partial E_{total}}{\\partial W_{j2k3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial W_{j3k1}} & \\frac{\\partial E_{total}}{\\partial W_{j3k2}} & \\frac{\\partial E_{total}}{\\partial W_{j3k3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2_{out1}}{\\partial h2_{in1}} * \\frac{\\partial h2_{in1}}{\\partial W_{j1k1}} & \\frac{\\partial E_{total}}{\\partial h2_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h2_{in2}} * \\frac{\\partial h2_{in2}}{\\partial W_{j1k2}}& \\frac{\\partial E_{total}}{\\partial h2_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h2_{in3}} * \\frac{\\partial h2_{in3}}{\\partial W_{j1k3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2\n",
    "_{out1}}{\\partial h2_{in1}} * \\frac{\\partial h2_{in1}}{\\partial W_{j2k1}}& \\frac{\\partial E_{total}}{\\partial h2_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h2_{in2}} * \\frac{\\partial h2_{in2}}{\\partial W_{j2k2}} & \\frac{\\partial E_{total}}{\\partial h2_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h2_{in3}} * \\frac{\\partial h2_{in3}}{\\partial W_{j2k3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2\n",
    "_{out1}}{\\partial h2_{in1}} * \\frac{\\partial h2_{in1}}{\\partial W_{j3k1}} & \\frac{\\partial E_{total}}{\\partial h2_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h2_{in2}} * \\frac{\\partial h2_{in2}}{\\partial W_{j3k2}} & \\frac{\\partial E_{total}}{\\partial h2_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h2_{in3}} * \\frac{\\partial h2_{in3}}{\\partial W_{j3k3}} \\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We have Already calculated the 2nd and 3rd term in each matrix. We need to check on the 1st term. If we see the matrix, the first term is common in all the columns. So there are only three values. Lets look into one value\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}} = \\frac{\\partial E_{1}}{\\partial h2_{out1}} + \\frac{\\partial E_{2}}{\\partial h2_{out1}} + \\frac{\\partial E_{3}}{\\partial h2_{out1}}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Lets see what each individual term boils down too.\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{1}}{\\partial h2_{out1}} = \\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O_{out1}}{\\partial O_{in1}} * \\frac{\\partial O_{in1}}{\\partial h2_{out1}}\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{2}}{\\partial h2_{out1}} = \\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O_{out2}}{\\partial O_{in2}} * \\frac{\\partial O_{in2}}{\\partial h2_{out1}}\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{3}}{\\partial h2_{out1}} = \\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O_{out3}}{\\partial O_{in3}} * \\frac{\\partial O_{in3}}{\\partial h2_{out1}}\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BY symmentry\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}}  \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out2}}   \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "(\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O_{out1}}{\\partial O_{in1}} * \\frac{\\partial O_{in1}}{\\partial h2_{out1}}) + (\\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O_{out2}}{\\partial O_{in2}} * \\frac{\\partial O_{in2}}{\\partial h2_{out1}}) + ( \\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O_{out3}}{\\partial O_{in3}} * \\frac{\\partial O_{in3}}{\\partial h2_{out1}})\\\\\n",
    "(\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O_{out1}}{\\partial O_{in1}} * \\frac{\\partial O_{in1}}{\\partial h2_{out2}}) + (\\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O_{out2}}{\\partial O_{in2}} * \\frac{\\partial O_{in2}}{\\partial h2_{out2}}) + (\\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O_{out3}}{\\partial O_{in3}} * \\frac{\\partial O_{in3}}{\\partial h2_{out2}})\\\\\n",
    "(\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O_{out1}}{\\partial O_{in1}} * \\frac{\\partial O_{in1}}{\\partial h2_{out3}}) + (\\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O_{out2}}{\\partial O_{in2}} * \\frac{\\partial O_{in2}}{\\partial h2_{out3}}) + (\\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O_{out3}}{\\partial O_{in3}} * \\frac{\\partial O_{in3}}{\\partial h2_{out3}})\\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Again the first two values are already calculated by us when dealing with derviatives of W_{kl}. We just need to calculate the third one, Which is the derivative of  input to each output layer wrt output of hidden layer-2. It is nothing but the corresponding weight which connects both the layers.\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial O_{inl}}{\\partial h2_{out1}} & \\frac{\\partial O_{in2}}{\\partial h2_{out1}} & \\frac{\\partial O_{in3}}{\\partial h2_{out1}} \\\\\n",
    "\\frac{\\partial O_{inl}}{\\partial h2_{out2}} & \\frac{\\partial O_{in2}}{\\partial h2_{out2}} & \\frac{\\partial O_{in3}}{\\partial h2_{out2}}\\\\\n",
    "\\frac{\\partial O_{inl}}{\\partial h2_{out3}} & \\frac{\\partial O_{in2}}{\\partial h2_{out3}} & \\frac{\\partial O_{in3}}{\\partial h2_{out3}} \\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    "W_{k1l1} & W_{k1l2} & W_{k1l3} \\\\\n",
    "W_{k2l1} & W_{k2l2} & W_{k2l3} \\\\\n",
    "W_{k3l1} & W_{k3l2} & W_{k3l3} \\\\ \\end{array} \\right] \n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}}  \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out2}}   \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "(\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O_{out1}}{\\partial O_{in1}} * W_{k1l1}) + (\\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O_{out2}}{\\partial O_{in2}} * W_{k1l2}) + ( \\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O_{out3}}{\\partial O_{in3}} * W_{k1l3})\\\\\n",
    "(\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O_{out1}}{\\partial O_{in1}} * W_{k2l1}) + (\\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O_{out2}}{\\partial O_{in2}} * W_{k2l2}) + ( \\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O_{out3}}{\\partial O_{in3}} * W_{k2l3})\\\\\n",
    "(\\frac{\\partial E_{1}}{\\partial O_{out1}} * \\frac{\\partial O_{out1}}{\\partial O_{in1}} * W_{k3l1}) + (\\frac{\\partial E_{2}}{\\partial O_{out2}} * \\frac{\\partial O_{out2}}{\\partial O_{in2}} * W_{k3l2}) + ( \\frac{\\partial E_{3}}{\\partial O_{out3}} * \\frac{\\partial O_{out3}}{\\partial O_{in3}} * W_{k2l3})\\\\\n",
    "\\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### All Values are calculated before we just need to impute the corresponding values for our example.\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}}  \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out2}}   \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "(-3.70644 *0.15911 * 0.1) + (-1.4755 * 0.2040 *0.4) + ( -1.6886 * 0.3685 * 0.8)\\\\\n",
    "(-3.70644 * 0.15911 * 0.3) + (-1.4755 * 0.2040 * 0.7) + ( -1.6886 * 0.3685 * 0.2)\\\\\n",
    "(-3.70644 * 0.15911  * 0.5) + (-1.4755 * 0.2040 * 0.2) + ( -1.6886 * 0.3685 * 0.9)\\\\\n",
    "\\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "(-0.0589) + (-0.2383) + ( -0.5931)\\\\\n",
    "(-0.1769) + (-0.417) + ( -0.14828)\\\\\n",
    "(-0.2948) + (-0.119) + ( -0.667)\\\\\n",
    "\\end{array} \\right] =\n",
    "\\left[ \\begin{array}{cccc}\n",
    "-0.8903\\\\\n",
    "-0.74218\\\\\n",
    "-1.0810\\\\\n",
    "\\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets look at the matrix\n",
    "### By symmetry\n",
    "\\begin{equation*}\n",
    "\\delta W_{jk} = \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial W_{j1k1}} & \\frac{\\partial E_{total}}{\\partial W_{j1k2}} & \\frac{\\partial E_{total}}{\\partial W_{j1k3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial W_{j2k1}} & \\frac{\\partial E_{total}}{\\partial W_{j2k2}} & \\frac{\\partial E_{total}}{\\partial W_{j2k3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial W_{j3k1}} & \\frac{\\partial E_{total}}{\\partial W_{j3k2}} & \\frac{\\partial E_{total}}{\\partial W_{j3k3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2_{out1}}{\\partial h2_{in1}} * \\frac{\\partial h2_{in1}}{\\partial W_{j1k1}} & \\frac{\\partial E_{total}}{\\partial h2_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h2_{in2}} * \\frac{\\partial h2_{in2}}{\\partial W_{j1k2}}& \\frac{\\partial E_{total}}{\\partial h2_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h2_{in3}} * \\frac{\\partial h2_{in3}}{\\partial W_{j1k3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2\n",
    "_{out1}}{\\partial h2_{in1}} * \\frac{\\partial h2_{in1}}{\\partial W_{j2k1}}& \\frac{\\partial E_{total}}{\\partial h2_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h2_{in2}} * \\frac{\\partial h2_{in2}}{\\partial W_{j2k2}} & \\frac{\\partial E_{total}}{\\partial h2_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h2_{in3}} * \\frac{\\partial h2_{in3}}{\\partial W_{j2k3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2\n",
    "_{out1}}{\\partial h2_{in1}} * \\frac{\\partial h2_{in1}}{\\partial W_{j3k1}} & \\frac{\\partial E_{total}}{\\partial h2_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h2_{in2}} * \\frac{\\partial h2_{in2}}{\\partial W_{j3k2}} & \\frac{\\partial E_{total}}{\\partial h2_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h2_{in3}} * \\frac{\\partial h2_{in3}}{\\partial W_{j3k3}} \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\delta W_{jk} =  \\left[ \\begin{array}{cccc}\n",
    "-0.8903 * 0.058156 * 1.35 & -0.74218 * 0.0564 * 1.35 & -1.0810 * 0.0196 * 1.35 \\\\\n",
    "-0.8903 * 0.058156  * 1.27 & -0.74218 * 0.0564 * 1.27 & -1.0810 * 0.0196 * 1.27 \\\\\n",
    "-0.8903 * 0.058156 * 1.8 & -0.74218 * 0.0564 * 1.8 & -1.0810 * 0.0196 * 1.8 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\delta W_{jk} =  \\left[ \\begin{array}{cccc}\n",
    "-0.06989 & -0.0565 & -0.0286 \\\\\n",
    "-0.06575 & -0.05316 & -0.0269 \\\\\n",
    "-0.0932 & -0.0753 & -0.03813 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Consider a learning rate (lr) of 0.01 We get our final Weight matrix as \n",
    "\\begin{equation*}\n",
    "\\acute{W_{jk}} = \\left[ \\begin{array}{cccc}\n",
    "W_{j1k1} - (lr*\\delta W_{j1k1}) & W_{j1k2} - (lr * \\delta W_{j1k2}) &W_{j1k3} - (lr * \\delta W_{j1k3}) \\\\\n",
    "W_{j2k1} - (lr* \\delta W_{j2k1}) & W_{j2k2} - (lr * \\delta W_{j2k2}) &W_{j2k3} - (lr * \\delta W_{j2k3}) \\\\\n",
    "W_{j3k1} - (lr * \\delta W_{j3k1}) & W_{j3k2} - (lr *\\delta W_{j3k2}) & W_{j3k3} - (lr * \\delta W_{j3k3}) \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\acute{W_{jk}} = \\left[ \\begin{array}{cccc}\n",
    "0.2 - (0.01*-0.06989) & 0.3 - (0.01 * -0.0565) & 0.5 - (0.01 * -0.0286) \\\\\n",
    "0.3 - (0.01* -0.06575) & 0.5 - (0.01* -0.05316) &0.7 - (0.01* -0.0269) \\\\\n",
    "0.6 - (0.01 * -0.0932) & 0.4 - (0.01 -0.0753) & 0.8 - (0.01 * -0.03813) \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\acute{W_{jk}} = \\left[ \\begin{array}{cccc}\n",
    "0.2006989 & 0.300565 & 0.500286 \\\\\n",
    "0.3006575 & 0.5005316 &0.700269 \\\\\n",
    "0.600932 & 0.400753 & 0.803813 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "#### Finally We made it, Lets jump to the next layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BackPropagating the error - (Input Layer - Hidden Layer1)  Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BP_layer1](Images/BP_layer1.png \"BP_layer1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets calculate a few handy derviatives before we actually calculate the error derviatives wrt Weights in this layer. \n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial h1_{out1}}{\\partial h1_{in1}} = \\frac{\\partial Relu(h1_{in1})}{\\partial h1_{in1}}\n",
    "\\end{equation*}\n",
    "\n",
    "#### We already know \n",
    "### Relu\n",
    "\\begin{equation*}\n",
    "relu = max(0, x) \n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\ if x >0 , \\frac{\\partial (relu)}{\\partial x} = 1\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\ Otherwise , \\frac{\\partial (relu)}{\\partial x} = 0\n",
    "\\end{equation*}\n",
    "\n",
    "### Since the inputs are positive\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial h1_{out1}}{\\partial h1_{in1}} = 1.0\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h1_{out1}}{\\partial h1_{in1}} \\\\\n",
    "\\frac{\\partial h1_{out2}}{\\partial h1_{in2}}   \\\\\n",
    "\\frac{\\partial h1_{out3}}{\\partial h1_{in3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " 1.0\\\\\n",
    "1.0\\\\\n",
    "1.0\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For each input to neuron lets calculate the derivative with respect to each weight.\n",
    "\n",
    "#### Now let us look at the final derivative\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial h1_{in1}}{\\partial W_{i1j1}} = \\frac{\\partial ((I_{out1}*W_{j1k1}) + (I_{out2} * W{i2j1}) + (I_{out3} * W{i3j1}) + b_{j1})}{\\partial W_{i1j1}}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Now let us look at the final derivative\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial h1_{in1}}{\\partial W_{i1j1}} = I_{out1}\n",
    "\\end{equation*}\n",
    "\n",
    "#### Using similarity we can write:\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h1_{in1}}{\\partial W_{i1j1}}  \\\\\n",
    "\\frac{\\partial h1_{in1}}{\\partial W_{i2j1}}   \\\\\n",
    "\\frac{\\partial h1_{in1}}{\\partial W_{i3j1}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " I_{out1}\\\\\n",
    " I_{out2}\\\\\n",
    " I_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 0.1\\\\\n",
    " 0.2\\\\\n",
    " 0.7\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h1_{in2}}{\\partial W_{i1j2}}  \\\\\n",
    "\\frac{\\partial h1_{in2}}{\\partial W_{i2j2}}   \\\\\n",
    "\\frac{\\partial h1_{in2}}{\\partial W_{i3j2}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " I_{out1}\\\\\n",
    " I_{out2}\\\\\n",
    " I_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 0.1\\\\\n",
    " 0.2\\\\\n",
    " 0.7\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial h1_{in3}}{\\partial W_{i1j3}}  \\\\\n",
    "\\frac{\\partial h1_{in3}}{\\partial W_{i2j3}}   \\\\\n",
    "\\frac{\\partial h1_{in3}}{\\partial W_{i3j3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    " I_{out1}\\\\\n",
    " I_{out2}\\\\\n",
    " I_{out3}\\\\ \\end{array} \\right] = \\left[ \\begin{array}{cccc}\n",
    " 0.1\\\\\n",
    " 0.2\\\\\n",
    " 0.7\\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Now we will calulate the change in \n",
    "\\begin{equation*}\n",
    "W_{j3k1}\n",
    "\\end{equation*} \n",
    "\n",
    "##### and generalize it for all other variables.\n",
    "\n",
    "#### Caution: Make sure that you have understood everything we discussed till here. \n",
    "\n",
    "##### This will be simply\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{total}}{\\partial W_{i2j1}}\n",
    "\\end{equation*}\n",
    "\n",
    "##### Using chain rule:\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial E_{total}}{\\partial W_{i2j1}} = \\frac{\\partial E_{total}}{\\partial h1_{out1}} * \\frac{\\partial h1\n",
    "_{out1}}{\\partial h1_{in1}} * \\frac{\\partial h1_{in1}}{\\partial W_{i2j1}}\n",
    "\\end{equation*}\n",
    "\n",
    "##### Now we will see each and every equation individually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets look at the matrix\n",
    "### By symmetry\n",
    "\\begin{equation*}\n",
    "\\delta W_{ij} = \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial W_{i1j1}} & \\frac{\\partial E_{total}}{\\partial W_{i1j2}} & \\frac{\\partial E_{total}}{\\partial W_{i1j3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial W_{i2j1}} & \\frac{\\partial E_{total}}{\\partial W_{i2j2}} & \\frac{\\partial E_{total}}{\\partial W_{i2j3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial W_{i3j1}} & \\frac{\\partial E_{total}}{\\partial W_{i3j2}} & \\frac{\\partial E_{total}}{\\partial W_{i3j3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}} * \\frac{\\partial h1_{out1}}{\\partial h1_{in1}} * \\frac{\\partial h1_{in1}}{\\partial W_{i1j1}} & \\frac{\\partial E_{total}}{\\partial h1_{out2}} * \\frac{\\partial h1\n",
    "_{out2}}{\\partial h1_{in2}} * \\frac{\\partial h1_{in2}}{\\partial W_{i1j2}}& \\frac{\\partial E_{total}}{\\partial h1_{out3}} * \\frac{\\partial h1\n",
    "_{out3}}{\\partial h1_{in3}} * \\frac{\\partial h1_{in3}}{\\partial W_{i1j3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}} * \\frac{\\partial h1_{out1}}{\\partial h1_{in1}} * \\frac{\\partial h1_{in1}}{\\partial W_{i2j1}}& \\frac{\\partial E_{total}}{\\partial h1_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h1_{in2}} * \\frac{\\partial h1_{in2}}{\\partial W_{i2j2}} & \\frac{\\partial E_{total}}{\\partial h1_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h1_{in3}} * \\frac{\\partial h1_{in3}}{\\partial W_{i2j3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}} * \\frac{\\partial h1_{out1}}{\\partial h1_{in1}} * \\frac{\\partial h1_{in1}}{\\partial W_{i3j1}} & \\frac{\\partial E_{total}}{\\partial h1_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h1_{in2}} * \\frac{\\partial h1_{in2}}{\\partial W_{i3j2}} & \\frac{\\partial E_{total}}{\\partial h1_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h1_{in3}} * \\frac{\\partial h1_{in3}}{\\partial W_{i3j3}} \\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We know the 2nd and 3rd derivatives in each cell in the above matrix. Lets look at how to get to derivative of 1st term in each cell.\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}}  \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out2}}   \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "(\\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2_{out1}}{\\partial h2_{in1}} * \\frac{\\partial h2_{in1}}{\\partial h1_{out1}}) \\\\\n",
    "(\\frac{\\partial E_{total}}{\\partial h2_{out2}} * \\frac{\\partial h2_{out2}}{\\partial h2_{in2}} * \\frac{\\partial h2_{in2}}{\\partial h1_{out2}})\\\\\n",
    "(\\frac{\\partial E_{total}}{\\partial h2_{out3}} * \\frac{\\partial h2_{out3}}{\\partial h2_{in3}} * \\frac{\\partial h2_{in3}}{\\partial h1_{out3}})\\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We have calculated all the values previously except the last one in each cell, which is a simple derivative of linear terms.\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}}  \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out2}}   \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "(\\frac{\\partial E_{total}}{\\partial h2_{out1}} * \\frac{\\partial h2_{out1}}{\\partial h2_{in1}} * W_{j1k1} \\\\\n",
    "(\\frac{\\partial E_{total}}{\\partial h2_{out2}} * \\frac{\\partial h2_{out2}}{\\partial h2_{in2}} * W_{j2k2}\\\\\n",
    "(\\frac{\\partial E_{total}}{\\partial h2_{out3}} * \\frac{\\partial h2_{out3}}{\\partial h2_{in3}} * W_{j3k3}\\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}}  \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out2}}   \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "-0.8903 *0.058156 * 0.2 \\\\\n",
    "-0.74218 * 0.0564 *0.5\\\\\n",
    "-1.0810 * 0.0196 * 0.8\\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "-0.01035 \\\\\n",
    "-0.0209\\\\\n",
    "-0.0169\\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets look at the matrix\n",
    "### By symmetry\n",
    "\\begin{equation*}\n",
    "\\delta W_{ij} = \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial W_{i1j1}} & \\frac{\\partial E_{total}}{\\partial W_{i1j2}} & \\frac{\\partial E_{total}}{\\partial W_{i1j3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial W_{i2j1}} & \\frac{\\partial E_{total}}{\\partial W_{i2j2}} & \\frac{\\partial E_{total}}{\\partial W_{i2j3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial W_{i3j1}} & \\frac{\\partial E_{total}}{\\partial W_{i3j2}} & \\frac{\\partial E_{total}}{\\partial W_{i3j3}} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}} * \\frac{\\partial h1_{out1}}{\\partial h1_{in1}} * \\frac{\\partial h1_{in1}}{\\partial W_{i1j1}} & \\frac{\\partial E_{total}}{\\partial h1_{out2}} * \\frac{\\partial h1\n",
    "_{out2}}{\\partial h1_{in2}} * \\frac{\\partial h1_{in2}}{\\partial W_{i1j2}}& \\frac{\\partial E_{total}}{\\partial h1_{out3}} * \\frac{\\partial h1\n",
    "_{out3}}{\\partial h1_{in3}} * \\frac{\\partial h1_{in3}}{\\partial W_{i1j3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}} * \\frac{\\partial h1_{out1}}{\\partial h1_{in1}} * \\frac{\\partial h1_{in1}}{\\partial W_{i2j1}}& \\frac{\\partial E_{total}}{\\partial h1_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h1_{in2}} * \\frac{\\partial h1_{in2}}{\\partial W_{i2j2}} & \\frac{\\partial E_{total}}{\\partial h1_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h1_{in3}} * \\frac{\\partial h1_{in3}}{\\partial W_{i2j3}} \\\\\n",
    "\\frac{\\partial E_{total}}{\\partial h1_{out1}} * \\frac{\\partial h1_{out1}}{\\partial h1_{in1}} * \\frac{\\partial h1_{in1}}{\\partial W_{i3j1}} & \\frac{\\partial E_{total}}{\\partial h1_{out2}} * \\frac{\\partial h2\n",
    "_{out2}}{\\partial h1_{in2}} * \\frac{\\partial h1_{in2}}{\\partial W_{i3j2}} & \\frac{\\partial E_{total}}{\\partial h1_{out3}} * \\frac{\\partial h2\n",
    "_{out3}}{\\partial h1_{in3}} * \\frac{\\partial h1_{in3}}{\\partial W_{i3j3}} \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\delta W_{ij} =  \\left[ \\begin{array}{cccc}\n",
    "-0.01035 * 1 * 0.1 & -0.0209 * 1 * 0.1 & -0.0169 * 1 * 0.1 \\\\\n",
    "-0.01035 * 1  * 0.2 & -0.0209* 1 * 0.2 & -0.0169 * 1 * 0.2 \\\\\n",
    "-0.01035 * 1 * 0.7& -0.0209 * 1 * 0.7 & -0.0169 * 1 * 0.7 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\delta W_{ij} =  \\left[ \\begin{array}{cccc}\n",
    "-0.001035 & -0.00209 & -0.00169 \\\\\n",
    "-0.00207 & -0.00418 & -0.00338 \\\\\n",
    "-0.007245 & -0.01463 & -0.01183 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Consider a learning rate (lr) of 0.01 We get our final Weight matrix as \n",
    "\\begin{equation*}\n",
    "\\acute{W_{ij}} = \\left[ \\begin{array}{cccc}\n",
    "W_{i1j1} - (lr*\\delta W_{i1j1}) & W_{i1j2} - (lr * \\delta W_{i1j2}) &W_{i1j3} - (lr * \\delta W_{i1j3}) \\\\\n",
    "W_{i2j1} - (lr* \\delta W_{i2j1}) & W_{i2j2} - (lr * \\delta W_{i2j2}) &W_{i2j3} - (lr * \\delta W_{i2j3}) \\\\\n",
    "W_{i3j1} - (lr * \\delta W_{i3j1}) & W_{i3j2} - (lr *\\delta W_{i3j2}) & W_{i3j3} - (lr * \\delta W_{i3j3}) \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\acute{W_{ij}} = \\left[ \\begin{array}{cccc}\n",
    "0.1 - (0.01*-0.001035) & 0.2 - (0.01 * -0.00209) & 0.3 - (0.01 * -0.00169) \\\\\n",
    "0.3 - (0.01* -0.00207) & 0.2 - (0.01* -0.00418) &0.7 - (0.01* -0.00338) \\\\\n",
    "0.4 - (0.01 * -0.0007245) & 0.3 - (0.01 -0.01463) & 0.9 - (0.01 * -0.01183) \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\acute{W_{ij}} = \\left[ \\begin{array}{cccc}\n",
    "0.10001035 & 0.2000209 & 0.3000169 \\\\\n",
    "0.3000207 & 0.2000418 &0.7000338 \\\\\n",
    "0.40007245 & 0.3001463 & 0.9001183 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "#### The End "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our Inital Weights \n",
    "\n",
    "\\begin{equation*}\n",
    "W_{ij} = \\left[ \\begin{array}{cccc}\n",
    "W_{i1j1} & W_{i1j2} & W_{i1j3} \\\\\n",
    "W_{i2j1} & W_{i2j2} & W_{i2j3} \\\\\n",
    "W_{i3j1} & W_{i3j2} & W_{i3j3} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "0.1 & 0.2 & 0.3 \\\\\n",
    "0.3 & 0.2 & 0.7 \\\\\n",
    "0.4 & 0.3 & 0.9 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "\\begin{equation*}\n",
    "W_{jk} = \\left[ \\begin{array}{cccc}\n",
    "W_{j1k1} & W_{j1k2} & W_{j1k3} \\\\\n",
    "W_{j2k1} & W_{j2k2} & W_{j2k3} \\\\\n",
    "W_{j3k1} & W_{j3k2} & W_{j3k3} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "0.2 & 0.3 & 0.5 \\\\\n",
    "0.3 & 0.5 & 0.7 \\\\\n",
    "0.6 & 0.4 & 0.8 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "\\begin{equation*}\n",
    "W_{kl} = \\left[ \\begin{array}{cccc}\n",
    "W_{k1l1} & W_{k1l2} & W_{k1l3} \\\\\n",
    "W_{k2l1} & W_{k2l2} & W_{k2l3} \\\\\n",
    "W_{k3l1} & W_{k3l2} & W_{k3l3} \\\\ \\end{array} \\right] =  \\left[ \\begin{array}{cccc}\n",
    "0.1 & 0.4 & 0.8 \\\\\n",
    "0.3 & 0.7 & 0.2 \\\\\n",
    "0.5 & 0.2 & 0.9 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "## Our final weights \n",
    "\\begin{equation*}\n",
    "\\acute{W_{ij}} = \\left[ \\begin{array}{cccc}\n",
    "0.10001035 & 0.2000209 & 0.3000169 \\\\\n",
    "0.3000207 & 0.2000418 &0.7000338 \\\\\n",
    "0.40007245 & 0.3001463 & 0.9001183 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\acute{W_{jk}} = \\left[ \\begin{array}{cccc}\n",
    "0.2006989 & 0.300565 & 0.500286 \\\\\n",
    "0.3006575 & 0.5005316 &0.700269 \\\\\n",
    "0.600932 & 0.400753 & 0.803813 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\acute{W_{kl}} = \\left[ \\begin{array}{cccc}\n",
    "0.105531 & 0.400576 & 0.805836 \\\\\n",
    "0.30055 & 0.700577 &0.2005849 \\\\\n",
    "0.5005779 & 0.2006017 & 0.9006098 \\\\ \\end{array} \\right]\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important Notes:\n",
    "- I have completely eliminated bias when differentiating. Do you know why ?\n",
    "- Backprop of bias should be straight forward. Try on your own.\n",
    "- I have taken only one example. What will happen if we take batch of examples?\n",
    "- Though I have not mentioned directly about vansihing gradients. Do you see why it occurs?\n",
    "- What would happen if all the weights are the same number instead of random ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important links:\n",
    "- http://csrgxtu.github.io/2015/03/20/Writing-Mathematic-Fomulars-in-Markdown/\n",
    "- https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/\n",
    "- http://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let me know your feedback. If you like it, please recommend and share it. Thank you."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}