05. Improvements to the RNN/5.02 Understanding the LSTM cell.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding the LSTM cell\n",
    "\n",
    "\n",
    "What makes the LSTM cells so special? How do the LSTM cells achieve long term\n",
    "dependency? How does it know what information to keep and what information to discard\n",
    "from the memory?\n",
    "\n",
    "This is all achieved by a special structure called gates. As shown in the following figure, a\n",
    "typical LSTM cell consists of three special gates called input gate, output gate, and forget\n",
    "gate:\n",
    "\n",
    "![image](images/1.png)\n",
    "\n",
    "These three gates are responsible for deciding what information to add; output and forget\n",
    "from the memory. With these gates, LSTM effectively keeps information in the memory\n",
    "only as long as they required. \n",
    "\n",
    "In an RNN cell, we used hidden state $h_t$ for two purposes, one for storing the information\n",
    "and other for making predictions. Unlike RNN, in the LSTM cell, we break the hidden\n",
    "states into two states called cell state and hidden state.\n",
    "\n",
    "* Cell state is also called internal memory where all the information will be stored.\n",
    "* Hidden state is used for computing the output. \n",
    "\n",
    "Both of these cell state and hidden states are shared across every time steps. Now we will\n",
    "deep dive into LSTM cell and see how exactly these gates are used and how hidden state is\n",
    "computed.\n",
    "\n",
    "\n",
    "## Forget Gate \n",
    "\n",
    "The forget gate $f_t$ is responsible for deciding what information should be removed from\n",
    "the cell state (memory). \n",
    "\n",
    "\n",
    "Consider the following sentences: Harry is a good singer. He lives in\n",
    "New York. Zayn is also a good singer.\n",
    "\n",
    "As soon as we start talking about Zayn, the network will understand that the subject has\n",
    "been changed from Harry to Zayn and the information about Harry is no longer required.\n",
    "Now, the forget gate will remove/forget information about Harry from the cell state.\n",
    "The forget gate is controlled by a sigmoid function. At a time step $t$ , we pass the\n",
    "input $x_t$ and previous hidden state ${h_{t-1}}$to the forget gate. It will return 0 if the particular\n",
    "information from the cell state should be removed and returns 1 if the information should\n",
    "not be removed. The forget gate $f$ at a time step $t$ is expressed as follows:\n",
    "\n",
    "$$f_{t}=\\sigma\\left(U_{f} x_{t}+W_{f} h_{t-1}+b_{f}\\right)$$\n",
    "\n",
    "Where:\n",
    "\n",
    "* $U_f$ is the input to hidden weights of the forget gate\n",
    "* $W_f$ is the hidden to hidden weights of the forget gate\n",
    "* $b_f$ is the bias of the forget gate\n",
    "\n",
    "The following figure shows the forget gate. As you can see, input $x_t$ is multiplied\n",
    "$U_f$ with and previous hidden state $h_{t-1}$ will be multiplied with $W_f$, both of them will get\n",
    "added together and sent to the sigmoid function which returns values from 0 to 1.\n",
    "\n",
    "![image](images/2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input Gate\n",
    "\n",
    "\n",
    "The input gate is responsible for deciding what information should be stored in the cell\n",
    "state.\n",
    "\n",
    "Let's consider the same example: Harry is a good singer. He lives in New York. Zayn is\n",
    "also a good singer.\n",
    "\n",
    "\n",
    "After the forget gate removes information from the cell state, the input gate decides what\n",
    "information it has to keep in the memory. Here, since the information about Harry is\n",
    "removed from the cell state by the forget gate, the input gate decides to update the cell state\n",
    "with the information about Zayn.\n",
    "Similar to forget gate, the input gate is controlled by a sigmoid function which returns\n",
    "either 0 or 1. If it returns 1 then the particular information will be stored/update to the cell\n",
    "state and if it returns 0 then we will not store the information to the cell state. The input\n",
    "gate $i$ at a time step $t$ is expressed as follows:\n",
    "\n",
    "$$ i_{t}=\\sigma\\left(U_{i} x_{t}+W_{i} h_{t-1}+b_{i}\\right)$$\n",
    "\n",
    "\n",
    "Where:\n",
    "* $U_i$ is the input to hidden weights of the input gate\n",
    "* $W_i$ is the hidden to hidden weights of the input gate\n",
    "* $b_i$ is the bias of the input gate\n",
    "\n",
    "\n",
    "The following figure shows the input gate:\n",
    "\n",
    "![image](images/3.png)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output gate\n",
    "\n",
    "We will have a lot of information in the cell state (memory). The output gate is responsible\n",
    "for deciding what information should be taken from the cell state to give as an\n",
    "output. \n",
    "\n",
    "Consider the following sentences. Zayn's debut album was a huge success. Congrats\n",
    "____.\n",
    "\n",
    "\n",
    "The output gate will look up all the information in the cell state and select the correct\n",
    "information to fill the blank. Here, congrats is an adjective which is used to describe a noun.\n",
    "So the output gate will predict Zayn (noun), to fill the blank. Similar to other gates, it is also\n",
    "controlled by a sigmoid function. The output gate $o$ at a time step $t$ is expressed as follows:\n",
    "\n",
    "$o_{t}=\\sigma\\left(U_{o} x_{t}+W_{o} h_{t-1}+b_{o}\\right)$\n",
    "\n",
    "Where:\n",
    "* $U_o$ is the input to hidden weights of the output gate\n",
    "* $W_o$ is the hidden to hidden weights of the output gate\n",
    "* $b_o$ is the bias of the output gate\n",
    "\n",
    "The output gate is shown in the following figure:\n",
    "\n",
    "![image](images/4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Updating the cell state\n",
    "\n",
    "\n",
    "We just learned how all the three gates in the LSTM works. But, the question is how can we\n",
    "actually update the cell state by adding the relevant new information and deleting the\n",
    "information that is not required from the cell state with the help of the gates?\n",
    "\n",
    "__First, we will see how to add new relevant information to the cell state:__\n",
    "\n",
    "\n",
    "To hold all the new\n",
    "information that can be added to the cell state, we create a new vector called $g_t$. It is called\n",
    "a candidate state or internal state vector. Unlike gates which is regulated by the sigmoid\n",
    "function, candidate state is regulated by the tanh function. But, why? Sigmoid function\n",
    "returns either 0 or 1 i.e it is always positive. We need to allow the values of $g_t$ to be either\n",
    "positive or negative. So, we use tanh function which returns either +1 or -1.\n",
    "The candidate state $g$ at a time $t$ is expressed as follows:\n",
    "\n",
    "\n",
    "$$g_{t}=\\tanh \\left(U_{g} x_{t}+W_{g} h_{t-1}+b_{g}\\right)$$\n",
    "\n",
    "\n",
    "Where:\n",
    "* $U_g$ is the input to hidden weights of the candidate state\n",
    "* $W_g$ is the hidden to hidden weights of the candidate state\n",
    "* $b_g$ is the bias of the candidate state\n",
    "\n",
    "Thus, the candidate state holds all the new information that can be added to the memory\n",
    "and it is shown in the following figure:\n",
    "\n",
    "![image](images/5.png)\n",
    "\n",
    "But how do we decide whether the information in the candidate state is relevant? How do\n",
    "we decide whether to add or not add new information in the candidate state to the cell\n",
    "sate? We learned that the input gate is responsible for deciding whether to add new\n",
    "information or not to the cell state. So if we multiply $g_t$ and $i_t$, we get only relevant\n",
    "information which should be added to the memory. \n",
    "\n",
    "\n",
    "That is, as we know input gate returns 0 if the information is not required and 1 if the\n",
    "information is required. Say, $i_t=0$ , then multiplying $g_t$ and $i_t$ gives 0 which means the\n",
    "information in $g_t$ is not required and we don't want to update the cell state with $g_t$. When\n",
    "$i_t=1$, then multiplying $g_t$ and $i_t$ gives $g_t$ which implies we can update the information\n",
    "in the $g_t$ to the cell state.\n",
    "\n",
    "\n",
    "Adding the new information to the cell state with the input gate $i_t$, and the candidate\n",
    "state $g_t$, is shown in the following figure\n",
    "\n",
    "![image](images/6.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__ Now, we will see how to remove information from the previous cell state which is not\n",
    "required anymore.__\n",
    "\n",
    "\n",
    "We learned that forget gate is used for removing information which is not required in the\n",
    "cell state. So if we multiply previous cell state $c_{t-1}$ and forget gate $f_t$ then we retain only\n",
    "relevant information in the cell state.\n",
    "\n",
    "Say,$f_t = 0$ , then multiplying $c_{t-1}$ and $f_t$ gives 0 which means the information in the cell\n",
    "state $c_{t-1}$ is not required and it should be removed (forgotten). When $f_t=1$ , then\n",
    "multiplying $c_{t-1}$ and $f_t$ gives $c_{t-1}$ which imples that information in the previous cell\n",
    "state is required and it should not be removed.\n",
    "Removing information from the previous cell state$c_{t-1}$ with the forget gate $f_t$ is shown in\n",
    "the following figure:\n",
    "\n",
    "![image](images/7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, in a nutshell we update our cell state by multiplying $g_t$ and $i_t$ to add new\n",
    "information and multiplying $c_{t-1}$ and $f_t$ to remove information. We can express the cell\n",
    "state equation as follows:\n",
    "\n",
    "$$c_{t}=f_{t} c_{t-1}+i_{t} g_{t} $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Updating hidden state \n",
    "\n",
    "We just learned how the information in the cell state will be updated. Now we will see,\n",
    "how the information in the hidden state $h_$ will be updated. We learned that the hidden state\n",
    " is used for computing the output. But how can we compute the output?\n",
    " \n",
    "We know that the output gate is responsible for deciding what information should be taken\n",
    "from the cell state to give as an output. Thus multiplying $o_t$ and tanh (to squash between -1\n",
    "and +1) of cell state $tanh(c_t)$, returns the output.\n",
    "Thus, hidden state $h_t$ is expressed as follows:\n",
    "\n",
    "$$h_{t}=o_{t} \\tanh \\left(c_{t}\\right)$$ \n",
    "\n",
    "\n",
    "The following figure shows how the hidden sate $h_t$ is computed by mutliplying $o_t$ and\n",
    "$tanh(c_t)$ :\n",
    "\n",
    "![image to be added](images/8.png)\n",
    "\n",
    "And finally, once we have the hidden state value, we can apply the softmax function and\n",
    "compute $\\hat{y}_t$ as shown:\n",
    "\n",
    "$$\\hat{y}_{t}=\\operatorname{softmax}\\left(V h_{t}\\right)$$\n",
    "\n",
    "Where, $V$ is the hidden to output layer weights. \n",
    "\n",
    "\n",
    "In the next section, we will see how exactly forward propgation is performed in the LSTM cell. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}