03.05 Gridworld.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gridworld"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Author:** ZHENG Wenjie\n",
    "\n",
    "**Last Update:** 2021-08-19"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is related to Example 3.5 of the book. It replicates Figure 3.2.\n",
    "\n",
    "For the plot renderer, I used 'notebook_connected' to reduce the file size. For personal use, replace it with 'notebook'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bellman equation for the state values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "v_{11} &= 0.5(-1 + \\gamma v_{11}) + 0.25 \\gamma v_{12} + 0.25 \\gamma v_{21} \\\\\n",
    "v_{12} &= 10 + \\gamma v_{52} \\\\\n",
    "v_{13} &= 0.25(-1 + \\gamma v_{13}) + 0.25 \\gamma v_{12} + 0.25 \\gamma v_{14} + 0.25 \\gamma v_{23} \\\\\n",
    "v_{14} &= 5 + \\gamma v_{34} \\\\\n",
    "v_{15} &= 0.5(-1 + \\gamma v_{15}) + 0.25 \\gamma v_{14} + 0.25 \\gamma v_{25} \\\\\n",
    "v_{21} &= 0.25(-1 + \\gamma v_{21}) + 0.25 \\gamma v_{11} + 0.25 \\gamma v_{22} + 0.25 \\gamma v_{31} \\\\\n",
    "v_{22} &= 0.25 \\gamma v_{12} + 0.25 \\gamma v_{21} + 0.25 \\gamma v_{23} + 0.25 \\gamma v_{32} \\\\\n",
    "v_{23} &= 0.25 \\gamma v_{13} + 0.25 \\gamma v_{22} + 0.25 \\gamma v_{24} + 0.25 \\gamma v_{33} \\\\\n",
    "v_{24} &= 0.25 \\gamma v_{14} + 0.25 \\gamma v_{23} + 0.25 \\gamma v_{25} + 0.25 \\gamma v_{34} \\\\\n",
    "v_{25} &= 0.25(-1 + \\gamma v_{25}) + 0.25 \\gamma v_{15} + 0.25 \\gamma v_{24} + 0.25 \\gamma v_{35} \\\\\n",
    "v_{31} &= 0.25(-1 + \\gamma v_{31}) + 0.25 \\gamma v_{21} + 0.25 \\gamma v_{32} + 0.25 \\gamma v_{41} \\\\\n",
    "v_{32} &= 0.25 \\gamma v_{22} + 0.25 \\gamma v_{31} + 0.25 \\gamma v_{33} + 0.25 \\gamma v_{42} \\\\\n",
    "v_{33} &= 0.25 \\gamma v_{23} + 0.25 \\gamma v_{32} + 0.25 \\gamma v_{34} + 0.25 \\gamma v_{43} \\\\\n",
    "v_{34} &= 0.25 \\gamma v_{24} + 0.25 \\gamma v_{33} + 0.25 \\gamma v_{35} + 0.25 \\gamma v_{44} \\\\\n",
    "v_{35} &= 0.25(-1 + \\gamma v_{35}) + 0.25 \\gamma v_{25} + 0.25 \\gamma v_{34} + 0.25 \\gamma v_{45} \\\\\n",
    "v_{41} &= 0.25(-1 + \\gamma v_{41}) + 0.25 \\gamma v_{31} + 0.25 \\gamma v_{42} + 0.25 \\gamma v_{51} \\\\\n",
    "v_{42} &= 0.25 \\gamma v_{32} + 0.25 \\gamma v_{41} + 0.25 \\gamma v_{43} + 0.25 \\gamma v_{52} \\\\\n",
    "v_{43} &= 0.25 \\gamma v_{33} + 0.25 \\gamma v_{42} + 0.25 \\gamma v_{44} + 0.25 \\gamma v_{53} \\\\\n",
    "v_{44} &= 0.25 \\gamma v_{34} + 0.25 \\gamma v_{43} + 0.25 \\gamma v_{45} + 0.25 \\gamma v_{54} \\\\\n",
    "v_{45} &= 0.25(-1 + \\gamma v_{45}) + 0.25 \\gamma v_{35} + 0.25 \\gamma v_{44} + 0.25 \\gamma v_{55} \\\\\n",
    "v_{51} &= 0.5(-1 + \\gamma v_{51}) + 0.25 \\gamma v_{41} + 0.25 \\gamma v_{52} \\\\\n",
    "v_{52} &= 0.25(-1 + \\gamma v_{52}) + 0.25 \\gamma v_{42} + 0.25 \\gamma v_{51} + 0.25 \\gamma v_{53} \\\\\n",
    "v_{53} &= 0.25(-1 + \\gamma v_{53}) + 0.25 \\gamma v_{43} + 0.25 \\gamma v_{52} + 0.25 \\gamma v_{54} \\\\\n",
    "v_{54} &= 0.25(-1 + \\gamma v_{54}) + 0.25 \\gamma v_{44} + 0.25 \\gamma v_{53} + 0.25 \\gamma v_{55} \\\\\n",
    "v_{55} &= 0.5(-1 + \\gamma v_{55}) + 0.25 \\gamma v_{45} + 0.25 \\gamma v_{54} \n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import plotly.express as px\n",
    "import plotly.figure_factory as ff\n",
    "import plotly.io as pio\n",
    "pio.renderers.default = 'notebook_connected' # or 'notebook' for personal use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.zeros((25, 25))\n",
    "\n",
    "A[0,0], A[0,1], A[0,5] = 2, 1, 1\n",
    "A[1,21] = 4\n",
    "A[2,1], A[2,2], A[2,3], A[2,7] = 1, 1, 1, 1\n",
    "A[3,13] = 4\n",
    "A[4,3], A[4,4], A[4,9] = 1, 2, 1\n",
    "\n",
    "A[5,0], A[5,5], A[5,6], A[5,10] = 1, 1, 1, 1\n",
    "A[6,1], A[6,5], A[6,7], A[6,11] = 1, 1, 1, 1\n",
    "A[7,2], A[7,6], A[7,8], A[7,12] = 1, 1, 1, 1\n",
    "A[8,3], A[8,7], A[8,9], A[8,13] = 1, 1, 1, 1\n",
    "A[9,4], A[9,8], A[9,9], A[9,14] = 1, 1, 1, 1\n",
    "\n",
    "A[10,5], A[10,10], A[10,11], A[10,15] = 1, 1, 1, 1\n",
    "A[11,6], A[11,10], A[11,12], A[11,16] = 1, 1, 1, 1\n",
    "A[12,7], A[12,11], A[12,13], A[12,17] = 1, 1, 1, 1\n",
    "A[13,8], A[13,12], A[13,14], A[13,18] = 1, 1, 1, 1\n",
    "A[14,9], A[14,13], A[14,14], A[14,19] = 1, 1, 1, 1\n",
    "\n",
    "A[15,10], A[15,15], A[15,16], A[15,20] = 1, 1, 1, 1\n",
    "A[16,11], A[16,15], A[16,17], A[16,21] = 1, 1, 1, 1\n",
    "A[17,12], A[17,16], A[17,18], A[17,22] = 1, 1, 1, 1\n",
    "A[18,13], A[18,17], A[18,19], A[18,23] = 1, 1, 1, 1\n",
    "A[19,14], A[19,18], A[19,19], A[19,24] = 1, 1, 1, 1\n",
    "\n",
    "A[20,15], A[20,20], A[20,21] = 1, 2, 1\n",
    "A[21,16], A[21,20], A[21,21], A[21,22] = 1, 1, 1, 1\n",
    "A[22,17], A[22,21], A[22,22], A[22,23] = 1, 1, 1, 1\n",
    "A[23,18], A[23,22], A[23,23], A[23,24] = 1, 1, 1, 1\n",
    "A[24,19], A[24,23], A[24,24] = 1, 1, 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "γ = 0.90\n",
    "L = np.eye(25) - 0.25*γ*A\n",
    "b = [-0.5, 10, -0.25, 5, -0.5,\n",
    "     -0.25, 0, 0, 0, -0.25,\n",
    "     -0.25, 0, 0, 0, -0.25,\n",
    "     -0.25, 0, 0, 0, -0.25,\n",
    "     -0.5, -0.25, -0.25, -0.25, -0.5\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.linalg.solve(L,b).reshape(5, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3.30899634,  8.78929186,  4.42761918,  5.32236759,  1.49217876],\n",
       "       [ 1.52158807,  2.99231786,  2.25013995,  1.9075717 ,  0.54740271],\n",
       "       [ 0.05082249,  0.73817059,  0.67311326,  0.35818621, -0.40314114],\n",
       "       [-0.9735923 , -0.43549543, -0.35488227, -0.58560509, -1.18307508],\n",
       "       [-1.85770055, -1.34523126, -1.22926726, -1.42291815, -1.97517905]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "        <script type=\"text/javascript\">\n",
       "        window.PlotlyConfig = {MathJaxConfig: 'local'};\n",
       "        if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}\n",
       "        if (typeof require !== 'undefined') {\n",
       "        require.undef(\"plotly\");\n",
       "        requirejs.config({\n",
       "            paths: {\n",
       "                'plotly': ['https://cdn.plot.ly/plotly-2.2.0.min']\n",
       "            }\n",
       "        });\n",
       "        require(['plotly'], function(Plotly) {\n",
       "            window._Plotly = Plotly;\n",
       "        });\n",
       "        }\n",
       "        </script>\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>                            <div id=\"e0a5c644-e2d1-4b9f-bb3d-08ee4d563ec1\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"e0a5c644-e2d1-4b9f-bb3d-08ee4d563ec1\")) {                    Plotly.newPlot(                        \"e0a5c644-e2d1-4b9f-bb3d-08ee4d563ec1\",                        [{\"coloraxis\":\"coloraxis\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\",\"name\":\"0\",\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"z\":[[3.308996335634639,8.789291862596121,4.427619182583304,5.3223675933702115,1.492178758740194],[1.5215880689552177,2.9923178561728165,2.250139950709492,1.9075717045592953,0.5474027057724851],[0.05082249014940586,0.7381705896183516,0.6731132598378815,0.35818621485579083,-0.4031411434164856],[-0.9735923036145049,-0.4354954300785404,-0.35488226701527265,-0.5856050882882878,-1.1830750812850597],[-1.8577005502986055,-1.3452312637820867,-1.2292672615389313,-1.4229181478367376,-1.9751790482770988]]}],                        {\"annotations\":[{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"3.31\",\"x\":0,\"y\":0},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"8.79\",\"x\":1,\"y\":0},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"4.43\",\"x\":2,\"y\":0},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"5.32\",\"x\":3,\"y\":0},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"1.49\",\"x\":4,\"y\":0},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"1.52\",\"x\":0,\"y\":1},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"2.99\",\"x\":1,\"y\":1},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"2.25\",\"x\":2,\"y\":1},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"1.91\",\"x\":3,\"y\":1},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"0.55\",\"x\":4,\"y\":1},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"0.05\",\"x\":0,\"y\":2},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"0.74\",\"x\":1,\"y\":2},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"0.67\",\"x\":2,\"y\":2},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"0.36\",\"x\":3,\"y\":2},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-0.4\",\"x\":4,\"y\":2},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-0.97\",\"x\":0,\"y\":3},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-0.44\",\"x\":1,\"y\":3},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-0.35\",\"x\":2,\"y\":3},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-0.59\",\"x\":3,\"y\":3},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-1.18\",\"x\":4,\"y\":3},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-1.86\",\"x\":0,\"y\":4},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-1.35\",\"x\":1,\"y\":4},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-1.23\",\"x\":2,\"y\":4},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-1.42\",\"x\":3,\"y\":4},{\"font\":{\"color\":\"Red\",\"size\":16},\"showarrow\":false,\"text\":\"-1.98\",\"x\":4,\"y\":4}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(247,251,255)\"],[0.125,\"rgb(222,235,247)\"],[0.25,\"rgb(198,219,239)\"],[0.375,\"rgb(158,202,225)\"],[0.5,\"rgb(107,174,214)\"],[0.625,\"rgb(66,146,198)\"],[0.75,\"rgb(33,113,181)\"],[0.875,\"rgb(8,81,156)\"],[1.0,\"rgb(8,48,107)\"]]},\"margin\":{\"t\":60},\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]],\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2}}},\"xaxis\":{\"anchor\":\"y\",\"constrain\":\"domain\",\"domain\":[0.0,1.0],\"scaleanchor\":\"y\"},\"yaxis\":{\"anchor\":\"x\",\"autorange\":\"reversed\",\"constrain\":\"domain\",\"domain\":[0.0,1.0]}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('e0a5c644-e2d1-4b9f-bb3d-08ee4d563ec1');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = px.imshow(v, color_continuous_scale='Blues')\n",
    "for i in range(5):\n",
    "    for j in range(5):\n",
    "        fig.add_annotation(x=j, y=i, text=np.round(v[i,j], 2), showarrow=False, font_size=16, font_color='Red')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>                            <div id=\"d4d871f4-dd7e-441c-90ab-d04943271250\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"d4d871f4-dd7e-441c-90ab-d04943271250\")) {                    Plotly.newPlot(                        \"d4d871f4-dd7e-441c-90ab-d04943271250\",                        [{\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"reversescale\":false,\"showscale\":false,\"type\":\"heatmap\",\"z\":[[-1.9,-1.3,-1.2,-1.4,-2.0],[-1.0,-0.4,-0.4,-0.6,-1.2],[0.1,0.7,0.7,0.4,-0.4],[1.5,3.0,2.3,1.9,0.5],[3.3,8.8,4.4,5.3,1.5]]}],                        {\"annotations\":[{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-1.9\",\"x\":0,\"xref\":\"x\",\"y\":0,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-1.3\",\"x\":1,\"xref\":\"x\",\"y\":0,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-1.2\",\"x\":2,\"xref\":\"x\",\"y\":0,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-1.4\",\"x\":3,\"xref\":\"x\",\"y\":0,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-2.0\",\"x\":4,\"xref\":\"x\",\"y\":0,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-1.0\",\"x\":0,\"xref\":\"x\",\"y\":1,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-0.4\",\"x\":1,\"xref\":\"x\",\"y\":1,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-0.4\",\"x\":2,\"xref\":\"x\",\"y\":1,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-0.6\",\"x\":3,\"xref\":\"x\",\"y\":1,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-1.2\",\"x\":4,\"xref\":\"x\",\"y\":1,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"0.1\",\"x\":0,\"xref\":\"x\",\"y\":2,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"0.7\",\"x\":1,\"xref\":\"x\",\"y\":2,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"0.7\",\"x\":2,\"xref\":\"x\",\"y\":2,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"0.4\",\"x\":3,\"xref\":\"x\",\"y\":2,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"-0.4\",\"x\":4,\"xref\":\"x\",\"y\":2,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"1.5\",\"x\":0,\"xref\":\"x\",\"y\":3,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"3.0\",\"x\":1,\"xref\":\"x\",\"y\":3,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"2.3\",\"x\":2,\"xref\":\"x\",\"y\":3,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"1.9\",\"x\":3,\"xref\":\"x\",\"y\":3,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"0.5\",\"x\":4,\"xref\":\"x\",\"y\":3,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"3.3\",\"x\":0,\"xref\":\"x\",\"y\":4,\"yref\":\"y\"},{\"font\":{\"color\":\"#000000\"},\"showarrow\":false,\"text\":\"8.8\",\"x\":1,\"xref\":\"x\",\"y\":4,\"yref\":\"y\"},{\"font\":{\"color\":\"#000000\"},\"showarrow\":false,\"text\":\"4.4\",\"x\":2,\"xref\":\"x\",\"y\":4,\"yref\":\"y\"},{\"font\":{\"color\":\"#000000\"},\"showarrow\":false,\"text\":\"5.3\",\"x\":3,\"xref\":\"x\",\"y\":4,\"yref\":\"y\"},{\"font\":{\"color\":\"#FFFFFF\"},\"showarrow\":false,\"text\":\"1.5\",\"x\":4,\"xref\":\"x\",\"y\":4,\"yref\":\"y\"}],\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]],\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2}}},\"xaxis\":{\"gridcolor\":\"rgb(0, 0, 0)\",\"showticklabels\":false,\"side\":\"top\",\"ticks\":\"\"},\"yaxis\":{\"showticklabels\":false,\"ticks\":\"\",\"ticksuffix\":\"  \"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('d4d871f4-dd7e-441c-90ab-d04943271250');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ff.create_annotated_heatmap(np.flipud(np.round(v, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all state values increase by 1, then all rewards increase by $1-\\gamma$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The $\\gamma<1$ in the Bellman equation is necessary. Otherwise, the matrix is not of full rank: the sum of every row is 0."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}