4-Keras-Higgs.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/apanin/anaconda/lib/python2.7/site-packages/pandas/computation/__init__.py:19: UserWarning: The installed version of numexpr 2.4.4 is not supported in pandas and will be not be used\n",
      "\n",
      "  UserWarning)\n",
      "Using gpu device 0: GeForce GTX TITAN X (CNMeM is disabled, cuDNN 5005)\n",
      "Using Theano backend.\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import theano\n",
    "import theano.tensor as T\n",
    "import matplotlib.pyplot as plt\n",
    "import keras \n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import LabelEncoder \n",
    "from keras.utils import np_utils\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, InputLayer, Dropout, BatchNormalization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Searching for Higgs boson\n",
    "\n",
    "discriminate two kinds of processes that possibly could be studied at the LHC."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img  src=\"higgs_feynman.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data description\n",
    "\n",
    "Dataset consists of 21 low-level features and 7 high-level features.\n",
    "- low-level features are momenta: lepton momentum\n",
    "- momenta and b-tags of 4 jets missing transverse momentum\n",
    "- high-level features are invariant masses\n",
    "\n",
    "feature named 'target' is present only in training. `1` corresponds to signal (Higgs) decay, `0` to background\n",
    "\n",
    "Evaluation metric: __ROC AUC__\n",
    "\n",
    "### Acknowledgment\n",
    "\n",
    "Thanks to P. Baldi, P. Sadowski, and D. Whiteson for publishing original dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img  src=\"roc_vs_significance.png\"/>\n",
    "Baldi et al., 2014 Searching for Exotic Particles in High-Energy Physics with Deep Learning (http://arxiv.org/abs/1402.4735)\n",
    "\n",
    "Dataset is hosted at https://archive.ics.uci.edu/ml/datasets/HIGGS\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "def load_data(data_path=\"higgs-data\", suffix=None):\n",
    "    df = pd.read_csv(os.path.join(data_path,\"HIGGS_1000000.csv\"), header=None, index_col=None)\n",
    "    df = df.astype(np.float32)\n",
    "    y = df[0].values==1\n",
    "    X = df[df.columns[1:]].values.astype(\"float32\")\n",
    "    return X, y\n",
    "\n",
    "X, y = load_data(\"higgs-data\")\n",
    "n_features = X.shape[1]\n",
    "N_EPOCH = 20 # theano.config.device == 'gpu'\n",
    "HOLDOUT_SIZE = 500000\n",
    "BATCH_SIZE = 2000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXl4HMWZ/z+l0S3rlizJlmTZli0f8c1NAEHAmCMxkA0E\nQkIS2JBNILv8sgkJ2RiTzS4LC1mWQFhCAuQgMYRchCPcCuawiQ9sbGTJ9yVZ1uiWZd31+6PV0mg0\nR89090yPpj7Po0dz9PS8M9P9rW+/9VaVkFKiUCgUivggIdoBKBQKhSJyKNFXKBSKOEKJvkKhUMQR\nSvQVCoUijlCir1AoFHGEEn2FQqGII0yJvhDicSFEkxDiwyDbnSqEGBRCXGXm/RQKhUJhDrNO/wlg\nVaANhBAu4B7gr4Aw+X4KhUKhMIEp0ZdSrgfagmx2K/As0GzmvRQKhUJhHltz+kKI6cBq4JGRh9Tw\nX4VCoYgidnfkPgB8R2pzPQhUekehUCiiSqLN+18BrBNCABQAlwghBqSUz3luJIRQVwAKhUIRBlLK\nkMy0rU5fSjlLSjlTSjkTLa//T96C77HtpP278847ox6D+nzq83n/rT+4HtbCVV+9KuqxqN8uvL9w\nMOX0hRC/Bc4DCoQQh4E7gaQREX/UzL4VCoW9bGrYBEBnX2eUI1FEElOiL6W8NoRtv2TmvRQKhbVs\natjE4qLFdNYr0Y8n1IjcCFBdXR3tEGxFfb7YZFPDJq6ouoLk2cnRDsU2JutvZwYRbl7I0iCEkE6I\nQ6GIFzr7Oim5v4RXrn+F216+jff/8f1oh6QIAyEEMsSOXLurdxQKhQPZ0riFJUVLqMip4HDn4ajE\nMFLVpzCIVcZYib5CEYdsatjEKdNOoXhKMS09LfQP9ZPsinyaR13hG8PKBlLl9BWKOEQXfVeCi+Ip\nxTR2NUY7JEWEUKKvUMQhuugDlGaVcqTzSJQjUkQKJfoKRZzRdrKNphNNVOVXAUr04w0l+gpFnLG5\ncTPLipfhSnABSvTjDSX6CkWc4ZnaAU30o1XBo4g8SvQVijjDW/TLssqU048jlOgrFHGGL6evRH+M\niooK7rvvPhYvXkxmZiY33ngjTU1NXHLJJWRnZ3PRRRfR3t4OwIYNGzjrrLPIzc1l6dKl/O1vfxvd\nzxNPPMGCBQvIyspi9uzZ/PSnPx19rqamhtLSUn70ox9RVFTEtGnTePLJJyPy+ZToKxRxRPOJZtp6\n26jMqxx9TIn+eIQQ/OEPf+D111+nrq6O559/nksuuYT/+q//4vjx4wwPD/Pggw9y9OhRLr/8ctas\nWUNbWxv33Xcfn/70p2lpaQGgqKiIF154gc7OTp544gluu+02tm7dOvo+TU1NdHZ20tDQwM9//nO+\n/vWv09HRYfvnU6KvUMQRmxs3s6JkBQli7NQvnlLM8RPHGRwejGJkExHCmr9wuPXWWyksLGTatGmc\nc845nHnmmSxZsoSUlBSuvPJKtm7dylNPPcWll17KqlXaMuEXXnghp5xyCi+88AIAl156KTNnzgTg\n3HPPZeXKlaxfv370PZKSklizZg0ul4tLLrmEKVOmUFdXZ+5LM4ASfYUijvBO7QAkuZIozCjkWPex\nKEXlGymt+QuHoqKi0dtpaWnj7qemptLd3c3Bgwf53e9+R25u7ujfO++8w7Fj2vf40ksvccYZZ5Cf\nn09ubi4vvvji6FUAQH5+PgkJYxKcnp5Od3d3eAGHgJqGQaGIIzY1bOK6RddNeFxP8ZRmlUYhKufj\nOV2EPiVCWVkZn//858fl6nX6+vr49Kc/za9//WtWr16Ny+XiyiuvdMS0E8rpKxRxhC+nDyNlmx2q\nbNMIunBff/31/OUvf+GVV15haGiI3t5eampqOHr0KP39/fT391NQUEBCQgIvvfQSr7zySpQj11Ci\nr1DECY1djfQM9DAzZ+aE51TZZmA8JzwTQiCEoLS0lD//+c/853/+J1OnTqW8vJz7778fKSWZmZk8\n+OCDXH311eTl5fHb3/6W1atX+91nJFHz6SsUccLz9c/z4MYHeeXzEx3nfe/eR2NXI/dffH/E4hmZ\nCz5i7xfL+PuuwplPXzl9hSJO8JfagZGcfpdy+vGAadEXQjwuhGgSQnzo5/nPCSG2CSG2CyHeEUIs\nNvueCoUidIKKvkrvxAVWOP0ngFUBnt8HnCulXAz8OzCxq1uhUNiKlFKJvgKwQPSllOuBtgDPvyel\n1IeZbQRUTZhCEWGOdh1lWA5TllXm8/lpmdM41n2MoeGhCEemiDSRzunfCLwY4fdUKOIe3eX7qxhJ\ndiWTl5ZH04mmCEemiDQRG5wlhDgf+DJwtq/n165dO3q7urqa6urqiMSlUFiFPo1BYoLzxjwGSu3o\n6CmeaZnTIhSVIlRqamqoqakxtQ9LSjaFEBXAX6SUi/w8vxj4A7BKSrnHx/OqZFMR83zjpW+Ql5bH\n2uq10Q5lAqt+vYqvnfo1PlX1Kb/bXPn0lXx+8ee5av5VEYlJlWwaJ6ZKNoUQ5WiCf70vwVcoJgMd\nvR08vvVx9rQ67xAP1omrU5qpOnPjAdPXoUKI3wLnAQVCiMPAnUASgJTyUWANkAs8MpJPHJBSnmb2\nfRUKJ/HEB09QkF7AoY5D0Q5lAgc7DpLsSg6atlEVPOZZu3Yte/fu5Ve/+lW0Q/GLadGXUl4b5Pmb\ngJvMvo9C4VSG5TAP//1hfnjBD/m3N/4t2uFMwIjLB030tzVti0BEsYkRQY/W1AqhoEbkKhQmeXnP\ny2SlZPGZBZ+hoavBcWWPoYi+Wit38qNEX6EwyY/f/zG3nHoLKYkp5KfnO25eeqOiX5atJl3Tueee\neygtLSUrK4t58+bx4osvcvfdd/P000+TmZnJsmXLANi/fz/nnXceWVlZrFy5ErfbHeXIg+O82jKF\nIobY3bKbTQ2b+P3VvwegPLucQx2HmJ41PcqRaUgp2dy42ZDoT8ucRkNXA8NyeNzKWvFGXV0dDz/8\nMJs2baK4uJhDhw4xODjIHXfcwd69e/nlL385uu11113H2WefzWuvvcaGDRu47LLLuOKKK6IYfXCU\n6CsUJnj47w9z47IbSUtKA8ZE/8yyM6Mcmcbetr1kpWQxNWNq0G1TE1PJTsmm+UQzRVOKgm5vN+Iu\na/Lj8s7QykJdLhd9fX3s3LmT/Px8ysvLtf1IOa5s8tChQ2zatIk33niDpKQkzjnnHD75yU86vgxV\nib5CESbd/d38avuv2Hrz2GLXZVlljqrgMZra0dEreJwg+qGKtVVUVlbywAMPsHbtWnbu3MnFF1/M\nj370ownbNTQ0kJubS1pa2uhjM2bM4PBhZ/eLxO81nEJhkl9u+yXVFdWUZ5ePPqY7faewqWETp5SE\nLvrxzrXXXsv69es5ePAgQghuv/32cevZApSUlNDW1kZPT8/oY/r2TkaJvkIRBlJKHnr/IW497dZx\nj5dnlzuqAiYcp++k+KNBfX09b7zxBn19faSkpJCamorL5aKoqIgDBw6Mpm9mzJjBKaecwp133snA\nwABvv/02zz//fJSjD44SfYUiDF7f/zquBBfnzThv3ONOcvrDcpgtjVtYMW2F4dcop68tav7d736X\nwsJCSkpKcLvd3H333XzmM58BID8/n1NO0RrS3/zmN2zcuJG8vDx+8IMfcMMNN0QzdEOonL5CEQY/\nfv/H3HrarRMu5Z2U069vqacwo5C8tDzDrynLKuPlvS/bGJXzWbRoERs3bvT53Pr168fdnzlzJm+9\n9VYkwrIM5fQVihDZ37afdw69w+cWfW7Cc4UZhXT3d9Mz0OPjlZEl1NQOKKcfDyjRVyhC5Cd//wlf\nXPpFMpIzJjyXIBIoyy7jcEf08+KhduKCEv14QIm+QhECPQM9PPHBE3zt1K/53cYpef1wnP70rOkc\n7Trq+FpzRfgo0VcoQuCp7U9xVtlZzMqd5XcbJ4j+4PAgHxz7gOUly0N6XXpSOulJ6bScbLEpMkW0\nUaKvUBhESjnagRsIJ3Tm7nLvYnrWdLJTs0N+bWlWqSPSUwp7UKKvUBjkrYNvMTA8wIWzLgy4nRNq\n9cNJ7eiUZamJ1yYzqmRToTCIPptmsBGX5dnlrNuxLkJR+SacTlydSHbmOn306mREOX2F4zjWfYxb\nXryFc584N9qhjHK44zBvHniTLyz5QtBtnZDTN+P0IyH63d3wzDOSz35Wkp0tyc2V/Nu/SVpb5ejE\nZupv/J9VKNFXOIb23nbueP0OFv5kIUkJSbx/9H36h/qjHRYAj2x6hOsXXU9mSmbQbcuyyjjceThq\nFTADQwN8ePxDlpUsC+v1pVmlHOmyXvRbW+EXv4DVq2HaNHj8cbjgAqirgy1b4NgxmDMH7roLOjos\nf3vFCEr0FVGnZ6CHe9+5lzk/nkNTdxNbb97K/6z6H8qyy9jftj/a4dE72MvPtvyMr5/2dUPbZyRn\nkJ6UjrsnOgtq7GzeSUVOBVOSp4T1em+nPzgI69bB2WfDsmVwxRXwjW/AfffBM8/Ahg3Q0ADDwxP3\n1dgIjzwCF10EM2fCc8/B1VfDoUPw0kvwj/8IRUVQUQGPPQYbN8L+/VBZCf/xH9DVFeaXECIHD8Ln\nPgfb4mC1SFM5fSHE48BlwHEp5SI/2zwIXAL0AF+UUm71tZ3Ceg62H+TVfa/y2r7XePvQ27xw3Qss\nKV4S7bBGGRga4PGtj/ODt37AmaVn8tYX32J+4fzR5yvzKtnTuoeqgqooRgnrdqxjxbQVzM2fa/g1\neoqnMKPQxsh8Yya1A2PVO52d8POfwwMPaKL87W9Daakm2Prfxo1jt1tbYfp0KC+HsjLYtw9qa+Gy\ny+BrX4M//xnS0wO/9+zZ8OSTUF8PP/iBdv+b34Svfx2mhNeGBeUvf4GbboJPfUprnP7wB/j4x+15\nLydgtiP3CeDHwC99PSmEuBSolFLOEUKcDjwCnGHyPRV+aO9t5839b/Lavtd4dd+rtPe2c+GsC7l4\n9sX0DPSwpXGLI0R/WA7zzM5n+P6b32dG9gz+eM0fOW36aRO2m5M3h92tu6MQ4Rh6mea/n//vIb1O\nF/1QJjuzCjOduAB0lrK/5QgVMyUrLxI8+yyceurY0yv8fKS+PjhyRGsAdOd8wQWQnBx6CHPnwq9/\nDR99pIl/ZSV861vwT/8UvOEwysAA3HEHPP00/PGPcNZZ8JnPwJVXwhNPwOWXW/M+TsOU6Esp1wsh\nKgJs8ingFyPbbhRC5AghiqSUTeG+587jO2nvbSc1MdXnX0piCokJsVmUNDA0wA/f+iEJIoGslCwy\nUzLJTM4kMyVTuz9yOzNZuy+EYMORDaMiv+P4Ds4qO4sLZ17IM595hsVFi0eXvWvoaqCupc7ymFt6\nWnjvyHskJSSR7Eom2ZVMksvj9sjj+mN/P/p37njjDhITEnnkskcClj9W5lVS31Jvecy7W3bT3NNM\namIqaYlpo8dOWpJ2O8WVMlpVsuHIBjp6O1hVuSqk97CyVl9KSa27FoEgJTGFFFcKya7kcbc9q2A2\nNWzihiWhz/a4dSvcfz+8+OIUXLem8Ma7bSytMj5ZW0qK5sxnzw75rf2yYIGWWvrwQy3Xf//92hXH\nTTeZc/6HDsE110Benva58/O1x1euhOef1/od7r0XvhC83z7msFsdpwOeBctHgFIgLNGXUnL242ez\noHABfUN99A72Tvg7OXCSBJFAamIq6UnpPH/d8z5dpBl+uvmn7Dy+k/+95H8t3W+tu5afbf0ZX176\nZQ52HKSrr4uu/i46+zrp6u+iq2/87WE5zJLiJVw06yJ+eP4PObv8bFITU33uuzyjiqcPPYWUYGWV\n3IMbH+SZj56hPLuc/qH+0b+BoYGx28Njt6dnTmdt9VqunHdl0HK9tJOV7Dz2guUxX/aby8hMyWRY\nDo8eM6PHz+BJBoYGSElMITUxlYGhAf7rwv8Kec1YKyt4tjRu4dwnz6Usq4y+oT76BvvoH+ofvT0w\nPDDayKa4UugZ6DF8RTc8DH/9qyamdXXwz/8MDz0E5/y2lIScI4Bx0beTRYvg2Wfhgw+0XP9//Afc\nfDPceqvWJxAKzz8PN96opY3+9V/Ba20UTj8d3nwTLr4YWlrgttus+xxOIBKW2Pt09VnSsHbt2tHb\n1dXVVFdXT9jmQPsBpiRP4d0b3/X7ZlJKBocH6R3s5duvfpu/HfibIdF/9FEtD3nppUE35Z3D73Cg\n/UDwDUOkzl3H6dNP598vMJZKGBweNHxV8z/fq+KDOXWk3ghTp2oniv7n6/6CBeByBd9vrbuW75/7\nfa5bdJ2hOIyydy987Zo5DF+3h7Q0KCnR8sXTp2t5Zf22/jdtmuY0g9E32MehjkN0freTZJfvvMOw\nHKZvsI+TgyfpH+qnKMOYqjz0kOZGFyyA5OXltEzdxJvJ2v2pU8NvuHYc38EV867gqaue8huv3qj2\nDfaRmJBIepKxHMjq1VpK5pvf1DpY9VSM3pm7uGhxeEHbxNKl8LvfwZ498KMfwfz5Wkrmm9/UUkKB\nGBiA731Pu3L4wx+0jml/zJ8Pb7+tOf/mZq2RccKQgpqaGmpqakztw27RPwqUedwvHXlsAp6i74/t\nTduDHoRCCJJcSSS5kjiz7Exe3P2ioUDfeksrFzMi+lsat9B8otnQfvfu1XKTd94ZfNv6lvqQOgtD\nSWO17q0k6ex9uFsHaXUn0tQETU1w/Lj2/8ABrVOuqQl27IB77oEvfzn4fmvdtXy34LuG4zDKsWOw\nfHYF2/KPcqy5H3dTMkePMvp36BC8997Y/dZW7eRM9X2hM8retr2UZ5f7FXzQZspMS0obXezcKHV1\nWs753HPhhe3l/KblEGvWwM6dmptcsED7W7hQ+3/qqZCVFXy/u9y7mJc/L2C8eooKAw2fJ++9p3W2\nFnr1N5dmOnu2zcpK+MlPtEb24Ye1jtezz9by/medNXH7w4fhs5+F7GytPLSgIPh7lJfD+vWaJrjd\nWhWSESNkJ96G+K677gp5H3aL/nPALcA6IcQZQLu/fH5fX3CnZkT0PVlespwfvvVDQ9u63ZCbG3y7\nnoEe9rbuBaCjtyPo3CbbtmmuwpDot9ZPWInJKlqb0ijKKOZ430Fmz5jNjBn+t73jDq3ULhhDw0Ps\nad0TUkNlFLcbigqSKM0q5Xj/AeZWzqWy0v/2paWa6JeV+d8GtKspu6qB3G4480y48EKYd1o5Tz92\niPXrQUqtMd25U+uY3LlTq4qZO1creQxGrbuW6xdfb3m8Q0PQ3q7ltb2JlSmWCwth7Votz//EE/D5\nz2tXhd/6Fnzyk1pj++KLmoG57Tbtce90TrD9v/GG1rl79dXw1FPBjYXTMVWnL4T4LfAuUCWEOCyE\n+LIQ4mYhxM0AUsoXgX1CiD3Ao4Df+WjdBkqatx8PTfTnFczjaNdROvs6g27b3Kz9BY2haTvzC+cz\nN3+uoU5Go/uF0J2+Ufr6oLcX5hdWGerMLSw0FvOB9gNMzZjqc155szQ3a25ML9sMRkGBsWMomGs2\nQ3PzmGMumVKCu8dN/1A/QkBxMXziE1oO+v/+Dx580FjDCprozy+YH3zDEGlthZwc3+411tbKTU/X\nyjrr6rQxBD/8oXY19aUvabn/Z5+F228PTfB1MjPhhRe01152WeTGDtiFKdGXUl4rpZwmpUyWUpZJ\nKR+XUj4qpXzUY5tbpJSVUsolUsot/vZl5ITddmwbS4qMlxwmJiSyuGgxWxuDDw1wu43FsKVxC8uL\nl1NVUGVI9PX9BhucKaXUXGi+9S7U7dZEsaqgijp3cNEPRUDtECPQ3r+wUBP93S3ByzaNNlR1LfY6\nfT1t4EpwUZJZwtFOn9lMw/H2D/VzsP0gc/LnWBipht6w+qIsOzYnXUtM1Bz5++9rjWtxsZbOMVt3\nn5Ki9QXMnQvnn6+lRWMVx4zIDXYCnOg/weHOwyE74eXFy9nS6LetATRBNurItzRuYXnJcubmzTXk\nmpubtQ6kziAXGy0nW5BICtINJBtDRD+5q/Ktdfq17lrmFdjnmgsKtFp9I04/JNG3oWGF8U4fAlfw\nFBQYi3d3y25m5MwI2AcRLnrD6otYSe/4Qwioroa77/b/GUPF5dL6ES69FM45RxuLEIvEjOjvbN7J\nvIJ5JLmSQtrvimkr2Ny4OeA2PT1a+sOw0y8Jzel7/veHntqxY9ZB/eSuKrBW9He5d9km+p5Of0+b\nNekd/WrKjpilHO/0IbDo5+Vp88sMDgber93fsT+nr4/KVStojUcIbbDYt74V/U7dcHGM6Ac7YUPt\nxNVZXhLc6Tc3ayV1wdIwfYN97HLvYnHRYubmG3f6nv/9YVc+X3/vggK0mC1O79jt9K1M7zT3aBvY\ncTXV1aWVO3p28gUaoOVyaYUDra2B92tXPh8mXpl4kpWSRYJIMNQfFo/cdJNWPBCLOEb0g52w25u2\nh5TP11lQuIAD7Qc40X/C7zZut/YDpqYGTsPsOL6DyrxK0pLSqMqvYnfL7qBOqLlZq30P9vnsyufD\nmGsuzSqlo6+Drr7APVG6gAYzeZHI6c/MncmRziMMDA0E3N6I6OuVO3ZdTXm75mCLqRiJ2U7RD+T0\nIfZTPArfxIzob2vaFpbTT3Yls3DqQj449kHA9y4oCJ5n1VM7ANmp2UxJnsLRLt8ddTputzbQI2h6\np9V+p58gEpiTNydoWio9XatUOOG/naT5RDNDcoipGVMtjnZk/yMxJ7uSmZY5LehgOCNXJ5HM50Pw\nUblG8vq1zbXjJqGzkkBOH2KvgkdhDMeIfqATVkoZdnoHgnfm6q6ysDBwHJ6iDwQt29Q7iOfPj256\nx7PDzmheP5iI6qkdu1Y+8ox5Tn7wzlyjTj+S+fFgoh8s5mE5TF1LZGP2RDn9yYljRD/QwX+k8wgp\nrpSwXeWKaSvYcsy/6Bt1+psbN7OiZGyKwar8wCWQJ05oHT8zZgQ/ufe07qEyL8DoIxN4ntzBYtYJ\nJkh2pnZ6e6G/X6uPBqjMDV6rb0T0d7XsirjTP9hx0G8KMFjMhzsOk5OaQ1aKgWG7YRDM6au1cicn\nMSH6Zlw+aJ25mxv8V/AYcfoDQwPsOL5j3ERWwZy+vt9grvlQxyHy0/LDXvQiGJ4nt9EO6GCCZGe5\npt5I6RcRlXmVQadYNpTesXk0rrdrzk7RRmt39PleBirYlaWd+XxQTj9ecYzoBzr4w+3E1fnY1I+x\np3UPJwdO+nzeiNOvddcyI2fGOGEOlirR9xtMQOtb6m1dKGSC07cwvWMH3oOGjKR38vO1KQWGhnw/\n3z/Uz6GOQ8zOtXDeXw98uWYhhNaZ2+E7Lx7syrK22V7RN5LTV6I/+XCM6Le2+l5uDcLvxNVJTUyl\nqqCKD49/6PN5I07fO58PwZ2+flIZEf25efbk8z3jAK2hMlJ1ZCS9Y3eNvo4Rp+9yaZNp+SuB3Nu6\nl7LsMlISQ5yRzCD+XHOgvL6Rqym7OnFBOf14xTGin5kJbW2+nzOb3gGtM9dfiseI09/csJnlxeNF\nf1buLA53HqZvsM/na/STKphrtrMTV0ptTnB9kQh9cZZgVUeBvouTAydp7G5kVu4si6PV8Hb6M3Nm\ncrjjcNCyzUDfs52VO+DfNZdnmRR9m5y+XpmVEWDaJCX6kxPHiL6/E7Z3sJf97ftNu8pAg7QMOf1j\nWyYsfZfsSqY8u5x9bft8vsao069rqbNN9NvbtRJMzyXrjHTmBvou6lvqmZU7y7YVyrydfkpiCiWZ\nJUEXJQn0Pds5DgL8u+aybP8DtILl9KM1GlcnJzWHgeGBoOM6FLGFY0Tf3wn7UfNHzMmbY/qyPFAF\nTzCnPzQ8xLZj21havHTCc4Fy5LroZ2drFSl9vi8IbM3p+5pfxUhnbiABtVOMwPdEYEZSPAFF38aJ\n1iCA0w8wQCtQvO4eNwNDAxRPKbYwyjGC5fNB65NQbn/y4XjRtyK1A7C4aDG1zbUTUjGDg9ocKHl5\n/p1XfUs9xVOKyUnNmfBcoLy+ZxVKfr7/K5nGrkYqcirC+VhB8SWgRpx+oPSOndMTg++GysjEa4HS\nO3Y3VOHk9PV4fXWv6IOy7BwHYWQhEVW2OflwjOj7O2G3HTPXiauTnpTOrNxZ7GzeOe5xzznF/Qmd\nr05cnUAC6umm/DVqe1v3UpFTEbFUCWidufWtgUflBko97GrZFfEORiPz6gd1+jaldwYGoLtbO468\nCST6KSna1B8dPio67S7XNOL0QeX1JyOOEX2/Tj/EhVMCsWLaigmduZ6imJOjzbjZ3z/+dd6DsjyZ\nmz/Xr4AaEX078/l6DOE4/UACWttsX40++I55Tt6csNM77h43Q8P2TRnR0qJdKfpaoGN65nQauhoY\nGvZdS+qvcY3WlYk3SvQnH44WfSmlZU4ffE/H4Ckw/tIwAZ1+gIVJPE8sf1cy9S31tncweju6mbkz\naexupHew1+/rcnI09zrgVTAzLIejErMRp+/vO7ZzojUI7JpTElPIT8/nWPcxn8/7a6iU01fYhWNE\n39cJe6z7GEIISqaUWPIey0uWT+jM9RYY75NwWA6z9dhWlpUs87nPkiklnBw8SdvJifWmRpy+neWa\negzeji4xIZGKnIqAIpqQoLnXlpbxjx/qOER+ej6ZKZk2RKvhK+aZuTM52H6QwWH/E9D7+46j7ZrD\nWUzFzonWIDSnryZdm1w4RvR9nbD6oCyrHNrS4qXsOL5jXL23t8B4Nz772vaRk5rjdw52IYTPztyB\nAW2OdX2x9WiJvr/VkYzMre8rZrtTO8PDWj+LtyClJqZSPKU4rAnMolWjrxPqAK0T/Sc4fuI4M3Nm\nWhjleAItleiJcvqTD9OiL4RYJYTYJYTYLYS43cfzBUKIvwohPhBC7BBCfNHXfnwd/NubtrN4qjWp\nHYDMlEzKssqoddeOPhbM6W9u2Ow3taPjS/S987x+Uw9RyOmDsekYfMVsd+VOe7s2YCjJxwJpwVI8\ngb7jaNTo6wQboOUdc11LHZV5lbgS7FuaKdBSiZ6o6p3JhynRF0K4gIeAVcAC4FohhPc16S3AVinl\nUqAauF8IMaFUxdcJa1W5picrpq0Yl9cP5vS3NG7x24mr40tAvd2fr0at9WQrfYN9ttVig/+Tuyo/\n+HKPvmLT9U4hAAAgAElEQVSORKrEnxgFW0XL3+Ivdk60BgZmq8wuC6lW3+7vGIw7/by0PE4Ongy4\nCJEitjDr9E8D9kgpD0gpB4B1wGqvbRoBfW7YLKBFSjkhMevP6XvOamkF3tMxBHP6W47578TV8eX0\njYj+7pbdtq2L6xmHT6dvYF59n4Jkc7lmIDEKVquflgaJiVoHtM7A0AAH2g/YNm01WJ/Tt3uiNTDu\n9PUBWsGm7VDEDmZFfzrgaWGOjDzmyWPAQiFEA7AN+GdfO9LnANHnBOkf6md3624WFC4wGeJ4vDtz\nAzl9KWXAyh0dX07fWwh8XcnYnc/X4/Dn9OvcdQEnXvMVs905/aBOP8Qplve17WN61nRSE1P9v8gk\nVuf07Z5obWhIS6Pl5RnbXuX1JxdmRwQFWUUVgDuAD6SU1UKI2cCrQoglUspxE3qsXbuWpCT43vfg\niiuqyZ2Xy8ycmZafrMtKlrHt2DaGhodwJbh8Ov3167XbBzsOkuJKCZp+mZM/h90tuxmWwyQIrR01\n4vTtzuf39WnTP2T5WIOjIL0AicTd46Yww7diFRZCnUdb1tLTQt9Qn2XVVL4I6PRDWEFr5kgfqN35\nfDDn9P2Kvo1O33NAohGU6DuHmpoaampqTO3DrOgfBco87pehuX1PzgL+A0BKuVcIsR+oAjZ5brR2\n7Vr+8he4/no45RT41bZfWZ7PB20SqeIpxdS11LGgcEFAp7+lceIka77ISskiJzWHI51HKM8uByaK\nfn7+2PTReudufUs9V8670qqPNgHvxUg8EUKMXqEEEv233x67ry/dZ2c6KpDTn5U7iwPtB0YbbF94\ni6jdE61BcKdfmF5Id383PQM9pCelj3/OqyN3cHiQva17o9K574/SzFK/awIoIkt1dTXV1dWj9++6\n666Q92E2vbMJmCOEqBBCJAPXAM95bbMLuBBACFGEJvg+p6X0PGHt6MTV0WfclHKiS/OMYUvjlgnT\nKfvDuwTSe79JSROnj45Gjb4nVQWBO3O9UyXR7mBMTUxlasbUkBYbj0TMwZy+EELrzPUhnN7x7mvb\nx7TMaaQlpdkQqYbRfL6OcvqTC1OiP9IhewvwMvAR8LSUslYIcbMQ4uaRzf4TOEUIsQ14Dfi2lNLn\nUheeIrP9uLnVsgKxokSr4NHXsPWcU9zb6QfL5+t4V8P4cn+eDcqwHGZ3627m5M8x81ECEuzkDjYd\ng7drrm2utbVcE4LHHCzF4+2c7Z5d05dx8IW/FE9mpjam4+TIom52D8qC0J1+WXYZR7qU6E8WTNfp\nSylfklJWSSkrpZR3jzz2qJTy0ZHbbinlJ6WUS6SUi6SUv/G3r0g6/c2Nm32erLroDw9LNjcGr9HX\n8Z6uOJjoH+08SlZKlm2LXusxBHT6QWr1J7jmluiXElbmBu7MnZDesTmn39WlrVWQGqTryZ/oCzE+\nZrvz+aCcfrzjmBG5MHbwHz9xnN7BXkqzSm15n+Uly9nauJWm48MTDv6UFK30b1dDA8Ny2HAM3qmS\nQA0K2D/njh5DoJM72Lz6BQXaIDO9wGeX295yTbDG6esC2tLTQv9Qv63jIIzOYWN0gFYkRD/knL4S\n/UmFo0RfF0Xd5dvVYZifnk9eWh7bj+zxefAXFsL63dqgLKMxhOr0I1GuGdQ151Wyv22/3/ls9Aaw\no0Ob9/9wx2HbFhbXMRKz0VG5usu3u+PZ0Lz0BgdoRSK9E6rTL0gvoKuvK+AEfYrYwVGirx/825vs\ny+frLC9ZztZjW3we/AUF8PejxvP5oK3j2tDVQO9gr988r6eji2aNvk5aUholmSXsb9vvdxs9xbOn\ndQ8VORUkuXzMj2AhwWIOVqvvKaB2j8SFEJy+gQFaUsqod5b7IkEkMC1zmnL7kwTHir5d+Xyd5SXL\nqW3f4tfpf+gOTfSTXEmjM1d2dGgOOcVrhUfPHLndNfpg7OQONh2D3lBFIrXT26utZZAZYALP2bmz\nR8s2fTFO9B1Qo69jpFa/oauBtKQ08tIMjpoKk1CdPqgUz2TCUaLvnd6xkxUlK9jfu9mv09/dbbwT\nV0cXUH/uzzu9E+2cPgTvzNVjtnuiNQg8rkAnLSmNgvQCv6kS7/ROJFyz0YnLDnce9jkCWm9YI5HP\nh9CdPoxU8CjRnxQ4SvQLC+F4ywC73LtYWLjQ1vdaXrKcRrGF/PyJJ2H61CZODp0IeWpbvVbfn9jq\nJ3f/UD9HOo8wM9e+qXPBmAsNNsWyfnVS67Z3+gUw7poDzcHjufrZLvcuxzj9jOQM0pPScfdMnAZU\nb1gjMecOhOn0M5XTnyw4SvRzc6ErqZ7SrDIykjOCv8AERVOKSBjMQOYcmPBcb85Wpg4uD7kDsCpf\nW3vWn5PSBXRf2z7KsstIdiWHGb0xjLjQYBOveaZ3nOKaA3Xm6qufHTs+yP62/bZOtAbGYwb/KZ5I\nNqxShuf0VXpn8uAo0U9IgPRZ26nKsTe1o5PWsZzmxM0THu9I30Jmd2ipHRhzzcHSO5HoxJVSK7fM\nzw+8nZH0zvHmYerc9qdKjLrmYFMsFxTAtkP7KckssXVkKxiPGfyLvmcKze5+k56eiQMSjaBEf/Lg\nKNEHSC7bzozUyIi+OLacI0NbJjzeKLaQ1BK66Ou1+v4un/V8cyTmg9EXI0kOcjExPWs6XX1ddPZ1\n+ny+oAAOtmoDybJTs22IdAyjDnRO3hz2tAWu1d92xP4rEwjR6fup1R9N70RoYFaoLh+U6E8mHCf6\ncup2iomM6PcfWMGeExNF/0DvZoaOBJ9ozZuijCL6h/o51Nzi88TKyNAc+EdNkSnXNHJyJ4gE5uTP\n8VvBU1gIh07an3YA47lmI4up7IpAwwqhiWhZdplf0T/e2U53f7dtAxJ1QmmkPFFr5U4eHCf6J7O2\nkdtvv+gPDkLP3uV82LJ5XEVF68lWOgdb6D4Yei5YCEFVQRX7O+t9nlj6kPuPjkdmYJbRkztQZ25h\nITQNRc41GxHQ2Xmz2d++32/ZZkEB7OuMjOiHmtP3JZy5udCZXEtVvr0zmEL4Tn9qxlTae9vpG+yz\nPihFRHGU6Lf0tDDk6oL2Ctvfq7UVclzTSBAJ4y5btzZuZfHUpbibw/tqqvKrONpX51cICgpgb7tz\nnD4EzusXFECba5ejqkrSk9LJT8v3u5pTYSEc7bV/YNbAgLboT7bBrJe/nL7LBekzapk5xVnpKE9c\nCS5KppTQ0NVgfVCKiOIo0f/w+IcUiUW43fa6HdAO/qmFYnTGTZ0tjVs4tWw5J09qi5CEytz8uTQP\n1/sV3NziTk4MdDE903uBMWsJpUIjkOgXFkJPurOcPgRO8RQWwnEZmSmV8/PH1kcIRqABWsnTdjEt\nOTINazhOH1Ref7LgKNHf3rSd8tTFE5boswP94Ndn3NTZ3LiZFSXLfS4VaISq/Co6Ev07/eSSeqYm\nzonIZbxRR1dV4H+K5cxMGMqrpSLTOTl9CFyrn5rbxqDstXWFLwhdQEumlODucdM/1D/xycJaCojM\nwKxwnD7AvIJ5fHDsA2sDUkQcR4n+tmPbqMpePGH5ODvQD359QRWdLY3aRGvhiv7c/Ln0ZvjO6QPI\nvHpyh+1N7UBornlu/lx2t2rLPXrT0deOSO4mtc/eDkYI3en7E/2T6XWk9ti74DyELqCuBBclmSUc\n7ZyYlurLrCWz19lO/6JZF/HqvletDUgRcRwl+vrCKZEQff3g90zvdPZ1crTrKFUFVT7XLjVCWcYc\nhnP2kJ7hu5OxP7OejF77RT8U15yVkkV2SrZPMapz15F6osr2lNvwsNbPElJ6x8/Ea+2JdbhanVW5\no+MrxdM72Etv8hFcHfbOYArmnP4nZn2Cvx38m+8rFUXM4BjRHxoe4qPmj1hR9rGIpHf0g788u5y+\noT4auxr54NgHLC5aTGJCYthO/2THFBL68jnS5bu8rTuljqSOyFSVhCJI/ubWr3XXkj0wz/aGWB9X\nkGRwEs9A8+o3De1isMmZnaK+RL++pZ4cZtLqtncGUzDn9AvSC5ibP5cNRzZYG5QiojhG9Pe07qF4\nSjEV0zIj6vSFEKMpns0Nm0fXxA3X6bvdkN7jP0feQj3S7SynD/6XTtzl3kWhmGd7QxxqvLNzZ7Ov\nbZ/PlNThnjpOHq5ieOJTlhKW0/cxQGuXexfTk+dHzOyEK/oAK2et5JW9r1gXkCLiOEb09Zk1dYft\nYzJCS/F0aStKVrC5cTNbjm1hxTRtUFa4Tr+5GbKH5voc7CSlpHGgnt6j9q2L6xlHKCe3vzl4drl3\nUZo63/aGONR4M5IzyE3L9ZmS2t1WR8bJKtrbLQzQB+E4fV+LqdQ21zIz0/7vGMKbbM2TlbOV6Mc6\npkVfCLFKCLFLCLFbCHG7n22qhRBbhRA7hBA1vrbZ1rSNxVMXk5KirTfa6XtWAMvwdGm60/dcCD1c\np9/cDFMTfAvose5jpCWm0daYayZ0Q4Tj9H01VLvcu5iVaX96Jxwx8tWZOzg8yL62fRS65kTk6sSK\nnH6tu5aqfPtFf2hIS6PlmZiu/8yyM6lrqaOlp8W6wBQRxZToCyFcwEPAKmABcK0QYr7XNjnAw8An\npZQfA/7B1762N21nSbG2Wla4ghsK3k7/vSPvsb9tPwsKFwDhO323G0rTfDv9upY65uRV2S5GfX3a\ngiRZIay57svp9w/1c6D9AHMLK22POZy0g69F0g+0H6Aoo4iivHTHXZ2Af9FfMs1+0W9t1aaedrnC\n30eyK5nzZpzHa/tesy4wRUQx6/RPA/ZIKQ9IKQeAdcBqr22uA34vpTwCIKX0KR+eC6dEQvQ9Xdqs\n3Fn0DfaxoHDB6HTHZpz+rCzfTr++pZ4FRXPp7NRGc9qFkcVIvKnIqaCxq5GTAydHH9vbupfy7HKm\nTU1xpNP31ZmrL5EYqWPIbEfu0PAQu1t2c0qF/WbAbD5fR6V4Yhuzoj8d8ExQHhl5zJM5QJ4Q4k0h\nxCYhxOd97cjd42ZW7iwgfJdtFH1Ocf2EFUKwrGTZuJWyzOT0Kwtn0NTdNE5AQV8tay55edq0x3YR\nzsmdmJDIzNyZ40RUn0Pfc5lHuwgnVeKrbFNfItHuYwjC+56zU7KRSDp6OwA42HGQgvQCZpRMsb0v\ny2w+X2fl7JW8su8Vn6uAKZxPosnXG/nVk4DlwCeAdOA9IcQGKeW4szVnQw4/uOsHAAwOVtPcXG0y\nNP+cODEyd3/62GOfXfjZcTMcmqneKZ6ayCw5i92tu8ct+1jfUs/ZZWePClJxsZlPETiGcE5ufTqG\nRUWLgDHR91zQ3S6am+FjHwvtNb5G5da561hctJjDNjt9KcNrqIQQo25/UeoibbWswvmjfVkdHVoK\nxg6scvpz8ubgEq6IzP+vGE9NTQ01NTWm9mFW9I8CZR73y9DcvieHAbeU8iRwUgjxFrAEGCf6l/3j\nZaz95FoAvv1te09YX1UXN59y87j7+fmaG5cytDSJfmLNHdTy+p6iX9cSmdRDuCe3d9lmrbuW6orq\niKfbjDI7bzZ7W/cyLIdJENpF666WXVy98Gp6C+Go7/nYLKGzk1GhDpVR0S9aNG4Off17tkv0rXL6\nQojRFI8S/chSXV1NdXX16P277ror5H2YTe9sAuYIISqEEMnANcBzXtv8Gfi4EMIlhEgHTgc+8t6R\npzjafWluRGBSUrQrgVDL/vQGxVtAB4YGONh+kNm5s20X0bCdfoG23KPOLrc2u2ZeHrS1adUfdhFO\n+eOU5Clkp2aPm/lRz+lH4hgKV0A9a/Vrm8fWKnCqGfCFnuJRxB6mRF9KOQjcAryMJuRPSylrhRA3\nCyFuHtlmF/BXYDuwEXhMShlQ9CNx8Bs5YcMRDl0M5ubPHSegB9oPMC1zGimJKbYLUrgnt+e8+lJK\nbWHxgioSE7Xpg9vaLA7Ug3BHinqmeDp6O+ju72Z65nRHC6hnZ66303dqQ+XNBTMvYP3B9Wp+/RjE\ndJ2+lPIlKWWVlLJSSnn3yGOPSikf9djmPinlQinlIinlg772o+eRwX7RNyowocYxNKQJY17exJkr\nPdfFdazTH8npSylp6GogLSmNvDStqNspDbE3nlMs6+kzIYRjv2MYG6ClN6x6isTuDnMrnX5eWh4L\nChfw7uF3rdmhImI4ZkRuTupYIjMSTtgOp9/aqjnixMSxuWz0Cge9qgScexlfkF6AQNDc0zzaiTv6\nnI2C1NsL/f3aNM6h4un0d7l3jX7HTr2agjGnf/zEca2BStcORic3VL5QpZuxiWNE35NYdfqeJ5V+\nIrt7NOXxdPpOzTfryz3WuetG8/k6dqYewhlXoFOZVzm6SLrngvNOFlBd9PXUjj4FtFPNgD9UXj82\niUvRt8vpe55UQggtrz8yMjeS6R0zJ7c+HYO307czZjMzP/pK74A2Y+fwMPT0WBXleMx8x9Mzp9PY\n3cjO4zsnfMdONAP+OH366ext3UvziQhMGqSwDEeKfmamdrnf22vP/u1y+t6NiecyhN6i79STW09L\n1bprI5beMSOglXmV7G3bO5of12MWwt4rKjPfcUpiCnlpebxx4I1xV1N2fsf6gEQrnX6SK4nqimo1\nJUOM4UjRt/uEtcvpewuBXg3T3d9N68lWyrLLRvdr18k9PKyNL8jPD+/1ekMV6fROuAKamZJJZnIm\nhzsPs7dtL3PyxmYwtfPqxKyAlmeX8/q+18fVudsZb0+Pdl5lZFi735WzV/Ly3pet3anCVhwp+uCM\ndEI4Tt9zv1X5Wt377pbdVOZVjg4g0gXUjlHsHR3aiZ2cHN7rqwqq2NywmbbettFGCpzr9EFz+6/v\ne53C9EIyksdUze5jyEyqpDy7nI6+jgkNq1O/Y3/onblOnJLhcMdhVv16FT0DNuX4YhTHin4sOn3v\n/epO3zO1A9g6fbSZ/DhoAtrQ1cDc/LmjjRQ41+mDNvHaC7tfGM3n69id3jHl9LPKSU1MZUbOjNHH\nnPwd+2N27mxSE1PZ2bzT+p2bQErJLS/dwsajG/m/Tf8X7XAchWNF3y7XMzioiW2ugSntzVTvgCZG\n+9r2UeuuHSf6YJ9zNrMGKkBqYioVORXj8vngbBdamVvJK3tfYV5+ZGM26/Sr8qvGNaxTpmjH58mT\nAV4YJnY5fSEEF8++2HGlm3/a9SfqW+p5+fqX+e93/1u5fQ/iTvRbWjTBNzKnuJnqHYD0pHSmZkzl\nlb2vjJYS6tj1+cw6UNBSPJ5pB7A3vWOF0+/q75rg9O36jgcGtEn7srPD38fppadz1fyrxj2m92XZ\ndVzY4fTBefX6nX2dfOOv3+DRyx/ltOmncWbpmTy66dHgL4wTHCv6dl2ahyKK2dlaBVGfwZHmvk6s\nqoIqNhzZ4NPp2/H5zDpQgK+u+CpXzLti3GN29kNYkdMHJjSsdh5D+fnaTK3hckbpGaw5b82Ex+1q\nqOxy+gDnzzyfdw6/Q++gTeV2IfK917/HxbMv5twZ5wKw5rw13Pvuvcrtj+BY0bfz4DcqikJoJ7dR\n4fC177l5c5HICaLvZKe/et7qcXMhgTb5XEKC5nCtxqwLHRX9CDl9K75jf9gZs11OPyc1h8VFi3n7\n0Nv2vEEIvH/0fZ6tfZZ7L7p39LGlxUuV2/cg7kQ/1BPWaBz+6qCrCqrIS8sjP318DaUTGrVQsTP1\nYEZEs1KyeHL1k5RllY17PBa/Y7s6c+10+gArZ0U/xTMwNMBX/vIV7l95/+icUTrK7Y/hWNF3SvrD\naBwnTmj9BJ4LswAsmrqIRVMXTdjeCemrULFDkPRxBWZjvmHpDaPTGejE4nccizl9cEZe/4END1A0\npYhrP3bthOeU2x/DsaIfa07fn5Oqrqjmxc+9GPZ+Q8VOR2dHzPq4gqQka/cLsev0nXDch8qp00/l\nYMdBjnUfs+9NArC/bT/3vHMPP7n0JxMaf501563hv9/97wnLmMYbcSf6djl9f/sVQpCelD7h8VjM\n3drhQu0U0NxcrVEZHLR2v7GY07fzewZtjeULZl4QlSkZpJR87cWv8c0zv8nsvNl+t1tavJQzSs/g\n0c3x7fYdK/p2rdZkl9MPVWztTF/FUnrHTgF1uTTht3oR+ljM6dvt9CF6ef1ndj7Dkc4j/OtZ/xp0\n2zXnreHed+6Na7fvWNG3a7UmO52+HY1JqNjp9O2I2W4HGmsNlR1XU0ND2rKfeXnBtzVDNKZkaDvZ\nxm0v38ZPL/8pSa7gOcKlxUs5vfT0uHb7jhV9sEdk7MzphyJedny2vj5tXEFWlrX71bFDkOx2oLHW\nUNkRb2urtti6kQGJZpiZO5PMlEy2N2239408+O7r32V11WrOLDvT8GvWnBvfbt/Rom9HCsQupx+q\nw87KGhNpqzCzGIkR7HDNdjv9WGuo7Gqk7E7t6EQyxfPOoXf4S/1fuPvCu0N63bKSZZxeejo/3fxT\nmyJzNqZFXwixSgixSwixWwhxe4DtThVCDAohrvK3jTdWnwBSRr56xx92TB9t98nthCuvUIm1hiov\nD7q6rO18trtc05NIrabVP9TPV57/Cg9c/MC4pVaNsubcNdzzzj1x6fZNib4QwgU8BKwCFgDXCiHm\n+9nuHuCvgGEfarXIdHf7rqUPhNnqnUBY/fnsPrljrXoHnGEcQiEhwfrO50g6/fNnns+GIxtsHwR1\n37v3UZFTwT8s+IewXh/Pbt+s0z8N2COlPCClHADWAat9bHcr8CwQ0ulntRMO52QtKNBOwOHh4PsO\nVbxi0enHUqcoWN9QdXZq02KnpFi3T2+sjjmSTj8rJYtlxctYf3C9be+xp3UPP3rvRzx86cN+a/KN\nEK9u36zoTwcOe9w/MvLYKEKI6WgNwSMjDxnu2rfapYXjKpOTtSuDjo7g+w5VvGLN6efkaFdLAwPW\n7TPWqnciUfpox3EfKacP9o7OlVLy1ee/ync+/h0qcipM7Ste3X6iydcbEfAHgO9IKaXQmmWfTfPa\ntWtHb1dXV1NdXU1hIWzaZDJCD8I9YfWTMNAc/E5I79h9cickaDlntxtKSqzZZ6xV79jdSIE9ZmDG\njODbWcXK2Su58bkbbdn3a/teo7G7kX85418s2d+ac9dw+W8v5ysrvkJaUpol+7STmpoaampqTO3D\nrOgfBTxnuSpDc/uerADWjVyGFQCXCCEGpJTPeW7kKfo6dqR3wjlh9TjmzvX9vD6/ek6I/Ul2fL6F\nC63bny9052yV6Mda9U6knL7Vab8VK6zbXzBWlKygoauBhq4GpmVOs3Tfv93xW25cdiOJCWalS2NZ\nyTJOnXYqj215jG+c/g1L9mknuiHWueuuu0Leh9n0ziZgjhCiQgiRDFwDjBNzKeUsKeVMKeVMtLz+\nP3kLvj+c4oSDxeF2aw441PnVnfL5QsFKEdXXKsjMtGZ/vrBDQO12+rGc0wdwJbj4xMxP8PIeaxdM\n7xvs4891f+bqhVdbut87z7szrnL7pkRfSjkI3AK8DHwEPC2lrBVC3CyEuNlscE7JeQdz5OEKgVM+\nXyhYGbPd4wpg7LezapCoyukb44p5V/D0zqct3ecre19hYeFCSrNKLd2vp9uPB0zX6UspX5JSVkkp\nK6WUd4889qiUcsI4Zynll6SUfzC6b6tPWDudvh2NSahE4uS20jlHopFKTdU6461ahD5Wc/qRdPqg\nif77R9+noavBsn2u27mOaxZeY9n+PNHd/tHOo7bs30k4ekSu1as12en07WhMQiUSJ7eVqYdIuGaw\nvqGKpYbV3+I+dpOelM5V86/i19t/bcn+egZ6eKH+hbDr8oOxrGQZt552K0sfXcpjmx+L6PxBkcbR\nog/WCqNd4uyE9I6+GEl+fvBtzWClIEXCNYP1x1AsNaw9PVr6LCPDmv2Fwg1LbuAX235hiYC+uPtF\nTp1+KkVTiiyIzDff+fh3eOMLb/DYlsf4xC8/wZ7WPba9VzRxvOhbmQKxy+mHu18rp4/WFyNJTja/\nr0DYkdO3m1i7OnGC0bGCj5d/nN7BXjY3bja9r3U71vHZhZ+1IKrALCpaxHs3vsflcy/njJ+dwf3v\n3s/gsMULMkQZx4u+E04AI04/nP0mJWkTr1kxfXSkTm4rBTSSTj+Wrk6s7MuKRj5fRwihuf0PfmFq\nP519nby671WunH+lRZEFxpXg4v+d+f/YeNNGXtj9Amf9/Cw+bPowIu8dCeJG9AcHtc68QAOs/BFM\n6MwIgVWfL1Ind6zlxyH2rk5SUoyNAjdCNJ0+wBeWfIF1O9fRP9Qf9j6eq3uOc8rPmbDYud3MzpvN\n6194na+s+AoX/PIC1ry5hr7BvojGYAeOF32r0jstLZrghzOneDChMyO4Vn2+WBTQSDl9q65O+vvD\nG4QXDlbFHE2nD1CRU8HCwoW8UP9C2Pt4eufTfPZj9qd2fCGE4KblN/HBzR+wrWkbyx5dxnuH34tK\nLFbheNF3ghMONve9GTdl1eeLlIDm51ubeoil6h29o9zOcQU6Vh4X0XT6oHXoPrntybBe23qylbcO\nvsWnqj5lbVAhMj1rOn+65k+srV7LVc9cxb/89V/o7u+OakzhEjeib+bgDzb3vdn0Tiw5fatTD7FU\nvROpeMEZZscq/mHBP/C3A3+j+UToH+iPtX/kolkXkZVi03JwISCE4OqFV7Pjn3bQ3tvO7pbd0Q4p\nLBwv+lamP8wc/P4ut6XUHGC4gmvVZXysClIsdT5HKl6wzgw4welnpmTyyapP8psPfxPya+0ckBUu\n+en5PHnFkywrWRbtUMLC8aLvBKevx+HrJGxv15xvuKWSsSagYI2I6uMKYim9E6sNa7SdPozV7IfC\n8RPH+fvRv3PZ3Mtsiio+iRvRt8vpmxWCWBUkszHr4wqSkqyJKRDx2rCCM5w+wPkV59Pc0xxS6eOz\nHz3LZXMvIz0phKXuFEFxvOhbld6xy+nb1ZiESqwJUiQbqayssRk9zaCcfvi4Elx8fvHnQ3L7kRqQ\nFW84XvStWq3JTqdvtjGJNUdnRcyRbKSsWoQ+0jn9WDsugnHDkht46sOnDI1wPdJ5hB3Hd7By9soI\nRKl34N8AAA9FSURBVBZfOF70ExLGygTN4GSn74SO6lCwIr0TSdcM1ohorKXQBge1NFpeZMc0+aWq\noIoZ2TMMLaX4u52/44p5V5CSaONixHGK40UfrHNpTs3pNzebq3vXxxBkRaiqzYr0TiRdM8RezFbE\n29qqXSmHMyDRLox26K7buS5qA7ImOzEh+la5NDucvtn9ZmRo6Qcz00dHYjEST2LNNUPsXZ3EWgrN\nKNd87Bpe3vMybSf9Tzi1r20f+9v2c8HMCyIYWfwQF6IvpfkTwJ/zsiKtYvZKJtJ5WysENNKCFGsi\nOmWKNvtqT0/4+4h0w2qEvLQ8Lpp9Ec/sfMbvNk/veJpPz/+0ZevgKsYTE6JvVhS7u7VL3HQTlV+B\nnL7ZE8usIEW6QsOqVEksxWyFcQgFIcw3rk50+hA8xRPNuXbigZgQfSeIYn6+NphoeHj841a4bLOf\nLxpOP9rptlAxK6CdndrSiykR7Fc021A5pVzTm4tnX8y+tn3Ut9RPeK62uZbmnmY+Xv7xKEQWH5gW\nfSHEKiHELiHEbiHE7T6e/5wQYpsQYrsQ4h0hxOJQ38MJopicrF1yt7ePf9wJ6Z1In9yZmVoJ7cmT\n4e8j0jFbcQxFWkCdcNzbQZIriesWXccvt/1ywnNP73yaqxdcjSvBQb3PkwxToi+EcAEPAauABcC1\nQoj5XpvtA86VUi4G/h34aajv4xRR9OW8nJDeifTJbUXde6RjtsI1R1pAnXCFaxc3LLmBX23/FcNy\n7NJZSsm6Heu45mPOmmtnsmHW6Z8G7JFSHpBSDgDrgNWeG0gp35NS6nMybgRKQ30Tp4iid4rg5EnN\n8U6ZYn6/sXZyx1q+2Wy80XL6sdSwhsKS4iXkpuZSc6Bm9LHtTdvpHezl9OmnRy+wOMCs6E8HDnvc\nPzLymD9uBF4M9U2cIoreblHfr9lSyVhzzWDuN4n0uAKw5hiK9Hc8WXP6Ot4duut2aLX5IlK1x3GK\n2Zoow0OKhBDnA18Gzvb1/Nq1a0dvV1dXU11dPXrfKaLo7byscn9OadRCwYwgRXpcAWijUltbtY74\nhDCsTrSc/qZN4b/eyU4f4HOLP8cPHvoB3f3dZCRlsG7nOv54zR+jHZajqampoaamxtQ+zIr+UaDM\n434Zmtsfx0jn7WPAKimlz1EZnqLvTUGBVjkjZXhC4XbD7Nmhv85XHJ5CZ2Vj4oT0VSiYST1EQ0A9\nF6HPzw/99Sqnbz1TM6ZyTvk5/P6j3zOvYB4prhSWFC2JdliOxtsQ33XXXSHvw2x6ZxMwRwhRIYRI\nBq4BnvPcQAhRDvwBuF5KuSecN0lO1mrsvStnjGKX07cybeSEjupQMCNI0aofNxNzrFXvSOl8pw9j\nKZ6ndz7NNQuvUamdCGBK9KWUg8AtwMvAR8DTUspaIcTNQoibRzZbA+QCjwghtgoh3g/nvcwIo1Wi\n6H0SOiG9oy9GEo57NYOZ9E60RopakZKKJGaupnp6tKvijAxrY7Kay+dezvam7fxi2y9U1U6EMD3O\nWUr5EvCS12OPety+CbjJ7PvowjhnTuivtcrxeDc8Vu03Nxe6urRKoFAXFdEXIwl35a5wMSNI0XT6\nsZSSMtuwOt3lA6QkpnDNwmt4+/DbLChcEO1w4oKYmdzCbDrBDkfudsMyC5bJTEjQOhpbWqC4OLTX\nRuvkjkWnH2spqbw8bSRwOGbA6fl8T75/3vc50jmhK1BhEzExDQOEn94ZGNBOnNxc62OwUrzCFaRo\nndyxJqAQew2VpxkIlVhx+gDFU4o5Zdop0Q4jbogZ0Q9XZFpbtRMnnDK9YDFYeWKF+/midXLHWqoE\nwo+5v1/LkWdnWx9TMMKNOZacviKyTHrRt1IUveecsfLECvdKJlond16eVv44NBT6a2Oteica4wp0\nwr06iSWnr4gsMSP6ThBF7zlnnJDeidbJnZioOd/W1tBfG2vVO9GcojjW0n4K5xMzou8UUdRFf2hI\nGzdg1fqjsXhym0k9xJLTj+ZiJE457hWTh0kv+laLoh6H1euPmrmSidbJHY5zllLrmIylfohoO/1o\nX+EqJhcxI/rhiqJdTt9q9xcvLrS9XRtXEGoJohXojVSoi9BH8ztWOX2F1cSM6DvN6Vt9UsWLC42m\nAw13Efpof8dOOO4Vk4eYEf0pU2BwMPTVmuxy+lafVGYcXSy50Gg70HAaqli7moLof88K5xIzoh/u\nak12On0npHdizYVG24GG01BF+zsO9ZgfHNSm57CqyEAxuYgZ0YfwRMbOnL7V+9WnjzZKNBYj8SRc\n1xxtpx/OMRRLV1NWFxkoJhcxJ/pOcfpW7zclBVJTNYdmlGgOGoLwXXM0nX64x1A0K6Tc7tDMQLS/\nY4WziSnRD1Vk7JhT3C6nD6G70Fh1zdGMOdx+iGiJaEpK6GtJRPs7VjibmBL9UEWmu1srDUxLsz4G\nO4Qg1D6LaDs6J1x5hUqox5A+riDS6xV4EmrM0bwyUTifmBP9UETGDseTn6/lTI8ft168Ys3ph1P3\nHu2YQz2GOjo005CSYl9MwQjnuFfpHYU/Ykr0Q700t8NVJiVpE6/t2RP99E60XXN6utZZGErde7Rd\naDjHULRdcyzGrHAuMSX6TnHCBQXaVLvRTu9E2zVD6IIUbRcazjEUbdccizErnItp0RdCrBJC7BJC\n7BZC3O5nmwdHnt8mhAh7ralQL3PtcsKFhdroTiv7CvT9xpLTh9jLN4dzDEW7YY2171jhbEyJvhDC\nBTwErAIWANcKIeZ7bXMpUCmlnAN8BXgk3PcLx1Xa5fTt2K9TrmRCIRQRjfa4Aoi9KxNQOX2FtZh1\n+qcBe6SUB6SUA8A6YLXXNp8CfgEgpdwI5AghisJ5M6c4nsJCe06qWKvegdBENNrjCkBbNrO7W1sM\nxwhOcM0qp6+wErOiPx047HH/yMhjwbYpDefN8vK0agqjqzXZ5XgKCuxLG8Wi0zcasxMcaKjrzjoh\nZpXTV1hJosnXGy3W8/Z2IU5uq+FyacPLr7lGy6cnJmrVNElJY7c9/2/YAJdfHs47Baaw0L4riL17\n4c47NSc6ODj23/O2/n/fvuif3MXF8L//C3/+s/b7JCRo//U/z/utrdFvpPSYL7pIm8TP3/Gj3964\nEW732VMVOQoLYds2+M53xuLyjNf7vhPMgMK5mBX9o0CZx/0yNCcfaJvSkcfGsXbt2tHb1dXVVFdX\n+3zDP/0JDh0aEz9vIfT8f+WVcPbZYX2ugHzmM3DBBdbvd8YM+OY3tdj1Oee9T27Px774RU3AosnX\nvqZ9x0ND2t/w8NhtX4/Nnx98n3bzwgvQ0OD/uPG8fd55sNo7YRlhFi+Gb39bm2FWNwAnTkw0A/rf\nV7+qHT+KyUdNTQ01NTWm9iFkqCtKeL5YiESgDvgE0AC8D1wrpaz12OZS4BYp5aVCiDOAB6SUZ3jt\nR5qJQ6FQKOIRIQRSypB6yUw5fSnloBDiFuBlwAX8XEpZK4S4eeT5R6WULwohLhVC7AFOAF8y854K\nhUKhCB9TTt+yIJTTVygUipAJx+nH1IhchUKhUJhDib5CoVDEEUr0FQqFIo5Qoq9QKBRxhBJ9hUKh\niCOU6CsUCkUcoURfoVAo4ggl+gqFQhFHKNFXKBSKOEKJvkKhUMQRSvQVCoUijlCir1AoFHGEEn2F\nQqGII5ToKxQKRRyhRF+hUCjiCCX6CoVCEUco0VcoFIo4Qom+QqFQxBFK9BUKhSKOCFv0hRB5QohX\nhRD1QohXhBA5PrYpE0K8KYTYKYTYIYT4hrlwFQqFQmEGM07/O8CrUsq5wOsj970ZAG6TUi4EzgC+\nLoSYb+I9Y5Kamppoh2Ar6vPFNpP5803mzxYuZkT/U8AvRm7/ArjCewMp5TEp5Qcjt7uBWmCaifeM\nSSb7gac+X2wzmT/fZP5s4WJG9IuklE0jt5uAokAbCyEqgGXARhPvqVAoFAoTJAZ6UgjxKlDs46nv\ned6RUkohhAywnynAs8A/jzh+hUKhUEQBIaVfrQ78QiF2AdVSymNCiBLgTSnlPB/bJQHPAy9JKR/w\ns6/wglAoFIo4R0opQtk+oNMPwnPADcA9I///5L2BEEIAPwc+8if4EHrQCoVCoQgPM04/D3gGKAcO\nAFdLKduFENOAx6SUlwkhPg68BWwH9Df6rpTyr6YjVygUCkXIhC36CoVCoYg9oj4iVwixSgixSwix\nWwhxe7TjsRohxAEhxHYhxFYhxPvRjscsQojHhRBNQogPPR4LOlAvFvDz2dYKIY6M/H5bhRCrohmj\nGfwNlpxEv5+/zzcpfkMhRKoQYqMQ4gMhxEdCiLtHHg/p94uq0xdCuIA64ELgKPB34FopZW3UgrIY\nIcR+YIWUsjXasViBEOIcoBv4pZRy0chj9wJuKeW9Iw13rpTS12A9R+Pns90JdEkpfxTV4CxACFEM\nFEspPxipqNuMNr7mS0yO38/f57uayfMbpkspe4QQicDbwL+ijZky/PtF2+mfBuyRUh6QUg4A64DV\nUY7JDiZNR7WUcj3Q5vVw0IF6sYCfzwaT5PfzM1hyOpPn9/P3+WDy/IY9IzeTARfa8RrS7xdt0Z8O\nHPa4f4SxH2myIIHXhBCbhBD/GO1gbCKkgXoxyK1CiG1CiJ/HaurDG6/BkpPu9/P4fBtGHpoUv6EQ\nIkEI8QHa7/SmlHInIf5+0Rb9eOhFPltKuQy4BG3uoXOiHZCdSC1fOJl+10eAmcBSoBG4P7rhmGck\n9fF7tMGSXZ7PTYbfz8dg0EnzG0oph6WUS4FS4FwhxPlezwf9/aIt+keBMo/7ZWhuf9IgpWwc+d8M\n/BEtpTXZaBrJpzIyUO94lOOxDCnlcTkC8DNi/PcbGSz5e+BXUkp9bM2k+f08Pt+v9c832X5DACll\nB/ACsIIQf79oi/4mYI4QokIIkQxcgzboa1IghEgXQmSO3M4AVgIfBn5VTKIP1AM/A/VilZGTSOdK\nYvj3CzBYclL8fv4+32T5DYUQBXpqSgiRBlwEbCXE3y/qdfpCiEuAB9A6JX4upbw7qgFZiBBiJpq7\nB23081Ox/vmEEL8FzgMK0PKHa4A/42OgXrRiDBcfn+1OoBotLSCB/cDNHvnTmMLfYEngfSbH7+fr\n890BXMsk+A2FEIvQOmoTRv5+JaX8b38DZf3uJ9qir1AoFIrIEe30jkKhUCgiiBJ9hUKhiCOU6CsU\nCkUcoURfoVAo4ggl+gqFQhFHKNFXKBSKOEKJvkKhUMQRSvQVCoUijvj/Wzjy3jzlrKwAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe4b829b0d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(X.mean(0),label=\"mean\")\n",
    "plt.plot(X.std(0), label=\"std\")\n",
    "_ = plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEACAYAAABPiSrXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFadJREFUeJzt3X/MXuV93/H3JxgodGBjmMwvQ9hmKlw1bUDFXdO6T0Qw\nTrQB6hC4W8HrrGmqtzbaH1uhf2AjsjT0j1GiDTRlBAxqKVZYwV2RsQtBq6aCoYOExPFsulmyH+IH\namLIumbCynd/3NcT33YcP5cNvu/H5v2Sjp7r/p7rXPd1jvycj885922nqpAkqcdHxj0BSdKJw9CQ\nJHUzNCRJ3QwNSVI3Q0OS1M3QkCR16wqNJPOSfCXJt5JsTbIkyfwkm5NsT7Ipybyh/nck2ZFkW5Jl\nQ/WrkrzW1t03VD89yeOt/kKSS4fWrWzvsT3JbR/UjkuSjl7vlcZ9wNNVdQXwMWAbcDuwuaouB55t\nr0myGLgFWAwsB+5PkjbOA8CqqloELEqyvNVXAXtb/V7gnjbWfOBO4Oq2rBkOJ0nSaM0YGknmAr9Y\nVV8GqKr9VfUOcD2wrnVbB9zY2jcAj1XVe1W1E3gdWJLkAuCsqtrS+j0ytM3wWE8A17T2dcCmqtpX\nVfuAzQyCSJI0Bj1XGpcBbyV5KMn/SPKlJD8OLKiqqdZnCljQ2hcCu4e23w1cdJj6ZKvTfu6CQSgB\n7yQ59whjSZLGoCc05gBXAvdX1ZXAX9NuRU2rwb9F4r9HIkknuTkdfXYDu6vqpfb6K8AdwJ4k51fV\nnnbr6c22fhJYOLT9xW2MydY+tD69zSXAG0nmAHOram+SSWBiaJuFwHPDk0tiWEnSMaiqzNzrYDNe\naVTVHmBXkstb6VPAN4E/Bla22krgydbeAKxIclqSy4BFwJY2zrvtk1cBbgWeGtpmeqybGDxYB9gE\nLGuf3joHuBZ45jBzdKlizZo1Y5/DbFk8Fh4Lj8WRl2PVc6UB8BvA7yc5DfhL4NeAU4D1SVYBO4Gb\n2wl8a5L1wFZgP7C6DsxwNfAwcAaDT2NtbPUHgUeT7AD2AivaWG8nuRuYvsq5qwYPxCVJY9AVGlX1\nNeBnD7PqUz+i/+eBzx+m/hfATx2m/v9ooXOYdQ8BD/XMU5J0fPmN8JPIxMTEuKcwa3gsDvBYHOCx\neP/yfu5tzQZJ6kTfB0katSTU8XgQLknSNENDktTN0JAkdTM0JEndDA1JUjdDQ5LUzdCQJHUzNCRJ\n3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LUzdCQJHUzNCRJ\n3eaMewKS9GGTZNxTOGaGhiSNRY35/Y8tuLw9JUnqZmhIkroZGpKkboaGJKlbV2gk2Znk60leSbKl\n1eYn2Zxke5JNSeYN9b8jyY4k25IsG6pfleS1tu6+ofrpSR5v9ReSXDq0bmV7j+1JbvtgdluSdCx6\nrzQKmKiqj1fV1a12O7C5qi4Hnm2vSbIYuAVYDCwH7s+Bz5c9AKyqqkXAoiTLW30VsLfV7wXuaWPN\nB+4Erm7LmuFwkiSN1tHcnjr081nXA+taex1wY2vfADxWVe9V1U7gdWBJkguAs6pqS+v3yNA2w2M9\nAVzT2tcBm6pqX1XtAzYzCCJJ0hgczZXGnyZ5Ock/b7UFVTXV2lPAgta+ENg9tO1u4KLD1CdbnfZz\nF0BV7QfeSXLuEcaSJI1B75f7PlFV307yt4HNSbYNr6yqSjLub6pIko6zrtCoqm+3n28l+SMGzxem\nkpxfVXvarac3W/dJYOHQ5hczuEKYbO1D69PbXAK8kWQOMLeq9iaZBCaGtlkIPHfo/NauXfuD9sTE\nBBMTE4d2kaQPuefb8v6k6sgXCEnOBE6pqu8m+XFgE3AX8CkGD6/vSXI7MK+qbm8Pwv+AQbBcBPwp\n8Pfa1ciLwG8CW4A/Ab5YVRuTrAZ+qqp+PckK4MaqWtEehL8MXMngmcpfAFe25xvT86uZ9kGSZpPB\nZ4PGfd4KVXXU/5ZIz5XGAuCP2geg5gC/X1WbkrwMrE+yCtgJ3AxQVVuTrAe2AvuB1UNn9dXAw8AZ\nwNNVtbHVHwQeTbID2AusaGO9neRu4KXW767hwJAkjdaMVxqznVcakk40J/KVht8IlyR1MzQkSd0M\nDUlSN0NDktTN0JAkdTM0JEndTor/I/zTn75lrO8/Zw787u+u5YorrhjrPCTpeDspQmPjxl8e6/uf\neeYa3nrrLUND0knvpAiNwX/fMT6nnvofx/r+kjQqPtOQJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQk\nSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LUzdCQJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQk\nSd0MDUlSN0NDktStKzSSnJLklSR/3F7PT7I5yfYkm5LMG+p7R5IdSbYlWTZUvyrJa23dfUP105M8\n3uovJLl0aN3K9h7bk9z2weyyJOlY9V5pfBbYClR7fTuwuaouB55tr0myGLgFWAwsB+5PkrbNA8Cq\nqloELEqyvNVXAXtb/V7gnjbWfOBO4Oq2rBkOJ0nS6M0YGkkuBj4D/GdgOgCuB9a19jrgxta+AXis\nqt6rqp3A68CSJBcAZ1XVltbvkaFthsd6Arimta8DNlXVvqraB2xmEESSpDHpudK4F/g3wPeHaguq\naqq1p4AFrX0hsHuo327gosPUJ1ud9nMXQFXtB95Jcu4RxpIkjcmcI61M8g+AN6vqlSQTh+tTVZWk\nDrdudNYOtSfaIkk64Pm2vD9HDA3g54Hrk3wG+DHg7CSPAlNJzq+qPe3W05ut/ySwcGj7ixlcIUy2\n9qH16W0uAd5IMgeYW1V7k0xy8Nl/IfDc4ae5dobdkKQPuwkOPqXedUyjHPH2VFX9dlUtrKrLgBXA\nc1V1K7ABWNm6rQSebO0NwIokpyW5DFgEbKmqPcC7SZa0B+O3Ak8NbTM91k0MHqwDbAKWJZmX5Bzg\nWuCZY9pLSdIHYqYrjUNN34b6ArA+ySpgJ3AzQFVtTbKewSet9gOrq2p6m9XAw8AZwNNVtbHVHwQe\nTbID2MsgnKiqt5PcDbzU+t3VHohLksYkB87pJ6bB85Tx7sPcuUvZsOFzLF26dKzzkHRiGNxwGfe5\nN1RVZu53ML8RLknqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSepmaEiSuhka\nkqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSepmaEiSuhka\nkqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKnbEUMjyY8leTHJq0m2JvmdVp+fZHOS7Uk2JZk3tM0d\nSXYk2ZZk2VD9qiSvtXX3DdVPT/J4q7+Q5NKhdSvbe2xPctsHu+uSpKN1xNCoqu8Bn6yqnwE+Bnwy\nyS8AtwObq+py4Nn2miSLgVuAxcBy4P4kacM9AKyqqkXAoiTLW30VsLfV7wXuaWPNB+4Erm7LmuFw\nkiSN3oy3p6rq/7bmacApwHeA64F1rb4OuLG1bwAeq6r3qmon8DqwJMkFwFlVtaX1e2Rom+GxngCu\nae3rgE1Vta+q9gGbGQSRJGlMZgyNJB9J8iowBXy1qr4JLKiqqdZlCljQ2hcCu4c23w1cdJj6ZKvT\nfu4CqKr9wDtJzj3CWJKkMZkzU4eq+j7wM0nmAs8k+eQh6ytJHa8J9lk71J5oiyTpgOfb8v7MGBrT\nquqdJH8CXAVMJTm/qva0W09vtm6TwMKhzS5mcIUw2dqH1qe3uQR4I8kcYG5V7U0yycFn/4XAc4ef\n3dre3ZCkD6kJDj6l3nVMo8z06anzph8+JzkDuBZ4BdgArGzdVgJPtvYGYEWS05JcBiwCtlTVHuDd\nJEvag/FbgaeGtpke6yYGD9YBNgHLksxLck5772eOaS8lSR+Ima40LgDWJfkIg4B5tKqeTfIKsD7J\nKmAncDNAVW1Nsh7YCuwHVlfV9K2r1cDDwBnA01W1sdUfBB5NsgPYC6xoY72d5G7gpdbvrvZAXJI0\nJjlwTj8xDZ6njHcf5s5dyoYNn2Pp0qVjnYekE8Pghsu4z72hqjJzv4P5jXBJUjdDQ5LUzdCQJHUz\nNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LUzdCQJHUz\nNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LUzdCQJHUz\nNCRJ3WYMjSQLk3w1yTeTfCPJb7b6/CSbk2xPsinJvKFt7kiyI8m2JMuG6lclea2tu2+ofnqSx1v9\nhSSXDq1b2d5je5LbPrhdlyQdrZ4rjfeAf11VPwn8HPAvk1wB3A5srqrLgWfba5IsBm4BFgPLgfuT\npI31ALCqqhYBi5Isb/VVwN5Wvxe4p401H7gTuLota4bDSZI0WjOGRlXtqapXW/v/AN8CLgKuB9a1\nbuuAG1v7BuCxqnqvqnYCrwNLklwAnFVVW1q/R4a2GR7rCeCa1r4O2FRV+6pqH7CZQRBJksbgqJ5p\nJPko8HHgRWBBVU21VVPAgta+ENg9tNluBiFzaH2y1Wk/dwFU1X7gnSTnHmEsSdIYzOntmORvMbgK\n+GxVfffAHSeoqkpSx2F+ndYOtSfaIkk64Pm2vD9doZHkVAaB8WhVPdnKU0nOr6o97dbTm60+CSwc\n2vxiBlcIk619aH16m0uAN5LMAeZW1d4kkxycAAuB5354hmt7dkOSPsQmOPh0etcxjdLz6akADwJb\nq+r3hlZtAFa29krgyaH6iiSnJbkMWARsqao9wLtJlrQxbwWeOsxYNzF4sA6wCViWZF6Sc4BrgWeO\nYT8lSR+AniuNTwC/Cnw9ySutdgfwBWB9klXATuBmgKrammQ9sBXYD6yuqulbV6uBh4EzgKeramOr\nPwg8mmQHsBdY0cZ6O8ndwEut313tgbgkaQxy4Hx+Yho8SxnvPsydu5QNGz7H0qVLxzoPSSeGwc2W\ncZ97Q1Vl5n4H8xvhkqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSp\nm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSp\nm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkrrNGBpJvpxkKslrQ7X5STYn2Z5kU5J5Q+vuSLIj\nybYky4bqVyV5ra27b6h+epLHW/2FJJcOrVvZ3mN7kts+mF2WJB2rniuNh4Dlh9RuBzZX1eXAs+01\nSRYDtwCL2zb3J0nb5gFgVVUtAhYlmR5zFbC31e8F7mljzQfuBK5uy5rhcJIkjd6MoVFVfwZ855Dy\n9cC61l4H3NjaNwCPVdV7VbUTeB1YkuQC4Kyq2tL6PTK0zfBYTwDXtPZ1wKaq2ldV+4DN/HB4SZJG\n6FifaSyoqqnWngIWtPaFwO6hfruBiw5Tn2x12s9dAFW1H3gnyblHGEuSNCbv+0F4VRVQH8BcJEmz\n3Jxj3G4qyflVtafdenqz1SeBhUP9LmZwhTDZ2ofWp7e5BHgjyRxgblXtTTIJTAxtsxB47vDTWTvU\nnjhkM0kSPN+W9+dYQ2MDsJLBQ+uVwJND9T9I8u8Z3EpaBGypqkrybpIlwBbgVuCLh4z1AnATgwfr\nAJuAz7eH3wGuBX7r8NNZe4y7IUkfFhMc/Bfqu45plBlDI8ljwC8B5yXZxeATTV8A1idZBewEbgao\nqq1J1gNbgf3A6nb7CmA18DBwBvB0VW1s9QeBR5PsAPYCK9pYbye5G3hpeg/bA3FJ0pjkwDn9xJSk\nxv1IZe7cpWzY8DmWLl061nlIOjEMvokw7nNvqKrM3O9gfiNcktTN0JAkdTM0JEndDA1JUjdDQ5LU\nzdCQJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LU\nzdCQJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUrdZHxpJ\nlifZlmRHkt8a93wk6cNsVodGklOA/wAsBxYDv5LkivHOavZ6/vnnxz2FWcNjcYDH4gCPxfs3q0MD\nuBp4vap2VtV7wB8CN4x5TrOWvxAHeCwO8Fgc4LF4/2Z7aFwE7Bp6vbvVJEljMGfcE5hB9XQ6++x/\neLzncUTf+943xvr+kjQqqeo6L49Fkp8D1lbV8vb6DuD7VXXPUJ/ZuwOSNItVVY52m9keGnOA/wlc\nA7wBbAF+paq+NdaJSdKH1Ky+PVVV+5P8K+AZ4BTgQQNDksZnVl9pSJJml9n+6akf6PmSX5IvtvVf\nS/LxUc9xVGY6Fkn+STsGX0/y35N8bBzzHIXeL38m+dkk+5P88ijnN0qdvyMTSV5J8o0kz494iiPT\n8TtyXpKNSV5tx+KfjmGax12SLyeZSvLaEfoc3Xmzqmb9wuDW1OvAR4FTgVeBKw7p8xng6dZeArww\n7nmP8Vj8fWBuay//MB+LoX7PAf8V+EfjnvcY/1zMA74JXNxenzfueY/xWKwFfmf6OAB7gTnjnvtx\nOBa/CHwceO1HrD/q8+aJcqXR8yW/64F1AFX1IjAvyYLRTnMkZjwWVfXnVfVOe/kicPGI5zgqvV/+\n/A3gK8Bbo5zciPUci38MPFFVuwGq6q9GPMdR6TkW3wbObu2zgb1VtX+EcxyJqvoz4DtH6HLU580T\nJTR6vuR3uD4n48nyaL/wuAp4+rjOaHxmPBZJLmJwwniglU7Wh3g9fy4WAfOTfDXJy0luHdnsRqvn\nWHwJ+MkkbwBfAz47ornNNkd93pzVn54a0vuLfuhnjk/GE0T3PiX5JPDPgE8cv+mMVc+x+D3g9qqq\nJOGH/4ycLHqOxanAlQw+wn4m8OdJXqiqHcd1ZqPXcyx+G3i1qiaS/F1gc5KfrqrvHue5zUZHdd48\nUUJjElg49Hohg0Q8Up+LW+1k03MsaA+/vwQsr6ojXZ6eyHqOxVXAHw7ygvOATyd5r6o2jGaKI9Nz\nLHYBf1VVfwP8TZL/Bvw0cLKFRs+x+Hng3wFU1V8m+d/ATwAvj2SGs8dRnzdPlNtTLwOLknw0yWnA\nLcChv/QbgNvgB98k31dVU6Od5kjMeCySXAL8F+BXq+r1McxxVGY8FlX1d6rqsqq6jMFzjV8/CQMD\n+n5HngJ+IckpSc5k8OBz64jnOQo9x2Ib8CmAdg//J4D/NdJZzg5Hfd48Ia406kd8yS/Jv2jr/1NV\nPZ3kM0leB/4a+LUxTvm46TkWwJ3AOcAD7W/Y71XV1eOa8/HSeSw+FDp/R7Yl2Qh8Hfg+8KWqOulC\no/PPxeeBh5J8jcFfnv9tVb09tkkfJ0keA34JOC/JLmANg9uUx3ze9Mt9kqRuJ8rtKUnSLGBoSJK6\nGRqSpG6GhiSpm6EhSepmaEiSuhkakqRuhoYkqdv/B1QphNf4SUz/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe49a5fb350>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = plt.hist(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "X_train,X_test,y_train,y_test = train_test_split(X,y, test_size=HOLDOUT_SIZE, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.58947229, -0.16173296,  0.33721739, ...,  3.89330411,\n",
       "         3.33545494,  2.58757854],\n",
       "       [ 0.94304585, -0.18218632,  0.85101897, ...,  0.758636  ,\n",
       "         1.22592688,  1.08016479],\n",
       "       [ 0.65535557, -1.21946299, -0.30652437, ...,  2.74427032,\n",
       "         1.49601126,  1.17035723],\n",
       "       ..., \n",
       "       [ 1.37787545, -1.61781621, -1.44010878, ...,  0.92217571,\n",
       "         0.90575123,  1.07951367],\n",
       "       [ 1.31620133, -0.95843947, -1.23813927, ...,  1.2260201 ,\n",
       "         1.04802704,  0.89940441],\n",
       "       [ 0.28549418, -1.88663161, -1.11218572, ...,  1.3549546 ,\n",
       "         0.86062706,  0.73824358]], dtype=float32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True, False, False, ..., False, False, False], dtype=bool)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start from logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 'dims')\n",
      "Building model...\n"
     ]
    }
   ],
   "source": [
    "dims = X.shape[1]\n",
    "\n",
    "def build_logistic_regression(dims):\n",
    "    print(dims, 'dims')\n",
    "    print(\"Building model...\")\n",
    "    model = Sequential()\n",
    "    model.add(Dense(1, input_shape=(dims,), init='zero'))\n",
    "    model.add(Activation('sigmoid'))\n",
    "    model.compile(optimizer='rmsprop', loss='binary_crossentropy')\n",
    "    return model\n",
    "model = build_logistic_regression(dims)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "500000/500000 [==============================] - 4s - loss: 0.6788     \n",
      "Epoch 2/5\n",
      "500000/500000 [==============================] - 4s - loss: 0.6639     \n",
      "Epoch 3/5\n",
      "500000/500000 [==============================] - 5s - loss: 0.6561     \n",
      "Epoch 4/5\n",
      "500000/500000 [==============================] - 4s - loss: 0.6517     \n",
      "Epoch 5/5\n",
      "500000/500000 [==============================] - 4s - loss: 0.6492     \n",
      "CPU times: user 20.6 s, sys: 3.57 s, total: 24.2 s\n",
      "Wall time: 24.5 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fe4f1712510>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model.fit(X_train, y_train, batch_size=BATCH_SIZE, nb_epoch=5)\n",
    "#from sklearn.metrics import roc_auc_score\n",
    "\n",
    "#print (\"train AUC:\", roc_auc_score(y_train, model.predict(X_train)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('train AUC:', 0.6616958203896921)\n",
      "('val AUC:', 0.66207023106054352)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "print (\"train AUC:\", roc_auc_score(y_train, model.predict(X_train)))\n",
    "print (\"val AUC:\", roc_auc_score(y_test, model.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callbacks\n",
    "\n",
    "A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. You can pass a list of callbacks (as the keyword argument "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class AucHistories(keras.callbacks.Callback):\n",
    "    def __init__(self, X_train=None, y_train=None):\n",
    "        self.X_train = X_train\n",
    "        self.y_train = y_train\n",
    "        \n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.aucs = []\n",
    "        self.train_aucs = []\n",
    "        self.losses = []\n",
    " \n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        self.losses.append(logs.get('loss'))\n",
    "        if self.model.validation_data is not None:\n",
    "            y_pred = self.model.predict(self.model.validation_data[0])\n",
    "            self.aucs.append(roc_auc_score(self.model.validation_data[1], y_pred))\n",
    "        if self.X_train is not None:\n",
    "            y_pred = self.model.predict(self.X_train)\n",
    "            self.train_aucs.append(roc_auc_score(self.y_train, y_pred))\n",
    "        return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 'dims')\n",
      "Building model...\n",
      "Train on 500000 samples, validate on 500000 samples\n",
      "Epoch 1/20\n",
      "500000/500000 [==============================] - 35s - loss: 0.6790 - val_loss: 0.6696\n",
      "Epoch 2/20\n",
      "500000/500000 [==============================] - 41s - loss: 0.6642 - val_loss: 0.6593\n",
      "Epoch 3/20\n",
      "500000/500000 [==============================] - 32s - loss: 0.6561 - val_loss: 0.6534\n",
      "Epoch 4/20\n",
      "500000/500000 [==============================] - 40s - loss: 0.6516 - val_loss: 0.6501\n",
      "Epoch 5/20\n",
      "500000/500000 [==============================] - 35s - loss: 0.6491 - val_loss: 0.6484\n",
      "Epoch 6/20\n",
      "500000/500000 [==============================] - 40s - loss: 0.6476 - val_loss: 0.6471\n",
      "Epoch 7/20\n",
      "500000/500000 [==============================] - 39s - loss: 0.6465 - val_loss: 0.6462\n",
      "Epoch 8/20\n",
      "500000/500000 [==============================] - 37s - loss: 0.6457 - val_loss: 0.6455\n",
      "Epoch 9/20\n",
      "500000/500000 [==============================] - 42s - loss: 0.6451 - val_loss: 0.6449\n",
      "Epoch 10/20\n",
      "500000/500000 [==============================] - 37s - loss: 0.6445 - val_loss: 0.6444\n",
      "Epoch 11/20\n",
      "500000/500000 [==============================] - 41s - loss: 0.6440 - val_loss: 0.6439\n",
      "Epoch 12/20\n",
      "500000/500000 [==============================] - 32s - loss: 0.6435 - val_loss: 0.6435\n",
      "Epoch 13/20\n",
      "500000/500000 [==============================] - 42s - loss: 0.6431 - val_loss: 0.6430\n",
      "Epoch 14/20\n",
      "500000/500000 [==============================] - 33s - loss: 0.6428 - val_loss: 0.6428\n",
      "Epoch 15/20\n",
      "500000/500000 [==============================] - 41s - loss: 0.6424 - val_loss: 0.6424\n",
      "Epoch 16/20\n",
      "500000/500000 [==============================] - 35s - loss: 0.6421 - val_loss: 0.6421\n",
      "Epoch 17/20\n",
      "500000/500000 [==============================] - 42s - loss: 0.6418 - val_loss: 0.6418\n",
      "Epoch 18/20\n",
      "500000/500000 [==============================] - 36s - loss: 0.6415 - val_loss: 0.6416\n",
      "Epoch 19/20\n",
      "500000/500000 [==============================] - 42s - loss: 0.6413 - val_loss: 0.6413\n",
      "Epoch 20/20\n",
      "500000/500000 [==============================] - 34s - loss: 0.6411 - val_loss: 0.6411\n"
     ]
    }
   ],
   "source": [
    "logistic_auc = AucHistories()\n",
    "model = build_logistic_regression(dims)\n",
    "model.reset_states()\n",
    "logistic_history = model.fit(X_train, y_train, nb_epoch=N_EPOCH, batch_size=BATCH_SIZE, callbacks=[logistic_auc], validation_data=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEACAYAAAC+gnFaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VdW5//HPQwAlGmYLyhRUFOcZqKLESkv01qntFVCR\nTpb24lDbKtr7s+ba1goWWtuqRQXLbbHWoWWwAnIrqUpBiVIBSRQVlICiDAIqSkKe3x9rh5wcMpPk\nTN/363Ve7Hk/OR7Xs/dae61t7o6IiGSuNokOQEREEkuJQEQkwykRiIhkOCUCEZEMp0QgIpLhlAhE\nRDJcvYnAzPLNrMTM1pjZhFq2yTOz5Wa2yswKo2VHR8sqP9vN7LpoXYGZlcasy2/Wv0pERBrM6upH\nYGZZwGvAcGADsAwY7e7FMdt0BhYDI9y91My6u/vmuOO0ifYf5O7rzew2YKe7T2n2v0hERBqlvjuC\nQcAb7r7O3cuAR4CL47a5HHjC3UsB4pNAZDjwpruvj1lmTYxZRESaUX2JoBcQW3iXRstiDQC6mtki\nMysyszE1HGcU8HDcsmvN7BUzmxbdVYiISALUlwgaMv5EO+BU4AJgBHCrmQ2oXGlm7YELgcdi9rkP\n6A+cDLwLTG5EzCIi0oza1rN+A9AnZr4P4a4g1npgs7vvAnaZ2bPAScCaaP35wEvu/kHlDu7+fuW0\nmT0IzK3p5GamgZBERBrJ3RtV9V7fHUERMMDMcqMr+5HAnLhtZgNDzSzLzLKBwcDqmPWjgT/H7mBm\nh8bMXgqsrC0Ad9enGT633XZbwmNIp4++T32fzfmpqHA2bHDmz3emTt2/YzVFnXcE7l5uZtcAC4As\nYJq7F5vZuGj9VHcvMbP5wAqgAnjA3VdHBf5BhIbiq+MOPdHMTiZUPa0FxjUpehGRFLNzJ6xaBStX\nVv9s3RrWm8EVV8BBB7VeTPVVDeHu84B5ccumxs3/EvhlDft+DHSvYflVjY5URCSFlJfD669XFfQr\nVoR/162rez93WL0azjijVcIEGpAIJD3k5eUlOoS0ou+zeaXy9+kO774bCvrKwn7lSiguht27G36c\nnBw44YTwOfjglou3JnV2KEs0M/Nkjk9EMsunn4ar9RUr4JVXqv7dsqXhx2jbFo4+uqrQP/HE8G/f\nvqFaaH+ZGd7IxmLdEYiIxHGHjRv3LfBfew327Gn4cfr0qSrwKz8DB0L79i0Xe1MoEYhIRquoCFf5\ny5ZVL/gbc5Wfk1N1ZV95lX/88dA5RbrKKhGISEZxh5ISWLQofAoLYXNNA+PU4sgjQ0F/0klV//br\nB21SeCxnJQIRSWvusGZN9YJ/06b696u8yo8t8I8/vvUbcluDEoGIpBV3eOut6gX/xo1179OtG5x9\nNpxySlXBn5vbPI23qUCJQERS3rp11Qv+9evr3r5LFxg2DM49N3yOOy61q3b2lxKBiKSULVugqCg0\n7lZ+3n237n06dYJzzqkq+E88MbML/nhKBCKStD76CF5+uXqh/9Zb9e+XkxMK/ry8UPCffDJkZbV4\nuClLiUBEksJnn4XHNmML/eLi8HhnfQ4+GIYOrSr4Tz01dNyShtFXJSIJsWsX/OMfMH8+vPBCSAIN\nGZKhffvQoDtoUBiP54wzQk9dXfE3nRKBiLSaTZvgySdhzhxYuDAkg7q0aQPHHltV4J9xRqjfT7ae\nualOiUBEWow7vPpqKPjnzg1X/nUNH3b44dWv9E85JT2f2082SgQi0qzKyuDZZ0PBP2cOrF1b+7YD\nB8JFF8EXvgCnnx6e55fWp0QgIvvtww9h3rxQ8M+bB9u317xdmzah49ZFF8GFF8KAATVvJ61LiUBE\nGm3PnjA4W2Eh/P3v4Q6gvLzmbXNy4PzzQ8F//vm66k9GSgQiUq/du+Gll0KB/+yz8PzzsGNH7dv3\n7Ruu+i+6KPTgVeNuclMiEJF97NoVGnYrC/4lS+CTT+re54wzqgr/E07InHF60oESgYiwc2co7P/5\nz1Dwv/hi/c/0H3ZY1Xg9//EfYV5SkxKBSAbaujVU71Re8b/8cv1v3urfPwzbMGxY+Pfww3XVny6U\nCEQyQGkpPPdc1WfVqvr3GTgwFPiVnz59Wj5OSQwlApE04w6vv1694K/rWX4IV/YnnlhV6J99NvTo\n0TrxSuLVmwjMLB/4NZAFPOjuE2vYJg/4FdAO2OzueWZ2NPBIzGaHA7e6+2/MrCvwF6AfsA64zN0/\n3M+/RSQjVT7KGVvwv/9+3ftkZcFpp4UCf9gwOOss6Nq1deKV5GNeR39vM8sCXgOGAxuAZcBody+O\n2aYzsBgY4e6lZtbd3TfHHadNtP8gd19vZpMICWOSmU0Aurj7zTWc3+uKTyQTucPKlWHMnueeg8WL\nQ2NvXTp0gCFDQsF/zjlh+qCDWideaV1mhrs3qvWmvjuCQcAb7r4uOsEjwMVAccw2lwNPuHspQHwS\niAwH3nT3yvcGXQQMi6ZnAIXAPolARKqsXQt//jM8/HAYv6cunTuHQr/yc+qpepZfaldfIugFxL70\nrRQYHLfNAKCdmS0CcoC73f2PcduMAh6Ome/h7pWvj94EqDZSpAabNsFjj4XCf8mS2rfr1at6wZ/p\nr16UxqkvETSkXqYdcCpwHpANLDGzpe6+BsDM2gMXAhNqPIG7m1mt5ykoKNg7nZeXR15eXgNCEkld\nO3bA3/4WCv9//KPmxzqzs0PHrfz8UPD3769HOTNVYWEhhYWF+3WM+toIhgAF7p4fzd8CVMQ2GEd1\n/B3cvSCafxCY7+6PR/MXA9+rPEa0rATIc/f3zOxQYJG7D6zh/GojkIzw6adhsLaHHw6jdn722b7b\ntG0LI0bA5ZeHJKDhmaUmLdFGUAQMMLNcYCMwEhgdt81s4HdRw/IBhKqjKTHrRwN/jttnDjAWmBj9\nO6sxQYukgz17YNGiUPj/9a+1j9h5zjmh8P/qV6F799aNUTJDnXcEAGZ2PlWPj05z91+Y2TgAd58a\nbfMj4BtABfCAu/8mWn4Q8DbQ3913xhyzK/Ao0Jc6Hh/VHYGkm82b4V//ClU+jz4K771X83YnnxwK\n/5EjwwBuIg3VlDuCehNBIikRSCpzhzffDEM5LF4c/i0pqX37I44Ihf/o0XDMMa0Xp6SXlqgaEpEG\nKiuDf/87FPiVhf+mTXXv06MHjBoVEsAZZ6jBVxJDiUCkiXbsCI90Vl7tv/BC/UM1t2sXXsl41lmh\n4ffcc0MvX5FEUiIQaYQ334T77gt1/CtWQEVF3dt36hQK/aFDw79nnBF6+YokEyUCkQZ46SWYNAke\nf7zuwj83t6rgHzoUjj1WHbsk+SkRiNTCHZ5+OiSAZ57Zd32bNnDSSVVX+2edBb17t36cIvtLiUAk\nTllZeLTzrrvCqJ7xvvQluPba8Hx/x46tH59Ic1MiEIl89BFMmwZTpsA771Rfl5UVnum/8cbwjL9I\nOlEikIz3/vvw29/CPffAtm3V12Vnw7e/DTfcEOr/RdKREoFkrDfegMmT4Q9/CGP9xOreHa67Dv7r\nv6Bbt4SEJ9JqlAgk4yxbFhqAn3giNAjHOuII+OEP4etf12OekjmUCCTtVVRAUVEY3fPvfw+JIN7p\np8OECXDppergJZlHiUDS0pYtsGBBKPwXLIAPPqh5u/PPh5tuCu/t1fAOkqmUCCQtVFTAyy/DU0+F\nwv/FF2vv+NW2bRjY7Uc/ghNPbN04RZKREoGkrK1bQ4evefNg/vzw9E9tPve5cPV//vnwxS9C166t\nF6dIslMikJThDsuXV131L11a+1V/mzYweDBccEEo/E85RUM9iNRGiUCS3ocfwowZYbC3116rfbtD\nDgnv8L3ggnDVr8c+RRpGiUCS1iuvwL33wp/+VPPwzmbhqr+yyue003TVL9IUSgSSVHbvDs/333NP\nGOc/Xk5OeHH7BReEMX/0Dl+R/adEIElh/XqYOhUeeKDmRt/jj4fx4+HKK+Hgg1s/PpF0pkQgCeMe\nXvBy770we/a+Db9t28JXvxoSwNChes5fpKUoEUir2749NP7ee2/Njb+9esG4cWGwt0MPbf34RDKN\nEoG0mhUrQt1/bY2/554brv4vuii821dEWocSgbSojz8OL3m5//7w3H+8nBwYOzaM8nnMMa0fn4hA\nvQ/bmVm+mZWY2Rozm1DLNnlmttzMVplZYczyzmb2uJkVm9lqMxscLS8ws9Jon+Vmlt9sf5EkhX//\nOxTuhx0G3/zmvknguONC1dCGDeFdAEoCIoljHj8Ob+xKsyzgNWA4sAFYBox29+KYbToDi4ER7l5q\nZt3dfXO0bgbwT3efbmZtgYPcfbuZ3QbsdPcpdQZn5nXFJ8ll50545JFw9V9UtO/6du3C6J7jx8PZ\nZ6vxV6QlmBnu3qj/u+qrGhoEvOHu66ITPAJcDBTHbHM58IS7lwLEJIFOwNnuPjZaXg5sj423MYFK\ncnIPhf4DD8DDD4eqoHgDBsB3vgNXXRXG/BGR5FJfIugFrI+ZLwUGx20zAGhnZouAHOBud/8j0B/4\nwMweAk4CXgKud/fKZsJrzewqoAj4obt/uH9/irSm7dtDwX///aEaKN4BB4RHP6++WkM8iyS7+hJB\nQ+pl2gGnAucB2cASM1saHftU4Bp3X2ZmvwZuBn4C3AfcHu3/U2Ay8K2aDl5QULB3Oi8vj7y8vAaE\nJC3BPdT1P/AA/OUvNT/5c8wx4ep/zBiN9SPSGgoLCyksLNyvY9TXRjAEKHD3/Gj+FqDC3SfGbDMB\n6ODuBdH8g8A84Hlgqbv3j5YPBW529y/HnSMXmOvuJ9RwfrURJIGPP4aHHgo9f1et2nf9gQfCZZeF\nBHDmmbr6F0mklmgjKAIGRIX1RmAkMDpum9nA76KG5QMIVUdT3H2Tma03s6Pc/XVCg/OrUaCHuvu7\n0f6XAisbE7S0js2b4Xe/C0/1bN267/oTTgiF/xVXQJcurR+fiDSPOhOBu5eb2TXAAiALmObuxWY2\nLlo/1d1LzGw+sAKoAB5w99XRIa4FZppZe+BN4BvR8olmdjKh6mktMK65/zBpurffhilT4MEH963+\nyc4Ob/e6+moYNEhX/yLpoM6qoURT1VDrWrUKJk2CP/8Zysurr+vfH37wg/DkT8eOiYlPROrXElVD\nkgEWL4Y774Qnn9x33Uknwc03w9e+FgaBE5H0o/+1M1RFRXjl45131jzuf14eTJgAI0ao+kck3SkR\nZJiystD7d9Kkmp8AuvTSkAAGx/cWEZG0pUSQIT7+GKZNg8mT4Z13qq9r1y688OXGGzXmj0gmUiJI\nc9u3w29+A3ffDVu2VF938MHh8c8bboDevRMTn4gknhJBmtqxIxT+U6bAh3GDdxxyCFx3XRgdtGvX\nxMQnIslDiSDN7NgR7gCmTIFt26qvy82FH/0IvvGN0B9ARASUCNLGjh2hB/DkyfsmgCOPhFtvhcsv\n1yOgIrIvFQspbufOqgQQPwzEEUeEBHDFFUoAIlI7FQ8paufOMA7QL3+pBCAi+0fFRIqpKwEcfnhI\nAFdeqQQgIg2n4iJF7NwJ99wTEkD8Y6D9+1clgHbtEhOfiKQuJYIk99FHIQHcdVfNCeD//b/wEhgl\nABFpKiWCJLV7d3gT2E9/Cps2VV+XmxsSwFVXKQGIyP5TIkgyFRVhLKBbb4W33qq+rl+/kADGjlUC\nEJHmo0SQJNxhwQK45ZZ9Xwbfuzf85CchAbRvn5j4RCR9KREkgRdeCGP+x79/uksX+PGPYfx46NAh\nIaGJSAZQIkigkhL47/+Gv/61+vIOHeD734ebboLOnRMTm4hkDiWCBCgthf/5H5g+PbQJVMrKgm9/\nO1QDHXZY4uITkcyiRNCKtm4NbwT77W/h00+rr7vssvCE0FFHJSY2EclcSgSt4JNPwoigEyfuOyT0\n8OHwi1/A6acnJjYRESWCFrRnT6j+KSiAjRurrzvttHB3MHx4QkITEdlLiaCFfPIJjBoFc+dWX37k\nkfDzn8PXvgZt2iQmNhGRWPUWRWaWb2YlZrbGzCbUsk2emS03s1VmVhizvLOZPW5mxWa22syGRMu7\nmtlCM3vdzJ42s7R6NmbzZjjvvOpJoGdPuO8+WL06tAcoCYhIsjB3r32lWRbwGjAc2AAsA0a7e3HM\nNp2BxcAIdy81s+7uvjlaNwP4p7tPN7O2wEHuvt3MJgGb3X1SlFy6uPvNNZzf64ovGa1dC/n58Prr\nVctuuCE0BB90UOLiEpHMYGa4uzVmn/quSwcBb7j7OncvAx4BLo7b5nLgCXcvBYhJAp2As919erS8\n3N23R/tcBMyIpmcAlzQm6GS1fDmceWZVEjCrem2kkoCIJKv6EkEvYH3MfGm0LNYAoKuZLTKzIjMb\nEy3vD3xgZg+Z2ctm9oCZVb4pt4e7Vw6ltgnosR9/Q1JYuBDOOQfeey/MH3AAPPooXHttYuMSEalP\nfY3FDamXaQecCpwHZANLzGxpdOxTgWvcfZmZ/Rq4GfhJtRO4u5nVep6CgoK903l5eeTl5TUgpNb1\npz+FF8KXl4f5zp1h9uyQGEREWlJhYSGF8ePTNFJ9bQRDgAJ3z4/mbwEq3H1izDYTgA7uXhDNPwjM\nA54Hlrp7/2j52cAEd/+ymZUAee7+npkdCixy94E1nD+p2wjcw3sCJsQ0offuDfPnw3HHJS4uEclc\nLdFGUAQMMLNcM2sPjATmxG0zGxhqZllR1c9goDiq+llvZpV9Zc8DXo2m5wBjo+mxwKzGBJ0M9uyB\n66+vngSOPx6WLFESEJHUUmfVkLuXm9k1wAIgC5jm7sVmNi5aP9XdS8xsPrACqAAecPfV0SGuBWZG\nSeRN4BvR8juBR83sW8A64LJm/rta1KefhreCPf541bJhw2DWLA0SJyKpp86qoURLxqqhbdvg4ovh\nueeqlv3nf8L//i8ceGDi4hIRgZapGpIY69fD0KHVk8D114c3iikJiEiqUiJooJUr4fOfDz2DK911\nF/zqV+olLCKpTWMNNUBhIVxyCWyPusO1awd/+ANcfnkioxIRaR5KBPV49NHQMLx7d5jPyYG//S2M\nJSQikg5UqVGHu+8OI4hWJoGePeHZZ5UERCS96KmhWsyaBZdeWjV/9NGho1hubkLCERFpkKY8NaRE\nUIM9e+CEE6A4GmP1zDNhzhzo1q3VQxERaRQ9PtpM/vKXqiSQk6MkICLpTYkgTnl5eLVkpe9/X0lA\nRNKbEkGcmTNhzZow3bkz/OAHiY1HRKSlKRHEKCuD22+vmv/hDzV2kIikPyWCGDNmwFtvhemuXeG6\n6xIbj4hIa1AiiHz2WXivcKWbboKOHRMXj4hIa1EiiEyfDu+8E6YPOQTGj09sPCIirUWJgPB+gZ//\nvGr+5pvh4IMTF4+ISGtSIgDuvx82bAjTPXvCd7+b2HhERFpTxieCTz6BO+6omv/xjyE7O3HxiIi0\ntoxPBPfeC5s2heneveHqqxMbj4hIa8voRPDRRzBxYtX8f/+33jQmIpknoxPBb38LmzeH6X794Jvf\nTGw8IiKJkLGJYMeO8KrJSrfeCu3bJy4eEZFEydhE8Otfw7ZtYfqII+CqqxIbj4hIomRkIti2DaZM\nqZr/yU/Ce4hFRDJRvYnAzPLNrMTM1pjZhFq2yTOz5Wa2yswKY5avM7MV0boXY5YXmFlptHy5meU3\ny1/TQFOmVL2I/uij9RJ6Eclsdb6hzMyygNeA4cAGYBkw2t2LY7bpDCwGRrh7qZl1d/fN0bq1wGnu\nvjXuuLcBO919CnVoiTeUbdkSXjf50Udh/uGHYfToZj2FiEjCtMQbygYBb7j7OncvAx4BLo7b5nLg\nCXcvBahMArFx1RZvYwJtLnfdVZUEjjsOLrssEVGIiCSP+hJBL2B9zHxptCzWAKCrmS0ysyIzGxOz\nzoH/i5bHd9W61sxeMbNp0V1Fi3v//fDIaKWCAsjKao0zi4gkr7b1rG9IvUw74FTgPCAbWGJmS919\nDTDU3Tea2SHAQjMrcffngPuAylfA/BSYDHyrpoMXxLw3Mi8vj7y8vAaEVLOJE8OQEgAnnQRf+UqT\nDyUikhQKCwspLCzcr2PU10YwBChw9/xo/hagwt0nxmwzAejg7gXR/IPAfHd/PO5YtwEfufvkuOW5\nwFx3P6GG8zdbG8HGjeEx0U8/DfOzZsHF8ZVcIiIpriXaCIqAAWaWa2btgZHAnLhtZgNDzSzLzLKB\nwcBqM8s2s5wosIOALwEro/lDY/a/tHJ5S7rzzqokcPrpcNFFLX1GEZHUUGfVkLuXm9k1wAIgC5jm\n7sVmNi5aP9XdS8xsPrACqAAecPfVZnY48FczqzzPTHd/Ojr0RDM7mVD1tBYY1xJ/XKX162Hq1Kr5\n228HS0hTtYhI8qmzaijRmqtq6Hvfg9//PkwPGQL/+pcSgYikp6ZUDaV9Ili3Do46CsrKwvzChTB8\n+P7HJiKSjFqijSDl/exnVUng7LPhvPMSG4+ISLJJ6zuCN96AgQNhz54wX1gIw4Y1T2wiIslIdwRx\nfvrTqiTwhS8oCYiI1CRt7whKSsIQEhUVYf755+Gss5oxOBGRJKQ7ghi3316VBEaMUBIQEalNWt4R\nrF8fXj1ZuesLL8CgQc0cnIhIEtIdQeS556qSwLBhSgIiInVJy0SwZEnV9H6MUScikhHSMhEsXVo1\n/fnPJy4OEZFUkHZtBLt2QceOUF4e5rduhS5dWiA4EZEkpDYC4KWXqpLAwIFKAiIi9Um7RKBqIRGR\nxkm7RBDbUDxkSOLiEBFJFWmVCNyrJwLdEYiI1C+tGovfeSd0JAPIyYFt2/RyehHJLBnfWBzbPjBo\nkJKAiEhDpG0iUPuAiEjDpFUiUEOxiEjjpU0bwWefhY5ku3eH+Q8+gO7dWzA4EZEklNFtBMuXVyWB\nI49UEhARaai0SQTqSCYi0jRpkwjUPiAi0jT1JgIzyzezEjNbY2YTatkmz8yWm9kqMyuMWb7OzFZE\n616MWd7VzBaa2etm9rSZdd7fP0RPDImINE2djcVmlgW8BgwHNgDLgNHuXhyzTWdgMTDC3UvNrLu7\nb47WrQVOc/etccedBGx290lRcuni7jfXcP4GNRZv3Ai9eoXpDh1gxw5o27be3URE0k5LNBYPAt5w\n93XuXgY8Alwct83lwBPuXgpQmQRi46rhuBcBM6LpGcAljQk6XuzdwBlnKAmIiDRGfYmgF7A+Zr40\nWhZrANDVzBaZWZGZjYlZ58D/Rcuvjlnew903RdObgB5NiH0vNRSLiDRdfdfODXmIvx1wKnAekA0s\nMbOl7r4GGOruG83sEGChmZW4+3PVTuDuZlbreQoKCvZO5+XlkVfDuyfVUCwimaqwsJDCwsL9OkZ9\nbQRDgAJ3z4/mbwEq3H1izDYTgA7uXhDNPwjMd/fH4451G7DT3aeYWQmQ5+7vmdmhwCJ3H1jD+ett\nIygrCx3JPv00zL/7LvTsWf8fLiKSjlqijaAIGGBmuWbWHhgJzInbZjYw1MyyzCwbGAysNrNsM8uJ\nAjsI+BKwKtpnDjA2mh4LzGpM0LFeeaUqCeTmKgmIiDRWnVVD7l5uZtcAC4AsYJq7F5vZuGj9VHcv\nMbP5wAqgAnjA3Veb2eHAX82s8jwz3f3p6NB3Ao+a2beAdcBlTf0D9NioiMj+Sfmxhq64Ah5+OEzf\nfTdcd10rBCYikqQycqwh3RGIiOyflL4jeP996BE9eHrAAaEjWfv2rRSciEgSyrg7gti7gdNOUxIQ\nEWmKtEkE6kgmItI0KZ0I1JFMRGT/pWwbQXk5dO4MH38c5tevh969WzE4EZEklFFtBKtWVSWB3r2V\nBEREmiplE4EeGxURaR4pmwhi2wfUUCwi0nQpmwh0RyAi0jxSsrF4yxbo3j1Mt2sXOpIdeGArByci\nkoQyprH4hReqpk85RUlARGR/pGQiULWQiEjzSclEoIZiEZHmk3JtBHv2QJcusHNnmF+7NryQRkRE\nMqSNoLi4Kgn07An9+iU2HhGRVJdyiSC+fcAalfdERCReSicCtQ+IiOy/lEsEGnFURKR5pVRj8Ycf\nhoZigKws2L4dDjooQcGJiCShtG8sfvHFqumTTlISEBFpDimVCNSRTESk+aVUIlBHMhGR5ldvIjCz\nfDMrMbM1Zjahlm3yzGy5ma0ys8K4dVnRurkxywrMrDRavtzM8uuLo6Ki+hhDuiMQEWkebetaaWZZ\nwO+A4cAGYJmZzXH34phtOgP3ACPcvdTMuscd5npgNZATs8yBKe4+paGBrlkD27aF6e7d4YgjGrqn\niIjUpb47gkHAG+6+zt3LgEeAi+O2uRx4wt1LAdx9c+UKM+sNXAA8CMS3YjeqVTv+sVF1JBMRaR71\nJYJewPqY+dJoWawBQFczW2RmRWY2Jmbdr4AbgYoajn2tmb1iZtOiu4o6qaFYRKRl1Fk1RKjCqU87\n4FTgPCAbWGJmS4GjgffdfbmZ5cXtcx9wezT9U2Ay8K2aDl5QUADA3/4GkAfkqaFYRCRSWFhIYWHh\nfh2jzg5lZjYEKHD3/Gj+FqDC3SfGbDMB6ODuBdH8g8B8QnIYA5QDBwIdCVVIV8WdIxeY6+4n1HB+\nd3d27oTOnUODcZs2oWNZTk781iIi0hIdyoqAAWaWa2btgZHAnLhtZgNDo6eDsoHBwGp3/7G793H3\n/sAo4JnKJGBmh8bsfymwsq4gli0LSQDg+OOVBEREmlOdVUPuXm5m1wALgCxgmrsXm9m4aP1Udy8x\ns/nACkJbwAPuvrqmw8VMTzSzk6Nla4FxdcWh9gGR1md6IiPpNdcQQSkx1tCFF8KTT4Zl06fDN76R\n2LhEMkFUxZDoMKQWtf33Scuxhtw19LSISEtK+kTw1luwOeqZ0LkzHHVUYuMREUk3SZ8I4juStUn6\niEVEUkvSF6tqKBYRaVlJnwg04qiIpLLnnnuOgQMHJjqMOiX9U0NZWc6ePWF+27bQTiAiLU9PDSW3\njHpqqDIJHHOMkoCI7Ku8vDwpj5VKkj4RVFK1kIhUys3NZdKkSZx44onk5OSwePFizjzzTLp06cLJ\nJ5/MP//5z73brl27lnPOOYeOHTvyxS9+kfHjxzNmTBgbc926dbRp04bp06fTr18/hg8fDsD06dM5\n9thj6drynKzvAAAJKklEQVS1K/n5+bzzzjt7j3fDDTfQo0cPOnXqxIknnsirr74KwFNPPcVxxx1H\nx44d6d27N5MnTwbCWEB9+vTZu39xcTF5eXl06dKF448/nrlz976qha9//euMHz+eL3/5y3Ts2JEh\nQ4bw1ltvtdwXWcndk/YDeOhJ4H7//S4irSgUD3Wtb95PY/Tr189POeUULy0t9Q0bNni3bt183rx5\n7u6+cOFC79atm2/evNnd3YcMGeI33nijl5WV+fPPP+8dO3b0MWPGuLv72rVr3cx87Nix/sknn/iu\nXbt81qxZfuSRR3pJSYnv2bPHf/azn/mZZ57p7u7z58/30047zbdv3+7u7iUlJf7uu++6u3vPnj39\n+eefd3f3Dz/80F9++WV3d1+0aJH37t3b3d13797tRxxxhP/iF7/wsrIyf+aZZzwnJ8dfe+01d3cf\nO3asd+vWzZctW+bl5eV+xRVX+KhRoxr13yda3riytrE7tOYnNhGsXFnj3ywiLSSZE0Fubq4/9NBD\n7u5+55137i3YK40YMcJnzJjhb7/9trdt29Z37dq1d92VV17pV155pbtXJYK1a9fuXZ+fn+/Tpk3b\nO79nzx7Pzs72t99+25955hk/6qijfOnSpb5nz55q5+zbt69PnTp1b5KoFJsInn32We/Zs2e19aNH\nj/aCggJ3D4ng6quv3rvuqaee8oEDB9b4HTRnIkiJqqGcnNBGICJSqbK65e233+axxx6jS5cuez+L\nFy/mvffeY+PGjXTt2pUDDzxwn/1qOlbl8a6//vq9x+rWrRsAGzdu5Nxzz+Waa65h/Pjx9OjRg3Hj\nxrFz504AnnjiCZ566ilyc3PJy8tjaeyz75GNGzfuc/5+/fqxceNGIDT09ujRY++6Dh068NFHHzX1\nK2qwlEgEgwdDVlaioxCRWM19T9BYlYPi9e3blzFjxrBt27a9n507d3LTTTdx6KGHsnXrVnbt2rV3\nv9j6/vhjVR7v/vvvr3a8jz/+mCFRR6Zrr72WoqIiVq9ezeuvv85dd90FwOmnn86sWbP44IMPuOSS\nS7jsssv2Oc9hhx3G+vXrK2s8gJB4evWKf99X60qJRKCOZCJSmyuvvJK5c+fy9NNPs2fPHj799FMK\nCwvZsGED/fr14/TTT6egoICysjKWLFnCk08+WefIqt/97ne54447WL06DKK8fft2HnvsMQCKiop4\n4YUXKCsrIzs7mwMPPJCsrCzKysqYOXMm27dvJysri5ycHLJquHodPHgw2dnZTJo0ibKyMgoLC3ny\nyScZNWoU0HyjiTaWEoGIpLTevXsze/Zs7rjjDj73uc/Rt29fJk+eTEX0EpOZM2eyZMkSunXrxq23\n3srIkSNp37793v3jk8Ill1zChAkTGDVqFJ06deKEE05gwYIFAOzYsYPvfOc7dO3aldzcXLp3786N\nN94IwJ/+9Cf69+9Pp06duP/++5k5c+Y+52jfvj1z585l3rx5HHLIIVxzzTX88Y9/5KhoEDUz2yee\n1hgOPOk7lIGzeTNE1XQi0krStUPZyJEjOfbYY7ntttsSHcp+yagOZQMGKAmISNMVFRXx5ptvUlFR\nwbx585gzZw6XXHJJosNKKvW9vD7h1JFMRPbHe++9x1e+8hW2bNlCnz59+P3vf89JJ52U6LCSStJX\nDd17r/O97yU6EpHMk65VQ+kio6qGdEcgItKykv6OoKzMaZv0FVgi6Ud3BMkto+4IlARERFqWilkR\nqVVrPMMuiVdvIjCzfODXQBbwoLtPrGGbPOBXQDtgs7vnxazLAoqAUne/MFrWFfgL0A9YB1zm7h/u\n598iIs1I1UKZo86qoagQ/x2QDxwLjDazY+K26QzcA1zo7scDX4s7zPXAaiD2V3UzsNDdjwL+Ec1L\nCyosLEx0CGlF32fz0veZWPW1EQwC3nD3de5eBjwCXBy3zeXAE+5eCuDumytXmFlv4ALgQSD2HvMi\nYEY0PQNQ744Wpv/Rmpe+z+al7zOx6ksEvYD1MfOl0bJYA4CuZrbIzIrMbEzMul8BNwIVcfv0cPdN\n0fQmoAciIpIQ9bURNKSSsB1wKnAekA0sMbOlwNHA++6+PGpDqPkE7h7GFBIRkUSosx+BmQ0BCtw9\nP5q/BaiIbTA2swlAB3cviOYfBOYTksMYoBw4EOhIqEK6ysxKgDx3f8/MDgUWufvAGs6vBCEi0kiN\n7UdQXyJoC7xGuNrfCLwIjHb34phtBhIalEcABwAvACPdfXXMNsOAH8U8NTQJ2OLuE83sZqCzu6vB\nWEQkAeqsGnL3cjO7BlhAeHx0mrsXm9m4aP1Udy8xs/nACkJbwAOxSSD2cDHTdwKPmtm3iB4f3f8/\nRUREmiKph5gQEZGWl5RDTJhZvpmVmNmaqA1C9oOZrTOzFWa23MxeTHQ8qcbMppvZJjNbGbOsq5kt\nNLPXzezpqD+N1KOW77LAzEqj3+fyqBOrNICZ9Yme2HzVzFaZ2XXR8kb9PpMuETSkE5s0mhMa509x\n90GJDiYFPUT4PcZSp8imqem7dGBK9Ps8xd3nJyCuVFUG3ODuxwFDgPFRedmo32fSJQIa1olNGk+D\nxjSRuz8HbItbrE6RTVDLdwn6fTaJu7/n7v+Opj8Cigl9vRr1+0zGRNCQTmzSOA78X9Th7+pEB5Mm\n1CmyeV1rZq+Y2TRVszWNmeUCpxCe3GzU7zMZE4Far5vfWe5+CnA+4dbx7EQHlE48PHGh323T3Qf0\nB04G3gUmJzac1GNmBwNPANe7+87YdQ35fSZjItgA9ImZ70O4K5Amcvd3o38/AP5GqH6T/bPJzHoC\nRJ0i309wPCnL3d/3CGFcMv0+G8HM2hGSwB/dfVa0uFG/z2RMBEXAADPLNbP2wEhgToJjSllmlm1m\nOdH0QcCXgJV17yUNMAcYG02PBWbVsa3UISqoKl2Kfp8NZuGFEdOA1e7+65hVjfp9JmU/AjM7n6p3\nIExz918kOKSUZWb9CXcBEDoQztT32Thm9mdgGNCdUN/6E2A28CjQF71To8Fq+C5vA/II1UIOrAXG\nxdRvSx3MbCjwLKFDb2VhfgthFIgG/z6TMhGIiEjrScaqIRERaUVKBCIiGU6JQEQkwykRiIhkOCUC\nEZEMp0QgIpLhlAhERDKcEoGISIb7/9X+0q2nW7klAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe4e31aef50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(logistic_auc.aucs, label=\"regression\", lw=3)\n",
    "_ = plt.legend(loc=\"lower right\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4FeXZ+PHvnZAEwp4EEwIJWwIBWQQREVGjKAbFpXVB\nsIitL6VWrbXVKr4WIuLrD1uttS5tlVpXUGtFikJBIYrIqiggBAgQspFACPsSQnL//pgJHkJ2kpxz\nkvtzXedyZp5nZu5zPJw7zzIzoqoYY4xpugK8HYAxxhjvskRgjDFNnCUCY4xp4iwRGGNME2eJwBhj\nmjhLBMYY08RZIjBeJSLpIjLC23E0NiKSKCKZ3o7D+AdLBMbb1H0ZY7zEEoExdUhEmnk7BmNqyhKB\n8RkiEiIiz4lItvv6k4gEu2URIjJPRPaJyF4R+cJjv4dFJEtEDopIqohcUcHx24rIGyKy2+2S+l9x\nhIjIfhE516NuBxE5KiIR7vpoEfnWPf8yEennUTddRH4nIuuAQyJyxr8rEUkQkUVu7KkicotH2T9F\n5K8istB9DykiEutRPkxEVrsxrhKRizzKwkTkNffzKhCRD8uc9zcikiciOSJyZ83+j5imwhKB8SX/\nCwwBBrivIcBjbtlvgUwgAjgHmAwgIr2Ae4DBqtoGGAmkV3D8vwCtgW7AZcAdwE9VtRD4ABjrUfdW\nIEVV80VkIDATmAiEAX8D5opIkEf924BRQDtVLfE8qYi0BBYBbwEd3LoviUhvj2rjgGnu+/sWeNvd\nNwz4GHjOPfezwMci0t7d702gOdDH/Vye9ThmFNAGiAbuAl4UkbYVfDamKVNVe9nLay9gB3CFu5wG\nJHmUjQR2uMuPA3OAHmX2jwPygBFAUCXnCQQKgQSPbT8HlrjLI4A0j7JlwE/c5ZeBaWWOlwpc4vEe\n7qzk3GOAL8ps+xswxV3+J/COR1lL4CTQGRgPrCiz71fABKAjUAy0LeecicBRIMBjWx4wxNv/z+3l\ney9rERhfEg3s9FjPcLcB/AEnUSwUkW0i8jCAqqYBvwaSgTwRmSUiHcs5dgQQVM7xO7nLKUCoiAwR\nka44LZLSbpYuwG/dbqF9IrIP50c62uNYlc3Q6QJcWGb/cUCkW65AVmllVT0CFLjH7+jG6WmnW9YZ\nKFDVAxWcd6+e3jo5CrSqJE7TRFkiML4kB+jqsR7rbkNVD6vqg6raA7ge+E3pWICqzlLVS3B+cBWY\nUc6x84Gico6f5R6jGHgPp3toLPAf9wcZnB/iJ1W1vcerlaq+63GsymY+ZQCfl9m/tare45YLEFNa\nWURa4XQDZbvvv0uZ43VxyzKBMOvuMWfLEoHxJbOAx9yB4QhgCk4feOlgbZyICHAQp0ukWER6isgV\nIhKC0/Vz3C07jccP/ZMi0kpEugAP4PTbl3oHp/9+nLtc6hXgF25rQUSkpYhc6/5gV8c8oKeI/ERE\ngtzXBSKS4FHnGhG52B0cfwJYrqrZwHx337Ei0kxExgAJwDxVzXXLXxKRdu5xL61mTMacYonA+JLp\nwBpgnfta424DZyxgEXAIp4/8RVX9HAgBngL2ALtwuoAmV3D8+4AjwHZgKc6A7Gulhaq6CjiM0x0z\n32P71zgDxS/gdNlsxRlortb1D6p6GGe84zacv+R3uTEHl1bBSTxTgb3AQOAn7r57gdE4g+X5wIPA\naFUtcPcdj9PSScUZA/iV56mrE58xolr5d0VEknBmLAQCr6rqGc1uEUkE/oTTB5uvqonu9sk4X+gS\nYD3uDA13JsS7OE3cdOBWVd1fN2/JGP8iIq8BWar6e2/HYpqmSlsEIhKI81dQEs70tLFlprwhIu2A\nF4HrVLUvcLO7vSvOX1GDVLUfTiK5zd3tEWCRqvYEPnPXjWmqxNsBmKatqq6hIThT6tJVtQiYDdxQ\nps444ANVLR10y3e3H8RpsoaKc7VlKE6zGJzBvtfd5deBG8/qXRjj3+w2G8arqrocvhOnT4vLAi4s\nUyceCBKRJTgX6/xZVd9U1QIReQZnxsQxYKGqfuruE6mqee5yHj9MozOmyVHVn3o7BtO0VdUiqM5f\nKUHAIOAa4Grg9yISLyI9cOZ3d8WZ89xSRG4/4wTOIIX9NWSMMV5SVYsgG4/5ze5yVpk6mTgDxMeA\nY+49YAbgJJmv3FkPiMi/gWE4MzXyRCRKVXPdi392l3dyEbEEYYwxNaSqNRp3qqpFsAaIF5Gu7vzm\nMcDcMnU+AoaLSKCIhOJ0HW0ENgNDRaSFO/f7Snc77jEmuMsTcG4dUNEbslcdvKZOner1GBrTyz5P\n+zx99VUblbYIVPWkiNwL/Bdn1s9MVd0kIpPc8r+paqqILMCZ910CvKKqGwFE5A2cZFICfAP83T30\n/wPeE5G7cKeP1ip6Y4wxZ63Ke6er6nw8Lq5xt/2tzPofgT+Ws+/TwNPlbC/AaSEYY4zxMruyuIlI\nTEz0dgiNin2edcs+T++q8spibxIR9eX4jDHG14gIWsPBYnusnjGmzjnzQ0x9q6s/lC0RGGPqhbXm\n61ddJlsbIzDGmCbO5xPBiRPejsAYYxo3n08EGzZ4OwJjjGncfD4RrFnj7QiMMY1F165d+eyzz7wd\nhs+xRGCMaTJExGY0lcPnE8HXX3s7AmOMadx8PhFs2gTHj3s7CmNMY3LixAl+/etf06lTJzp16sQD\nDzzACXdmSn5+PqNHj6Z9+/aEh4dz6aWXntpvxowZdO7cmTZt2pCQkMDixYu99RbqlM9fRxAfD+vX\nwwUXeDsSY0xjoKpMnz6dVatW8d133wFwww03MH36dKZNm8YzzzxDTEwM+fnOwxZXrFgBwObNm3nx\nxRdZs2YNUVFRZGRkcPLkSa+9j7rk8y2CwYOte8iYxkakbl619c477zBlyhQiIiKIiIhg6tSpvPnm\nmwAEBweza9cu0tPTCQwM5OKLLwYgMDCQwsJCvv/+e4qKioiNjaV79+518XF4nV8kAhswNqZxUa2b\nV23l5OTQpUuXU+uxsbHk5OQA8NBDDxEXF8fIkSPp0aMHM2bMACAuLo7nnnuO5ORkIiMjGTt2LLt2\n7Tqrz8FX+HwiOP98SwTGmLoVHR1Nenr6qfWMjAyio6MBaNWqFX/84x/Ztm0bc+fO5dlnnz01FjB2\n7FiWLl3Kzp07EREefvhhb4Rf53w+EfTvD1u2wLFj3o7EGNNYjB07lunTp5Ofn09+fj7Tpk1j/Pjx\nAMybN4+0tDRUlTZt2hAYGEhgYCBbtmxh8eLFFBYWEhISQvPmzQkMDPTyO6kbPp8ImjeHXr1g3Tpv\nR2KMaQxEhMcee4zBgwfTv39/+vfvz+DBg3nssccASEtL46qrrqJ169YMGzaMe+65h8suu4zCwkIm\nT55Mhw4d6NixI/n5+Tz11FNefjd1wy+eRzBxIgwcCL/8pbcjMsZUh3tPfG+H0ahV9BnX5nkEPt8i\nABswNsaY+lRlIhCRJBFJFZGtIlLuyIiIJIrIWhHZICIp7rZe7rbS1wER+ZVbliwiWR5lSZXFYAPG\nxhhTfyrtGhKRQGAzzoPms4HVwFhV3eRRpx2wDLhaVbNEJEJV88scJ8Ddf4iqZorIVOCQqj5baXBu\n11BhIbRvD/n5EBpay3dqjGkw1jVU/xqya2gIkKaq6apaBMwGbihTZxzwgapmAZRNAq4rgW2qmukZ\nb3WDDAmBPn3AvQjQGGNMHaoqEXQCPH+8s9xtnuKBMBFZIiJrRGR8Oce5DXinzLb7ROQ7EZnptioq\nZd1DxhhTP6pKBNVp2wUBg4BrgKuB34tIfGmhiAQD1wHve+zzMtANOA/YBTxT1UnsVhPGGFM/qrrp\nXDYQ47Eeg9Mq8JQJ5KvqMeCYiHwBDAC2uuWjgK9VdU/pDqq6u3RZRF4F/lNRAMnJyQDs2gWff54I\nJFYRsjHGNB0pKSmkpKSc1TGqGixuhjNYPALIAVZx5mBxAvACTmsgBFgJjFHVjW75bGC+qr7usU9H\nVd3lLj8AXKCq48o5v5bGd+IEtGsHe/ZAy5Zn9Z6NMfXMBovrX10OFlfaIlDVkyJyL/BfIBCYqaqb\nRGSSW/43VU0VkQXAOqAEeMUjCbTEGSieWObQM0TkPJyupx3ApKoCDQ6Gvn3h22/BvRmgMcaYOuAX\nVxaXuvtuSEiA++/3YlDGmCo1thZBSkoK48ePJzMzs9J6Xbt2ZebMmYwYMaLeY2pyVxaXsgFjY4wv\n89dnIvtdIrAppMYYU7f8KhH06QM7d8KhQ96OxBjjj2bMmMEtt9xy2rb777+f+++/n3/+85/06dOH\nNm3a0KNHD/7+97+f1bkKCwv95rnIPv/MYk9BQdCvnzNgfMkl3o7GGFNb8njddJ/o1JqNQ4wdO5Zp\n06Zx+PBhWrVqRXFxMe+//z5z5swhPz+fjz/+mG7duvHFF18watQoLrjgAgYOHFir2J588km/eS6y\nXyUC+KF7yBKBMf6rpj/gdSU2NpZBgwbx4YcfMn78eBYvXkxoaChDhgw5rd6ll17KyJEjWbp0aa0T\nwTvvvMMLL7xAREQEAFOnTmXSpElMmzbttOci9+jRo9znIoeHhxMbG3t2b7ia/KprCOxWE8aYszNu\n3DhmzZoFOD/Wt99+OwDz589n6NChhIeH0759ez755BP27t1b6/P403OR/S4R2MwhY8zZuPnmm0lJ\nSSE7O5s5c+Ywbtw4CgsLuemmm/jd737H7t272bdvH9dcc81ZTYH1p+ci+10i6N0bMjPh4EFvR2KM\n8UcdOnQgMTGRO++8k+7du9OrVy9OnDjBiRMniIiIICAggPnz57Nw4cKzOo8/PRfZ7xJBs2YwYAB8\n8423IzHG+Ktx48bx2WefMW6cc2eb1q1b8/zzz3PrrbcSFhbGrFmzuOGG0++4X9PrA/zpuch+dWVx\nqV/9Crp0gd/+1gtBGWOq1NiuLPZFTfbK4lI2YGyMMXXHLxOBDRgbY7whIyOD1q1bn/Fq06YNWVll\n79DvP/yya6i42LkldWam819jjG+xrqH61+S7hgIDbcDYGGPqil8mArDuIWOMqSt+d4uJUoMHw38q\nfMClMcbb/PF2zE2VX44RAGzaBKNHw7ZtDRyUMcb4sCYzRgDQs6fz/OJ9+7wdiTHG+De/TQSBgXDe\neTZOYIwxZ6vKRCAiSSKSKiJbRaTcux+JSKKIrBWRDSKS4m7r5W4rfR0QkV+5ZWEiskhEtojIQhGp\n1SRQGzA2xpizV2kiEJFA4AUgCegDjBWR3mXqtANeBK5T1b7AzQCqullVB6rqQOB84CjwobvbI8Ai\nVe0JfOau15g9utIYY85eVS2CIUCaqqarahEwG7ihTJ1xwAeqmgWgqvnlHOdKYJuqZrrr1wOvu8uv\nAzfWJni71YQxxpy9qhJBJyDTYz3L3eYpHggTkSUiskZExpdznNuAdzzWI1U1z13OAyJrEPMPJ46H\nggI4i2dHGGNMk1fVdQTVmVsaBAwCRgChwHIRWaGqWwFEJBi4Dih3fEFVVUQqPE9ycvKp5cTERBIT\nE0+tBwTAoEHOOMHIkdWI1BhjGpmUlBRSUlLO6hiVXkcgIkOBZFVNctcnAyWqOsOjzsNAC1VNdtdf\nBRao6r/c9RuAu0uP4W5LBRJVNVdEOgJLVDWhnPNXeB1BqQcfhLAwePTR6r5lY4xpvOrjOoI1QLyI\ndHX/sh8DzC1T5yNguIgEikgocCGw0aN8LDCrzD5zgQnu8gRgTk2C9mQzh4wx5uxUeWWxiIwCngMC\ngZmq+pSITAJQ1b+5dR4EfgqUAK+o6vPu9pbATqCbqh7yOGYY8B4QC6QDt6rq/nLOXWWLIC0NRoyA\nnTur9X6NMaZRq02LwG9vMVFKFdq3h61boUOHBgrMGGN8VKO8xURRcVGl5SLONFLrHjLGmNrx+UTw\nVeZXVdax6wmMMab2fD4RfLL1kyrr2ICxMcbUnu8ngrTqJQJrERhjTO34fCLIPZxLxoGMSut06wZH\njkBeXqXVjDHGlMPnE0FSXBLzt86vtI4NGBtjTO35fCK4Ju4a6x4yxph65POJYGSPkaSkp1B4srDS\nejZzyBhjasfnE0F4aDh9z+nLFzu/qLSezRwyxpja8flEAE730MdbP660TpcucPw45OQ0UFDGGNNI\n+EciiL+myusJRKxVYIwxteEXieC8qPM4fOIwW/durbSeJQJjjKk5v0gEIsKouFHMT6t8GqkNGBtj\nTM35RSKA6nUPlbYIfPiGqsYY43P8JhFc2f1KlmUu48iJIxXWiYmB4mIbMDbGmJrwm0TQtnlbLoi+\ngCXpSyqsU3qFsXUPGWNM9flNIoCadQ8ZY4ypHr9MBJU9tcxuNWGMMTXjV4mgd0RvADblb6qwTmnX\nkA0YG2NM9VSZCEQkSURSRWSriDxcQZ1EEVkrIhtEJMVjezsR+ZeIbBKRjSJyobs9WUSy3H3WikhS\ndYIVkSq7hzp1goAAyMqqzhGNMcZUmghEJBB4AUgC+gBjRaR3mTrtgBeB61S1L3CzR/GfgU9UtTfQ\nH0h1tyvwrKoOdF8LqhvwNfGV327CBoyNMaZmqmoRDAHSVDVdVYuA2cANZeqMAz5Q1SwAVc0HEJG2\nwCWq+g93+0lVPeCxn9Qm4Mu7Xs6anDUcOH6gwjo2TmCMMdVXVSLoBGR6rGe52zzFA2EiskRE1ojI\neHd7N2CPiLwmIt+IyCsiEuqx330i8p2IzHRbFdXSMrglF8dczKfbP62wjs0cMsaY6mtWRXl1hlyD\ngEHACCAUWC4iK9xjDwLuVdXVIvIc8AgwBXgZmObu/wTwDHBXeQdPTk4+tZyYmEhiYuKpcYKb+txU\nbkCeA8ZSq3aHMcb4h5SUFFJSUs7qGFLZVEwRGQokq2qSuz4ZKFHVGR51HgZaqGqyu/4qMB/4Elih\nqt3c7cOBR1R1dJlzdAX+o6r9yjm/lhdfWkEal7x2CTm/yUEq+KWPjobly53bUxtjTFMhIqhqjf4E\nrqpraA0QLyJdRSQYGAPMLVPnI2C4iAS6XT8XAptUNQ/IFJGebr0rge/dQDt67P8jYH1Ngo4Li6NN\nSBu+zf22wjo2TmCMMdVTaSJQ1ZPAvcB/gY3Au6q6SUQmicgkt04qsABYB6wEXlHVje4h7gPeFpHv\ncGYN/Z+7fYaIrHO3XwY8UNPAr4mrfBqpzRwyxpjqqbRryNsq6hoCWLRtEcmfJ7PsZ8vKLf/4Y3ju\nOVi0qD4jNMYY31IfXUM+69Iul7Jh9wb2Ht1bbvnw4bB6NRQUNHBgxhjjZ/w2EYQ0CyGxayILty0s\nt7xtWxg1CmbPbuDAjDHGz/htIoCqH2o/YQK8/noDBmSMMX7IrxPBqPhRLEhbQHFJcbnlV10FmZmw\nqeJ71BljTJPn14kgtm0sHVt3ZHXO6nLLAwPhJz+xVoExxlTGrxMBVD2NdMIEeOst5xGWxhhjzuT3\nieDantdWmgjOPRc6doTPPmvAoIwxxo/4fSK4qPNFbNu3jdzDuRXWsUFjY4ypmN8ngqDAIK7qfhUL\n0ip+pMHYsc4FZgcqvnO1McY0WX6fCKDqh9qHh8MVV8D77zdgUMYY4ycaRSJIikti0fZFFBUXVVjH\nuoeMMaZ8jSIRRLWKokf7HizPWl5hnVGjYPNm2LatAQMzxhg/0CgSAVTdPRQcDOPGwRtvNGBQxhjj\nBxpVIqjsdhPwQ/dQSUkDBWWMMX6g0SSCC6IvYNehXWQcyKiwznnnQZs28MUXDRiYMcb4uEaTCAID\nAkmKS2L+1vkV1hGxQWNjjCmr0SQCgGvjr+WTtIrHCQBuvx3mzIEjRxooKGOM8XGNKhGM7DGSlPQU\nCk8WVlgnKgqGDYN//7sBAzPGGB/WqBJBeGg4fc/pyxc7Kx8EuPNO6x4yxphSVSYCEUkSkVQR2Soi\nD1dQJ1FE1orIBhFJ8djeTkT+JSKbRGSjiAx1t4eJyCIR2SIiC0WkXV29oaruRgpw3XWwdi1kVDyu\nbIwxTUaliUBEAoEXgCSgDzBWRHqXqdMOeBG4TlX7Ajd7FP8Z+ERVewP9gdJHxDwCLFLVnsBn7nqd\nuCb+mirHCZo3h1tvhTffrKuzGmOM/6qqRTAESFPVdFUtAmYDN5SpMw74QFWzAFQ1H0BE2gKXqOo/\n3O0nVbX0tm/XA6WdM68DN571O3GdF3UehwoPkVaQVmm90tlDqnV1ZmOM8U9VJYJOQKbHepa7zVM8\nECYiS0RkjYiMd7d3A/aIyGsi8o2IvCIioW5ZpKrmuct5QORZvIfTiAij4kZVOo0U4MILnemkK1bU\n1ZmNMcY/NauivDp/LwcBg4ARQCiwXERWuMceBNyrqqtF5DmcLqApp51AVUWkwvMkJyefWk5MTCQx\nMbHKgK6Jv4ZX177KfRfeV2EdkR8GjS+6qMpDGmOMT0pJSSElJeWsjiFaSd+IO7ibrKpJ7vpkoERV\nZ3jUeRhooarJ7vqrwHzgS2CFqnZzt18CPKyqo0UkFUhU1VwR6QgsUdWEcs6vlcVXkQPHDxDzpxh2\n/XYXLYNbVlgvKwv694fsbGjRosanMcYYnyMiqKrUZJ+quobWAPEi0lVEgoExwNwydT4ChotIoNv1\ncyGwye36yRSRnm69EcD37vJcYIK7PAGYU5Ogq9K2eVsGRw9mSfqSSut17gznnw9zy74jY4xpQipN\nBKp6ErgX+C+wEXhXVTeJyCQRmeTWSQUWAOuAlcArqrrRPcR9wNsi8h3OrKH/c7f/P+AqEdkCXOGu\n16mq7kZaym45YYxp6irtGvK22nYNAWzZu4VLX7uUbb/aVmn30JEjTstg40bnIffGGOPP6qNryG/1\nDO/JZV0v48XVL1Zar2VL+NGP4O23GygwY4zxMY22RQCwac8mLvvnZaT9Ko02IW0qrPfFF3DPPbBu\nnTObyBhj/JW1CMro3aE3SXFJ/HnFnyutN3y400X0zTcNFJgxxviQRp0IAKZcNoU/r/wz+47tq7BO\nQADccYcNGhtjmqZG3TVUauLciUS2imT6FdMrrLN9u3O1cXa283xjY4zxR9Y1VIHHLn2Ml9e8zJ4j\neyqs07079O4Nn1Q949QYYxqVJpEIurTrwm3n3sbTy56utJ5dU2CMaYqaRNcQQM6hHPq93I8Nd2+g\nY+vyLxg4eBBiY2HrVujQoU5Oa4wxDcq6hioR3TqaCQMm8NSXT1VYp00bGD0aZs1qwMCMMcbLmkwi\nAHhk+CO8vf5tMg9kVljHuoeMMU1Nk0oE57Q8h58P+jnTv6h49tAVV0BeHmzY0ICBGWOMFzWpRADw\n0MUP8cGmD9i+b3u55YGBMH68tQqMMU1Hkxks9pSckkz6/nT+eeM/yy1PTYXLL4fMTGhW1aN7jDHG\nh9hgcTU9MPQBPtn6CZvzN5dbnpAAXbrAwoUNHJgxxnhBk0wEbZu35YGhD5D8eXKFdWzQ2BjTVDTJ\nriGAwycOE/d8HJ/e8Sl9z+l7RnlBgXO18Y4d0L59vYRgjDF1zrqGaqBVcCt+d/HvmJoytdzysDC4\n6iq7psAY0/g12RYBwLGiY8T9JY7/jP0PgzoOOqN85Uq48UZYvx4iIuotDGOMqTPWIqihFkEteHT4\no0xZMqXc8gsvhNtvh3vvbeDAjDGmAVWZCEQkSURSRWSriDxcQZ1EEVkrIhtEJMVje7qIrHPLVnls\nTxaRLHf7WhFJqpN3Uwv/M+h/WL97Pcszl5db/sQTsHYtfPBBAwdmjDENpNKuIREJBDYDVwLZwGpg\nrKpu8qjTDlgGXK2qWSISoar5btkO4HxVLShz3KnAIVV9ttLg6rlrqNSr37zKu9+/y6Lxi8ot/+or\nuOkm6yIyxvi++ugaGgKkqWq6qhYBs4EbytQZB3ygqlkApUnAM66K4q1JoPVpwoAJ7Ni3g8/TPy+3\nfNgwGDcO7ruvgQMzxpgGUFUi6AR43qEty93mKR4IE5ElIrJGRMZ7lCnwqbt9Ypn97hOR70Rkptuq\n8JqgwCCmXDaF3y/5PRW1QKZPd55p/O9/N3BwxhhTz6q6gUJ1+mWCgEHACCAUWC4iK1R1KzBcVXNE\npAOwSERSVXUp8DIwzd3/CeAZ4K7yDp6cnHxqOTExkcTExGqEVHO397udp758ikXbFzGyx8gzylu0\ngH/8A26+GS691LqIjDG+ISUlhZSUlLM6RlVjBEOBZFVNctcnAyWqOsOjzsNAC1VNdtdfBRao6r/K\nHGsqcFhVnymzvSvwH1XtV875G2SMoNS7G97l2RXPsuKuFYiU33P1m9/Arl12fYExxjfVxxjBGiBe\nRLqKSDAwBphbps5HwHARCRSRUOBCYKOIhIpIazewlsBIYL277vmIsB+Vbve2W869hWNFx5i3ZV6F\ndaZPh6+/ti4iY0zjUWkiUNWTwL3Af4GNwLuquklEJonIJLdOKrAAWAesBF5R1Y1AFLBURL51t89T\n1dLbuM1wp5V+B1wGPFAP763GAiSAaZdPY0rKFEq0pNw6oaHw2mvOtQV79zZwgMYYUw+a9JXF5VFV\nLnjlAiYPn8xNfW6qsN5vfgO5ufDOOw0YnDHGVMGuLK4DIsITlz/BlJQpFJcUV1hv+nRYswY+/LAB\ngzPGmHpgiaAcSXFJtA1py7vfv1thndBQZxbRPfdYF5Exxr9Z11AFFu9YzC/m/YKN92ykWUDFs2wf\neAB274a3327A4IwxpgLWNVSHruh2BZ3adGLmNzMrrffkk7BqFcyZ00CBGWNMHbMWQSXW561nxBsj\neO+W90jsmlhhvS+/hFtvde5FFB7ecPEZY0xZtWkRWCKowuIdi7ntX7exaPwiBkQNqLCedREZY3yB\ndQ3Vgyu6XcGL17zIte9cy459OyqsZ11Exhh/ZS2Canpp9Us8t+I5vvzZl5zT8pxy6yxdCmPGWBeR\nMcZ7rGuonk1ZMoX5afNZfMdiWoe0LrfOr38N+fnw1lsNHJwxxmCJoN6pKpPmTSJ9fzrzxs0jODD4\njDpHj8KAAfDHP8INZZ/cYIwx9cwSQQM4WXKSW96/hRbNWvDWj98iQM4cZintItqwAcLCvBCkMabJ\nskTQQI4VHePqt65mUMdB/OnqP5V7y2rrIjLGeIPNGmogLYJaMHfsXBbvWMzTy54ut86TT8KKFfDR\nRw0cnDF7PsqzAAAVQ0lEQVTG1JAlglpq17wdC36ygL9+/VdeW/vaGeUtWzr3IvrlL2HPHi8EaIwx\n1WSJ4CxEt45mwe0LeHTxo+U+zObSS+HnP4dhw2DzZi8EaIwx1WCJ4Cz1iujFR7d9xM8++hlfZX51\nRvnUqfDII05S+PRTLwRojDFVsERQB4Z0GsIbP3qDH737Izbu2XhG+V13wbvvwu23w1//6oUAjTGm\nEjZrqA69te4tHv3sUZb9bBkxbWPOKE9Lg9Gj4eqr4ZlnoFnFd7c2xphasemjPuCZr55h5tqZfPmz\nLwlrceZFBPv2OXcqDQqCWbOgbVsvBGmMabTqZfqoiCSJSKqIbBWRhyuokygia0Vkg4ikeGxPdx9S\nv1ZEVnlsDxORRSKyRUQWiki7mgTty3477LdcG38to98ZzdGio2eUt28Pn3wCXbs6g8g7Kr6PnTHG\nNIhKWwQiEghsBq4EsoHVwFhV3eRRpx2wDLhaVbNEJEJV892yHcD5qlpQ5rhPA/mq+rSbXNqr6iPl\nnN/vWgQAJVrCnXPupOBYAR+O+ZCgwKAz6qjCCy/A//0fvP8+DB/uhUCNMY1OfbQIhgBpqpquqkXA\nbKDsHXTGAR+oahZAaRLwjKuc414PvO4uvw7cWJOgfV2ABDDz+pmUaAk/n/dzyktmInDfffDaa/Dj\nH8Mbb3ghUGOMoepE0AnI9FjPcrd5igfCRGSJiKwRkfEeZQp86m6f6LE9UlXz3OU8ILIWsfu0oMAg\n3r/lfbbs3cJVb17F97u/L7deUhKkpMDjj8PkyVBS0rBxGmNMVfNWqtMvEwQMAkYAocByEVmhqluB\n4aqaIyIdgEUikqqqS087gaqKSIXnSU5OPrWcmJhIYmJiNULyDS2DW/L5nZ/z8uqXufz1y7mt7208\nnvg47Vu0P61enz7O7Sh+/GO4+WZ4803nymRjjKlKSkoKKSkpZ3WMqsYIhgLJqprkrk8GSlR1hked\nh4EWqprsrr8KLFDVf5U51lTgkKo+KyKpQKKq5opIR2CJqiaUc36/HCMoT/7RfH6/+Pf8O/XfPJ74\nOBMHTSQwIPC0OoWFMGkSrFsHc+dC585eCtYY47fqY4xgDRAvIl1FJBgYA8wtU+cjYLiIBIpIKHAh\nsFFEQkWktRtYS2AksMHdZy4wwV2eADT6BzxGhEbw8uiXWfiThczaMIvz/34+X+z84rQ6ISHOmMGY\nMTB0KKxe7aVgjTFNSpXXEYjIKOA5IBCYqapPicgkAFX9m1vnQeCnQAnwiqo+LyLdgX+7h2kGvK2q\nT7n1w4D3gFggHbhVVfeXc+5G0yLwpKq8v/F9Hlr0EEM7D+UPV/2B2Laxp9WZMwcmToSXXoJbbvFS\noMYYv2MXlPmZo0VH+cOyP/CXVX/hviH38dDFDxEaFHqqfO1a5yln//M/zkBy0JmzUI0x5jT2PAI/\nExoUytTEqXwz6Rs25m+kz4t9eP/7909NNx04EFauhC+/hF69nNtaFxV5OWhjTKNjLQIf8nn659y/\n4H7aNW/H86Oep39k/1NlS5c6U0y3b4f//V+44w5rIRhjzmRdQ41AcUkxr3zzClNTpnJT75t44vIn\nCA8NP1X+5ZdOQkhL+yEhBAd7MWBjjE+xrqFGIDAgkF8M/gWb7tlEoATS+8Xe/GXlX07dt2j4cFi0\nyHkW8nvvQc+e8Pe/w4kTXg7cGOO3rEXg4zbs3sCjnz3KV5lfcceAO/jF4F/QM7znqfKvvnJaCJs3\nOwPKP/2ptRCMacqsa6gR27FvB3//+u/849t/0D+yP3cPvpvre11PswDn4vDly52EsGnTDwkhJMTL\nQRtjGpwlgiag8GQhH2z6gJdWv0T6/nQmDprIxPMnEt06GnBuVfH44/D9905C+NnPLCEY05RYImhi\n1uWt469r/sqsDbMY0W0Edw++myu6XYGIsHKlkxA2bHCemXzXXZYQjGkKLBE0UYcKD/HWurd4ac1L\nnCg+wd2D72bCgAm0b9GeVauchLBmjfNktLFj4aKLnNtgG2MaH0sETZyqsixzGS+veZmPt3zMj3v/\nmF9e8EsGRw9m61aYPdt5PObRo3DbbU5S6N/fkoIxjYklAnPK7iO7+cfaf/DXNX+lQ8sOTBw0ket6\nXkdUq46sW+ckhNmzITTUSQhjx0JcnLejNsacLUsE5gzFJcUsSFvAG+veYOG2hcSFxTE6fjTX9ryW\ngVGDWLkigFmznMdlxsQ4LYUxY+wW2Mb4K0sEplJFxUUsy1zGvC3zmLdlHgcKD3Bt/LWM7jmaxNgr\nWb2sFbNmOXc+7dfPaSXcfDNERHg7cmNMdVkiMDWSVpDGx1s+Zt7WeazIWsHFMRczuudoruxyLanL\nuzF7NsyfDxdf7CSF66+Htm29HbUxpjKWCEytHSw8yKJti5i3dR6fbP2EiNAIRseP5oqY0ez++iLe\nm92MlBQYMACuugpGjoQLLoBmVT3s1BjToCwRmDpRoiWsyVlzqgtp54GdJMUlkRgzkuBdl7B+aTcW\nLRQyMuDyy52kMHIkdO/u7ciNMZYITL3IPpjNx1s/ZvGOxSzNWArA8Njh9G97Cey8hE1f9OXThYG0\navVDa+Hyy6FdOy8HbkwTZInA1DtVZcf+HSzduZSlGc5r95HdDOs8jLjgSyjecQmbFw9mxbIQ+vX7\nobUwZIh1IxnTECwRGK/IO5zHlxlfnkoMm/M3MzDyfGJ0OEXbLiF10TCytrUhMREuvRSGDoVBg+yW\nF8bUh3pJBCKSxA8Pr39VVWeUUycR+BMQBOSraqJHWSCwBshS1evcbcnA/wB73GqTVXVBOce1ROCH\nDhYeZHnmcpZmLOXLjC9Zk7OG7m170rHoEkp2XkTO6iHsWNuN/v2Eiy5ybnkxdKhzHYNd5WzM2anz\nROD+iG8GrgSygdXAWFXd5FGnHbAMuFpVs0QkQlXzPcp/A5wPtFbV691tU4FDqvpsFW/IEkEjUHiy\nkK93fc3SnUtZmb2SVdmrOHbyOPEthtDqwBAObx5C2ucXEFLc4VRSuOgiOP98aN7c29Eb419qkwiq\n6rUdAqSparp7gtnADcAmjzrjgA9UNQugTBLoDFwDPAn8pmy8NQnU+K+QZiEMixnGsJhhp7ZlH8xm\ndc5qVmWvYlXssxSdu5rQoHCyGcKc3CG8+viFZKwcSN9eoae1Grp0sVaDMXWtqkTQCcj0WM8CLixT\nJx4IEpElQGvgz6r6plv2J+AhoE05x75PRO7A6Tb6rarur2nwxn91atOJTm06cWPCjYAzZXXL3i1O\nYshexaqes5HE79kf0pOVx4ew9NMhZDwxhMCCPgweFMjAgZx6detmycGYs1FVIqhOv0wQMAgYAYQC\ny0VkBdAL2K2qa90xBE8vA9Pc5SeAZ4C7yjt4cnLyqeXExEQSE8seyjQGARJAQkQCCREJ3DHgDsDp\nUvou7ztWZq1kVU4KR857mqwD2aQG9STjaALvL+xF/vMJFOUkcF5sTwYPaMGgQU5y6NXLZimZpiEl\nJYWUlJSzOkZVYwRDgWRVTXLXJwMlngPGIvIw0EJVk931V4EFOMlhPHASaI7TKvhAVe8oc46uwH9U\ntV8557cxAnOag4UH2Zy/mdT8VOe1N5WNeZvZvm8bLYki5FACx7N6cSQjgR5tEhjSPYFh/SMZNEjo\n18/GHEzjVx+Dxc1wBotHADnAKs4cLE4AXgCuBkKAlcAYVd3oUecy4EGPWUMdVXWXu/wAcIGqjivn\n/JYITLWcLDlJ+v70Uwli3a5Uvs1MZfvBzZwoLiJofy+OZyfQQRJICO/NkC79Gd63K/36BtClCwQE\nePsdGFM36mv66Ch+mD46U1WfEpFJAKr6N7fOg8BPgRLgFVV9vswxLsMZByidNfQGcB5O19MOYJKq\n5pVzbksE5qzlH81nc/5m1u9K5astqXybs5H0o+s5qvsIzO9Hya7+RDfrx7nh/bmoRz8G923Huefa\ndFbjn+yCMmNqYN+xfazfvZ4V6etYtnUd63evI+vEBgILw9G8/uiu/sSG9GdAVH+G9oynf99mnHsu\nREdbgjC+yxKBMWepREvYvm876/PWszJ9Hct3rGNTwTr2ncym+eEETmb3h7z+dGnZmz6R8Qzq3oVz\newfRqxf06GFXSxvvs0RgTD05fOIw3+/+nnV561i5cx3fZaWy42Aa+0/mEFIYAwXxHM+Jo73G0a1N\nPH2j4xgc15VzE4Lp1QuioqwVYRqGJQJjGljhyUJ27N9BWkEam/ek8c3ONDbmbiXjcBr7S7IIOtaJ\nkvx4KIgjMiiOHu3j6d8pjgt7diMhPoRu3aB9e0sSpu5YIjDGh5woPkH6/nTSCtJYl5XG1+lb2bwn\njawjaRwgg8DjHSnZE0fA/ng6BMYR0zKOXh3iGBDbnYQeLejWDbp2hRYtvP1OjD+xRGCMnygqLiLj\nQAZb9m5lXVYa32U6LYrMI2kUlKQTdOIcZF8chbviCC2MIzokjriwOPp26kGvbi3p1s25orpzZ7tw\nzpzOEoExjUBxSTGZBzNJK0hjy940vs3YysbcNHYcSGP3ie0EFYcRdCiOk7u7czy/I20Do4hsFUls\nWCQ9IqPoHRtJ767t6NZNiImB4GBvvyPTkCwRGNPIlWgJ2QezSStIY9u+bWQfyGVbXh4Ze/PYdSiX\nvcfzOKi5nOQ4AUcjKTkYSUhRFG0CI+nQIpLoNlF07RBJfHQkfbtGMSi+I5HtyrsVmPFXlgiMMQAc\nKzpG3pE8sg/kkZqZR2pWLtt355G1P5fdR/LYX5THEcmlqHkOos0IPh5DG40hPCiG6FYxdGkfQ0JU\nLP1iYxgU35nIMBuo8BeWCIwxNVJSomzL3s832zP4PjOTrbszydifSe6xTApOZnKkWQZFzbORE20I\nLoyhdUkMEUExRLeMpVtYDD2jYugb25mBcdFEdQi22U8+wBKBMabOFZeUsC13N99sy+T7rEzS8jLZ\neSCD3KOZFBRncjQwi6KQPDjenpDjnWmlnQkP6kRUaGdi23ciPrIz58Z0YmCPznSNbmX3dapnlgiM\nMV5RXFLMjj15fLs9i42Z2WzNyyJjfza7jmRRUJTNoYAsToRkQ3EQQcc607KkE+0DOxMZ2pmYtp3o\nGt6JuHOi6dUpmj6x5xAeFmAJo5YsERhjfJaqsmv/Pr7dls2GjCy25GaTXpDlJoscDpHD8aAcioP2\nw5FIgo51IrQ4mjYB0U4Lo2U0Me2i6R7RifioaLpHtyUqSoiIgKAgb78732GJwBjj9wpPFrKzIJdN\nmTmk5mSzIz+HzP057DqcTf6JHA6U5HAkIJsSigk4HE3xgWiCCzvSKiCctiFhhIeGcU7rMKLbhxEb\nEU63qDDiO4fRK7Y97do0/oxhicAY02QcKjzErsO7yNifzdZdu8jYU0DW3gJyDxaQf7iAfYUFHCoq\n4BgFFAYUUBK8D062oFlRGM01jJYBYbQJCiMsNIxzWoXRsW04XcKj6BEZRXx0JLFhUYS1CCNA/KuP\nyhKBMcZUoKREydpziC2ZBaTlFLAzz0kcu/YXsOdwAfsK93CwJI9jgbmcCM6FVnkQfIigog60KI6i\ntUTSPiiKiBaRdGwVRUz7KLpERBIXFUWvzpHERLQjIMD706YsERhjTB1QhQMHICfvBFtzdpOWl0vG\n3lyy9+eRe8S5cG//yVwOk0thszyKmudCYCEBxyMIPhlB85IIWgaE06ZZBO1DIogIDSeydQTR7SLo\nHBZO13Mi6N4xgs7nhBIcXLfJwxKBMcZ4yb5Dx0jLyWd7Xj4Z+flk78sn7+Bedh/Op+B4PgdO7OVQ\nST7HyKcwMJ+TIfmAIsciCCqKIKQknFAiWHD/nzkvLqrWcVgiMMYYP3K48Cg78vLZnptPRv5esgry\neXD09XRo17LWx6yvZxYn8cMzi19V1Rnl1EkE/gQEAfmqmuhRFgisAbI8Hl4fBrwLdAHSgVtVdX85\nx7VEYIwxNVCbRFDpcLj7I/4CkAT0AcaKSO8yddoBLwLXqWpf4OYyh7kf2IjzoPpSjwCLVLUn8Jm7\nbupRSkqKt0NoVOzzrFv2eXpXVfOihgBpqpquqkXAbOCGMnXGAR+oahaAquaXFohIZ+Aa4FXAM0Nd\nD7zuLr8O3Fjrd2Cqxf6h1S37POuWfZ7eVVUi6ARkeqxnuds8xQNhIrJERNaIyHiPsj8BDwElZfaJ\nVNU8dzkPiKxZ2MYYY+pKVc82qk4HfRAwCBgBhALLRWQF0AvYrapr3TGE8k+gqiJiAwHGGOMllQ4W\ni8hQIFlVk9z1yUCJ54CxiDwMtFDVZHf9VWABTnIYD5wEmgNtcLqQ7hCRVCBRVXNFpCOwRFUTyjm/\nJQhjjKmhOp01JCLNgM04f+3nAKuAsaq6yaNOAs6A8tVACLASGKOqGz3qXAY86DFr6Glgr6rOEJFH\ngHaqagPGxhjjBZV2DanqSRG5F/gvzvTRmaq6SUQmueV/U9VUEVkArMMZC3jFMwl4Hs5j+f8B74nI\nXbjTR8/+rRhjjKkNn76gzBhjTP3zydvqiUiSiKSKyFZ3DMKcBRFJF5F1IrJWRFZ5Ox5/IyL/EJE8\nEVnvsS1MRBaJyBYRWeheT2OqUMFnmSwiWe73c617EaupBhGJcWdsfi8iG0TkV+72Gn0/fS4RVOci\nNlNjijM4P1BVh3g7GD/0Gs730ZNdFFk75X2WCjzrfj8HquoCL8Tlr4qAB1T1XGAocI/7e1mj76fP\nJQKqdxGbqTnv3x/XT6nqUmBfmc12UWQtVPBZgn0/a0VVc1X1W3f5MLAJ51qvGn0/fTERVOciNlMz\nCnzqXvA30dvBNBJ2UWTduk9EvhORmdbNVjsi0hUYiDNzs0bfT19MBDZ6XfcuVtWBwCicpuMl3g6o\nMXHvjGjf29p7GegGnAfsAp7xbjj+R0RaAR8A96vqIc+y6nw/fTERZAMxHusxOK0CU0uqusv97x7g\nQ5zuN3N28kQkCsC9KHK3l+PxW6q6W1049yWz72cNiEgQThJ4U1XnuJtr9P30xUSwBogXka4iEgyM\nAeZ6OSa/JSKhItLaXW4JjATWV76XqYa5wAR3eQIwp5K6phLuD1WpH2Hfz2oTEQFmAhtV9TmPohp9\nP33yOgIRGcUPz0CYqapPeTkkvyUi3XBaAeBcQPi2fZ41IyKzgMuACJz+1inAR8B7QCyVPFPDnK6c\nz3IqkIjTLaTADmCSR/+2qYSIDAe+wLmgt/THfDLOXSCq/f30yURgjDGm4fhi15AxxpgGZInAGGOa\nOEsExhjTxFkiMMaYJs4SgTHGNHGWCIwxpomzRGCMMU2cJQJjjGni/j84jHOLU3eCKwAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe4e31b8150>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(logistic_history.history['loss'], label=\"loss\")\n",
    "plt.plot(logistic_history.history['val_loss'], label=\"val_loss\")\n",
    "plt.legend()\n",
    "_ = plt.title(\"loss over epoch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Neural Network\n",
    "\n",
    "with Adam optimization\n",
    "(Kingma & Ba, 2014 Adam: A Method for Stochastic Optimization (http://arxiv.org/abs/1412.6980))\n",
    "\n",
    "\"... based on adaptive estimates of lower-order moments ...\"\n",
    "\n",
    "https://www.quora.com/Can-you-explain-basic-intuition-behind-ADAM-a-method-for-stochastic-optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_basic_NN(dims):\n",
    "    model = Sequential()\n",
    "\n",
    "    model = Sequential()\n",
    "    #model.add(InputLayer(input_shape=(X.shape[1],)))\n",
    "\n",
    "    #a fully-connected layer above input\n",
    "    model.add(Dense(300,input_dim=dims, activation='tanh'))\n",
    "\n",
    "    #a fully-connected layer above other layer\n",
    "    model.add(Dense(300, activation='tanh'))\n",
    "\n",
    "    #a fully-connected layer with sigmoid output - this layer will be interpreted as NN output\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "    #logloss, see https://www.kaggle.com/wiki/LogarithmicLoss\n",
    "    loss = keras.objectives.binary_crossentropy\n",
    "\n",
    "    # optimizer that will train the NN to minimize loss above. \n",
    "    # may or may not be suboptimal choice. See dir(keras.optimizers)\n",
    "    optimizer = keras.optimizers.adam(lr=0.001)\n",
    "    model.compile(loss=loss, optimizer=optimizer,\n",
    "              metrics=[keras.metrics.binary_accuracy] #also monitor accuracy\n",
    "             )\n",
    "    return model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = build_basic_NN(X.shape[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class AucHistory(keras.callbacks.Callback):\n",
    "    def __init__(self, X_train=None, y_train=None):\n",
    "        self.X_train = X_train\n",
    "        self.y_train = y_train\n",
    "        \n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.aucs = []\n",
    "        self.train_aucs = []\n",
    "        self.losses = []\n",
    " \n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        self.losses.append(logs.get('loss'))\n",
    "        if self.model.validation_data is not None:\n",
    "            y_pred = self.model.predict(self.model.validation_data[0])\n",
    "            self.aucs.append(roc_auc_score(self.model.validation_data[1], y_pred))\n",
    "        if self.X_train is not None:\n",
    "            y_pred = self.model.predict(self.X_train)\n",
    "            self.train_aucs.append(roc_auc_score(self.y_train, y_pred))\n",
    "        return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 500000 samples, validate on 500000 samples\n",
      "Epoch 1/20\n",
      "500000/500000 [==============================] - 49s - loss: 0.6466 - binary_accuracy: 0.6200 - val_loss: 0.6369 - val_binary_accuracy: 0.6353\n",
      "Epoch 2/20\n",
      "500000/500000 [==============================] - 38s - loss: 0.6288 - binary_accuracy: 0.6471 - val_loss: 0.6235 - val_binary_accuracy: 0.6534\n",
      "Epoch 3/20\n",
      "500000/500000 [==============================] - 49s - loss: 0.6122 - binary_accuracy: 0.6650 - val_loss: 0.6013 - val_binary_accuracy: 0.6773\n",
      "Epoch 4/20\n",
      "500000/500000 [==============================] - 44s - loss: 0.5922 - binary_accuracy: 0.6828 - val_loss: 0.5838 - val_binary_accuracy: 0.6904\n",
      "Epoch 5/20\n",
      "500000/500000 [==============================] - 42s - loss: 0.5823 - binary_accuracy: 0.6911 - val_loss: 0.5772 - val_binary_accuracy: 0.6949\n",
      "Epoch 6/20\n",
      "500000/500000 [==============================] - 50s - loss: 0.5744 - binary_accuracy: 0.6969 - val_loss: 0.5684 - val_binary_accuracy: 0.7015\n",
      "Epoch 7/20\n",
      "500000/500000 [==============================] - 36s - loss: 0.5676 - binary_accuracy: 0.7024 - val_loss: 0.5668 - val_binary_accuracy: 0.7023\n",
      "Epoch 8/20\n",
      "500000/500000 [==============================] - 49s - loss: 0.5612 - binary_accuracy: 0.7070 - val_loss: 0.5580 - val_binary_accuracy: 0.7091\n",
      "Epoch 9/20\n",
      "500000/500000 [==============================] - 36s - loss: 0.5563 - binary_accuracy: 0.7106 - val_loss: 0.5558 - val_binary_accuracy: 0.7112\n",
      "Epoch 10/20\n",
      "500000/500000 [==============================] - 46s - loss: 0.5513 - binary_accuracy: 0.7147 - val_loss: 0.5502 - val_binary_accuracy: 0.7160\n",
      "Epoch 11/20\n",
      "500000/500000 [==============================] - 36s - loss: 0.5473 - binary_accuracy: 0.7179 - val_loss: 0.5468 - val_binary_accuracy: 0.7195\n",
      "Epoch 12/20\n",
      "500000/500000 [==============================] - 50s - loss: 0.5437 - binary_accuracy: 0.7208 - val_loss: 0.5425 - val_binary_accuracy: 0.7223\n",
      "Epoch 13/20\n",
      "500000/500000 [==============================] - 40s - loss: 0.5402 - binary_accuracy: 0.7234 - val_loss: 0.5398 - val_binary_accuracy: 0.7241\n",
      "Epoch 14/20\n",
      "500000/500000 [==============================] - 48s - loss: 0.5360 - binary_accuracy: 0.7263 - val_loss: 0.5370 - val_binary_accuracy: 0.7266\n",
      "Epoch 15/20\n",
      "500000/500000 [==============================] - 43s - loss: 0.5330 - binary_accuracy: 0.7286 - val_loss: 0.5335 - val_binary_accuracy: 0.7286\n",
      "Epoch 16/20\n",
      "500000/500000 [==============================] - 46s - loss: 0.5305 - binary_accuracy: 0.7303 - val_loss: 0.5301 - val_binary_accuracy: 0.7308\n",
      "Epoch 17/20\n",
      "500000/500000 [==============================] - 48s - loss: 0.5274 - binary_accuracy: 0.7326 - val_loss: 0.5290 - val_binary_accuracy: 0.7317\n",
      "Epoch 18/20\n",
      "500000/500000 [==============================] - 37s - loss: 0.5252 - binary_accuracy: 0.7345 - val_loss: 0.5292 - val_binary_accuracy: 0.7309\n",
      "Epoch 19/20\n",
      "500000/500000 [==============================] - 46s - loss: 0.5228 - binary_accuracy: 0.7365 - val_loss: 0.5248 - val_binary_accuracy: 0.7346\n",
      "Epoch 20/20\n",
      "500000/500000 [==============================] - 42s - loss: 0.5198 - binary_accuracy: 0.7386 - val_loss: 0.5244 - val_binary_accuracy: 0.7353\n"
     ]
    }
   ],
   "source": [
    "basic_NN_aucs = AucHistories()\n",
    "basic_NN_histories = model.fit(X_train, y_train, nb_epoch=N_EPOCH, batch_size=BATCH_SIZE, \n",
    "          validation_data=[X_test,y_test], callbacks=[basic_NN_aucs])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEKCAYAAADzQPVvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VFX6x/HPk4RepITeAkhfQJCaoIYigoCslSIIq4vs\nLtj9obgWFnGVXXV17YjiSrMLqDQpQXpTCL2odAiGXgMh5/fHHTBAEhJIZpLM9/163Vdm7jl35plh\neObMueeeY845REQkdwsJdAAiIpL1lOxFRIKAkr2ISBBQshcRCQJK9iIiQUDJXkQkCCjZi1+Y2RYz\naxvoOHIbM4s2s+2BjkOyPyV78Rfn20QkAJTsRS6DmYUFOgaRjFCyF78zs3xm9pqZ7fRt/zGzvL6y\ncDP71swOmNk+M/sh2XFPmNkOMztsZuvNrE0qj3+VmX1sZnt93Ud/N08+MztoZvWS1S1lZsfNLNx3\nv7OZrfA9/3wzq5+s7hYzG2RmscARM7vo/4+Z1Taz732xrzezO5OVfWRm75rZdN9riDGzysnKI81s\nqS/GJWbWMllZCTMb5Xu/9pvZ1xc876NmFmdmu8ysb8b+RSQYKNlLIPwdaAY09G3NgKd9ZY8B24Fw\noDQwGMDMagEDgCbOuaJAe2BLKo//BlAEqArcANwD/Mk5lwB8CfRIVvcuIMY5F29mjYAPgH5ACeA9\nYJKZ5UlWvzvQESjmnEtK/qRmVgj4HhgDlPLVfdvM6iSr1hMY6nt9K4CxvmNLAN8Br/me+1XgOzMr\n7jtuNJAfqOt7X15N9phlgaJAeeA+4C0zuyqV90aClXNOm7Ys34BfgTa+25uBDsnK2gO/+m7/A5gA\nVL/g+KuBOKAtkCeN5wkFEoDayfbdD8z23W4LbE5WNh/o5bv9DjD0gsdbD1yX7DX0TeO5uwE/XLDv\nPeBZ3+2PgHHJygoBiUBFoDew6IJjFwB9gHLAGeCqFJ4zGjgOhCTbFwc0C/S/ubbstallL4FQHtia\n7P423z6Af+N9GUw3s5/N7AkA59xm4GFgCBBnZuPNrFwKjx0O5Enh8Sv4bscABc2smZlF4P2yONsl\nUgV4zNeFc8DMDuAl4vLJHiutkS9VgOYXHN8TKOMrd8COs5Wdc8eA/b7HL+eLM7mtvrKKwH7n3KFU\nnnefO/9XxnGgcBpxShBSspdA2AVEJLtf2bcP59xR59zjzrnqwC3Ao2f75p1z451z1+ElVQcMT+Gx\n44HTKTz+Dt9jnAE+w+vK6QF840u64CXbF5xzxZNthZ1znyZ7rLRGFG0D5lxwfBHn3ABfuQGVzlY2\ns8J4XTY7fa+/ygWPV8VXth0ooa4ZuRJK9hII44GnfSdjw4Fn8fqkz54gvdrMDDiM131xxsxqmlkb\nM8uH101z0ld2nmTJ/AUzK2xmVYBH8PrRzxqH15/e03f7rPeBv/ha/WZmhcysky8pp8e3QE0z62Vm\neXxbUzOrnazOzWYW5Tsh/Tyw0Dm3E5jiO7aHmYWZWTegNvCtc26Pr/xtMyvme9zr0xmTCKBkL4Ex\nDFgGxPq2Zb594PXNfw8cweuzfss5NwfIB7wI/AbsxuuuGZzK4z8AHAN+AebinQQddbbQObcEOIrX\ndTIl2f7leCdn38TrXtmEd3I3XdcHOOeO4p1/6I7XIt/tiznv2Sp4Xy7PAfuARkAv37H7gM54J6jj\ngceBzs65/b5je+P9YlmP1yf/YPKnTk98EtzMOX1ORPzBzEYBO5xzzwQ6Fgk+atmL+I8FOgAJXkr2\nIv6jKSMkYNSNIyISBNSyFxEJAgGfzMnM9NNCROQyOOfSfR4oW7TsA30ZcW7annvuuYDHkJs2vZ96\nL7PrllHZItmLiEjWUrIXEQkCSva5THR0dKBDyFX0fmYevZeBFfChl2bmAh2DiEhOY2a4DJygDfho\nHBHJmby56sQfMqNBrGQvIpdNv8qzXmZ9qarPXkQkCFwy2ZtZB9/CyZvOrhqUQp1oM/vJzFabWUyy\n/VvMLNZXtiQT4xYRkQxI8wStmYUCG4B2ePNzLwV6OOfWJatTDG8dz5ucczvMLNw5F+8r+xW41v0+\nJ3dKz6ETtCI5kO8EYaDDyPVSe58zeoL2Ui37ZniLM29xzp0GPgG6XlCnJ/Clc+7ssm/xF8aU3mBE\nRDJDREQEM2fODHQY2cqlkn0Fzl9geQe/L9x8Vg289TFnm9kyM+udrMwBM3z7+115uCIil2ZmGi10\ngUuNxknPb7Q8QGOgLVAQWGhmi5xzm4BWzrldZlYK+N7M1jvn5l5ZyCIiklGXSvY7gUrJ7lfCa90n\ntx2Id86dAE6Y2Q9AQ2CTc24XgHPuNzP7Gq9b6KJkP2TIkHO3o6OjdaWdiGSKU6dOMWjQID7//HMA\n7rrrLoYPH07evHmJj4+nb9++zJ8/n5CQEOrVq8cPP/wAwPDhw3njjTc4fPgw5cuX5+2336ZNmzaB\nfCnExMQQExNz2cdf6gRtGN4J2rbALmAJF5+grY23QPNNeItCLwa6AVuAUOfcETMrBEwH/uGcm37B\nc+gErUgOlJ1P0FatWpWRI0cyZ84cZsyYwaRJkwDo2rUrbdu2ZejQoQwePJiDBw/y5ptvArBo0SKi\noqLYsGEDN954I0uWLKFs2bJs27aNxMREqlWrFpDX4pcTtM65RGAgMA1YC3zqnFtnZv3NrL+vznpg\nKhCLl+jfd86tBcoCc81shW//txcmehHJ3cwyZ7tc48aN49lnnyU8PJzw8HCee+45Ro8eDUDevHnZ\nvXs3W7ZsITQ0lKioKABCQ0NJSEhgzZo1nD59msqVKwcs0WeqbDAnsxORnCc7/9+NiIhwM2bMcAUK\nFHBr1649t3/dunUub968zjnnjhw54h577DFXrVo1V61aNffSSy+dqzdu3DjXqlUrV7x4cde9e3e3\na9cuv7+Gs1J7n337051rs8UVtAcOBDoCEcmNypcvz5YtW87d37ZtG+XLlwegcOHCvPzyy/z8889M\nmjSJV199lVmzZgHQo0cP5s6dy9atWzEznngixetJc5RskewffDDQEYhIbtSjRw+GDRtGfHw88fHx\nDB06lN69vdHh3377LZs3b8Y5R9GiRQkNDSU0NJSNGzcya9YsEhISyJcvH/nz5yc0NDTAr+TKZYtk\nv2gRTJgQ6ChEJDcxM55++mmaNGlCgwYNaNCgAU2aNOHpp58GYPPmzdx4440UKVKEyMhIBgwYwA03\n3EBCQgKDBw+mVKlSlCtXjvj4eF588cUAv5orly3ms58713HnnbBqFYSHBzQcEUmn7DwaJzfJrNE4\n2SLZO+d47DHYsQM+/TSg4YhIOinZ+4e/5sbxm2HDIDYWPvss0JGIiOQ+2aZlD7B4MXTtCitXQpky\nAQ1LRC5BLXv/yHUte4DmzeHee6F/f9BnSEQk82SrZA/w3HPw888wdmygIxERyT2yVTfOWT/+CB06\nwE8/QYULJ1QWkWxB3Tj+kSu7cc5q3Bj+9jfo10/dOSIimSFbJnuAp56C3bth1KhARyIikvNly26c\ns1atgjZtYPlyqFzZz4GJSJrUjeMfuaobJ7UPTP368MgjcN996s4RkawXExNDpUqVLlkvJ65xmy2S\n/UvzXkq1bNAgOHQI3n3XjwGJiKQhJ65xmy2S/YgfR/DRio9SLAsLg//9D555Bn75xb9xiYjkFtki\n2U+9eypPzniSKZumpFhepw48+aR3wVVSkp+DE5EcZ/jw4dx5553n7XvooYd46KGH+Oijj6hbty5F\nixalevXqjBgx4oqeKyEhgYcffpgKFSpQoUIFHnnkEU6dOgVAfHw8nTt3pnjx4pQsWZLrr7/+vBgr\nVqxI0aJFqV279rm59LPKpRYc94ta4bX4utvX3PLJLXzX8zuaVWh2UZ1HHvGmQX7jDXjooQAEKSIZ\nZv/InK4O91zGTtr16NGDoUOHcvToUQoXLsyZM2f4/PPPmTBhAvHx8Xz33XdUrVqVH374gY4dO9K0\naVMaNWp0WbG98MILLFmyhJUrVwLeOrfDhg1j6NChvPLKK1SqVIn4+HjAW+cWYMOGDbz11lssW7bs\nvHVus1RGlrXKio1kS25NWj/JlX25rNsYvzHFZbg2bnSuZEnnNmxIsVhE/IhsvCyhc861atXKffzx\nx84556ZPn+6qV6+eYr0//vGP7vXXX3fOOTd79mxXsWLFSz52RESEmzlzpnPOuerVq7spU6acK5s2\nbZqLiIhwzjn37LPPuq5du7rNmzefd/ymTZtc6dKl3YwZM9ypU6fSfK7U3mdy4rKEZ3Wp1YXnWz/P\nTWNuYs/RPReV16gBzz4LffvCmTP+j09Eco6ePXsyfvx4wFt4/O677wZgypQptGjRgpIlS1K8eHEm\nT57Mvn37Lvt5du3aRZUqVc7dr1y5Mrt27QLg//7v/7j66qtp37491atXZ/jw4QBcffXVvPbaawwZ\nMoQyZcrQo0cPdu/efdkxpEe2SvYAf278Z/pe05ebx97MkYQjF5UPHAj58sGrrwYgOBHJMe644w5i\nYmLYuXMnEyZMoGfPniQkJHD77bczaNAg9u7dy4EDB7j55puv6HqBnLLObbZL9gDPXP8MTcs35bbP\nbuPUmVPnlYWEwIcfwr/+BWvXBihAEcn2SpUqRXR0NH379qVatWrUqlWLU6dOcerUKcLDwwkJCWHK\nlClMnz79ip4np6xzmy2TvZnxVqe3KJSnEPdOvJckd/4QnKpVvcVO+vSBrD6nISI5V8+ePZk5cyY9\ne/YEoEiRIvz3v//lrrvuokSJEowfP56uXbued0xGx8/nlHVus/V0CSdOn6Dd6HZEVozk3+3/fV6Z\nc3DTTXDDDfD3v/sjUhFJTtMl+EeuW4M2NftP7KfVh63o17gfj7R85Lyy7du9GTJnzICGDbM6UhFJ\nTsneP3LV3DhpKVGgBFN7TeXVRa/yyepPziurVAlefhnuvhtOnAhQgCKS62zbto0iRYpctBUtWpQd\nO3YEOrzLku1b9metiltF24/b8skdn9Cmaptz+52Dnj2heHF4++2sjFREklPL3j+CpmV/Vv0y9fns\nzs/o/kV3VuxZcW6/mTdJ2pQp3hW2IiJysRyT7AGiI6J56+a36DyuM1sObjm3/6qrYNw4b6HyHPoL\nS0QkS+WYbpzk3lj8Bm8tfYt5984jvGD4uf3DhsHMmd4J2ywesioS9HLaFL85WVCMxknNkzOeZM7W\nOcy8ZyYF8xQEvCkU2raF9u29ZQ1FRHKroEn2zjn6TuzL/hP7+brb14SFeBN4bt8OTZrApEnQvHlm\nRysikj3k2hO0FzIzRnYZSUJiAsN+GHZuf6VK8M473gidw4cDGKCISDaSY1v2Z205uIVrR1zLmr+t\noWzhsuf29+8Px47BmDGZEaWISPYSNC37syKKRdCnYR+Gzhl63v7//Ad+/BFGjw5QYCIi2UiOb9kD\n7Du+j1pv1mLBfQuoWbLmuf0rV0K7drBwIVx99ZVGKiKSfQRdyx6gZMGSPNbyMZ6e9fR5+xs29BYq\n79kTTp1K5WARkSCQK5I9wEMtHmLB9gUs2bnkvP0PPAClS3srXImIBKtck+wL5inIkOghDPp+0HkX\nIJjBqFFe3/3MmQEMUEQkgHJNsgfoe01f4o7FMXXz1PP2lyrlJfw+fcC3yLuISFDJVck+LCSMF9u+\nyBMznuBM0vkrkrdvDz16wL33ejNliogEk1yV7AG61upKkXxFGLtq7EVlL7wAu3ZpKmQRCT65Yujl\nheZtm8fdX93NhoEbyB+W/7yyTZsgMhJmzYL69TP1aUVE/CYoh15eqFXlVlxT9hreXnpxE75GDfj3\nv70uHa1uJSLBIle27AHW/raW6I+i2fjARorlL3ZemVa3EpGcTi17n7ql6nJLrVsYPm/4RWXJV7ea\nODEAwYmI+FmubdkD7Di8g4bvNiT2L7FUKFrhovKFC+HWW2H5cqhwcbGISLalln0yFYtWpF/jfjwX\n81yK5S1bwoAB0Lu3t/CJiEhulauTPcCTrZ5k0oZJrP1tbYrlTz3lzZvz/vt+DkxExI9ydTfOWa8s\neIW52+YyofuEFMtjY73ZMdeuhfDwFKuIiGQr6sZJwYBmA/hpz0/M3zY/xfIGDbzROYMH+zkwERE/\nCYqWPcD/VvyPET+OYN6f5mF28ZfhoUNQpw5MmADNmmV5OCIiV0Qt+1T0atCLIwlHmLRhUorlV10F\nL73knbDVyVoRyW2CJtmHhoTyUruXGDxzMIlJiSnW6d0b8uWDkSP9HJyISBYLmmQP0PHqjpQuVJqP\nVnyUYrkZvPWWt9DJvn3+jU1EJCsFTZ/9WUt2LuG2T29j4wMbKZinYIp1HnwQEhLgvff8FpaISIao\nz/4SmlVoRstKLfnv4v+mWmfoUJg0CZYt82NgIiJZKOha9gAb920k6sMo1g9YT8mCJVOs89FH3iRp\nixZBSNB9JYpIdqeWfTrULFmTO+veyT/n/jPVOvfcA3nywAcf+DEwEZEsEpQte4A9R/dQ7+16/Hj/\nj1QpViXFOitWwE03wbp1UKKEnwMUEUlDRlv2QZvsAZ6d/SxbDm7h41s/TrXOwIHeuPt33vFjYCIi\nl5Dp3Thm1sHM1pvZJjN7IpU60Wb2k5mtNrOYjBwbSI9HPs70n6ezcs/KVOs8/zx8/bU3DbKISE6V\nZsvezEKBDUA7YCewFOjhnFuXrE4xYD5wk3Nuh5mFO+fi03Os7/iAtewB3lj8BlM2T2Hy3ZNTrfPh\nhzBiBCxYoJO1IpI9ZHbLvhmw2Tm3xTl3GvgE6HpBnZ7Al865HQDOufgMHBtw/Zv0Z+O+jXyz4ZtU\n6/Tt611wNWqU/+ISEclMl0r2FYDtye7v8O1LrgZQwsxmm9kyM+udgWMDLm9oXj645QP6f9uf+OPx\nKdYJCfGurP3732H/fj8HKCKSCcIuUZ6e/pU8QGOgLVAQWGhmi9J5LABDhgw5dzs6Opro6Oj0Hpop\nboi4gZ71e/LX7/7KZ3d8luKsmI0bw223wTPPeIlfRMSfYmJiiImJuezjL9Vn3wIY4pzr4Ls/GEhy\nzg1PVucJoIBzbojv/khgKl5LPs1jffsD2md/1snEk1w74lr+ft3f6Vm/Z4p19u+HunVh8mQv+YuI\nBEpm99kvA2qYWYSZ5QW6ARfOETwRaGVmoWZWEGgOrE3nsdlG/rD8jL51NA9PfZgdh3ekWKdECXjh\nBW8a5KQkPwcoInIF0kz2zrlEYCAwDS+Bf+qcW2dm/c2sv6/OeryWfCywGHjfObc2tWOz7qVcucbl\nGvNg8we5d+K9pPZr409/Aufgf//zc3AiIlcgqC+qSkliUiJRH0bRp2Ef/tb0bynWWbYMOnf2rqwt\nXtzPAYqIoCtoM8WG+A1EfRjFwvsWUqNkjRTr/OUv3tw5b7zh5+BERFCyzzRvLH6DcavHMfdPcwkL\nuXjQ0r593snaadPgmmsCEKCIBDXNeplJBjQbQKE8hfjX/H+lWF6yJAwbppO1IpIzKNmnIsRCGNV1\nFK8teo0Ve1akWOe+++D0aRg92s/BiYhkkLpxLmH0ytH8a8G/WNZvGfnC8l1UvnQp3HKLd7K2WLEA\nBCgiQUl99pnMOccdn9/B1cWvZviNw1Osc//9kD8//Df1lQ5FRDKVkn0W+O3YbzR4twGf3/k5rSq3\nuqg8Ph6aN4c77/QuugoNDUCQIhJUdII2C5QqVIp3O71Lnwl9OJJw5KLy8HBYvNhbr/aPf4TDhwMQ\npIhIGpTs06lr7a7cUOUGHp/+eIrl4eEwfTqULw+RkfDLL34OUEQkDUr2GfBah9eY9vM0Jm9KeaGT\nvHnh3Xe9C64iI+EKJqgTEclUSvYZUDRfUT7640f0+6Yf+47vS7GOmbdu7Zgx0K2bt8KViEig6QTt\nZXh02qPsOrKLT+74JM16mzZ5wzLbtYP//AfCLrV6gIhIOukErR+80OYFYuNi+WR12sm+Rg1YuBA2\nb4YOHbTKlYgEjpL9ZSiQpwAf3/oxD055kJ2Hd6ZZt1gx+PZbaNjQG565fr2fghQRSUbJ/jI1Kd+E\nAU0HcN+k+1Kd+/6s0FB45RUYPBiuvx6mTPFTkCIiPkr2V+Cp655i34l9vLf8vXTVv/de+Oor7++r\nr3qLoIiI+INO0F6hdb+t47pR17Hoz4u4usTV6Tpm61bo2hUaNfKGaua7eModEZE06QStn9UpVYdn\nrn+G7l90Z/P+zek6pkoVmDcPDh2CNm0gLi6LgxSRoKdknwkeaP4Ad9S9gxYjW/Dw1IdTHYOfXOHC\n8MUX0Latd+J25Uo/BCoiQUvJPhOEWAhPtnqStQPWcvrMaWq/VZuXF7zMycSTaR8XAkOHwvDh3lj8\nTz/1U8AiEnTUZ58F1sev58kZT7IybiX/bPNPuv2hGyGW9vfq8uXQs6c3RPPNN6F0aT8FKyI5kqY4\nzkbmbJnD498/jmG83P5lrq9yfZr1T5yAf/wDRo3yrrjt0cObfkFE5EJK9tlMkkvik9Wf8NTMp2hU\nrhHD2w2nZsmaaR6zdKk3PLNaNXjnHW8mTRGR5DQaJ5sJsRB61u/J+oHraVmxJVEfRvHA5Af47dhv\nqR7TtKnXrXPNNd42apTG5IvIlVHL3s/ij8fz/JznGbtqLI9HPs5DzR+iQJ4CqdZfuRL+9CcoVQre\nfx8qV/ZjsCKSballn82FFwzn9Y6vs/C+hSzdtZTab9VmTOwYklxSivUbNvRWwYqOhmuv9S7CSkq5\nqohIqtSyD7B52+bx2PTHSExK5PUOr6e4xu1Z69Z5ffn588PIkVC9uh8DFZFsRS37HKZV5VYsum8R\ngyIHcdfnd/Hs7GdJTEpMsW6dOt6Vt126eBdivfYanDnj54BFJEdSyz4b2XN0D72/7s3JxJOMu20c\nla6qlGrdTZvgz3+G06fhww+hdm0/BioiAaeWfQ5WtnBZpvWaRucanWnyfhMmrJ+Qat0aNWD2bOjV\nC667zrsKNzHlHwQiImrZZ1eLdiyix5c96FSjEy+3f5n8YflTrbtlC9x/v7cS1ogR0Lix/+IUkcBQ\nyz6XaFGxBT/1/4m9x/bSfGRz1v22LtW6EREwbRoMGACdOnlDNXemvYCWiAQZJftsrFj+Ynx6x6cM\nbDqQ6z+6nlE/jUp1VSwzL8lv2OBdcduwIQwZAseO+TdmEcme1I2TQ6zZu4ZuX3SjQZkGvNv5XYrm\nK5pm/a1b4amnYM4cGDYM7rnHm2VTRHIHdePkUvVK12Npv6Vcle8qGr3XiKU7l6ZZv0oVGDvWmzN/\nxAho0gRiYvwTq4hkP2rZ50Bfrv2Sv373VwZFDeLRlo9ecvpk5+Dzz+GJJ7y5dv71L280j4jkXGrZ\nB4Hb697Okn5L+GrdV3Qa14m9x/amWd8M7rrLuwK3ZUtve+QRb/SOiAQHJfscKqJYBHP6zqFx2cY0\neq8RM3+Zeclj8ueHQYNg7VpISPAuxHrtNTh1yg8Bi0hAqRsnF5j5y0zumXAPfRr24enrn6ZgnoLp\nOm7NGvi///Ouxv33v6FrVy2WIpJTaPGSILX32F7u/+Z+pv08jeL5i1OteDWql6hOtWK+v8WrUb14\ndUoXKo1dkNGnTYPHHoPwcHj1VV2UJZITKNkHuSSXxK4ju/h5/8/8cuAXfj5w/t8Tp09QrXi1c8n/\n7JdC5SLViJkQwfND8tKmDTz/vLdSlohkT0r2kqZDJw/x68FfU/wy2HF4B+ULV+DaQ/9k9n+707Mn\nPP00lCkT6KhF5EJK9nLZEpMSWbJzCX0n9KVZmRsoOv91Ph1TkIEDvW6eomlfxyUifqShl3LZwkLC\niKwUyfL7l2N5ThJTswljvl/Fr79CzZrw+uveKB4RyXmU7OUiRfIVYfSto3my1ZPcM7MNkQ++y7Rp\nju+/94Zrjh6tRVNEchp140iaNsRvoPuX3alevDojbxnJqqXFeOIJOHoU/vlPb5ZNDdcU8T9140im\nqhVei4X3LaR8kfI0eq8RYRELmT/fG63zxBNw/fWwYEGgoxSRS1HLXtJt4vqJ3P/t/TzS4hEGRQ3C\nJYXw8cfw3HPQqJHX0q9XL9BRigQHjcaRLLX90HZ6ftWT/GH5GX3raMoWLsvJk/D22/DSS3DzzfCP\nf3izbopI1lE3jmSpSldVYnaf2URWjKTxe42Z/vN08ueHRx/1pl2oWNFr5d95J8ya5c24KSKBp5a9\nXLbZv86m99e96dWgF8+3fp48oXkAOHwYxoyBd97xJlnr3x/69oUSJQIbr0huom4c8avfjv1G34l9\n2Xd8H+NvH0/V4lXPlTkHCxd6Sf+bb7yJ1v76V2jeXCN4RK6UunHEr0oVKsU3Pb6hW71uNB/ZnM/X\nfH6uzAwiI71x+Zs3Q/360KuX183z3ntw5EgAAxcJMmrZS6ZZtmsZ3b/oTvOKzbmx2o3UL12fuqXq\nUiBPgXN1kpJg5kx4912YPRu6dfNa+w0aBDBwkRxI3TgSUIcTDjNi+QhW7FnBqr2r2LhvI1WuqkKD\nMg2oX7q+97dMfSKKRbBndwgjR8L770OlSl7Sv/NOb5EVEUmbkr1kK6fOnGLjvo3ExsWyKm4VsXu9\nvwdPHqRe6Xo0KN2AuuH1OfZLA2aNr8/KxcXp08c7qat1ckVSp2QvOcKBEwdYvXe19yWwdxWxcbGs\n3ruaQmFXUfBIfeIWt6ZFnvt5+C9X0bEjhIYGOmKR7EXJXnKsJJfE1oNbiY2L5dPVX/DNuikU3tiP\nPMsf4sF7y3LvvRq+KXKWkr3kGlsObuGVBa/w8YqxlIm/iz1fPc5d7a5mwABvRI9IMFOyl1xn77G9\nvLH4Dd5e+g4VEtoR99UT1CjciIED4bbbIG/eQEco4n9K9pJrHUk4wojlI3h14auU4g/Y/CfZsyia\n+/sZ/ftD+fKBjlDEfzI92ZtZB+A1IBQY6ZwbfkF5NDAR+MW36yvn3PO+si3AYeAMcNo51yyFx1ey\nlwxJSExg7KqxDJ8/nHxJxajw65Ms/KgrN7UPYeBAaNVKV+hK7pepyd7MQoENQDtgJ7AU6OGcW5es\nTjTwqHPulhSO/xW41jm3P43nULKXy3Im6QwTN0zkpXkvcejEEZqcGsTSD+6mQN68DBwI3btDkSKB\njlIka2SwKOpbAAANzUlEQVT2dAnNgM3OuS3OudPAJ0DXlJ43rZjSG4xIRoSGhHJbndtY/OfFvNPl\nLeLLfsKJ+6vT8tH/MHHKUSpU8Obj+fhjOHgw0NGKBNalkn0FYHuy+zt8+5JzQKSZrTSzyWZW94Ky\nGWa2zMz6XXm4IhczM9pUbcO0XtOY2H0iBwotYnHLqtw+6i9U7TiRzyYcpXJl6NgRPvgA4uMDHbGI\n/4Vdojw9/Ss/ApWcc8fNrCMwAajpK4tyzu02s1LA92a23jk398IHGDJkyLnb0dHRREdHpyd2kYs0\nLteYT+/4lF8O/MLX675myrE3WNy0F9d2bkbJkx35dHZHHnm0Ls2aGrffDrfeCmXLBjpqkUuLiYkh\nJibmso+/VJ99C2CIc66D7/5gIOnCk7QXHJNiP72ZPQccdc69csF+9dlLljp66iizfp3FlE1TmLJ5\nCmeSkqgd1oGENR2JndCWhrWLcvvt3jDOihUDHa1I+mT2CdowvBO0bYFdwBIuPkFbBtjrnHNm1gz4\nzDkXYWYFgVDn3BEzKwRMB/7hnJt+wXMo2YvfOOfYsG/DucS/cMdCquZtQt5tHdk0pSN1Sv6BO273\nWv1Vq1768UQCJSuGXnbk96GXHzjnXjSz/gDOuffMbADwVyAROI43MmeRmVUDvvI9TBgw1jn3YgqP\nr2QvAXPs1DFmb5nNlE1TmLx5CkePn6bkgQ7s/qEjVV07butUlC5d4JprNJxTshddVCVymZxzbNy3\nkSmbpzB50xTmb11I2ZPRHFvYi7Cfu9ClYwG6dIE2bTQNswSekr1IJjmccJiv133N2FVjWbx9KTWT\nbuXk0rvZOieaNtGhdOkCnTrpBK8EhpK9SBbYdWQXn6z+hDGxY9h9JI7GYT1J/LEXi79pQK2aRpcu\n0KWLt+KWunvEH5TsRbLYmr1rGLtqLGNXjaVI3qK0LNSLpBU9iZlUicRE6NzZS/ytW0O+fIGOVnIr\nJXsRP0lySczfNp8xsWP4Yt0X1C9dnzbhvUhafQczvi3GqlXQtq2X+G++GcqUCXTEkpso2YsEQEJi\nApM3TWbMqjHM+GUGN1a7kS4Rd5O47mamfZeP6dOhVi2v1d+5s0b3yJVTshcJsAMnDvDlui8ZEzuG\nn/b8RLMKzWheLpKih6LYOr8507+5ihMnvJO7nTt7rf+CBQMdteQ0SvYi2ci+4/tYuGMh87fNZ8GO\nBSzftZzqJapTr0gUtiOSX+ZEsXpeBDdcb3Tu7H0BVKoU6KglJ1CyF8nGTp05xYo9K84l//nb5pOU\n5KgSGkXir5FsnhVFlbyNuKVTXjp3hqZNtdi6pEzJXiQHcc6x9dBWL/lvX8D87fPZGL+Zkqcbc3Jj\nJKd+jqJzw5b8qVs4rVsr8cvvlOxFcrjDCYdZsnMJ87fNZ8bG+SzbvRh3tBwhOyJpWSmSfu2juLNN\nLUJDLjVDueRmSvYiucyZpDOs+W0NXy2bz9fLFrDuyALO5D1A1bCW3Fw/ituaRtK0fFMK5S0U6FDF\nj5TsRXI55+D7Rbt5e9JCZm5cwJkK80ksGUvtknVpXT2SyEqRRFWOomJRzdecmynZiwSRpCSYPx9G\njz/J5/OWU6TefK76wwJ2hy2gYN78RFbykn+7au2oE14H0+D+XEPJXiRInT4NM2fCuHEw6RtHncjN\n1LlpAYnl5zFnx3RCLITONTrTqWYnoiOiyR+mqTtzMiV7EeHECfjuOxg/HmbMgMbXOuresJrEat+x\n6tS3rN4bS+uqrelUoxOdanSiQtELl5aW7E7JXkTOc+QIzJ0LMTHetnYtNIrcR5moaRwq8y0/HppG\n5WKVzyX+ZhWaERqiMZ7ZnZK9iKTp0CGvn3/2bC/5r9uQSK22iyh0zXfsKPQtR5L20LFGRzrV6MRN\nV99EsfzFAh2ypEDJXkQy5NAhmDcvWct/11Yqtp6Mq/EtO0Pncm35xnSp1YmONTpSr1Q9neTNJpTs\nReSKnE3+s2fDrLnHWXdyNiVbfMvRslMJzXuadlVv4vaGHbixWjuKFyge6HCDlpK9iGSqgwe9bp+l\nyxxzVm1i+aGpnKw4laSK8ygXWp/rynWgR7Ob6NjgWsI0n4PfKNmLSJZyDnbtggVLTjLxp7nM3zuV\n7XmnklQwjnLH29OsRAdua9iedi3KUq5coKPNvZTsRcTvnIPF67bz8YJpzNo2lZ/dTNz+CArs7ECD\nQh1oV6slkc3zEhUFhQsHOtrcQcleRALu9JnTLNqxmM+WT2PKpqnsOLGJAvubc3RXBcoWKU3dymVo\nVq8MkQ3LUKl4GcoUKkPJgiUJMU3ull5K9iKS7ew9tpclO5ewdd8elq6LY/Wve/l1bxyHEuPIHx5H\nUsE4TtlhShUMp0zhMt5WyLcV/v1v4byFSXJJnEk6Q5JL8m67M6nuS77/jDuDc46WlVpSs2TNQL8l\nV0zJXkRyjIMHYc4cmDULZsw+xY79v3FNqzhqN4mjUu04worFsfdYHHHH4og7Gsex08cItVBCLITQ\nEO9viIWke1+SS2LWr7OoclUVejfoTbc/dCO8YHig34bLomQvIjlWXJxvyOcsbztyBFq3hjZtvK16\n9StfqD0xKZEZv8zg45UfM3nTZG6IuIHeDXrTuWbnHDVfkJK9iOQaW7f+nvhnzYLERGjY8Pytdm3I\nk+fyHv9wwmG+WvcVo2NHs2LPCu6ocwe9G/YmqlJUtr94TMleRHIl52DnTli58vxt61aoVeviL4Hw\nDPbObD+0nbGrxjI6djQnTp+gV4Ne9G7Qmxola2TNC7pCSvYiElSOH4c1a87/AoiNhUKFoEGD878A\nataEsLC0H885x4+7f2R07GjGrx5PteLVvP79et0oWbCkf15UOijZi0jQcw62bbv4V8DOnVC/PjRp\nAk2ben/r1El9IffTZ07z/S/fMzp2NJM3TaZ1RGu61etGuSLlyB+W/9yWLzTf+ffD8mX5MFIlexGR\nVBw9CitWwNKlsGyZt+3cCddc83vyb9IEatSAC9dzP3TyEF+u+5KJGyZy6OQhTiaeTHVLOJNA3tC8\n530BJN+alm/Kmze/eUWvRcleRCQDDh2C5ct/T/7LlsH+/dC48e/Jv2lTiIhI/0igJJfEqTOnUv0y\nyB+Wn2vKXnNFcSvZi4hcofj4878Ali71Vv86m/ivvx6iorzzAoGiZC8ikgV27/a+ABYv9i4E+/FH\n76Rv69YQHQ2RkVCwoP/iUbIXEfGD48dh4cLfV/xasQIaNfISf+vW0LIlFCiQdc+vZC8iEgDHjsGC\nBb8n/9hYuPba35N/ixaQPxMv0FWyFxHJBo4ePX+t39Wrvf7+6Gho395r+V8JJXsRkWzo8GEv+cfE\nwJkz8PLLV/Z4SvYiIkEgo8leKwWIiAQBJXsRkSCgZC8iEgSU7EVEgoCSvYhIEFCyFxEJAkr2IiJB\nQMleRCQIKNmLiAQBJXsRkSCgZC8iEgSU7EVEgoCSvYhIEFCyFxEJAkr2IiJBQMleRCQIKNmLiAQB\nJXsRkSCgZC8iEgSU7EVEgoCSvYhIELhksjezDma23sw2mdkTKZRHm9khM/vJtz2d3mNFRMQ/0kz2\nZhYKvAl0AOoCPcysTgpV5zjnGvm2YRk8VjJRTExMoEPIVfR+Zh69l4F1qZZ9M2Czc26Lc+408AnQ\nNYV6dgXHSibSf6jMpfcz8+i9DKxLJfsKwPZk93f49iXngEgzW2lmk82sbgaOFRERPwi7RLlLx2P8\nCFRyzh03s47ABKDmFUcmIiKZxpxLPZ+bWQtgiHOug+/+YCDJOTc8jWN+Ba7FS/iXPNbM0vOFIiIi\nF3DOpdSFnqJLteyXATXMLALYBXQDeiSvYGZlgL3OOWdmzfC+QPab2SWPzWiwIiJyedJM9s65RDMb\nCEwDQoEPnHPrzKy/r/w94A7gr2aWCBwHuqd1bNa9FBERSU2a3TgiIpI7BPQKWl10lbnMbIuZxfou\nblsS6HhyEjP70MzizGxVsn0lzOx7M9toZtPNrFggY8xJUnk/h5jZjmQXYHYIZIw5hZlVMrPZZrbG\nzFab2YO+/Rn6fAYs2euiqyzhgGjfxW3NAh1MDjMK77OY3JPA9865msBM331Jn5TeTwe8muwCzKkB\niCsnOg084pyrB7QABvhyZYY+n4Fs2euiq6yhE96XwTk3Fzhwwe5bgP/5bv8P+KNfg8rBUnk/QZ/P\nDHPO7XHOrfDdPgqsw7tmKUOfz0Ame110lfkcMMPMlplZv0AHkwuUcc7F+W7HAWUCGUwu8YDvAswP\n1C2Wcb7RjY2AxWTw8xnIZK8zw5kvyjnXCOiI91PvukAHlFs4bySDPrNX5h2gKnANsBt4JbDh5Cxm\nVhj4EnjIOXckeVl6Pp+BTPY7gUrJ7lfCa93LZXLO7fb9/Q34Gq+rTC5fnJmVBTCzcsDeAMeToznn\n9jofYCT6fKabmeXBS/SjnXMTfLsz9PkMZLI/d9GVmeXFu+hqUgDjydHMrKCZFfHdLgS0B1alfZRc\nwiSgj+92H7ypQOQy+RLSWbeiz2e6mJkBHwBrnXOvJSvK0OczoOPsfXPpvMbvF129GLBgcjgzq4rX\nmgfvYrmxej/Tz8zGAzcA4Xj9n88CE4HPgMrAFuAu59zBQMWYk6Twfj4HRON14TjgV6B/sj5nSYWZ\ntQJ+AGL5vatmMLCEDHw+dVGViEgQ0LKEIiJBQMleRCQIKNmLiAQBJXsRkSCgZC8iEgSU7EVEgoCS\nvYhIEFCyFxEJAv8PTjVFbp884VkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe4a73a98d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(basic_NN_histories.history['loss'], label=\"loss\")\n",
    "plt.plot(basic_NN_histories.history['val_loss'], label=\"val_loss\")\n",
    "plt.legend()\n",
    "_ = plt.title(\"loss over epoch\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEKCAYAAADzQPVvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYXePd//H31yQROZKEIAkJoo1QEkQcqlMNmRCCepCo\npodUnlZK1CGi7WN+11WUFlE8rUa0eFqJikhUI7Q6RVskKg45IAhyIGcJgiTz/f1xr7H37Mzsw2TP\nrH34vK5rX7OOe92zM/nMPd91r7XM3RERkdK2U9wNEBGR5qewFxEpAwp7EZEyoLAXESkDCnsRkTKg\nsBcRKQMKe5EyZmbVZnZv3O2Q5qewl6yY2VIz+9jMNpnZe2Z2r5l1StnmGDN7wsw2mtkGM5tlZv1S\ntulkZpPM7O3ovZaY2c1m1jXNsc3M3jSzBY2062spy75lZk8lzbeJQu01M/vQzN4ysylmtm/TP5GS\noQttyoTCXrLlwHB37wgcChwC/KRupZkdDcwBZgB7AX2AF4F/mlmfaJs2wN+AfsDQ6L2OBtYAg9Ic\n+3hgZ2B3MzuigXZlCqwHgOHASKBT1P55wNfS7dSczKxVXMdOYXE3QFqGwl5y5u7vA48B/ZMW3wDc\n7e63uvtH7r7e3X8KPANUR9t8E+gFnOHui6P3Wu3u17j77DSHHA1MB2ZG01kzsyHAEGCEuz/v7rXu\nvtHdf+3udzWyTz8zqzGz9Wb2ipmdGi0/ysxWmpklbXuGmb0YTe9kZldGf62sMbNpZrZbtK63mdWa\n2XfM7G3gr40ce7iZzY+O/U8zOyRp3dLo/ReY2Tozu8vMdk5a/z0ze93M1prZTDPbK2ldfzN7PFr3\nnplNjFY50MbM7o7+InvFzA7P5TOW4qCwl1wYgJn1BKqAZ6P5doQe+p8a2Od+4MRoeggw290/zvqA\n4b2/DkyL3utcM2udQ5uHAM+6+/Isj9caeBh4FNgd+CHwBzPr6+7PAh9R/y+CUcAfoukfAqcR/hLZ\nC1gP3J5yiOOBLwJDGzj2AGAK8D2gC3AHMCvl+x0FnATsDxxI9NeVmZ0AXAv8V3Tst4Gp0bqOhF8u\nf4nWHUD4CwvCv+lpwH1AZ2AWcFumz0mKj8JesmXAQ2a2EXgHeAP4WbSuC+FnaWUD+70HdIumuzay\nTTpnAhvd/Z/AE9GyU3LYv2vUhmwNBtq7+8/dfau7/x34MyFkIYTiSPg8RIdFywDGAj9x9xXuvgX4\nf8BZZpb8/6za3Te7+6cNHPsC4A53n+vBPcCnUZsg9MJvc/fl7r4euKauLcB5wBR3n+/unwETgaOj\n8xLDgRXufrO7f+buH7r7c0nHfcrdH/Vwo6z/I5S5pMQo7CVbTiiFdAIqgROAuvr5eqCW0GtMtRew\nOppeA+yd43FHAw8CuPs24CHql3K2Aqk9/dbAlmh6bSPtaszewLspy94GekTT9wFnRucfzgSed/e6\n7XsDM6ISzHpgYdS+7knvlfreyfYFLq3bP3qPntT/zJL3fydpXV1vHgB3/4jwvfeI3uPNNMd9P2n6\nY6Btyi8oKQH6B5WcufuTwK3A9dH8R8C/gbMb2PxsEiWDvwJDo9JMRlG56ARgdFQrXxm938lm1iXa\n7B3CyeBkfYClScccZGY9yM4KoFdyXZ4QwssA3H0hIVSHEXr7f0za7h2gyt13S3q1c/fkv2bSnUx+\nB7gmZf8O7j4taZt9UqbrylMrCL9sADCz9oS/apYRfkHs18gxNRqnXLi7XnplfAFvASckzXcj1K+P\niuaPBT4k1K07ArsRyjzrgP2jbdoAzwGzgS8QOhtdgauAYQ0ccyKwANgj6dWdUEIaF21zAbA4ej8j\n/LWxEjgp6X1mRscdCLSK2vffwLcbOGbr6P0nRNOVwEbgwKRtrgD+TugFd0laPj5avk80vztwWjTd\nm/DXz05pPuPDCYE/KPpe2hNKVh2i9UsJI5x6EEpnTwM/i9Z9DVhFKMHsDNwCPBmt60j4ZXBxtK4j\nMChaVw3cm9SGjO3UqzhfsTdAr+J4pYZ9tOx/gQeT5o+Nwm4T8AHhROdBKft0Am6OQm0TsAT4JbBb\nA8dcBFzYwPLLgeeiaYuC+bXomAtSQzwK7WrgdcIvpKXAb4GejXyvBwE1wAbgFUL5Knl9L2Ab8HDK\ncgMuIfzy2Rh9b3Vh3DvaJ22IEk7cPkcoja0gnJhun/RvMCH6HtcDvwPaJu07NjrmWsKJ1r2T1vUn\n/JWzjvDL8Ipo+dXAPUnbZdVOvYrvZdE/cFpmVgVMAiqAO939+pT13QgndvYk9Jx+6e6/N7NewD2E\nHpkDv3X3X2U8oIhsx8zeAr7r7k9k3FgkRcaavZlVEIZiVRF6PCNTr4oExgEvuPthhD97b4wuGtkC\nXOLu/QkjCi5sYF8REWlm2ZygHQQscfelHoaTTQVGpGyzkvDnOdHXtR6Grb3n7vMB3P1Dwp/luY7G\nEBGRHZTNJds9qD/caxlwVMo2k4EnzGwF4eTPdqMyzKw3MIDoQhwRyY27p446EslaNj37bIZmXQXM\nd/e9gcOA26MLTgAwsw6E+5NcHPXwRUSkBWXTs19OGH1QpxfRmOMkxxCu5sPd34hOJH0BmBdd6j0d\n+D93fyj1zc1M43xFRJrA3bO+kV02Pft5QN/oRk5tgHMIw7qSLSbcgwQz604I+jejC1OmAAvdfVKa\nBuuVp9fVV18dextK6aXPU59lob5ylTHs3X0rYbTNHMLl39PcfZGZjTWzsdFm1wJHRHf/+ythDO86\nwrjrbwBfNbMXoldVzq0UEZEdktU9tT3cfnZ2yrI7kqbXAKc2sN/T6JYMIiKxUxCXmMrKyribUFL0\neeaPPst4ZXUFbbM2wMzjboOISLExMzyHE7SF8mg0EZGS5g4bN8KqVeHlDscd13LHV9iLiDTRp5/C\n6tWJAG/s9f774etnnyX2PeQQeOmllmurwl5EJAN3WLAAHnoI/vY3WLEihPeGDU1/z1Wr8te+bCjs\nRUQasG0bPPMMzJgRQv6NN3b8Pdu3hz32CK+99gq/RCzrqvuOUdiLiEQ++ST03B96CGbNytz73mkn\n2H33EN7duyeCPPXVvXvYrn37lvk+GqKwF5GytmEDPPJICPjZs+GjjxrerkMHGDYMRoyAAQNCiHfp\nEgK/GCjsRaTsLFsWeu4PPQR//zts3drwdnvsEcL99NPhhBOgbduWbWc+KexFpOhs2QKbN4fXxx8n\nptO9Pv4YNm0K4T53buPvfcABcMYZIeCPOgoqKlru+2pOCnsRKQi1taFG/u67oee9bFliuu7rmjUh\nuLdty++xjzwyhPvpp0O/fi130rQlKexFpNm5h7HmqeGdPL18eeixt4RWreCrXw3hftpp0LNnyxw3\nTgp7Eckrd3jrLXj+efjPfxJf167N3zEqKmCXXaBdu/A13St1m/33h6oq2G23/LWnGCjsRaTJamvD\n+PPUYG/qxUZdukCvXqGn3bNnYrrua/fuIbxbt87v91EOFPYikpVt2+D11+sH+wsvhPu9ZKNjR+jT\nZ/sATw73du2a93soZwp7EWnQ8uXhCtJnnoFnnw0B39gY9FS77QaHHx5eAweGr/vtV5onPouFwl5E\n2Lw59NTrgv2ZZ8JJ02x067Z9sO+7r4K90CjsRcqMe6iz1/Xan3kGXnyx8QuLknXvvn2w9+ypYC8G\nCnuRErdpEzz3XP1wX7Mm837t24fx54MHh4uLBg2Cvfdu/vZK81DYi5Qgd3jqKbjjDnjggfr3UW/M\nF78Ygr3u1b9/GI8upUH/lCIlZO1auOce+O1vYfHixrfbddf6wT5oUPmNOy83CnuRIucOTz+d6MV/\n+un22xx8cHgEXl249+1bPHdrlPxQ2IsUqXXrEr34RYu2X9+xI5x3HlxwQbglr5S3jL/bzazKzBab\n2etmNqGB9d3M7FEzm29mr5jZt7LdV0RyU9eLP//8cLL0kku2D/ojjoDJk8Oj8379awW9BObuja80\nqwBeBYYAy4G5wEh3X5S0TTWws7tPNLNu0fbdAc+0b7S/p2uDiIRe/L33hl78woXbr+/QIdGLHziw\n5dsnLc/McPesB71mKuMMApa4+9LozacCI4DkwF4JfCma7gSsdfetZnZ0FvuKSCM2bAjDJP/4R/jT\nn8Ij81IdfjiMHQvnnhvKNiKNyRT2PYB3k+aXAUelbDMZeMLMVgAdgbNz2FdECOWZ116Df/0L/v3v\n8HXhwrA8VYcOMGpU6MUffnjLt1WKU6awz6a+chUw390rzWx/4HEzOzSXRlRXV38+XVlZSWVlZS67\nixSdDz8MT0uqC/d//zuUatIZODD04keOVC++HNXU1FBTU9Pk/TPV7AcD1e5eFc1PBGrd/fqkbf4C\nXOPu/4zm/wZMIPwiSbtvtFw1eylp7rB0af1e+0svZX7aUkUFHHpoGDL5zW+qFy/15btmPw/oa2a9\ngRXAOcDIlG0WE07C/tPMugNfAN4ENmaxr0jJ2bgR5s0LPfdnnw3h/v77mffr2hWOPjq8jjkm3Kqg\nffvmb6+Uh7RhH51oHQfMASqAKe6+yMzGRuvvAK4FfmdmLxKGcl7h7usAGtq3+b4VkZb3ySfhJmJz\n54bXc8/Bq682XGtPZhYudKoL9qOPDhc66YZi0lzSlnFapAEq40iR2LYt3ILguecSwf7SS9k9N7VT\np/q99kGDoHPn5m+zlK58l3FEytY774Shj3W99uefDydWM6mogEMOCYF+5JEh4Pv10+0JJF4Ke5EU\nTz0F114Ljz6a3fZ9+4ZQrwv3ww7T4/Wk8CjsRQg19jlz4Jprwu0IGrPXXolQHzQo3JpAd4uUYqCw\nl7JWWwszZoSe/H/+U3+dGVRWhrtE1oV7jx6xNFNkhynspSxt2RJuQ/Dzn29/3/fWrcO49iuugAMP\njKd9IvmmsJeysnkz/O53cMMN8Pbb9dftsgt873tw2WXQq1c87RNpLgp7KQubNsFvfgM33rj9BU6d\nOsGFF8L48bDHHvG0T6S5KeylpK1dC7/6Fdx6K6xfX39dt27hfvA/+EF4TJ9IKVPYS0lasQJuuin0\n5j/6qP66Hj3g8sthzBjdjkDKh8JeSsLKlWF8/JNPhtfLL2+/zf77w5VXhqc87bxzy7dRJE4Keyk6\n7uHkal2wP/kkvP5649sfcghcdRWcdRa00k+8lCn96EvBcw83F0sO93ffTb9PRQUce2wYWXPKKbpV\ngYjCXgrOtm2hDJMc7qtXp99n553DxU/HHx9egweHJzqJSKCwl4Kxbh1Mngy33565596hQ+i514X7\nkUeqDi+SjsJeYrdwYRgeec894aKnhnTpAl/+ciLcDztM9XeRXOi/i8SithZmz4ZbboHHH99+fdeu\nMGRIItwPOkh1d5EdobCXFrVpE/z+9+Eip4ZG0Bx6KFx8cXiodtu2Ld48kZKlsJcW8eabcNttMGVK\neEZrsp12ghEjQsgff7wezSfSHBT20mzcoaYmlGpmzdr+uaydO8N3vwvjxkGfPrE0UaRsKOwl7z75\nJNw++JZbwjNaUx14IFx0EYwereGRIi1FYS95NXlyuFp1zZrt1w0dGko1Q4fqZKtIS1PYS164w09+\nEp74lKxdu/AgkIsuCg/dFpF4KOxlh23dCv/93+Hka5199gm1+DFj9IxWkUKQMezNrAqYBFQAd7r7\n9SnrLwPOS3q/fkA3d99gZhOBbwC1wMvAt9390zy2X2K2eTOce244AVvnlFPg/vtDr15ECoN56hCJ\n5JVmFcCrwBBgOTAXGOnuixrZfjgw3t2HmFlv4Amgn7t/ambTgL+4+90p+3i6NkjhWr8eTjsNnn46\nsWz06FC3b906vnaJlAMzw92zHqic6TTZIGCJuy919y3AVGBEmu1HAfdF0xuBLUA7M2sFtCP8wpAS\nsGJFGBOfHPRXXBGe76qgFyk8mcK+B5B8S6pl0bLtmFk7YCgwHcDd1wE3Au8AK4AN7v7XHW2wxO/V\nV+GYY+CVVxLLbrwRrr9eF0SJFKpMNftc6iunAk+7+wYAM9sfGA/0Bj4A/mRm57n7H1J3rK6u/ny6\nsrKSysrKHA4rLWnuXDj55MTQylatQm/+G9+It10ipa6mpoaampom75+pZj8YqHb3qmh+IlCbepI2\nWjcDmObuU6P5c4AT3X1MNH8+MNjdL0zZTzX7IvHYY3DmmYlnurZrBw88AMOGxdsukXKU75r9PKCv\nmfU2szbAOcCs1I3MrDNwPDAzafFiYLCZ7WJmRjjJuzDbhklhue++MMqmLui7dIEnnlDQixSLtGHv\n7luBccAcQlBPc/dFZjbWzMYmbXo6MMfdNyft+yJwD+EXRt1F87/NZ+OlZdxyC4waFcbTA/TqFU7M\nHnVUvO0SkeylLeO0SANUxilY7vDjH8N11yWWHXQQzJkDPXvG1y4Ryb2MoytopUFbt8LYsXDXXYll\nxxwDDz8cSjgiUlwU9rKdhq6KHT4cpk3TVbEixUr3HpR61q+Hk06qH/Tf+hY8+KCCXqSYqWdfptzh\ngw9g+fJwNWzd1z/+ERYsSGw3YUKo2etiKZHiprAvQZ98AitXhgBPDfO6rytWwMcfp3+fm26CSy5p\nmTaLSPNS2JeIJUtg/Hh45hlYu3bH3ktXxYqUHoV9CZg9O4yD37Aht/3atYMePcJr770TX085Bfr2\nbZ62ikg8FPZFrLY21NN/+tP6D/OuqIC99to+xFO/duqkWrxIuVDYF6lNm8K942fMSCzr2TMMjzzq\nqBD4IiJ1FPZF6NVX4YwzYFHSI2S+8pXwdKg99oivXSJSuDTOvsjMmgWDBtUP+osvhscfV9CLSOMU\n9kWithaqq2HECNi4MSxr2xbuvRcmTdLToUQkPZVxisAHH4RhkH/+c2LZvvuGev2AAfG1S0SKh8K+\nwC1cGOrzr72WWPa1r8HUqdCtW3ztEpHiojJOAXvwwTCyJjnoL78cHn1UQS8iuVHYF6Bt2+AnP4Gv\nfx0+/DAsa9cuPC3qhhvCFa4iIrlQbBSY9evhvPPCVbF19tsv1Oe/9KX42iUixU1hX0BeeQVOPx3e\neCOxbOjQcCdKPTBERHaEyjgFYto0GDy4ftBfdRU88oiCXkR2nHr2MVuzBsaNC2Ffp317uPvuULMX\nEckHhX2Mpk+H738fVq9OLOvbN9Tn+/ePr10iUnpUxonB6tVwzjlw1ln1g/7b34a5cxX0IpJ/6tm3\nsAcegB/8oH7I9+gBkyfDsGHxtUtESlvGnr2ZVZnZYjN73cwmNLD+MjN7IXq9bGZbzWzXaN2uZvaA\nmS0ys4VmNrg5volisHo1nH02/Nd/1Q/673wnjMJR0ItIczJPfupF6kqzCuBVYAiwHJgLjHT3RY1s\nPxwY7+5Dovm7gX+4+11m1gpo7+4fpOzj6dpQCv70p9CbX7Mmsaxnz9Cbr6qKr10iUrzMDHfP+vFD\nmXr2g4Al7r7U3bcAU4ERabYfBdwXNaQz8GV3vwvA3bemBn2pW7Uq9OTPPrt+0H/3u6E3r6AXkZaS\nKex7AO8mzS+Llm3HzNoBQ4Hp0aI+wGoz+52Z/cfMJkfblDz38CCR/v1Djb5Oz57hvjZ33gmdO8fX\nPhEpP5lO0OZSXzkVeNrd6x573QoYCIxz97lmNgm4Evif1B2rq6s/n66srKSysjKHwxaW99+HCy8M\nwyqTjRkDv/ylQl5Emqampoaampom75+pZj8YqHb3qmh+IlDr7tc3sO0MYJq7T43m9wT+7e59ovnj\ngCvdfXjKfiVRs3cPF0aNGwdr1yaW9+oVevInnRRf20Sk9OS7Zj8P6Gtmvc2sDXAOMKuBg3YGjgdm\n1i1z9/eAd83swGjREGBBtg0rJqtWhTHzI0fWD/oLLgi1eQW9iMQtbRnH3bea2ThgDlABTHH3RWY2\nNlp/R7Tp6cAcd9+c8hY/BP4Q/aJ4A/h2XltfALZsCWH+4ouJZb16wZQpcOKJ8bVLRCRZ2jJOizSg\nyMs4N98MP/pRYv6CC+AXv4BOneJrk4iUvlzLOAr7HfDee3DggbBpU5i/7jq48sp42yQi5SHfNXtJ\nY8KERNB/8Yv1e/giIoVEPfsm+te/4NhjE/OPPaYavYi0HPXsW8C2bWGIZZ0zz1TQi0hhU9g3weTJ\n8MILYbptW7jppnjbIyKSicI+R2vXwo9/nJi/6irYd9/42iMikg2FfY5+/GNYty5M77cfXH55vO0R\nEcmGTtDm4Pnn4cgjw60RAGbNglNPjbdNIlKedIK2mdTWhpOydUF/8skwfHj6fURECoXCPkv33APP\nPBOm27SBSZPAsv6dKiISL4V9Fj74IFxAVefSS6Fv3/jaIyKSK4V9Fqqrw50tITyAJHk0johIMdAJ\n2gxeeQUOOyxcSAXhnvVnnx1vm0REdII2j9zhhz9MBP1XvxqeKSsiUmwU9mncfz/UPQWsogJuvVUn\nZUWkOCnsG/Hhh+FEbJ2LLgoPEBcRKUYK+0Zccw0sXx6mu3eHq6+Otz0iIjtCJ2gb8NprcPDB4ZGD\nAL//PYweHWuTRETq0QnaHeQOF1+cCPqjj4bzz4+3TSIiO0phn+Lhh+HRR8O0Gdx2G+ykT0lEipxi\nLMnmzTB+fGJ+7FgYODC+9oiI5IvCPskvfgFvvRWmu3SBn/0s3vaIiOSLwj6ydClcd11i/tproWvX\n2JojIpJXGcPezKrMbLGZvW5mExpYf5mZvRC9XjazrWa2a9L6imjdw/lufD796EfwySdheuBAGDMm\n3vaIiORT2qGXZlYBvAoMAZYDc4GR7r6oke2HA+PdfUjSsh8BhwMd3f20BvaJfejlnDlQVZWY/9e/\nwigcEZFCle+hl4OAJe6+1N23AFOBEWm2HwXcl9SYnsDJwJ1AQd5o4LPPwtWxdUaPVtCLSOnJFPY9\ngHeT5pdFy7ZjZu2AocD0pMU3A5cDtTvQxmZ1++3hIiqATp3g5z+Ptz0iIs2hVYb1udRXTgWedvcN\n8HlJZ5W7v2Bmlel2rK6u/ny6srKSysq0m+fVnXcmtwP23LPFDi0ikrWamhpq6u7M2ASZavaDgWp3\nr4rmJwK17n59A9vOAKa5+9Ro/lrgfGAr0BboBEx392+m7Bdbzf611+ALXwjTbdvCmjXQvn0sTRER\nyUm+a/bzgL5m1tvM2gDnALMaOGhn4HhgZt0yd7/K3Xu5ex/gXOCJ1KCP24wZiemhQxX0IlK60pZx\n3H2rmY0D5gAVwBR3X2RmY6P1d0Sbng7McffN6d4uHw3Op+SwP+OM+NohItLcyvaul8uXh+fJQngw\nyapV4apZEZFioLteZmnmzMR0ZaWCXkRKW9mGvUo4IlJOyrKMs24d7LFH4kHiy5ZBjwavHhARKUwq\n42Thz39OBP2gQQp6ESl9ZRn2KuGISLkpuzLOxx9Dt27hQSUAixcnLqwSESkWKuNkMGdOIuj79VPQ\ni0h5KLuwVwlHRMpRWZVxtmwJo3A2bAjzc+fCEUe0yKFFRPJKZZw0/vGPRND36gWHHx5ve0REWkpZ\nhX1qCccK8nEqIiL5VzZhX1sLDz2UmFe9XkTKSdmE/dy5sGJFmO7aFY47Lt72iIi0pLIJ+wcfTEyf\ndhq0yvSMLhGRElIWYe+uIZciUt7KYujlggVw8MFhun378PjBtm2b9ZAiIs1KQy8bkNyrHzZMQS8i\n5afswl4lHBEpRyVfxnn7bejdO0y3bg2rV0Pnzs12OBGRFqEyTorksfUnnKCgF5HyVPJhn1zCOfPM\n+NohIhKnki7jrF4Ne+4Zrp41g5UroXv3ZjmUiEiLUhknycMPh6AHOOYYBb2IlK+swt7MqsxssZm9\nbmYTGlh/mZm9EL1eNrOtZrarmfUys7+b2QIze8XMLsr/t9C45KtmNQpHRMpZxjKOmVUArwJDgOXA\nXGCkuy9qZPvhwHh3H2JmewJ7uvt8M+sAPA+cnrxvc5VxNm0Kjx/87LMw/8YbsN9+eT+MiEgsmqOM\nMwhY4u5L3X0LMBUYkWb7UcB9AO7+nrvPj6Y/BBYBe2fbuB0xe3Yi6L/0JQW9iJS3bMK+B/Bu0vyy\naNl2zKwdMBSY3sC63sAA4NlcG9kUupBKRCQhm3s/5lJjORV42t03JC+MSjgPABdHPfx6qqurP5+u\nrKyksrIyh0Nu79NP4ZFHEvMKexEpdjU1NdTU1DR5/2xq9oOBaneviuYnArXufn0D284Aprn71KRl\nrYE/A7PdfVID++S9Zj97Npx8cpju0yfU6/VUKhEpJc1Rs58H9DWz3mbWBjgHmNXAgTsDxwMzk5YZ\nMAVY2FDQNxc9flBEpL6MYe/uW4FxwBxgIaHnvsjMxprZ2KRNTwfmuPvmpGXHAt8Avpo0NLMqj+3f\nzrZtMHNmYl4lHBGREryC9umn4ctfDtPdu8Py5VBRkbe3FxEpCGV/BW1yCWfECAW9iAiUWNjr8YMi\nIg0rqTLO/PkwYECY7tQp3AitTZu8vLWISEEp6zJOcq/+lFMU9CIidUo27FXCERFJKJkyzhtvwAEH\nhOmdd4Y1a6BDhx1+WxGRglS2ZZzkXv2JJyroRUSSlWTYq4QjIlJfSZRx3nsP9t47DL3caacwv/vu\neWqgiEgBKssyzsyZIegBjj9eQS8ikqokwl4lHBGR9Iq+jPPBB6Env2VLmH/7bdhnnzw1TkSkQJVd\nGeeRRxJBf/jhCnoRkYYUfdg/+GBiWiUcEZGGFXUZZ/Nm6NYNPv44zC9YAAcdlMfGiYgUqLIq48yZ\nkwj6Aw+Efv3ibY+ISKEq6rC/5ZbE9Ne/rscPiog0pmjLOPPmwZFHhumKCnjzTZ2cFZHyUTZlnF/8\nIjF97rkKehGRdIqyZ//mm9C3L9TWhvn58+HQQ5uhcSIiBaosevY335wI+hNPVNCLiGRSdD37NWtC\nyWbz5jD/2GMh8EVEyknJ9+z/938TQX/YYTBkSLztEREpBhnD3syqzGyxmb1uZhMaWH+Zmb0QvV42\ns61mtms2++Zq82a47bbE/GWXabiliEg20pZxzKwCeBUYAiwH5gIj3X1RI9sPB8a7+5Bs982ljPOb\n38D3vx+m99kHliyB1q2z2lVEpKTku4wzCFji7kvdfQswFRiRZvtRwH1N3DetbdvgxhsT85dcoqAX\nEclWprD9Fp17AAAGlUlEQVTvAbybNL8sWrYdM2sHDAWm57pvNmbODD15gF13hTFjmvpOIiLlp1WG\n9bkM1TkVeNrdN+S6b3V19efTlZWVVFZW1m+Eww03JOa//309UFxEyktNTQ01NTVN3j9TzX4wUO3u\nVdH8RKDW3a9vYNsZwDR3n5rLvtnU7J96KjxuEKBNG1i6FPbaK9tvUUSk9OS7Zj8P6Gtmvc2sDXAO\nMKuBg3YGjgdm5rpvNpJvjXD++Qp6EZFcpS3juPtWMxsHzAEqgCnuvsjMxkbr74g2PR2Y4+6bM+2b\nawMXLYKHH07MX3ppru8gIiIFfwXtmDEwZUqYPu20cKJWRKTc5VrGKeiwX7kSeveGzz4L8089Bccd\n13JtExEpVCV1u4Rbb00E/eDBcOyx8bZHRKRYFWzPftOmcJXshmgg5/TpcOaZLdw4EZECVTI9+zvv\nTAT9AQfAiCZfeysiIgUZ9lu2hHvW17n00vDoQRERaZqCDPv774d3oxst7L47jB4db3tERIpdwYW9\ne/2LqMaNg112ia89IiKloOBO0D7+OJx0Uphu1w7eeQe6do2pcSIiBaroT9Am9+q/8x0FvYhIPhRU\nz37+fBgwICzfaadwS+M+fWJsnIhIgSrqnn1yr/6ssxT0IiL5UjA9+7ffhv33D0+kApg7F444Itam\niYgUrKLt2U+alAj6ykoFvYhIPhVEz37dOqdXL/joo7DskUfg5JNjbZaISEEryp79b36TCPr+/WHY\nsHjbIyJSagoi7H/1q8T05ZeDZf27SkREslEQZZy6Z5P36AFvvhmeMysiIo0ryjJOnYsvVtCLiDSH\ngunZd+wYbn7WuXOszRERKQpF27MfO1ZBLyLSXAqiZ9+qlfPWW9CzZ6xNEREpGkXZsx81SkEvItKc\nMoa9mVWZ2WIze93MJjSyTaWZvWBmr5hZTdLyiWa2wMxeNrM/mtnODe1/xRVNbr+IiGQhbdibWQVw\nG1AFHASMNLN+KdvsCtwOnOruBwNnRct7A98DBrr7IUAFcG5Dx+nff4e+B0lSU1MTdxNKij7P/NFn\nGa9MPftBwBJ3X+ruW4CpQOqjv0cB0919GYC7r4mWbwS2AO3MrBXQDliet5ZLg/QfKr/0eeaPPst4\nZQr7HsC7SfPLomXJ+gJdzOzvZjbPzM4HcPd1wI3AO8AKYIO7/zU/zRYRkVxkCvtshuq0BgYCJwND\ngZ+aWV8z2x8YD/QG9gY6mNl5O9BWERFporRDL81sMFDt7lXR/ESg1t2vT9pmArCLu1dH83cCjxJ+\nkZzk7mOi5ecDg939wpRjxDv2U0SkSOUy9LJVhvXzgL7RydYVwDnAyJRtZgK3RSdzdwaOAm4i9Pj/\nx8x2AT4BhgDP7UhjRUSkadKGvbtvNbNxwBzCaJop7r7IzMZG6+9w98Vm9ijwElALTHb3hQBmdg/h\nF0Yt8B/gt833rYiISGNiv4JWRESaX6xX0GZzwZZkz8yWmtlL0QVu25XMpHFmdpeZvW9mLyct62Jm\nj5vZa2b2WHRNiWShkc+z2syWRT+fL5hZVZxtLCZm1isa8bggunj1omh51j+jsYV9NhdsSc4cqHT3\nAe4+KO7GFJnfEX4Wk10JPO7uBwJ/i+YlOw19ng7cFP18DnD3R2NoV7HaAlzi7v2BwcCFUV5m/TMa\nZ88+mwu2JHc64d0E7v4UsD5l8WnA3dH03cDpLdqoItbI5wn6+WwSd3/P3edH0x8CiwjXPGX9Mxpn\n2GdzwZbkxoG/Rhe3fS/uxpSA7u7+fjT9PtA9zsaUiB+a2YtmNkVlsaaJRkcOAJ4lh5/ROMNeZ4bz\n71h3HwAMI/yZ9+W4G1QqPIxk0M/sjvk10Ac4DFhJuMJecmBmHYDpwMXuvil5Xaaf0TjDfjnQK2m+\nF6F3L03k7iujr6uBGYRSmTTd+2a2J4CZ7QWsirk9Rc3dV3kEuBP9fObEzFoTgv5ed38oWpz1z2ic\nYf/5BVtm1oZwwdasGNtT1MysnZl1jKbbAycBL6ffSzKYBYyOpkcDD6XZVjKIwqjOGejnM2tmZsAU\nYKG7T0palfXPaKzj7M1sGDCJxAVb18XWmCJnZn0IvXkIF8v9QZ9n9szsPuArQDdC7fN/CFeH3w/s\nAywFznb3DXG1sZg08HleDVQSSjgOvAWMTao3SxpmdhzwJOHi1brQnki4K0FWP6O6qEpEpAwUxGMJ\nRUSkeSnsRUTKgMJeRKQMKOxFRMqAwl5EpAwo7EVEyoDCXkSkDCjsRUTKwP8HwVOZ9bCVaiEAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe4a734db50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(basic_NN_aucs.aucs, lw=3)\n",
    "_ = plt.title(\"ROC AUC over epoch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So far so good ...\n",
    "\n",
    "A few obvious ideas to improve performance:\n",
    "\n",
    "- batch normalization\n",
    "- dropout\n",
    "- make it deeper\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch normalization\n",
    "\n",
    "- Consider a neuron in any layer beyond first\n",
    "- At each iteration we tune it's weights towards better loss function\n",
    "- But we also tune it's inputs. Some of them become larger, some – smaller\n",
    "- Now the neuron needs to be re-tuned for it's new inputs\n",
    "\n",
    "Idea: _It's usually a good idea to normalize linear model inputs_\n",
    "- We normalize activation of a hidden layer\n",
    "(zero mean unit variance)\n",
    "- Now all neurons have ~same output range\n",
    "\n",
    "<small>Ioffe &amp; Szegedi, 2015 Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (http://arxiv.org/abs/1502.03167)</small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to BatchNorm in Keras\n",
    "\n",
    "```python\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "\n",
    "BatchNormalization(epsilon=1e-06, mode=0, \n",
    "                   axis=-1, momentum=0.99, \n",
    "                   weights=None, beta_init='zero', \n",
    "                   gamma_init='one')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The **dropout layers** have the very specific function to *drop out* a random set of activations in that layers by setting them to zero in the forward pass. Simple as that. \n",
    "\n",
    "It allows to avoid *overfitting* but has to be used **only** at training time and **not** at test time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img  src=\"dropout.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hinton et al., 2012 Improving neural networks by preventing co-adaptation of feature detectors (http://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def build_model_dropout_batch(n_features=28,num_layers=1, size=100, lr=1e-3, keep_prob=1.0,nonlinearity='relu'):\n",
    "    model = Sequential()\n",
    "    model.add(InputLayer([n_features]))\n",
    "    \n",
    "    for i in range(num_layers): \n",
    "        model.add(Dense(size)) #fully-connected linear\n",
    "        model.add(BatchNormalization()) #batchnorm\n",
    "        model.add(Activation(nonlinearity)) #nonlinearity\n",
    "        if keep_prob<1.0:\n",
    "            model.add(Dropout(1.-keep_prob)) #dropout (if any)\n",
    "    \n",
    "    \n",
    "    #a fully-connected layer with sigmoid output - this layer will be interpreted as NN output\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "    \n",
    "    #logloss, see https://www.kaggle.com/wiki/LogarithmicLoss\n",
    "    loss = keras.objectives.binary_crossentropy\n",
    "\n",
    "    # optimizer that will train the NN to minimize loss above. \n",
    "    # may or may not be suboptimal choice. See dir(keras.optimizers)\n",
    "    optimizer = keras.optimizers.adam(lr=lr)\n",
    "\n",
    "    model.compile(loss=loss, optimizer=optimizer,\n",
    "                  metrics=[keras.metrics.binary_accuracy], #also monitor accuracy\n",
    "                      \n",
    "                 )\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "model = build_model_dropout_batch(n_features=n_features,num_layers=6,size=600,keep_prob=0.9)\n",
    "dropout_batch_aucs = AucHistories()\n",
    "dropout_batch_histories = model.fit(X_train, y_train, nb_epoch=int(N_EPOCH*1.5), batch_size=BATCH_SIZE, \n",
    "          validation_data=[X_test,y_test], callbacks=[dropout_batch_aucs], verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot(dropout_batch_histories.history['loss'], label=\"loss\")\n",
    "plt.plot(dropout_batch_histories.history['val_loss'], label=\"val_loss\")\n",
    "plt.legend()\n",
    "_ = plt.title(\"loss over epoch\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot(dropout_batch_aucs.aucs, lw=3)\n",
    "_ = plt.title(\"ROC AUC over epoch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  further steps\n",
    "  - Increased learning rate (with batch normalization, on ImageNet authors were able to use 30x the original learning rate)\n",
    "  - Decreased learning rate towards the end of the training - fine tuning\n",
    "  - Averaging parameters of the models (between current and e.g. previous epoch or a few previous iterations) - this helps on image models\n",
    "  - Averaging predictions - this almost always helps. If you optimize likelihood of some distributions, replacing predictions with an average of a few models is still an estimator but has typically lower variance\n",
    "  - Reducing batch size and training for more time once we find good parameters (potentially also increasing dropout rate)\n",
    "  - Or ... any tuning of the parameters - we didn't do almost any!\n",
    "  - Different optimizer/params to train faster/more stable\n",
    "  - Different nonlinearities for hidden layers\n",
    "  - deeper networks"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}