Rapport Projet.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import time # pour comparer les algorithmes en fonction de leur durée d'éxecution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Etant donné que nous sommes un groupe constitué de trois personnes, nous vous proposons le sujet 1 enrichi du sujet 3. Concrètement, nous allons implémenter 4 méthodes d'optimisations, puis les comparer sur la base de données \"titanic3.xls\" que nous décrirons ultérieurement  :\n",
    "- Batch Proximal Gradient Descent (avec accéleration Nesterov, aka FISTA)\n",
    "- SDCA\n",
    "- SVRG\n",
    "- SAGA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# SDCA sur SVM avec régularisation LASSO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Nous allons en premier lieu nous concentrer sur l'algorithme SVM avec régularisation LASSO. \n",
    "\n",
    "Commençons par définir les notations afin de fixer les idées :\n",
    "- les données seront notées par la matrice $X$ de taille (n_samples, n_features)\n",
    "- le paramètre à estimer est $\\theta$ de taille n_features\n",
    "- les valeurs à prédire sont dans le vecteur $b$ de taille n_samples\n",
    "\n",
    "Afin de pouvoir fixer de manière optimale les pas de descente, nous allons utiliser une version smooth de la hinge loss : la square hinge loss (smoothness $\\gamma = \\| X \\|_2$), mais qui n'est pas strongly convex. On s'attend donc à avoir une convergence en $\\frac{\\gamma}{t} $.\n",
    "\n",
    "$$ l(b,X^T \\theta) = \\frac{1}{n\\_samples} \\sum_{i=1}^{n\\_samples} max(0, 1 - b_i X_i^T \\theta)^2 $$\n",
    "\n",
    "Le gradient s'écrit\n",
    "\n",
    "$$ \\nabla_\\theta l(b, X^T \\theta) = \\frac{1}{n\\_samples}  \\sum_{i=1}^{n\\_samples} \\left\\{\n",
    "    \\begin{array}{ll}\n",
    "        0 & \\mbox{si } b_i X_i^T \\theta > 1 \\\\\n",
    "        -2 b_i X_i^T (1 - b_i X_i^T \\theta) & \\mbox{sinon.}\n",
    "    \\end{array}\n",
    "\\right. $$\n",
    "\n",
    "Enfin, la perte considéré étant le LASSO, ie la régularisation l1, le fontion $\\psi$ considérée s'écrit ici :\n",
    "$$ \\psi (\\theta) = \\lambda |\\theta|_1 $$ \n",
    "\n",
    "Le prox se calcule donc de cette manière :\n",
    "\n",
    "$$ \\theta_{t+1} = \\theta_t - \\eta \\nabla_\\theta l(b, X^T \\theta_t) - \\eta \\lambda \\text{sgn}(\\theta_t) $$\n",
    "(en principe il faudrait utiliser sgn($\\theta$) mais nous ferons une approximation...)\n",
    "\n",
    "Enfin, au cours de ce notebook, nous ne lancerons pas les différents algorithmes sur un nombre d'itérations fixé à l'avance, mais sur un budget 'temps' fixé à l'avance (par exemple on donnera 5 secondes à tous les algorithmes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "*[Partie 0 : Définition des données](#section0)\n",
    "\n",
    "*[Partie 1 : Batch Proximal Gradient Descent](#section1)\n",
    "\n",
    "*[Partie 2 : FISTA](#section2)\n",
    "\n",
    "*[Partie 3 : SVRG](#section3)\n",
    "\n",
    "*[Partie 4 : SAGA](#section4)\n",
    "\n",
    "*[Partie 5 : Primal Coordinate descent](#section5)\n",
    "\n",
    "*[Partie 6 : SDCA](#section6)\n",
    "\n",
    "*[Partie 7 : Comparaison](#section7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id='section0'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Définition des données"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Commençons par définir les données de test qui seront plus tard remplacées par les vraies données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "\n",
    "def binarize(x):\n",
    "    return np.sign(x)\n",
    "\n",
    "n_samples, n_features = 2000, 40\n",
    "X = np.random.randn(n_samples, n_features)\n",
    "w = np.random.randn(n_features)\n",
    "b = binarize(X.dot(w) + 0.1 * np.random.randn(n_samples))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Nous utiliserons par la suite le dataset Titanic. L'idée est qu'à partir des caractéristiques d'un passager, on cherche à prédire si celui-ci survit ou pas. \n",
    "\n",
    "Afin de tester l'algorithme dans plusieurs conditions, on va s'autoriser à dupliquer les données en lignes ou en colonnes (on rajoute des passagers fictifs ou des variables supplémentaires). Ainsi, on ne change pas la 'difficulté intrinsèque' du problème (l'optimum ne bouge pas !) mais on va pouvoir observer comment se comportent les algorithmes lorsqu'on scale. Voici les 4 cas considérés :\n",
    "- n_samples petit, n_features petit (c'est le cas de base : 1045 individus, 6 dimensions)\n",
    "- n_samples grand, n_features petit (on fait n_samples * 100)\n",
    "- n_samples petit, n_features grand (on fait n_features * 100)\n",
    "- n_samples grand, n_features grand (on fait n_samples \\* 100, n_features * 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# on importe la bdd initiale\n",
    "data = pd.read_excel('titanic3.xls')\n",
    "\n",
    "data = data[['pclass','sex','age','sibsp','parch','fare','survived']]\n",
    "data = data.dropna()\n",
    "X = data[['pclass','sex','age','sibsp','parch','fare']]\n",
    "X = pd.get_dummies(X)\n",
    "X = X[['pclass','sex_female','age','sibsp','parch','fare']]\n",
    "b=data.survived\n",
    "\n",
    "X = X.values\n",
    "b = b.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# on scale les données\n",
    "def grand_petit(X,b):    \n",
    "    X = np.concatenate([X for i in range(100)], axis=0)\n",
    "    b = np.concatenate([b for i in range(100)], axis=0)\n",
    "    return X,b\n",
    "    \n",
    "def petit_grand(X,b):\n",
    "    X = np.concatenate([X for i in range(100)], axis=1)\n",
    "    return X,b\n",
    "    \n",
    "def grand_grand(X,b):\n",
    "    X,b = petit_grand(X,b)\n",
    "    X,b = grand_petit(X,b)\n",
    "    return X,b\n",
    " \n",
    "# Ici il faut choisir dans quel cas on se met, on laisse tout en commentaire si on veut rester dans le cas de base 'petit_petit'\n",
    "#X,b = grand_petit(X,b)\n",
    "X,b = petit_grand(X,b)\n",
    "#X,b = grand_grand(X,b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1045, 600)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "n_samples, n_features = X.shape\n",
    "\n",
    "import sklearn.preprocessing\n",
    "X = sklearn.preprocessing.normalize(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "On définit aussi les fonctions de couts et les gradients (partiels ou non) pour la fonction hinge et la squared hinge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# On commence avec la hinge normale\n",
    "def f_hinge(theta, lam):\n",
    "    cost = 1-np.multiply(b,X.dot(theta))\n",
    "    cost[cost <= 0] = 0\n",
    "    return np.mean(cost) + lam * np.linalg.norm(theta, ord=1)\n",
    "\n",
    "def partial_grad_hinge(theta, i):\n",
    "    if b[i] * X[i].dot(theta) < 1:\n",
    "        return -b[i] * X[i]\n",
    "    return 0 * X[i] \n",
    "\n",
    "def grad_hinge(theta):\n",
    "    gradient = -np.multiply(b, X.T)\n",
    "    gradient[:,np.multiply(b, X.dot(theta)) >= 1] = 0 * gradient[:,np.multiply(b, X.dot(theta)) >= 1]\n",
    "    return np.mean(gradient, axis=1)\n",
    "\n",
    "\n",
    "# On continue avec la squared hinge\n",
    "def f(theta, lam):\n",
    "    cost = (1-np.multiply(b,X.dot(theta))) ** 2\n",
    "    cost[cost <= 0] = 0\n",
    "    return np.mean(cost) + lam * np.linalg.norm(theta, ord=1)\n",
    "\n",
    "def partial_grad(theta, i):\n",
    "    if b[i] * X[i].dot(theta) < 1:\n",
    "        return -2 * b[i] * X[i]* (1 - b[i] * X[i].dot(theta))\n",
    "    return 0 * X[i] \n",
    "\n",
    "def partial_coordinate_grad(theta, i):\n",
    "    gradient = np.multiply( np.multiply(-2*b, X[:,i].T) , (1 - np.multiply(b, X[:,i]*theta[i])) )\n",
    "    gradient[np.multiply(b, X[:,i]*theta[i]) >= 1] = 0 * gradient[np.multiply(b, X[:,i]*theta[i]) >= 1]  \n",
    "    return np.mean(gradient)\n",
    "\n",
    "def grad(theta):\n",
    "    gradient = np.multiply( np.multiply(-2*b, X.T) , (1 - np.multiply(b, X.dot(theta))) )\n",
    "    gradient[:,np.multiply(b, X.dot(theta)) >= 1] = 0 * gradient[:,np.multiply(b, X.dot(theta)) >= 1]   \n",
    "    return np.mean(gradient, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "et les différentes constantes :\n",
    "- pénalisation : lambda = $\\frac{1}{\\text(n\\_features)}$\n",
    "- pas de descente : eta = $\\frac{1}{\\|X\\|_2}$  ($\\frac{1}{\\gamma}$, avec $\\gamma=\\|X\\|_2$)\n",
    "- nombre d'itérations : T = 300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "lam = 1./n_features\n",
    "eta = 1./(np.linalg.norm(X))\n",
    "T = 300"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id='section1'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Batch Proximal Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Commençons avec l'algorithme le plus simple : la descente de gradient proximale. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def PGD(T, lam, eta):\n",
    "    # on déclenche le timer\n",
    "    t0 = time.time()\n",
    "    \n",
    "    # initialisation\n",
    "    theta = np.zeros(n_features)\n",
    "    \n",
    "    # historique des fonctions de perte\n",
    "    cost_history = np.empty(shape=[T])\n",
    "    time_history = np.empty(shape=[T,2])\n",
    "    \n",
    "    for t in range(0,T):\n",
    "        cost_history[t] = f(theta, lam)\n",
    "        time_history[t] = (time.time()-t0, f(theta, lam))\n",
    "        \n",
    "        # compute gradient\n",
    "        gt = grad(theta)\n",
    "        \n",
    "        # compute prox\n",
    "        theta = theta - eta * gt - eta * lam * np.sign(theta)\n",
    "               \n",
    "    return theta, cost_history, time_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "theta, cost_history_PGD, time_history_PGD = PGD(T=T,lam=lam,eta=eta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmY3VWd5/H3p7YslSJ7CsgeCEvYApSANqOFqB3oaYO2\nC2ir49idpkfUnnnGZ2idme4Ze2bscbDHBSfGXbuVsUdB1DSrFqDNkoAhK4GiCFkIWUhIUpVKavvO\nH/eX5N5bVambSv3q3rr1eT1PPXXP+Z1z7/fkEr455/wWRQRmZmYDqSh2AGZmNjI4YZiZWUGcMMzM\nrCBOGGZmVhAnDDMzK4gThpmZFcQJw8zMCuKEYWZmBXHCMDOzglQVO4ChNG3atJg3b96g+ra1tVFb\nWzu0ARWJx1J6ymUc4LGUqsGO5emnn94bEdMLaVtWCWPevHmsXr16UH2bmppobGwc2oCKxGMpPeUy\nDvBYStVgxyLp5ULbeknKzMwK4oRhZmYFccIwM7OCOGGYmVlBnDDMzKwgqSUMSd+WtFvS+n6OS9KX\nJTVLWivpiqxjSyRtTo7dnlaMZmZWuDRPq/0u8FXg+/0cvwFYmPxcDfwf4GpJlcCdwNuB7cAqSfdG\nxMahDvB3W/fzwye3suP1dpp3HubGgxv463deNNQfY2ZWFlJLGBHxqKR5J2myFPh+ZJ4R+4SkSZLO\nAuYBzRHRAiDprqTtkCeM3YeO8o9Pbz9efmlv21B/hJlZ2SjmhXszgW1Z5e1JXV/1V/f3JpKWAcsA\n6uvraWpqKjiAVw9255Rf2LH3lPqXqtbW1rIYB5TPWMplHOCxlKrhGMuIv9I7IlYAKwAaGhriVK50\n3N/WwV/984Mnyh0VvOUtb0HSUIc5rHz1aukpl3GAx1KqhmMsxTxLagcwO6s8K6nrr37ITRpfzfia\nyuPl9s5u9h/uTOOjzMxGvGImjHuBDydnS10DHIiIncAqYKGk+ZJqgJuTtkNOEjMnjcup27G/PY2P\nMjMb8VJbkpL0I6ARmCZpO/BXQDVARCwHVgI3As3AYeCjybEuSbcB9wOVwLcjYkNacc6cPI4Xdrce\nL+94vZ1LZk1M6+PMzEasNM+SumWA4wF8vJ9jK8kklNSdnT/DeN0zDDOzvoz6K729JGVmVphRnzBm\nTc6fYRwuUiRmZqVt1CeMXjMML0mZmfVp1CeMWZPH55S37XPCMDPry6hPGDPqxlBTdeKP4UB7JweP\n+FoMM7N8oz5hVFSIWXnLUtv2eR/DzCzfqE8YALOmeFnKzGwgThjA7Lwzpbbv9wzDzCyfEwYwO2+G\nsdVLUmZmvThhALN7nSnlhGFmls8JA5jjGYaZ2YCcMOidMLbtb6enJ4oUjZlZaXLCACaOr6a2+kS5\no6uHVw8eKV5AZmYlyAkjMWN87h/Fy695WcrMLJsTRmLGuNzHsm7d11akSMzMSpMTRmJGbe4fxRbP\nMMzMcjhhJHrNMJwwzMxypJowJC2RtFlSs6Tb+zg+WdLdktZKekrSxVnHtkhaJ2mNpNVpxgl97GF4\nScrMLEeaz/SuBO4E3g5sB1ZJujciNmY1+wywJiLeJemCpP31Wcevi4i9acWYrX587gzj5b2HiQgk\n9dPDzGx0SXOGcRXQHBEtEdEB3AUszWuzCPgVQEQ8B8yTVJ9iTP2aOEaMq648Xj50tIv9h32bczOz\nY1KbYQAzgW1Z5e3A1XltngXeDTwm6SpgLjAL2AUE8JCkbuDrEbGirw+RtAxYBlBfX09TU9Oggm1r\na2PqmAq2Z+WIux98jHMmVfbfqUS1trYO+s+h1JTLWMplHOCxlKrhGEuaCaMQnwe+JGkNsA74HdCd\nHLs2InZImgE8KOm5iHg0/w2SRLICoKGhIRobGwcVSFNTE4vmTGD7xl3H66bOvYDGy2cO6v2Kqamp\nicH+OZSachlLuYwDPJZSNRxjSTNh7ABmZ5VnJXXHRcRB4KMAymwWvAS0JMd2JL93S7qbzBJXr4Qx\nlOZOzb1FiC/eMzM7Ic09jFXAQknzJdUANwP3ZjeQNCk5BvAnwKMRcVBSraS6pE0t8A5gfYqxAjBn\nam1O2WdKmZmdkNoMIyK6JN0G3A9UAt+OiA2Sbk2OLwcuBL4nKYANwMeS7vXA3ckZSlXADyPivrRi\nPWaeZxhmZv1KdQ8jIlYCK/Pqlme9fhw4r49+LcBlacbWl7lTcmcYW/Z6hmFmdoyv9M4yc/I4aipP\n/JG81tbBgXafWmtmBk4YOSor1Gvj+yXPMszMACeMXuZPy12WatnTWqRIzMxKixNGngXTJ+SUPcMw\nM8twwsizoNcMwwnDzAycMHpZMD0vYXiGYWYGOGH0kr+H8dLeVnp6okjRmJmVDieMPFNqa5g4rvp4\n+UhnD68ePFLEiMzMSoMTRh5JfcwyvCxlZuaE0Yde+xg+tdbMzAmjL73OlPIMw8zMCaMv+ddi+NRa\nMzMnjD71utp7r5ekzMycMPowf1otmTurZ2zf3057R3f/HczMRgEnjD6Mra5kzpQTNyGMgBe98W1m\no5wTRj/OzdvHaN7thGFmo1uqCUPSEkmbJTVLur2P45Ml3S1praSnJF1caN+0nVvvhGFmli21hCGp\nErgTuAFYBNwiaVFes88AayLiUuDDwJdOoW+qFs6oyym/sPvQcH68mVnJSXOGcRXQHBEtEdEB3AUs\nzWuzCPgVQEQ8B8yTVF9g31QtnJE7w3jBMwwzG+XSTBgzgW1Z5e1JXbZngXcDSLoKmAvMKrBvqs7J\nSxgvv3aYo10+U8rMRq+qIn/+54EvSVoDrAN+B5zS/5UlLQOWAdTX19PU1DSoQFpbW3v1nTpWvHYk\nc6fa7p7gH//pEWbVlf55An2NZaQql7GUyzjAYylVwzGWNBPGDmB2VnlWUndcRBwEPgogScBLQAsw\nbqC+We+xAlgB0NDQEI2NjYMKtqmpify+F7c8xSPP7zlenjT3AhovPXtQ7z+c+hrLSFUuYymXcYDH\nUqqGYyxp/nN5FbBQ0nxJNcDNwL3ZDSRNSo4B/AnwaJJEBuw7HM7N38fY5X0MMxu9UpthRESXpNuA\n+4FK4NsRsUHSrcnx5cCFwPckBbAB+NjJ+qYVa3/yN759aq2ZjWap7mFExEpgZV7d8qzXjwPnFdp3\nuC2szz9TyqfWmtnoVfo7uEV0bt61GC172ujo6ilSNGZmxeWEcRITx1Vz1sSxx8tdPeE715rZqOWE\nMYALzsydZTy308tSZjY6OWEM4Pwzz8gpP/eqE4aZjU5OGAO48Ky8GcarB4sUiZlZcTlhDOCC/BmG\nl6TMbJRywhjAgum1VFeeePzeqweP8PrhjiJGZGZWHE4YA6iurOCcvIcpeR/DzEYjJ4wCXHhW7rLU\nZicMMxuFnDAKcH7+qbXe+DazUcgJowD512Js8sa3mY1CThgFyF+Sen7XIbp7okjRmJkVhxNGAWbU\njWFKbc3x8uGObra81lbEiMzMhp8TRgEkcdHZubOM9TsOFCkaM7PicMIo0EVnT8wpb3jFG99mNro4\nYRQof4ax4RXPMMxsdHHCKNDFM3NnGOt3HCTCG99mNnqkmjAkLZG0WVKzpNv7OD5R0s8lPStpg6SP\nZh3bImmdpDWSVqcZZyHmThnPhDEnHlB4oL2T7fvbixiRmdnwSi1hSKoE7gRuABYBt0halNfs48DG\niLgMaATukFSTdfy6iFgcEQ1pxVmoigqx6CwvS5nZ6JXmDOMqoDkiWiKiA7gLWJrXJoA6SQImAPuA\nrhRjOi0XzcxPGN74NrPRI82EMRPYllXentRl+ypwIfAKsA74VEQce2h2AA9JelrSshTjLFj+mVI+\ntdbMRpOqgZuk6veBNcBbgXOAByU9FhEHgWsjYoekGUn9cxHxaP4bJMlkGUB9fT1NTU2DCqS1tXXA\nvu2HenLKz2zZO+jPS1MhYxkpymUs5TIO8FhK1XCMJc2EsQOYnVWeldRl+yjw+cicbtQs6SXgAuCp\niNgBEBG7Jd1NZomrV8KIiBXACoCGhoZobGwcVLBNTU0M1Leru4e/efJ+jnZlEseBo8EFl1/DmRPH\nDuoz01LIWEaKchlLuYwDPJZSNRxjSXNJahWwUNL8ZCP7ZuDevDZbgesBJNUD5wMtkmol1SX1tcA7\ngPUpxlqQqsqKXtdjrNn2epGiMTMbXqkljIjoAm4D7gc2AT+OiA2SbpV0a9Lsc8CbJK0DHgb+Q0Ts\nBeqB30h6FngK+GVE3JdWrKfistmTcsrPbnfCMLPRIdU9jIhYCazMq1ue9foVMrOH/H4twGVpxjZY\ni/MThmcYZjZK+ErvU3TZrNyEsXb7AXp8q3MzGwWcME7R3KnjmTiu+ni59WgXLXtbixiRmdnwcMI4\nRZJ67WOs2ebrMcys/DlhDMLiWbkX8Hkfw8xGAyeMQVg8x2dKmdno44QxCJfmbXxv2nmQI53dRYrG\nzGx4OGEMwrQJY5g9Zdzxcmd3sHa79zHMrLwVlDAknSNpTPK6UdInJU0aqF85u3LO5Jzy6pf3FSkS\nM7PhUegM4ydAt6Rzydy3aTbww9SiGgGunDclp/zMy/uLFImZ2fAoNGH0JLf6eBfwlYj4NHBWemGV\nvoa5uTOMp1/e70e2mllZKzRhdEq6BfgI8Iukrvok7cveefV11GU9snX/4U5e3NNWxIjMzNJVaML4\nKPBG4L9FxEuS5gM/SC+s0ldZIS7vNcvwPoaZla+CEkZEbIyIT0bEjyRNBuoi4m9Tjq3k5S9Lrd7i\nfQwzK1+FniXVJOkMSVOAZ4BvSPpiuqGVviv72McwMytXhS5JTUwem/pu4PsRcTXwtvTCGhkWz55E\nZYWOl1v2trG39WgRIzIzS0+hCaNK0lnA+zix6T3q1Y6p6vUEvqde8j6GmZWnQhPGfyXz5LwXI2KV\npAXAC+mFNXJcs2BqTvnxF18rUiRmZukqdNP7HyPi0oj486TcEhF/NFA/SUskbZbULOn2Po5PlPRz\nSc9K2iDpo4X2LRXXLMi9gO+JFicMMytPhW56z5J0t6Tdyc9PJM0aoE8lcCdwA7AIuEXSorxmHwc2\nRsRlQCNwh6SaAvuWhIZ5U8jaxuCF3a3exzCzslToktR3gHuBs5Ofnyd1J3MV0JzMRjqAu4CleW0C\nqJMkYAKwD+gqsG9JOGNsNRfPzH0+xpMt3scws/JTaMKYHhHfiYiu5Oe7wPQB+swEtmWVtyd12b4K\nXAi8AqwDPhURPQX2LRn5+xheljKzclQ1cBMAXpP0x8CPkvItwFD8X/H3gTXAW4FzgAclPXYqbyBp\nGbAMoL6+nqampkEF0traOui+41u7csoPr9/K9ZP2Duq9hsLpjKXUlMtYymUc4LGUquEYS6EJ418D\nXwH+jswy0j8D/2qAPjvI3NX2mFlJXbaPAp+PzF37miW9BFxQYF8AImIFmTvo0tDQEI2NjQOPpg9N\nTU0Mtu8VRzr58u8eoCe59+ArrcGiK69hRt3YQb3f6TqdsZSachlLuYwDPJZSNRxjKfQsqZcj4p0R\nMT0iZkTETcBAZ0mtAhZKmi+pBriZzD5Itq3A9QCS6oHzgZYC+5aMM8ZWc0nePsZvm4s3wzAzS8Pp\nPHHv353sYHI79NvIXL+xCfhxRGyQdKukW5NmnwPeJGkd8DDwHyJib399TyPW1P2LhblbOo8974Rh\nZuWl0CWpvmigBhGxEliZV7c86/UrwDsK7VvKrl04ja/+uvl4+bHmvUQEmRPAzMxGvtOZYfhpQVmu\nmDOZ8TWVx8t7Dh1l865DRYzIzGxonTRhSDok6WAfP4fIXI9hiZqqCt6Yd3qtl6XMrJycNGFERF1E\nnNHHT11EnM5yVlm6duG0nPJj3vg2szJyOktSlid/4/vJltc40tldpGjMzIaWE8YQOmd6LWdPPHHt\nxdGuHl/1bWZlwwljCEmi8YIZOXW/em53kaIxMxtaThhD7Pq8hPHwpt1kLmQ3MxvZnDCG2JvOmcaY\nqhN/rDteb+f5Xa1FjMjMbGg4YQyxcTWV/N65uWdLPfzcriJFY2Y2dJwwUvDW/H2MTd7HMLORzwkj\nBfkJ45mt+9nX1lGkaMzMhoYTRgrOnjSOC88643i5J+ChjV6WMrORzQkjJe9YVJ9T/qf1O4sUiZnZ\n0HDCSMkNl5yZU/5t82scPNJZpGjMzE6fE0ZKzq+vY97U8cfLHd09/NoX8ZnZCOaEkRJJLLn4rJy6\n+9a/WqRozMxOnxNGim64OHdZqmnzHto7fDNCMxuZUk0YkpZI2iypWdLtfRz/tKQ1yc96Sd2SpiTH\ntkhalxxbnWacabl01sScmxG2d3b7Ij4zG7FSSxiSKoE7gRuARcAtkhZlt4mIL0TE4ohYDPwl8EhE\n7Mtqcl1yvCGtONMkiRsvyV2W+tmaV4oUjZnZ6UlzhnEV0BwRLRHRAdwFLD1J+1uAH6UYT1EsXTwz\np9y0eTcHDvtsKTMbedJMGDOBbVnl7UldL5LGA0uAn2RVB/CQpKclLUstypRdPPMMFkyrPV7u7A5f\nk2FmI1KpPGb1D4Hf5i1HXRsROyTNAB6U9FxEPJrfMUkmywDq6+tpamoaVACtra2D7juQSyd10JL1\ntNbvNW3gzMMtqXwWpDuW4VYuYymXcYDHUqqGYyxpJowdwOys8qykri83k7ccFRE7kt+7Jd1NZomr\nV8KIiBXACoCGhoZobGwcVLBNTU0Mtu9A5lzUyj13PHK8/Nz+Hs6//GrOmjgulc9LcyzDrVzGUi7j\nAI+lVA3HWNJckloFLJQ0X1INmaRwb34jSROBtwA/y6qrlVR37DXwDmB9irGmasH0CVw6a+LxcgT8\n9Jn+cqeZWWlKLWFERBdwG3A/sAn4cURskHSrpFuzmr4LeCAi2rLq6oHfSHoWeAr4ZUTcl1asw+Hd\nl+du3/zj6m1+Ep+ZjSip7mFExEpgZV7d8rzyd4Hv5tW1AJelGdtwW7p4Jv995XN0dPcAsOW1w6za\nsp+r5k8pcmRmZoXxld7DZHJtDW+/KPcOtj9eva2f1mZmpccJYxi9r2F2TvmXa3dyyHewNbMRwglj\nGF177jTOyrtVyD2+8tvMRggnjGFUWSHee+WsnLofPL7Fm99mNiI4YQyzW66eQ2WFjpef39XKUy/t\nO0kPM7PS4IQxzM6aOI63X5i7+f39J14uUjRmZoVzwiiCD71xbk75/vWvsuvgkSJFY2ZWGCeMInjT\nOVM5Z/qJGxJ29QTff3xL0eIxMyuEE0YRSOIjb5qXU/f3T2yl7WhXcQIyMyuAE0aRvOfKWUwaX328\nfKC90xfymVlJc8IokvE1VXzomty9jG/95iW6kluHmJmVGieMIvrwG+dRU3XiK9i+v51frPXDlcys\nNDlhFNH0ujH80RW5d7H96q+b6e7xhXxmVnqcMIrsz958Ts6FfM27W1m5zrMMMys9ThhFNm9aLUsX\nn51T95VfvUCPZxlmVmKcMErAbdedS9Ykg+d3tfLztb4poZmVFieMErBg+gTeeVnuLON/PbCZo13d\nRYrIzKy3VBOGpCWSNktqlnR7H8c/LWlN8rNeUrekKYX0LTefett5VGVNM7bta+cHj/seU2ZWOlJL\nGJIqgTuBG4BFwC2SFmW3iYgvRMTiiFgM/CXwSETsK6RvuZk/rZYPXj0np+4rv2rmwGE/YMnMSkOa\nM4yrgOaIaImIDuAuYOlJ2t8C/GiQfcvCJ69fyIQxJx6zfqC9k681NRcxIjOzE9JMGDOB7HtdbE/q\nepE0HlgC/ORU+5aTqRPG8OeN5+TUfeeft7Bt3+EiRWRmdkLVwE2GxR8Cv42IU36SkKRlwDKA+vp6\nmpqaBhVAa2vroPsOpXN7gsljxP6jmdNqO7p6uO07j/IXV4xB0gC9M0plLEOhXMZSLuMAj6VUDcdY\n0kwYO4DZWeVZSV1fbubEctQp9Y2IFcAKgIaGhmhsbBxUsE1NTQy271A7OHEbn/5/a4+Xn93TzdHp\nF7Dk4rMK6l9KYzld5TKWchkHeCylajjGkuaS1CpgoaT5kmrIJIV78xtJmgi8BfjZqfYtV390xSwa\n5k7Oqfurezdw6Ig3wM2seFJLGBHRBdwG3A9sAn4cERsk3Srp1qym7wIeiIi2gfqmFWupqagQ//3d\nl+ScZrvr4FHueOD5IkZlZqNdqnsYEbESWJlXtzyv/F3gu4X0HU3Oq69j2ZsX8LWmF4/Xfe/xLSxd\nfDaXz5ncf0czs5T4Su8S9om3LmT2lHHHyxHwb//vGj+Zz8yKwgmjhI2rqeRvbrokp27La4f53C82\nFikiMxvNnDBK3FvOm84fX5N7Bfhdq7Zx3/pXixSRmY1WThgjwGdvXMQ502tz6m7/6VpePXCkSBGZ\n2WjkhDECjKup5Es3X0515Ymzpl4/3Mmtf/+072hrZsPGCWOEuHjmRP7d28/PqVuz7XX+0z3rifDD\nlswsfU4YI8iyNy+g8fzpOXU/Xr2dv3/Ct0E3s/Q5YYwglRXiSzdfzvxpufsZ/+XnG/lt894iRWVm\no4UTxggzcVw1Kz50JbU1lcfrunqCZd9fzbrtB4oYmZmVOyeMEWhhfR1ffP/inLq2jm7+1XeeomVP\na5GiMrNy54QxQv3+RWfy2RsvzKl7ra2DD33rKfYd6SlSVGZWzpwwRrA/ffMC/uwtC3Lqdrzezv94\n8ogfumRmQ84JY4S7fckFvPfKWTl1e9qD9339cS9PmdmQcsIY4STxP959CTdcfGZO/c4DR3jf159g\nwyveCDezoeGEUQaqKiv4yi2Xs3Tx2Tn1e1uP8t7lj/PQxl1FiszMyokTRpmoqqzgi+9bzPsacpen\nDnd086c/WM03H2vxFeFmdlqcMMpIZYX4/Lsv5R1zc5+LFQF/88tNfpaGmZ2WVBOGpCWSNktqlnR7\nP20aJa2RtEHSI1n1WyStS46tTjPOclJRIT5w4Rg+t/QiKrMe8Qpwz5pX+MOv/oZNOw8WKTozG8lS\nSxiSKoE7gRuARcAtkhbltZkEfA14Z0RcBLw3722ui4jFEdGQVpzl6kNvnMe3PtLAhDG5s42WPW3c\ndOdv+eZjLXT3eInKzAqX5gzjKqA5IloiogO4C1ia1+YDwE8jYitAROxOMZ5Rp/H8Gdzz8Tdxfn1d\nTv3Rrh7+5pebeN/XH6d5t0+9NbPCpJkwZgLbssrbk7ps5wGTJTVJelrSh7OOBfBQUr8sxTjL2rkz\n6rjn47/H+xtm9zr29Mv7ufHLj/HFBzbT3uHnapjZySmtM2ckvQdYEhF/kpQ/BFwdEbdltfkq0ABc\nD4wDHgf+ICKelzQzInZImgE8CHwiIh7t43OWAcsA6uvrr7zrrrsGFW9raysTJkwYVN9S099Y/vmV\nLn6w8Sjtfex7Txkr3nd+DVefWYmk3g2KpFy+l3IZB3gspWqwY7nuuuueLnTZv2rgJoO2A8j+Z+2s\npC7bduC1iGgD2iQ9ClwGPB8ROyCzTCXpbjJLXL0SRkSsAFYANDQ0RGNj46CCbWpqYrB9S01/Y2kE\nPnbwCJ+9ex0Pbcpd/dt3JFj+7FEe3X0Gf/G2hbx9UX1JJI5y+V7KZRzgsZSq4RhLmktSq4CFkuZL\nqgFuBu7Na/Mz4FpJVZLGA1cDmyTVSqoDkFQLvANYn2Kso0b9GWP5xocb+N/vX8y0CTW9jm/ceZBl\nP3iaP/jyb7h/w6v0eGPczBKpzTAiokvSbcD9QCXw7YjYIOnW5PjyiNgk6T5gLdADfDMi1ktaANyd\n/Au3CvhhRNyXVqyjjSRuunwmb71wBl/9VTPf+e1LdHbnJoaNOw/yZz94mnlTx/OhN87jPVfOYuK4\n6iJFbGalIM0lKSJiJbAyr255XvkLwBfy6lrILE1Zis4YW81nbryQm98wmzseeJ5frtvZq82W1w7z\nuV9s5I4HNnPT5TN575WzWDx7UkksV5nZ8Eo1YdjIsGD6BO784BV88tVDfPnhF/pMHIc7uvnhk1v5\n4ZNbmTd1PEsXz+Smy2f2elysmZUvJww77vwz644njuWPvMgv1r7Sa6kKMrOOLz38Al96+AXOr6/j\n+gtn8LZF9SyeNYmKCs88zMqVE4b1cv6Zdfzd+xfzmRsv5K6ntvIPT27l1YNH+my7edchNu86xNea\nXmTahBreeM40rlkwhWsWTGXBtFovXZmVEScM69f0ujF84vqF3Np4Dg9v2s09v9vBr57bTUd334+A\n3dvawc+ffYWfP/vK8f5Xz5/C5XMmc8nMiVx09hnUjvF/cmYjlf/22oCqKytYcvGZLLn4TA4c7uSf\n1u/kZ2te4akt+056P6o9h47yi7U7+cXazJ6IBOdOn8AlMydy/pl1LKyfwMIZdcycNM5LWWYjgBOG\nnZKJ46u5+ao53HzVHF4/3EHT5j08tGkXj2zew6EBbp0eAS/sbuWFvPtXjauu5NwZE5g7dTyzp4xn\nzpTxzJ6c+d3l60DMSoYThg3apPE13HR55mypzu4e1u04wBMtr/FEyz5Wb9nH4QLvT9Xe2c26HQdY\nt6P342QFzHjiIerPGMuMurHUnzEmeZ35PaW2hsnja5hUW03dmCrvmZilyAnDhkR1ZQVXzJnMFXMm\n828aobO7h7XbD/C7rftZv+MAa3ccoGVP2ym/bwC7Dh5l18GjwMmfT15VISaNr2bS+BomJ7/PGFvN\nhDGV1I6pYsLYKiaMqaK2poraMVXUjc38rq2pZGz1sZ8KxlVXUlXpZ4uZ5XPCsFRUV1Zw5dzJXDl3\n8vG6Q0c62fDKQTa+cpAXdrfSvPsQz+9q5UB755B8ZldPsLe1g72tHaf9XlUVYlx1JWNrMklkbFUl\n1ZUVVFdVUFMpaqoqMuXKCmoqK6jOr6vK1FVXVlAp8fLLHWzWi1RW6PhPhURVhaioEJVSn8cqs45L\nmX2gCgmReVhWhTJX7otMfUV+O4HI/IbMjO2Y3MmYetXntj1R2tXWw5a9bXlte/fPd7xtVoP+4jn2\nfrl1fYab89nZgj6WM/OqXj/Sw64+zgDs656s+e/Xd5s+6mLgfkPx+bvaeujo6qGmKr1/7Dhh2LCp\nG1vNNQumcs2CqcfrIoI9rUd5cXcb2/YfZtu+zM/WfYfZtr+dPYeOFiXWrp7g0NGuAfdlTsnzzw3d\nexXbY02mVBG4AAAJbElEQVTFjmDoND1c7AiGzJVvOMy5M9K7+64ThhWVJGbUZfYn3sjUXscfePjX\nXLD4anYfOsKug0dP/D54hN2HjrKvrYPXD3ew/3An7Z1+podZmpwwrKTVVIo5U8czZ+r4Adse6ezm\nQHsn+w93sL+tk9cPd3DoSBetR7toO5r5feJ1N61HO2k72s3hji6OdPZwpLOb9uQnpcfEmI1oThhW\nNo5tXNefMfa03ici6OjuOZFEOro50tVNV3emvrOrh87uoLO7J1PuzqwdZ8qRHD9R7ukJWra8zKzZ\ns+nqDnoi6O4Junoyx7qTcvex192Z3z3H2kTy05NZx+6JTIwRJMeSMmS1y9T1JO0gd309e109JzcW\n0La9vZ1x48ZlvW/W8aDP1wN9bk6/5EhuXX/vm/vZvfdPeu9vZLfp6Oigpqamz12QvvZi8vdL+m7T\n13sNfPZefpNT/fz29naqK9M9S9AJwyyPJMZUVTKmqnLIbune1LSTxsYLh+S9is0PHSpNTU1NzJ2a\n7s1Afe6gmZkVxAnDzMwK4oRhZmYFSTVhSFoiabOkZkm399OmUdIaSRskPXIqfc3MbPiktuktqRK4\nE3g7sB1YJeneiNiY1WYS8DVgSURslTSj0L5mZja80pxhXAU0R0RLRHQAdwFL89p8APhpRGwFiIjd\np9DXzMyGUZoJYyawLau8PanLdh4wWVKTpKclffgU+pqZ2TAq9nUYVcCVwPXAOOBxSU+cyhtIWgYs\nS4qtkjYPMpZpwN5B9i01HkvpKZdxgMdSqgY7lrmFNkwzYewAZmeVZyV12bYDr0VEG9Am6VHgsqR+\noL4ARMQKYMXpBitpdUQ0nO77lAKPpfSUyzjAYylVwzGWNJekVgELJc2XVAPcDNyb1+ZnwLWSqiSN\nB64GNhXY18zMhlFqM4yI6JJ0G3A/UAl8OyI2SLo1Ob48IjZJug9YC/QA34yI9QB99U0rVjMzG1iq\nexgRsRJYmVe3PK/8BeALhfRN2Wkva5UQj6X0lMs4wGMpVamPRflPgzIzM+uLbw1iZmYFGfUJY6Tf\ngkTSFknrkturrE7qpkh6UNILye/JA71PMUj6tqTdktZn1fUbu6S/TL6nzZJ+vzhR962fsfy1pB3J\nd7NG0o1Zx0p5LLMl/VrSxuSWPZ9K6kfUd3OScYy470XSWElPSXo2Gct/SeqH9zvJPIhldP6Q2VB/\nEVgA1ADPAouKHdcpjmELMC2v7n8Ctyevbwf+tthx9hP7m4ErgPUDxQ4sSr6fMcD85HurLPYYBhjL\nXwP/vo+2pT6Ws4Arktd1wPNJzCPquznJOEbc90LmuUwTktfVwJPANcP9nYz2GUa53oJkKfC95PX3\ngJuKGEu/IuJRYF9edX+xLwXuioijEfES0Ezm+ysJ/YylP6U+lp0R8Uzy+hCZU91nMsK+m5OMoz8l\nOQ6AyGhNitXJTzDM38loTxjlcAuSAB5Kbq1y7Ir3+ojYmbx+FagvTmiD0l/sI/W7+oSktcmS1bHl\nghEzFknzgMvJ/It2xH43eeOAEfi9SKqUtAbYDTwYEcP+nYz2hFEOro2IxcANwMclvTn7YGTmpyPy\nVLiRHHvi/5BZ7lwM7ATuKG44p0bSBOAnwF9ExMHsYyPpu+ljHCPye4mI7uTv+izgKkkX5x1P/TsZ\n7QmjkNuXlLSI2JH83g3cTWbauUvSWQDJ7939v0PJ6S/2EfddRcSu5C95D/ANTiwJlPxYJFWT+Z/s\nP0TET5PqEffd9DWOkfy9AETE68CvgSUM83cy2hPGiL4FiaRaSXXHXgPvANaTGcNHkmYfIXMLlpGi\nv9jvBW6WNEbSfGAh8FQR4ivYsb/IiXeR+W6gxMciScC3gE0R8cWsQyPqu+lvHCPxe5E0XZnnByFp\nHJlnBT3HcH8nxd79L/YPcCOZsydeBD5b7HhOMfYFZM6EeBbYcCx+YCrwMPAC8BAwpdix9hP/j8gs\nCXSSWWP92MliBz6bfE+bgRuKHX8BY/kBsI7MrW/uBc4aIWO5lszSxlpgTfJz40j7bk4yjhH3vQCX\nAr9LYl4P/Oekfli/E1/pbWZmBRntS1JmZlYgJwwzMyuIE4aZmRXECcPMzArihGFmZgVxwrCyIikk\n3ZFV/veS/nqI3vu7kt4zFO81wOe8V9ImSb8usP1n0o7JDJwwrPwcBd4taVqxA8km6VSebvkx4E8j\n4roC2zth2LBwwrBy00XmUZX/Nv9A/gxBUmvyu1HSI5J+JqlF0uclfTB5/sA6Sedkvc3bJK2W9Lyk\nf5n0r5T0BUmrkhva/VnW+z4m6V5gYx/x3JK8/3pJf5vU/WcyF5x9S9IX8tqfJenR5BkO6yX9C0mf\nB8Yldf+QtPvjJPY1kr4uqfLYeCX9XfI8hYclTT+NP2cbhZwwrBzdCXxQ0sRT6HMZcCtwIfAh4LyI\nuAr4JvCJrHbzyNx76A+A5ZLGkpkRHIiINwBvAP40uR0DZJ6R8amIOC/7wySdDfwt8FYyN8F7g6Sb\nIuK/AquBD0bEp/Ni/ABwf2RuQHcZsCYibgfaI2JxRHxQ0oXA+4HfS9p1Ax9M+tcCqyPiIuAR4K9O\n4c/HjFOZJpuNCBFxUNL3gU8C7QV2WxXJbaIlvQg8kNSvA7KXhn4cmZvWvSCpBbiAzD28Ls2avUwk\nc++eDuCpyDyPIN8bgKaI2JN85j+QeQjTPSeLEfh2ckO9eyJiTR9trgeuBFZlbqXEOE7ckK4H+L/J\n678Hftqrt9lJOGFYufrfwDPAd7Lqukhm1ZIqyDxl8ZijWa97sso95P49yb+XTpB5GtonIuL+7AOS\nGoG2wYXfW0Q8mty+/g+A70r6YkR8P6+ZgO9FxF8W8pZDFZuNDl6SsrIUEfuAH5NZLjpmC5l/fQO8\nk8xTy07VeyVVJPsaC8jc2O1+4M+Tf/kj6bzk7sEn8xTwFknTkj2GW8gsE/VL0lxgV0R8g8xS2RXJ\noc5jn03mRnTvkTQj6TMl6QeZv+/HZkEfAH5T2JDNMjzDsHJ2B3BbVvkbwM8kPQvcx+D+9b+VzP/s\nzwBujYgjkr5JZm/jmeSW2nsY4LG4EbFT0u1knmsg4JcRMdBt6BuBT0vqBFqBDyf1K4C1kp5J9jH+\nI/BAMovqBD4OvExmvFclx3eT2eswK5jvVms2SkhqjYgJxY7DRi4vSZmZWUE8wzAzs4J4hmFmZgVx\nwjAzs4I4YZiZWUGcMMzMrCBOGGZmVhAnDDMzK8j/B2iee6yv/D+HAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266a20ecdd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_history_PGD, lw=4)\n",
    "plt.grid()\n",
    "#plt.yscale('log')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Number of step')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id='section2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# FISTA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Une accélération est proposée dans le cours, implémentons-la aussi : c'est l'accélération Nesterov."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def FISTA(T, lam, eta):\n",
    "    # on déclenche le timer\n",
    "    t0 = time.time()\n",
    "    \n",
    "    # initialisation des s\n",
    "    st_moins2 = 1.\n",
    "    st_moins1 = 1.\n",
    "    st = 1.\n",
    "    \n",
    "    # initialisation\n",
    "    xt = np.zeros(n_features)\n",
    "    xt_moins1 = np.zeros(n_features)\n",
    "    yt = np.zeros(n_features)\n",
    "    \n",
    "    # historique des fonctions de perte\n",
    "    cost_history = np.empty(shape=[T])\n",
    "    time_history = np.empty(shape=[T,2])\n",
    "    \n",
    "    for t in range(0,T):\n",
    "        cost_history[t] = f(xt, lam)\n",
    "        time_history[t] = (time.time()-t0, f(xt, lam))\n",
    "        \n",
    "        # compute gradient\n",
    "        gt = grad(xt)\n",
    "        \n",
    "        # compute prox\n",
    "        xt_moins1 = xt\n",
    "        xt = yt - eta * gt - eta * lam * np.sign(yt)   \n",
    "        \n",
    "        # compute st\n",
    "        st_moins2 = st_moins1\n",
    "        st_moins1 = st\n",
    "        st = (1 + np.sqrt(1 + 4 * st_moins1**2)) / 2 \n",
    "        \n",
    "        # compute nestorov acceleration\n",
    "        yt = xt + (st_moins2 / st) * (xt - xt_moins1)  \n",
    "               \n",
    "    return xt, cost_history, time_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "theta, cost_history_FISTA, time_history_FISTA = FISTA(T=T,lam=lam,eta=eta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8XXWd//HX597kZmuapLQN3aALRSlLEUpBQIkbixuu\nI7iguFQcUWcc/Q3qzOgs/kaHBzr+RsZateKGDKMwVO2wuIRdukChLVAoLW2T7m2aNutNbj6/P+5p\ncu/NdpPmJDe57+fjkUdyvud87/1+eyHvfL/nnO8xd0dERGQwkbFugIiIjA8KDBERyYoCQ0REsqLA\nEBGRrCgwREQkKwoMERHJigJDRESyosAQEZGsKDBERCQrBWPdgJE0depUnzt37rDqNjc3U1ZWNrIN\nGiPqS+6ZKP0A9SVXDbcv69evP+ju07I5dkIFxty5c1m3bt2w6tbW1lJTUzOyDRoj6kvumSj9APUl\nVw23L2a2I9tjNSUlIiJZUWCIiEhWFBgiIpIVBYaIiGRFgSEiIlkJLTDMbKWZ7TezTf3sNzP7f2a2\n1cyeMbPzUvZdaWZbgn03hdVGERHJXpiX1d4GfBf4aT/7rwIWBl8XAt8DLjSzKHAr8CagDlhrZqvc\n/dmRbuBTOxu4/Ymd1B9pZeueFt58dDNfe/uZI/02IiITQmiB4e4PmdncAQ65GvipJ58R+2czqzSz\nGcBcYKu7bwMwszuCY0c8MPYfa+e/19d1b28/2DzSbyEiMmGM5Y17s4BdKdt1QVlf5Rf29yJmtgxY\nBlBdXU1tbW3WDdh7NJG2/WL9wSHVz1VNTU0Toh8wcfoyUfoB6kuuGo2+jPs7vd19BbACYMmSJT6U\nOx0bmuN89bEHerbjES677DLMbKSbOap092rumSj9APUlV41GX8byKql6YE7K9uygrL/yEVdZWkhp\nLNq93dqRoKGlI4y3EhEZ98YyMFYB1wVXS10ENLr7HmAtsNDM5plZDLgmOHbEmRmzKkvSynYfaQ3j\nrURExr3QpqTM7JdADTDVzOqArwKFAO6+HFgNvBnYCrQA1wf7Os3sRuA+IAqsdPfNYbVzZmUJL+5v\n6t6ua2jlrFkVYb2diMi4FeZVUtcOst+BT/ezbzXJQAndrKr0EUa9RhgiIn3K+zu9NSUlIpKdvA+M\n2ZkjjAYFhohIX/I+MGZmjjAaFRgiIn3J+8CYXl6Utn2oKT5GLRERyW15HxiVpbG07SMtCgwRkb7k\nfWBMLi4gGum5s7s5nqCtIzFADRGR/JT3gWFmVPUaZehubxGRTHkfGABVpYVp2w2alhIR6UWBAVSV\npY8wGpoVGCIimRQYwJSMKanDGmGIiPSiwACqyjKnpHQOQ0QkkwIDep301pSUiEhvCgz6CAxNSYmI\n9KLAQCe9RUSyocAApmScwziscxgiIr0oMNDyICIi2VBg0MdltZqSEhHpJdTAMLMrzWyLmW01s5v6\n2F9lZneb2TNmtsbMzkrZ97KZbTSzDWa2Lsx26hyGiMjgQgsMM4sCtwJXAYuAa81sUcZhXwY2uPs5\nwHXAdzL2v87dz3X3JWG1E5ILEKasP0hzPEF7pxYgFBFJFeYIYymw1d23uXscuAO4OuOYRcAfAdz9\neWCumVWH2KY+mRmT0s97awFCEZEMBSG+9ixgV8p2HXBhxjFPA+8CHjazpcCpwGxgH+DA780sAXzf\n3Vf09SZmtgxYBlBdXU1tbe2wGlta4ByN9wwz7n/wMeaUj89TPE1NTcP+d8g1E6UvE6UfoL7kqtHo\nS5iBkY1vAN8xsw3ARuAp4Phc0KXuXm9m04EHzOx5d38o8wWCIFkBsGTJEq+pqRlWQyY/8b/sbenq\n3l6w6BwuXjB1WK811mpraxnuv0OumSh9mSj9APUlV41GX8IMjHpgTsr27KCsm7sfBa4HMDMDtgPb\ngn31wff9ZnY3ySmuXoExUspjlrbd0KwpKRGRVGHOuawFFprZPDOLAdcAq1IPMLPKYB/Ax4GH3P2o\nmZWZWXlwTBlwObApxLZSVpgRGLoXQ0QkTWgjDHfvNLMbgfuAKLDS3Teb2Q3B/uXAGcBPzMyBzcDH\ngurVwN3JQQcFwO3ufm9YbQUozwwMXVorIpIm1HMY7r4aWJ1Rtjzl58eB0/uotw1YHGbbMk3KmJLS\nMzFERNKNz8uAQqDLakVEBqbACGSe9NbyICIi6RQYgcwpKZ30FhFJp8AITNJVUiIiA1JgBHQfhojI\nwBQYgZICiKasQNjU3km8s2uAGiIi+UWBEYiYUVmSfqmUHqQkItJDgZEi87kYuhdDRKSHAiNFVWn6\nCEPnMUREeigwUlRlPKpVV0qJiPRQYKSYkjklpZv3RES6KTBSVGaMMHTSW0SkhwIjxZSy9HMYh3UO\nQ0SkmwIjhc5hiIj0T4GRQoEhItI/BUaKzPsw9BAlEZEeoQaGmV1pZlvMbKuZ3dTH/iozu9vMnjGz\nNWZ2VrZ1w9DrPgw9E0NEpFtogWFmUeBW4CpgEXCtmS3KOOzLwAZ3Pwe4DvjOEOqOuMyrpBpbFRgi\nIseFOcJYCmx1923uHgfuAK7OOGYR8EcAd38emGtm1VnWHXGTi9OfWHu0rYOuLg/7bUVExoUwA2MW\nsCtluy4oS/U08C4AM1sKnArMzrLuiCuIRiiLRbu33eFYe2fYbysiMi4UDH5IqL4BfMfMNgAbgaeA\nxFBewMyWAcsAqqurqa2tHVZDmpqaqK2tpSjSRXNK+f1/ephppePr2oDjfZkIJkpfJko/QH3JVaPR\nlzADox6Yk7I9Oyjr5u5HgesBzMyA7cA2oGSwuimvsQJYAbBkyRKvqakZVmNra2upqalh+oaHOLz3\nWHf5GYvP56xZFcN6zbFyvC8TwUTpy0TpB6gvuWo0+hLmn85rgYVmNs/MYsA1wKrUA8ysMtgH8HHg\noSBEBq0bloqMZ2Ic1YlvEREgxBGGu3ea2Y3AfUAUWOnum83shmD/cuAM4Cdm5sBm4GMD1Q2rraky\nA0NXSomIJIV6DsPdVwOrM8qWp/z8OHB6tnVHw2QFhohIn8bX2dxRoBGGiEjfFBgZFBgiIn1TYGRQ\nYIiI9E2BkUGBISLSNwVGhl6X1bbpTm8REVBg9DK5JP3CMY0wRESSFBgZdOOeiEjfFBgZdB+GiEjf\nFBgZ+jrp7a4lzkVEFBgZigqiFBf2/LMkupzm+JAW0BURmZAUGH3QeQwRkd4UGH2YXKzzGCIimRQY\nfdDNeyIivSkw+qDAEBHpTYHRBwWGiEhvCow+ZN6LoZPeIiIKjD7pKikRkd5CDQwzu9LMtpjZVjO7\nqY/9FWb2GzN72sw2m9n1KfteNrONZrbBzNaF2c5MuttbRKS30B7RamZR4FbgTUAdsNbMVrn7symH\nfRp41t3fZmbTgC1m9gt3jwf7X+fuB8NqY390DkNEpLcwRxhLga3uvi0IgDuAqzOOcaDczAyYBBwG\nxnw9cQWGiEhvYQbGLGBXynZdUJbqu8AZwG5gI/A5d+8K9jnwezNbb2bLQmxnLwoMEZHeQpuSytIV\nwAbg9cAC4AEze9jdjwKXunu9mU0Pyp9394cyXyAIk2UA1dXV1NbWDqshTU1N3XXrjnWl7dtzqHHY\nrzsWUvsy3k2UvkyUfoD6kqtGoy9hBkY9MCdle3ZQlup64BueXA52q5ltB14JrHH3egB3329md5Oc\n4uoVGO6+AlgBsGTJEq+pqRlWY2trazled09jK3/36B+793VGYgz3dcdCal/Gu4nSl4nSD1BfctVo\n9CXMKam1wEIzm2dmMeAaYFXGMTuBNwCYWTXwCmCbmZWZWXlQXgZcDmwKsa1pNCUlItJbaCMMd+80\nsxuB+4AosNLdN5vZDcH+5cA/A7eZ2UbAgL9194NmNh+4O3kunALgdne/N6y2ZiopjFIYNToSyedg\nxDu7aOtIUFwYHa0miIjknFDPYbj7amB1RtnylJ93kxw9ZNbbBiwOs20DMTMqSgo52BTvLmts7VBg\niEhe053e/dDNeyIi6RQY/dB5DBGRdAqMfvR6iFKLAkNE8psCox+9FiBsU2CISH5TYPRDU1IiIukU\nGP1QYIiIpFNg9EOBISKSLqvAMLMFZlYU/FxjZp81s8pwmza2FBgiIumyHWH8GkiY2Wkk122aA9we\nWqtywOSS9Hsa9dQ9Ecl32QZGl7t3Au8E/sPdvwjMCK9ZY6/3c73H/DEdIiJjKtvA6DCza4EPA78N\nygoHOH7c05SUiEi6bAPjeuDVwNfdfbuZzQN+Fl6zxp4CQ0QkXVaLDwbP4f4sgJlVAeXu/s0wGzbW\nFBgiIumyvUqq1swmm9kU4EngB2b2rXCbNrYmFRUQjVj3dmtHgnhn1wA1REQmtmynpCqCx6a+C/ip\nu18IvDG8Zo09M2NycfoATKMMEcln2QZGgZnNAP6CnpPeE56mpUREemQbGP9E8sl5L7n72uCJeC+G\n16zcoMAQEemRVWC4+3+7+znu/qlge5u7v3uwemZ2pZltMbOtZnZTH/srzOw3Zva0mW02s+uzrTsa\net+LocAQkfyV7Unv2WZ2t5ntD75+bWazB6kTBW4FrgIWAdea2aKMwz4NPOvui4Ea4BYzi2VZN3Qa\nYYiI9Mh2SurHwCpgZvD1m6BsIEuBrcFoJA7cAVydcYwD5WZmwCTgMNCZZd3QKTBERHpkGxjT3P3H\n7t4ZfN0GTBukzixgV8p2XVCW6rvAGcBuYCPwOXfvyrJu6BQYIiI9srpxDzhkZh8EfhlsXwscGoH3\nvwLYALweWAA8YGYPD+UFzGwZsAygurqa2traYTWkqampV92De+Jp25tf2EZttH5Yrz+a+urLeDVR\n+jJR+gHqS64ajb5kGxgfBf4D+DbJaaTHgI8MUqee5Kq2x80OylJdD3zD3R3YambbgVdmWRcAd19B\ncgVdlixZ4jU1NYP3pg+1tbVk1t1bupM7t2zs3i6fejI1NYuH9fqjqa++jFcTpS8TpR+gvuSq0ehL\ntldJ7XD3t7v7NHef7u7vAAa7SmotsNDM5plZDLiG5HmQVDuBNwCYWTXwCmBblnVDpykpEZEeJ/LE\nvc8PtDNYDv1GkvdvPAfc6e6bzewGM7shOOyfgYvNbCPwB+Bv3f1gf3VPoK3DosAQEemR7ZRUX2yw\nA9x9NbA6o2x5ys+7gcuzrTvadB+GiEiPExlh+Ii1IkdphCEi0mPAEYaZHaPvYDCgJJQW5ZCKUgWG\niMhxAwaGu5ePVkNy0aRYARGDriAyW+IJOhJdFEZPZGAmIjI+6TffACIR63UeQ6MMEclXCoxB6DyG\niEiSAmMQCgwRkSQFxiAUGCIiSQqMQeheDBGRJAXGIDTCEBFJUmAMoldgtCgwRCQ/KTAGoRGGiEiS\nAmMQlQoMERFAgTEojTBERJIUGINQYIiIJCkwBqGlQUREkhQYg9AIQ0QkSYExCC1xLiKSFGpgmNmV\nZrbFzLaa2U197P+imW0IvjaZWcLMpgT7XjazjcG+dWG2cyDlRQVEIz0PF2yJJ2jvTIxVc0RExkxo\ngWFmUeBW4CpgEXCtmS1KPcbdb3b3c939XOBLwIPufjjlkNcF+5eE1c7BmBlVGaOMI7p5T0TyUJgj\njKXAVnff5u5x4A7g6gGOvxb4ZYjtGbbK0lja9uHm+Bi1RERk7IQZGLOAXSnbdUFZL2ZWClwJ/Dql\n2IHfm9l6M1sWWiuzMCUjMBpaFBgikn8GfETrKHob8GjGdNSl7l5vZtOBB8zseXd/KLNiECbLAKqr\nq6mtrR1WA5qamvqt29nSlrb9yNoNxHflyj9dbwP1ZbyZKH2ZKP0A9SVXjUZfwvytVw/MSdmeHZT1\n5RoypqPcvT74vt/M7iY5xdUrMNx9BbACYMmSJV5TUzOsxtbW1tJf3XsPPcOT+3sGSzPnLqTmolOH\n9T6jYaC+jDcTpS8TpR+gvuSq0ehLmFNSa4GFZjbPzGIkQ2FV5kFmVgFcBtyTUlZmZuXHfwYuBzaF\n2NYBZZ7DOKIpKRHJQ6GNMNy908xuBO4DosBKd99sZjcE+5cHh74TuN/dm1OqVwN3m9nxNt7u7veG\n1dbBTClLv0rqcLOukhKR/BPqRLy7rwZWZ5Qtz9i+Dbgto2wbsDjMtg2FRhgiIrrTOytVmZfVKjBE\nJA8pMLKQOSXVoBv3RCQPKTCykDkl1aAb90QkDykwsqAb90REFBhZmVxSiPWsP8ixtk46El1j1yAR\nkTGgwMhCNGK9nouhBQhFJN8oMLKUOS2lS2tFJN8oMLJUWZp5854CQ0TyiwIjS1PKMk98a0pKRPKL\nAiNLvS6t1ZSUiOQZBUaWMp+6p8AQkXyjwMhSVeaUlM5hiEieUWBkKXM9KZ3DEJF8o8DIUq/A0AhD\nRPKMAiNLOochIvlOgZGlzHMYutNbRPKNAiNLeiaGiOS7UAPDzK40sy1mttXMbupj/xfNbEPwtcnM\nEmY2JZu6oy3zTu/G1g46tQChiOSR0ALDzKLArcBVwCLgWjNblHqMu9/s7ue6+7nAl4AH3f1wNnVH\nW2E0khYa7hpliEh+CXOEsRTY6u7b3D0O3AFcPcDx1wK/HGbdUTF1UlHa9sFjCgwRyR9hBsYsYFfK\ndl1Q1ouZlQJXAr8eat3RNHVS+nmMA03tY9QSEZHRVzDWDQi8DXjU3Q8PtaKZLQOWAVRXV1NbWzus\nBjQ1NQ1at6ulLW37kbUb8N2F/Rw9drLpy3gxUfoyUfoB6kuuGo2+hBkY9cCclO3ZQVlfrqFnOmpI\ndd19BbACYMmSJV5TUzOsxtbW1jJY3QePbWbN3pe7t6fOnk/NZQuG9X5hyqYv48VE6ctE6QeoL7lq\nNPoS5pTUWmChmc0zsxjJUFiVeZCZVQCXAfcMte5o63UOQ1NSIpJHQhthuHunmd0I3AdEgZXuvtnM\nbgj2Lw8OfSdwv7s3D1Y3rLZma1qvwNBJbxHJH6Gew3D31cDqjLLlGdu3AbdlU3esTS1PP+mtEYaI\n5BPd6T0EmVNSB44pMEQkfygwhqD3OQxNSYlI/lBgDEFmYBxubifR5aPeDnensbWDlngnXWPw/iKS\nn3LlPoxxIVYQoaq0sPvhSV2ePI9RPbk49Pfef6yN1c/s4Q/P72f9jgZa4gkAzGDe1DKWnFrFZadP\n502LqkNvi4jkJwXGEFVPLk572t7exrZQA6O5vZNb7n+Bnz+xg3hn78UO3WHbgWa2HWjmznV1TJ0U\n4+Jq5/yLOigvzr2bCkVk/NKU1BCdXJEeDnuPtvVz5Ilbs/0wV33nYVY+ur3PsOjLwaY4q17q4PW3\nPMhdT9bhrikrERkZCowhOjljNLEvpMD41fo6rlnxODsPt/S5v7gwQlFB/x/fgWPtfP7Op/n07U9y\ntE0PexKRE6cpqSHKnH7a2zjygfGzx1/m7+/pfZ/irMoSrnv1qbzhjGoWTCvDzGhu72TDriPcv3kv\ndz1Zz7H2zrQ6qzfuZWN9Iz+87gJecXL5iLdVRPKHRhhDFPaU1K/X1/UZFh+5eC4PfP61fPKyBZw2\nfRJmBkBZUQGXnDaVf7z6LP785Tdw4+tOo8DS6+463Mp7lj/GYy8dHNG2ikh+UWAMUZhTUk/tbOBL\nd29MKyuMGt/7wHl87e1nUhobeEBYVlTAF654Bf9yaQlnzpyctu9YWycfXrmG/924Z8TaKyL5RYEx\nRNMnp9+LMVJTUvuPtvHJn61PO7kdK4iw4rolXHX2jCG91sllEe76y4t5/4WnpJV3JJwbf/kUv31m\n94i0WUTyiwJjiHqPME58eRB354u/eob9GUuN/Nu7z+F1r5g+rNcsKojy9XecxU1XvTKtPNHlfO6O\nDax6WqEhIkOjwBiiKWUxYilXJzW1d57wVUi3r9nJgy8cSCv75GXzecerTuwhg2bGDZct4Jb3LiaS\ncl4j0eX81R1P8RuFhogMgQJjiMyM2ZUlaWW7+rn0NRs7DjXz9d89l1a2dN4U/s8Vr+ynxtC9+/zZ\nfPt956aFRpfDX//Xhl5BJSLSHwXGMMyeUpq2vetw67Bex935+3s2dy/zAVAWi3LLexcTjdgANYfu\n6nNn8e/XvCrtdTu7nE/9fD1P7WwY0fcSkYlJgTEMc6rSRxh1DcMbYdz/7D4eyvgL/+/fuog5GYE0\nUt6+eCbf+ovFaWUt8QQfvW0tW/cfC+U9RWTiUGAMQ+Yv9OFMSbXGE/zTb55NK7to/hTed8GcfmqM\njKvPncVX37YorayhpYMP/WgNu48Mb6QkIvkh1MAwsyvNbIuZbTWzm/o5psbMNpjZZjN7MKX8ZTPb\nGOxbF2Y7h2pOVUZgNAz9F+0PHt5Gfcov6GjE+Me3n9V9Q16Yrr9kHje+7rS0sj2NbXx45RoaW7SM\niIj0LbTAMLMocCtwFbAIuNbMFmUcUwn8J/B2dz8TeG/Gy7zO3c919yVhtXM4Zp/glFRjawc/eHhb\nWtlHLp47qkt3/M3lp3Pt0vTRzIv7m/jET9fR1pHop5aI5LMwRxhLga3uvs3d48AdwNUZx7wfuMvd\ndwK4+/4Q2zNiek9JtQ5pVdiVj2znWFvPmk+VpYV87o0LR6x92TAz/uUdZ3PlmSenla95+TCfv3PD\nmDwYSkRyW5iBMQvYlbJdF5SlOh2oMrNaM1tvZtel7HPg90H5shDbOWRVpYVMKupZpqO1I5H1DXyN\nrR2sfHR7WtknXjOfyWPw7IpoxPj3a87lgrlVaeWrN+7ln3/7rJZGF5E0Y71abQFwPvAGoAR43Mz+\n7O4vAJe6e72ZTQceMLPn3f2hzBcIwmQZQHV1NbW1tcNqSFNT05DqTi/uoiklI379wKOcOTU6aL27\nX4ynjS7KCmF+Yhe1tXVDae6AhtqXD8936g8Yu5t7AuK2x16m5VA9b54XG7F2DcdQ+5KrJko/QH3J\nVaPRlzADox5InSSfHZSlqgMOuXsz0GxmDwGLgRfcvR6S01RmdjfJKa5egeHuK4AVAEuWLPGampph\nNba2tpah1P3tgafZtr7nl3zpjPnUXDJvwDqNLR185k9/TCv7y9e/gqsyTkCfqKH2BeBVS1t5138+\nmjZSunNLBxefe+YJ33F+IobTl1w0UfoB6kuuGo2+hDkltRZYaGbzzCwGXAOsyjjmHuBSMysws1Lg\nQuA5Myszs3IAMysDLgc2hdjWITtt+qS07Rf3Nw1a50ePbEt7XkVVaSEfvnjuSDdtWGZVlnDb9Usp\nL0r/G+KLv3qaR17UsugiEmJguHsncCNwH/AccKe7bzazG8zshuCY54B7gWeANcAP3X0TUA08YmZP\nB+W/c/d7w2rrcCzMCIyt+wYOjCMtcX786MtpZZ947fy0cyFj7YwZk/n+dedTGO25tLcj4dzw8/Vs\n3t04hi0TkVwQ6m8rd18NrM4oW56xfTNwc0bZNpJTUzlr4fT0S2Bf2H8Md+/3PoofPbK99+ji1XPD\nbOKwXLxgKrf8xbl89pdPdZc1tXfykR+v5a5PXRzaXegikvt0p/cwza4qobiw55/vSEtHv0/f62t0\nsey1CyjLodFFqrcvnsnfveWMtLIDx9r50I+eYE+j7gYXyVcKjGGKRIxFM9KfavfUziN9HvvDh7fT\nlDK6mFIW47pXnxpq+07Ux18zn49dmn4S/+VDLVyz4s8KDZE8pcA4Aeedkn7/wpM7eq/62tAc58cZ\n910se+38nB1dpPrKm8/greekP+1vh0JDJG8pME7AeaemB8ZTu3qPMFY8vI3mlOXLx8Po4rhIxPj2\n+87lijOr08p3HGrhPd97XCvciuSZ3P8zN4dljjA21jfS1pGguDB5A9/uI62sfCR9dPHJ186nNDZ+\n/tkLoxG++/7zuPH2J7lv877u8vojrbz7e4+z4kPnc+H8k8awhSOjq8vZe7SNg03tNLV30tKeIOFO\nWayAsqIolaUxZlYWU1Qw+M2ZIhPV+PnNlYNOrihmVmVJ96qz8c4uHnzhAFcE6zPdfN8W2ju7uo+f\nOqmID42T0UWq46Hxmduf4t7Ne7vLG1s7eP8Pn+CmK1/Jx18zb1RW2h0J7s6OQy08svUgj287xIv7\njrHjUEvaZ9UXM5hZUcKpJ5Vy5szJnDO7ksWzK5kzpWTc9F3kRCgwTtAbz5jOTx7f0b39vxv3cMWZ\nJ/P0riPc/VT6je1fuPz0cTW6SJUMjVfxd/+ziTvW9iwRluhyvr76OR568QBff8fZnHJSbl52u/9Y\nG4+/dIhHtx7k0a2H0paWz5Z7cmRVf6SVx1461F0+dVIR559ayfmnVnH+qVWcNatiQo1E2joSHDjW\nTntngraOLl5sSFD00iFiBUYsGqW4MEJVWYyq0tiIPylScsv4/O2VQ9589oy0wHjg2X3sbWzjprs2\nph33ypPLee+ScB+OFLaCaIR/fdfZzKws4VsPvJC27+EXD/Kmbz/IRy6Zy0cvmUf15OKsXzfe2cW+\no200xztp6+iiuDBCWayAaeVFw27rsbYO1mw/zCNbD/LY1kNs2Rfe+ZaDTe3ct3lf95RdLBrh7NkV\n3QGSOXWZa9o6Euw+0squhlbqGlqoa2ilrqGVXYeTPx9s6mNhzSf+3KvIDKpKY5xUFuPkimJOnlzM\njIpiZlSWcHJFMTMrkt8nFxeMuxGZu9PU3smRlg4aWuI0tHRwpCXO0dYOOrucRJfT5U6XQ1FBhJLC\nKCWxKMWFUUoKo1SUFDKlLEZVWYyyWHTc9f84BcYJWjJ3CtPKizhwLPk/VXM8wUX/+odex33lLWdM\niL++zIzPvmEhZ8yYzN/cuYGjKQsptnd28f0Ht7Hyke1cctpUak6fxunV5VRXJMOjpT1BXUMLuxpa\n2Hm4hR2HWnj5UDP1Da30t5p6ZZFx2nOPccqUUuZMKeWUKaWcclIpU8pilMaiJLqc1niCvUfb2HGo\nhU31jWzYdYQX9h3r9zX7U1FSyKzKEiYVF1AWixKNGM3tCZrjnRw41s6exr7vs8kUT3SxfkcD61Ou\nmptealy6bwMLpk9iZmUxMypKmFxcSGksSmksSpdDR6KLzi6nM9FFPNFFR8LpSHTR0ZncjncmywCi\nkeRnETUjGjHMkqsPR82S5RFLO6Y53snBpjgHjrUHfekJhf3HsltpeTDucLg5zuHm+IBL5ZTFoskA\nqSxJhkqM7uh5AAAMsklEQVRlSTJYKoopLy6kqCBCUUGEWPAVMcMdHA++J3+Bpy6m3Of+oJy08p7j\nOhJdbDqYoOmZ3TS2dnR/HWlOhkJqODS2xrv/7U9ULBqhqqyQqtIYlaWFVJQUUlkSoyL4OfWrrKiA\nWDRCQdQojEaIRSMUFiQ/00xH2rroTHRREA3vWiYFxgmKRoyPXjKPb977fL/HvG3xTF6zcNootip8\nb1pUze8++xq+8N9P88T2w2n7OhJO7ZYD1G450E/t7B1pd9btaGBdH5csn4hYQYQL5lZx8YKpXDhv\nCgumTaKqbOCVeds6koH3/N5jPFPXyNO7jvBMXSOtWTxwan+Lc9dTmWtv5qfmeIKXDjTz0oHmsW5K\n0rqnBj9mBMUTXew72p71IxGG4qzzWnqtczeSFBgj4KOXzuWXa3ays49ne8+bWsb/fedZY9Cq8M2Z\nUsodyy7i10/W86+rn+NQc3ysm9SviMHZsyq45LSpXHLaVM4/tar7arZsFRdGOW16OadNL+et58wE\noDPRxfN7j7E+CLUndzQM6/xILosYTC8vprQoSnFBlPaWJk6aUklHMOppiSc43BynsVWP953oFBgj\noKggyvIPns/1t61J+6vhjBmT+fFHLqB8DB6ONFrMjPecP5u3nD2DXz1Zx8pHtrP94ND/cpxeXkRF\nSSHFhVHaOxM0tnaw/1g7J/IMpwXTyroD4qL5J1FRMvKfQ0E0wlmzKjhrVkX3ysN7Glt5cscR1u04\nzJM7Gti8+yidOfwEQzOYMbmY2VWlzK4qYfaU4HtVCXOqSjm5opjClGmO5DLar+71Oh2JLhqa4+w/\n1s6+o23sbmxjb2Mrexrb2HOkjb1H29h9pHXQq9FyVXFhJJhGilFVWhhMJ8WIRY1IMB0YiRjtHQla\nOxK0dnTRGk/QEu+ksbWDhuY4h5rj47b/oMAYMYtmTuaBz1/Gzx7fwaGmOK86pZIrzzo57X+0iawk\nFuVDF53KBy88hRf3N/Gn5/ezafdR6hpaONQUJ2LJaaAZFSXMmZL8RTR3ahlzTyrjlCmllMR6/7Xf\n3png7vseZObCs9l5uIVdh5PnPuoaWjnW1kFzPEFBxCgujDJ1UoxZlSUsmDaJxXMqOWd2BZWlY/Pw\npxkVJbzlnBLeEtwl3xpP8NPf1lI4fT67j7Syu7GVvY1tNLcnaOnopDWeIGLJOeqCqFEQCearC4I5\n62iEwoIIsahREIlgBl3uJLoITrSmnHTtgoQ7HpQlPDmfH4tGmFZelPyaVMT0yUXdATGjooRYwYn/\nd1oYjTB9cjHTJxdz1qyKPo9xd460dHT/G6SGyt7GNlriCeKdXbR3JrrP23Q5GMlgMyz4TveJY7O+\n9xlAxnbqcQVRI9HaxNyZ06koKWRycN6gsrTn/ELq96GOSPvTGk9wuCVOQ3PyPEljawdHWuM951Fa\nes6ntHYkgvNYwfmsruTPnV1O5mmMeDwe+nlSBcYImlxcyKdH+GFI442ZcXp1OadXlw9+8CCKCqKc\nXBbhtaeP7/M/JbEor5gSpSZjba58ZGbJS3DLYpw5s+9QGU3J0dL5o/qeJbEos2IlzKosGdHXra2t\nZd7UshF9zUz58eeviIicMAWGiIhkRYEhIiJZCTUwzOxKM9tiZlvN7KZ+jqkxsw1mttnMHhxKXRER\nGT2hnfQ2syhwK/AmoA5Ya2ar3P3ZlGMqgf8ErnT3nWY2Pdu6IiIyusIcYSwFtrr7NnePA3cAV2cc\n837gLnffCeDu+4dQV0RERlGYgTEL2JWyXReUpTodqDKzWjNbb2bXDaGuiIiMorG+D6MAOB94A1AC\nPG5mvZfBHICZLQOWBZtNZrZlmG2ZChwcZt1co77knonSD1BfctVw+5L1Q3rCDIx6IHU979lBWao6\n4JC7NwPNZvYQsDgoH6wuAO6+Alhxoo01s3XuvuREXycXqC+5Z6L0A9SXXDUafQlzSmotsNDM5plZ\nDLgGWJVxzD3ApWZWYGalwIXAc1nWFRGRURTaCMPdO83sRuA+IAqsdPfNZnZDsH+5uz9nZvcCzwBd\nwA/dfRNAX3XDaquIiAwu1HMY7r4aWJ1Rtjxj+2bg5mzqhuyEp7VyiPqSeyZKP0B9yVWh98X8RNaP\nFhGRvKGlQUREJCt5HxjjfQkSM3vZzDYGy6usC8qmmNkDZvZi8L1qrNvZFzNbaWb7zWxTSlm/bTez\nLwWf0xYzu2JsWt23fvryNTOrDz6bDWb25pR9udyXOWb2JzN7Nliy53NB+bj6bAbox7j7XMys2MzW\nmNnTQV/+MSgf3c/Eg4et5OMXyRPqLwHzgRjwNLBorNs1xD68DEzNKPs34Kbg55uAb451O/tp+2uB\n84BNg7UdWBR8PkXAvOBzi451Hwbpy9eAL/RxbK73ZQZwXvBzOfBC0OZx9dkM0I9x97mQfA7UpODn\nQuAJ4KLR/kzyfYQxUZcguRr4SfDzT4B3jGFb+uXuDwGHM4r7a/vVwB3u3u7u24GtJD+/nNBPX/qT\n633Z4+5PBj8fI3mp+yzG2WczQD/6k5P9APCkpmCzMPhyRvkzyffAmAhLkDjw+2BpleN3vFe7+57g\n571A9dg0bVj6a/t4/aw+Y2bPBFNWx6cLxk1fzGwu8CqSf9GO288mox8wDj8XM4ua2QZgP/CAu4/6\nZ5LvgTERXOru5wJXAZ82s9em7vTk+HRcXgo3ntse+B7J6c5zgT3ALWPbnKExs0nAr4G/cvejqfvG\n02fTRz/G5efi7ong//XZwFIzOytjf+ifSb4HRjbLl+Q0d68Pvu8H7iY57NxnZjMAgu/7+3+FnNNf\n28fdZ+Xu+4L/ybuAH9AzJZDzfTGzQpK/ZH/h7ncFxePus+mrH+P5cwFw9yPAn4ArGeXPJN8DY1wv\nQWJmZWZWfvxn4HJgE8k+fDg47MMkl2AZL/pr+yrgGjMrMrN5wEJgzRi0L2vH/0cOvJPkZwM53hcz\nM+BHwHPu/q2UXePqs+mvH+PxczGzaZZ8fhBmVkLyWUHPM9qfyVif/R/rL+DNJK+eeAn4yli3Z4ht\nn0/ySoingc3H2w+cBPwBeBH4PTBlrNvaT/t/SXJKoIPkHOvHBmo78JXgc9oCXDXW7c+iLz8DNpJc\n+mYVMGOc9OVSklMbzwAbgq83j7fPZoB+jLvPBTgHeCpo8ybgH4LyUf1MdKe3iIhkJd+npEREJEsK\nDBERyYoCQ0REsqLAEBGRrCgwREQkKwoMmVDMzM3slpTtL5jZ10botW8zs/eMxGsN8j7vNbPnzOxP\nWR7/5bDbJAIKDJl42oF3mdnUsW5IKjMbytMtPwZ8wt1fl+XxCgwZFQoMmWg6ST6q8q8zd2SOEMys\nKfheY2YPmtk9ZrbNzL5hZh8Inj+w0cwWpLzMG81snZm9YGZvDepHzexmM1sbLGj3yZTXfdjMVgHP\n9tGea4PX32Rm3wzK/oHkDWc/MrObM46fYWYPBc9w2GRmrzGzbwAlQdkvguM+GLR9g5l938yix/tr\nZt8OnqfwBzObdgL/zpKHFBgyEd0KfMDMKoZQZzFwA3AG8CHgdHdfCvwQ+EzKcXNJrj30FmC5mRWT\nHBE0uvsFwAXAJ4LlGCD5jIzPufvpqW9mZjOBbwKvJ7kI3gVm9g53/ydgHfABd/9iRhvfD9znyQXo\nFgMb3P0moNXdz3X3D5jZGcD7gEuC4xLAB4L6ZcA6dz8TeBD46hD+fUQYyjBZZFxw96Nm9lPgs0Br\nltXWerBMtJm9BNwflG8EUqeG7vTkonUvmtk24JUk1/A6J2X0UkFy7Z44sMaTzyPIdAFQ6+4Hgvf8\nBcmHMP3PQG0EVgYL6v2Pu2/o45g3AOcDa5NLKVFCz4J0XcB/BT//HLirV22RASgwZKL6d+BJ4Mcp\nZZ0Eo2ozi5B8yuJx7Sk/d6Vsd5H+/0nmWjpO8mlon3H3+1J3mFkN0Dy85vfm7g8Fy9e/BbjNzL7l\n7j/NOMyAn7j7l7J5yZFqm+QHTUnJhOTuh4E7SU4XHfcyyb++Ad5O8qllQ/VeM4sE5zXmk1zY7T7g\nU8Ff/pjZ6cHqwQNZA1xmZlODcwzXkpwm6peZnQrsc/cfkJwqOy/Y1XH8vUkuRPceM5se1JkS1IPk\n/+/HR0HvBx7JrssiSRphyER2C3BjyvYPgHvM7GngXob31/9Okr/sJwM3uHubmf2Q5LmNJ4MltQ8w\nyGNx3X2Pmd1E8rkGBvzO3Qdbhr4G+KKZdQBNwHVB+QrgGTN7MjiP8XfA/cEoqgP4NLCDZH+XBvv3\nkzzXIZI1rVYrkifMrMndJ411O2T80pSUiIhkRSMMERHJikYYIiKSFQWGiIhkRYEhIiJZUWCIiEhW\nFBgiIpIVBYaIiGTl/wMAvHrOp1Kh4gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266a1878470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_history_FISTA, lw=4)\n",
    "plt.grid()\n",
    "#plt.yscale('log')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Number of step')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id='section3'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# SVRG "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "L'algorithme SVRG est aussi proposé dans le cours comme une accélération de la descente de gradient. Implémentons-le :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def SVRG(T, m, lam, eta):\n",
    "    # on déclenche le timer\n",
    "    t0 = time.time()\n",
    "    \n",
    "    # initialisation\n",
    "    theta_hat = np.zeros(n_features)\n",
    "    \n",
    "    # historique des fonctions de perte\n",
    "    cost_history = np.empty(shape=[T])\n",
    "    time_history = np.empty(shape=[T,2])\n",
    "    \n",
    "    for t in range(0,T): \n",
    "        cost_history[t] = f(theta_hat, lam)\n",
    "        time_history[t] = (time.time()-t0, f(theta_hat, lam))\n",
    "        \n",
    "        # vrai gradient\n",
    "        g_hat = grad(theta_hat)\n",
    "    \n",
    "        theta = np.zeros([m+1, n_features])\n",
    "        theta[0,:] = theta_hat\n",
    "        for k in range(0,m):\n",
    "            # random sample an indice\n",
    "            idx = np.random.randint(0, n_samples) \n",
    "\n",
    "            # compute variance reduced gradient\n",
    "            g = partial_grad(theta[k], idx) - partial_grad(theta_hat, idx) + g_hat\n",
    "\n",
    "            # compute prox\n",
    "            theta[k+1] = theta[k] - eta * g - eta * lam * np.sign(theta[k])\n",
    "        \n",
    "        # on retire la première ligne\n",
    "        theta = theta[1:]\n",
    "        \n",
    "        theta_hat = np.mean(theta, axis=0)\n",
    "        \n",
    "    return theta_hat, cost_history, time_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "theta, cost_history_SVRG, time_history_SVRG = SVRG(T=T, m=20, lam=lam, eta=eta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+cXXV95/HXOzNJCCHkh4QpkCyJNFSiBYpjUJfqIAWD\nrUZZ2wXt4rK22fgQddstj0a7D6u2+yiWh3bdio3RIuKqrC1S4poHEdRLtGJJwJBfEBgGJBPQAEHC\nBPJjJp/9454kZ869M3NmMufeO/e+n4/HPHLP95zvud/PHMIn3+/3nO9RRGBmZjaSSfVugJmZTQxO\nGGZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVku7fVuwHg65ZRTYsGC\nBWOqu2/fPqZPnz6+DaoTx9J4miUOcCyNaqyx3H///c9GxNw8xzZVwliwYAEbN24cU91SqURXV9f4\nNqhOHEvjaZY4wLE0qrHGIunneY/1kJSZmeXihGFmZrk4YZiZWS5OGGZmlosThpmZ5VJYwpB0k6Td\nkrYOsV+S/rekbkmbJV2Q2rdU0o5k38qi2mhmZvkVeVvtzcDngVuG2H85sCj5uRD4B+BCSW3AjcCl\nQC+wQdKaiNg+3g3c8YsX+eR3ttF/ONjz/Mus27OZv7ni3PH+GjOzplBYwoiI9ZIWDHPIMuCWKL8j\n9qeSZkk6DVgAdEdED4CkW5Njxz1h9B3o5yePPXd0+6SnXxzvrzAzaxr1fHDvDGBnars3KatWfuFQ\nJ5G0HFgO0NHRQalUyt2AnhcGBm2/8MLeUdVvVH19fU0RBzRPLM0SBziWRlWLWCb8k94RsRpYDdDZ\n2RmjedLxlF0vwL0/Pro9bfpJdHX99ng3seb89GrjaZY4wLE0qlrEUs+EsQuYn9qel5RNHqJ83LW3\nadD2wOEo4mvMzJpCPW+rXQNcndwt9XrghYh4GtgALJK0UNIU4Mrk2HHXpsEJo//w4SK+xsysKRTW\nw5D0TaALOEVSL/CXlHsPRMQqYC3wNqAbeAm4JtnXL+laYB3QBtwUEduKaGPbJPcwzMzyKvIuqatG\n2B/AB4fYt5ZyQilU+6TBHayBcMIwMxtKSz/p3ZadwxhwwjAzG0pLJ4z2Sdk5DCcMM7OhtHTC8ByG\nmVl+rZ0wKu6ScsIwMxtKaycMP4dhZpZbSyeM7ByGE4aZ2dBaOmF4DsPMLL+WThjZ5zD8pLeZ2dBa\nOmFkOhgcDjjsXoaZWVUtnTAkVQ5L+WlvM7OqWjphgOcxzMzyavmE4TulzMzyafmEke1h+OE9M7Pq\nWj5huIdhZpZPyyeMyh6Gb601M6vGCcM9DDOzXFo+YVS8RMkJw8ysqkIThqSlknZI6pa0ssr+2ZJu\nl7RZ0n2SXpPa94SkLZI2SdpYVBvdwzAzy6fId3q3ATcClwK9wAZJayJie+qwjwGbIuJdkl6VHH9J\nav/FEfFsUW0Ev0TJzCyvInsYS4DuiOiJiIPArcCyzDGLgR8ARMTDwAJJHQW2qYJ7GGZm+RTWwwDO\nAHamtnuBCzPHPAhcAfxI0hLgTGAe8EsggLslDQBfjIjV1b5E0nJgOUBHRwelUmlUjXz5pZcGbf/0\n3+7jqZPbRnWORtPX1zfq30OjapZYmiUOcCyNqhaxFJkw8rge+JykTcAW4GfAQLLvoojYJelU4C5J\nD0fE+uwJkkSyGqCzszO6urpG1YCZm39Eb9/eo9u/dUEnvzlv5lhiaRilUonR/h4aVbPE0ixxgGNp\nVLWIpciEsQuYn9qel5QdFRF7gWsAJAl4HOhJ9u1K/twt6XbKQ1wVCeN4VTy458UHzcyqKnIOYwOw\nSNJCSVOAK4E16QMkzUr2AfwRsD4i9kqaLmlGcsx04DJgaxGNrJzD8IN7ZmbVFNbDiIh+SdcC64A2\n4KaI2CZpRbJ/FXAO8FVJAWwD3p9U7wBuL3c6aAe+ERF3FtHOipcoDbiHYWZWTaFzGBGxFlibKVuV\n+nwvcHaVej3AeUW27YhMvvBdUmZmQ/CT3hWvaXXCMDOrpuUThp/DMDPLp+UThpc3NzPLp+UThl+g\nZGaWT8snjPY29zDMzPJo+YQxSX6BkplZHi2fMDyHYWaWT8snjDa/QMnMLJeWTxjuYZiZ5dPyCaOt\nzXdJmZnl4YQh9zDMzPJwwvBzGGZmubR8wqicw/BttWZm1bR8wsjOYQw4X5iZVdXyCcM9DDOzfFo+\nYWSfw/AchplZdYUmDElLJe2Q1C1pZZX9syXdLmmzpPskvSZv3fHiu6TMzPIpLGFIagNuBC4HFgNX\nSVqcOexjwKaIOBe4GvjcKOqOi+zig+5hmJlVV2QPYwnQHRE9EXEQuBVYljlmMfADgIh4GFggqSNn\n3XHhFyiZmeVTZMI4A9iZ2u5NytIeBK4AkLQEOBOYl7PuuPDSIGZm+bTX+fuvBz4naROwBfgZMDCa\nE0haDiwH6OjooFQqjaoBjz9xaND2z5/cSam0e1TnaDR9fX2j/j00qmaJpVniAMfSqGoRS5EJYxcw\nP7U9Lyk7KiL2AtcASBLwONADTBupbuocq4HVAJ2dndHV1TWqRu689wl4eNvR7V87/XS6un5zVOdo\nNKVSidH+HhpVs8TSLHGAY2lUtYilyCGpDcAiSQslTQGuBNakD5A0K9kH8EfA+iSJjFh3vEzykJSZ\nWS6F9TAiol/StcA6oA24KSK2SVqR7F8FnAN8VVIA24D3D1e3iHZm5zD6B5wwzMyqKXQOIyLWAmsz\nZatSn+8Fzs5btwgVL1AKJwwzs2pa/klv3yVlZpZPyycML29uZpaPE0a2h+E5DDOzqpww3MMwM8ul\n5ROGlzc3M8un5RNGxZCUOxhmZlW1fMJoz95W6x6GmVlVLZ8wKuYw3MUwM6vKCcPPYZiZ5eKE4buk\nzMxyafmE4Se9zczyafmE4R6GmVk+LZ8wprQP/hX0D/guKTOzalo+YUxuG/wrOOSEYWZWlRNG2+Ah\nqUO+rdbMrKqWTxhTMj2Mg+5hmJlV1fIJw0NSZmb5FJowJC2VtENSt6SVVfbPlPQdSQ9K2ibpmtS+\nJyRtkbRJ0sai2jg5M+l9qN8Jw8ysmsJe0SqpDbgRuBToBTZIWhMR21OHfRDYHhFvlzQX2CHp6xFx\nMNl/cUQ8W1QbofI5DM9hmJlVV2QPYwnQHRE9SQK4FViWOSaAGZIEnATsAfoLbFOF7JDUwYHDhN/r\nbWZWociEcQawM7Xdm5SlfR44B3gK2AJ8JCKOjAkFcLek+yUtL6qRbZPkh/fMzHIobEgqp7cCm4C3\nAGcBd0n6UUTsBS6KiF2STk3KH46I9dkTJMlkOUBHRwelUmnUjWgjGEht/+CH9zC1XUMe3+j6+vrG\n9HtoRM0SS7PEAY6lUdUiliITxi5gfmp7XlKWdg1wfZTHgLolPQ68CrgvInYBRMRuSbdTHuKqSBgR\nsRpYDdDZ2RldXV2jbujU0joO7j82Evb6N17EzBMnj/o8jaJUKjGW30MjapZYmiUOcCyNqhaxFDkk\ntQFYJGmhpCnAlcCazDFPApcASOoAfgPokTRd0oykfDpwGbC1qIb6WQwzs5EV1sOIiH5J1wLrgDbg\npojYJmlFsn8V8FfAzZK2AAL+PCKelfRK4PbyXDjtwDci4s6i2upnMczMRlboHEZErAXWZspWpT4/\nRbn3kK3XA5xXZNvSJrdnb611wjAzy2r5J73BPQwzszycMKgyh9Hv22rNzLKcMHAPw8wsDycMqi1x\n7oRhZpblhEH15UHMzGwwJwwqX9PqBQjNzCo5YVBlDsNLnJuZVciVMCSdJWlq8rlL0oclzSq2abXj\nOQwzs5Hl7WHcBgxI+nXK6zbNB75RWKtqzHMYZmYjy5swDkdEP/Au4O8j4jrgtOKaVVvZ5zA8h2Fm\nVilvwjgk6SrgfcD/S8om7nKuGX4Ow8xsZHkTxjXAG4D/GRGPS1oIfK24ZtWW15IyMxtZrsUHk/dw\nfxhA0mxgRkR8usiG1VLFHIbvkjIzq5D3LqmSpJMlzQEeAL4k6bPFNq12PIdhZjayvENSM5PXpl4B\n3BIRFwK/U1yzastzGGZmI8ubMNolnQb8AccmvZtGNmH0O2GYmVXImzA+RfnNeY9FxIbkjXiPFtes\n2spOeh/0kJSZWYVcCSMi/ikizo2IDyTbPRHxH0aqJ2mppB2SuiWtrLJ/pqTvSHpQ0jZJ1+StO54q\n5zDcwzAzy8o76T1P0u2Sdic/t0maN0KdNuBG4HJgMXCVpMWZwz4IbI+I84Au4DOSpuSsO248h2Fm\nNrK8Q1JfAdYApyc/30nKhrME6E56IweBW4FlmWMCmCFJwEnAHqA/Z91x44RhZjayvAljbkR8JSL6\nk5+bgbkj1DkD2Jna7k3K0j4PnAM8BWwBPhIRh3PWHTfZxQf9ilYzs0q5HtwDnpP0h8A3k+2rgOfG\n4fvfCmwC3gKcBdwl6UejOYGk5cBygI6ODkql0qgb0f1U/6DtXU8/Tan0/KjP0yj6+vrG9HtoRM0S\nS7PEAY6lUdUilrwJ478Afw/8HeVhpJ8A/3mEOrsor2p7xLykLO0a4PqICKBb0uPAq3LWBSAiVlNe\nQZfOzs7o6uoaOZqMl7Y8DZsfOLo9+xVz6ep67ajP0yhKpRJj+T00omaJpVniAMfSqGoRS967pH4e\nEe+IiLkRcWpEvBMY6S6pDcAiSQslTQGupDwPkvYkcAmApA7gN4CenHXHjecwzMxGdjxv3PvT4XYm\ny6FfS/n5jYeAb0XENkkrJK1IDvsr4I2StgDfB/48Ip4dqu5xtHVYFXMYfg7DzKxC3iGpajTSARGx\nFlibKVuV+vwUcFneukWpeA7Diw+amVU4nh5G0/wzfHK7h6TMzEYybA9D0otUTwwCphXSojpon+T3\nYZiZjWTYhBERM2rVkHqqfKd303SezMzGzfEMSTWNqZkhqQP9A3VqiZlZ43LCAE6Y3DZo+8AhD0mZ\nmWU5YeAehplZHk4YwFT3MMzMRuSEQWUPY797GGZmFZwwKCeM9I21hwaCgcO+U8rMLM0JA5DE5Mxv\nwvMYZmaDOWEkMtMY7Pc8hpnZIE4YicmZp73dwzAzG8wJIzHFPQwzs2E5YSSycxj7D7mHYWaW5oSR\nmFIxJOUehplZmhNGonLS2z0MM7M0J4yEh6TMzIZXaMKQtFTSDkndklZW2X+dpE3Jz1ZJA5LmJPue\nkLQl2bexyHYCTGnzkJSZ2XCO5xWtw5LUBtwIXAr0AhskrYmI7UeOiYgbgBuS498O/ElE7Emd5uKI\neLaoNqa5h2FmNrwiexhLgO6I6ImIg8CtwLJhjr8K+GaB7RlW5XMY7mGYmaUVmTDOAHamtnuTsgqS\nTgSWAreligO4W9L9kpYX1spE9jmMA+5hmJkNUtiQ1Ci9HfjXzHDURRGxS9KpwF2SHo6I9dmKSTJZ\nDtDR0UGpVBpbCwYOQWoJwq0PP0rpwBNjO1ed9fX1jf330GCaJZZmiQMcS6OqRSxFJoxdwPzU9ryk\nrJoryQxHRcSu5M/dkm6nPMRVkTAiYjWwGqCzszO6urrG1Nh/fuR7wKGj2/PPXEBX16IxnaveSqUS\nY/09NJpmiaVZ4gDH0qhqEUuRQ1IbgEWSFkqaQjkprMkeJGkm8GbgjlTZdEkzjnwGLgO2FtjWKpPe\nnsMwM0srrIcREf2SrgXWAW3ATRGxTdKKZP+q5NB3Ad+LiH2p6h3A7ZKOtPEbEXFnUW2Fygf3vPig\nmdlghc5hRMRaYG2mbFVm+2bg5kxZD3BekW3Lyi4N4h6GmdlgftI74ecwzMyG54SRmOwnvc3MhuWE\nkZjiHoaZ2bCcMBKVk97uYZiZpTlhJConvd3DMDNLc8JIVLwPwz0MM7NBnDAS2bukvJaUmdlgThiJ\nqZm7pF52wjAzG8QJI3FCZkhq34H++jTEzKxBOWEkTmgf3MPoc8IwMxvECSMxtQ2Uyhn7Dx3m0IAn\nvs3MjnDCSEjipKmDl9bysJSZ2TFOGCnZhOFhKTOzY5wwUpwwzMyG5oSRctIJHpIyMxuKE0ZKtofx\n4n4nDDOzI5wwUjwkZWY2tEIThqSlknZI6pa0ssr+6yRtSn62ShqQNCdP3SL4Likzs6EVljAktQE3\nApcDi4GrJC1OHxMRN0TE+RFxPvBR4J6I2JOnbhGme0jKzGxIRfYwlgDdEdETEQeBW4Flwxx/FfDN\nMdYdFzNO8JCUmdlQikwYZwA7U9u9SVkFSScCS4HbRlt3PGV7GB6SMjM7pn3kQ2ri7cC/RsSe0VaU\ntBxYDtDR0UGpVBpTA/r6+nhqT8+gskce30mptHtM56unvr6+Mf8eGk2zxNIscYBjaVS1iKXIhLEL\nmJ/anpeUVXMlx4ajRlU3IlYDqwE6Ozujq6trTI0tlUq8dt4ibtm+6WjZjDlz6eq6YEznq6dSqcRY\nfw+NplliaZY4wLE0qlrEUuSQ1AZgkaSFkqZQTgprsgdJmgm8GbhjtHXH2/QpHpIyMxtKYT2MiOiX\ndC2wDmgDboqIbZJWJPtXJYe+C/heROwbqW5RbT0i+6S3J73NzI4pdA4jItYCazNlqzLbNwM356lb\nND/pbWY2ND/pnZK9rdYJw8zsGCeMlFnTpgza/tVLB+vUEjOzxuOEkTLjhHbaJh177d6+gwMc6B+o\nY4vMzBqHE0bKpEli9omTB5X96qVDdWqNmVljccLImHXi4GGpPfs8LGVmBk4YFeZkEsbzThhmZoAT\nRoVZmSGp5z0kZWYGOGFUmDM9MyTlO6XMzAAnjAqzp3tIysysGieMjOxdUs+7h2FmBjhhVJjtSW8z\ns6qcMDKyCWOPJ73NzAAnjArZOQwvD2JmVuaEkZG9S+q5PicMMzNwwqhw6oypg7Z/sXc//QOH69Qa\nM7PG4YSRMX1qO69I9TIGDge/2Lu/ji0yM2sMThhVzJs9bdB27/Mv16klZmaNo9CEIWmppB2SuiWt\nHOKYLkmbJG2TdE+q/AlJW5J9G4tsZ9a82ScO2nbCMDMr8BWtktqAG4FLgV5gg6Q1EbE9dcws4AvA\n0oh4UtKpmdNcHBHPFtXGocybM7iHsXPPS7VugplZwymyh7EE6I6Inog4CNwKLMsc8x7g2xHxJEBE\n7C6wPbm5h2FmVqnIhHEGsDO13ZuUpZ0NzJZUknS/pKtT+wK4OylfXmA7K1TOYbiHYWZW2JDUKL7/\ntcAlwDTgXkk/jYhHgIsiYlcyTHWXpIcjYn32BEkyWQ7Q0dFBqVQaU0P6+vqO1n26b/BttI8+/fyY\nz1sP6VgmumaJpVniAMfSqGoRS5EJYxcwP7U9LylL6wWei4h9wD5J64HzgEciYheUh6kk3U55iKsi\nYUTEamA1QGdnZ3R1dY2psaVSiSN1D/QP8PGfrKP/cACwZ39w/pI3VryNr1GlY5nomiWWZokDHEuj\nqkUsRQ5JbQAWSVooaQpwJbAmc8wdwEWS2iWdCFwIPCRpuqQZAJKmA5cBWwts6yBT29v49VNPGlS2\n7am9tfp6M7OGVFjCiIh+4FpgHfAQ8K2I2CZphaQVyTEPAXcCm4H7gC9HxFagA/ixpAeT8u9GxJ1F\ntbWaV58+c9D2tqdeqOXXm5k1nELnMCJiLbA2U7Yqs30DcEOmrIfy0FTdvPr0k7ntgWPb7mGYWavz\nk95DePXpJw/a3rLLPQwza21OGENYfPrJTNKx7Z5n9vn2WjNraU4YQ5hxwmQ6z5wzqOwHDzfEc4Vm\nZnXhhDGMS84ZvFLJXdt/WaeWmJnVnxPGMLIJ48fdz9LzTF+dWmNmVl9OGMM4a+5JLD7t2OR3BHyh\n9FgdW2RmVj9OGMOQxIquswaV/fP9vdzzyDN1apGZWf04YYzgba/5NV55yvRBZSu+dj+33d/LIb+6\n1cxaSL0XH2x47W2T+Nt3n8sffPFekqWlePnQAP/9nx7kr7+7nUsXd3DOaSfz7+acyMxpkzl52mSm\nTW5DKvdQBOXPCAkORxBRXor3cHLCiKQciIjkeyIphzj6ufwnHPt8rF65bgA79gxwQs9zyfck35f6\nPFpjqEKM5YuqfNeWZ/qJHSPcnTa2r6qpzc/0E01yl51jyS9q+B/n5t39vO5AP9OnFve/dSeMHDoX\nzOG6t76KT9/58KDy5186xLc29tapVSO476f1bsH4uX9DvVswPpolDnAsDer3uvZXrIM3njwkldMH\nus7iU8tezZQ2/8rMrDX5/36jcPUbFnDXn76Jq5bMZ+a0yfVujplZTXlIapTOfMV0/uaKc/nkO17D\nT3ueY9tTe+ne3cczfQfY+/Ih9u4/xIFDh4/OJwyaRwAmJfMZk5I5DoBJk47NcUxK5j1IfT5SDuU6\n5brpY5OyZP+Le19g1sxZyTmOnTt9nlrQGL8rXWvPnj3MmTNnyGOPfdeYvqpm9jy3hzmvGDmOicCx\njE6t/tN8bs8eTpzSVuh3OGGM0ZT2Sbzp7Lm86ey59W5KhfKLVN5Q72aMi3IsS+rdjOPWLHGAY2lU\npVKJ02dNG/nA4+AhKTMzy8UJw8zMcnHCMDOzXApNGJKWStohqVvSyiGO6ZK0SdI2SfeMpq6ZmdVO\nYZPektqAG4FLgV5gg6Q1EbE9dcws4AvA0oh4UtKpeeuamVltFdnDWAJ0R0RPRBwEbgWWZY55D/Dt\niHgSICJ2j6KumZnVUJEJ4wxgZ2q7NylLOxuYLakk6X5JV4+irpmZ1VC9n8NoB14LXAJMA+6VNKpF\nkCQtB5Ynm32SdoyxLacAz46xbqNxLI2nWeIAx9KoxhrLmXkPLDJh7ALmp7bnJWVpvcBzEbEP2Cdp\nPXBeUj5SXQAiYjWw+ngbK2ljRHQe73kagWNpPM0SBziWRlWLWIocktoALJK0UNIU4EpgTeaYO4CL\nJLVLOhG4EHgoZ10zM6uhwnoYEdEv6VpgHdAG3BQR2yStSPavioiHJN0JbAYOA1+OiK0A1eoW1VYz\nMxtZoXMYEbEWWJspW5XZvgG4IU/dgh33sFYDcSyNp1niAMfSqAqPRWN9M5qZmbUWLw1iZma5tHzC\nmOhLkEh6QtKWZHmVjUnZHEl3SXo0+XN2vdtZjaSbJO2WtDVVNmTbJX00uU47JL21Pq2ubohYPiFp\nV3JtNkl6W2pfI8cyX9IPJW1Pluz5SFI+oa7NMHFMuOsi6QRJ90l6MInlk0l5ba9JRLTsD+UJ9ceA\nVwJTgAeBxfVu1yhjeAI4JVP2t8DK5PNK4NP1bucQbX8TcAGwdaS2A4uT6zMVWJhct7Z6xzBCLJ8A\n/qzKsY0ey2nABcnnGcAjSZsn1LUZJo4Jd10ov4fppOTzZODfgNfX+pq0eg+jWZcgWQZ8Nfn8VeCd\ndWzLkCJiPbAnUzxU25cBt0bEgYh4HOimfP0awhCxDKXRY3k6Ih5IPr9I+Vb3M5hg12aYOIbSkHEA\nRFlfsjk5+QlqfE1aPWE0wxIkAdydLK1y5In3joh4Ovn8C6CjPk0bk6HaPlGv1YckbU6GrI4MF0yY\nWCQtAH6L8r9oJ+y1ycQBE/C6SGqTtAnYDdwVETW/Jq2eMJrBRRFxPnA58EFJb0rvjHL/dELeCjeR\n2574B8rDnecDTwOfqW9zRkfSScBtwH+LiL3pfRPp2lSJY0Jel4gYSP6uzwOWSHpNZn/h16TVE0ae\n5UsaWkTsSv7cDdxOudv5S0mnASR/7h76DA1nqLZPuGsVEb9M/pIfBr7EsSGBho9F0mTK/5P9ekR8\nOymecNemWhwT+boARMSvgB8CS6nxNWn1hDGhlyCRNF3SjCOfgcuArZRjeF9y2PsoL8EyUQzV9jXA\nlZKmSloILALuq0P7cjvyFznxLsrXBho8FkkC/hF4KCI+m9o1oa7NUHFMxOsiaa7K7w9C0jTK7wp6\nmFpfk3rP/tf7B3gb5bsnHgP+ot7tGWXbX0n5TogHgW1H2g+8Avg+8ChwNzCn3m0dov3fpDwkcIjy\nGOv7h2s78BfJddoBXF7v9ueI5WvAFspL36wBTpsgsVxEeWhjM7Ap+XnbRLs2w8Qx4a4LcC7ws6TN\nW4GPJ+U1vSZ+0tvMzHJp9SEpMzPLyQnDzMxyccIwM7NcnDDMzCwXJwwzM8vFCcOaiqSQ9JnU9p9J\n+sQ4nftmSe8ej3ON8D2/L+khST/MefzHim6TGThhWPM5AFwh6ZR6NyRN0mjebvl+4I8j4uKcxzth\nWE04YViz6af8qso/ye7I9hAk9SV/dkm6R9IdknokXS/pvcn7B7ZIOit1mt+RtFHSI5J+L6nfJukG\nSRuSBe3+a+q8P5K0BthepT1XJeffKunTSdnHKT9w9o+Sbsgcf5qk9ck7HLZK+m1J1wPTkrKvJ8f9\nYdL2TZK+KKntSLyS/i55n8L3Jc09jt+ztSAnDGtGNwLvlTRzFHXOA1YA5wD/CTg7IpYAXwY+lDpu\nAeW1h34XWCXpBMo9ghci4nXA64A/TpZjgPI7Mj4SEWenv0zS6cCngbdQXgTvdZLeGRGfAjYC742I\n6zJtfA+wLsoL0J0HbIqIlcDLEXF+RLxX0jnAfwT+fXLcAPDepP50YGNEvBq4B/jLUfx+zBhNN9ls\nQoiIvZJuAT4MvJyz2oZIlomW9BjwvaR8C5AeGvpWlBete1RSD/Aqymt4nZvqvcykvHbPQeC+KL+P\nIOt1QCkinkm+8+uUX8L0L8O1EbgpWVDvXyJiU5VjLgFeC2woL6XENI4tSHcY+L/J5/8DfLuittkw\nnDCsWf0v4AHgK6myfpJetaRJlN+yeMSB1OfDqe3DDP57kl1LJyi/De1DEbEuvUNSF7BvbM2vFBHr\nk+Xrfxe4WdJnI+KWzGECvhoRH81zyvFqm7UGD0lZU4qIPcC3KA8XHfEE5X99A7yD8lvLRuv3JU1K\n5jVeSXlht3XAB5J/+SPp7GT14OHcB7xZ0inJHMNVlIeJhiTpTOCXEfElykNlFyS7Dh35bsoL0b1b\n0qlJnTlJPSj/fT/SC3oP8ON8IZuVuYdhzewzwLWp7S8Bd0h6ELiTsf3r/0nK/7M/GVgREfslfZny\n3MYDyZLazzDCa3Ej4mlJKym/10DAdyNipGXou4DrJB0C+oCrk/LVwGZJDyTzGP8D+F7SizoEfBD4\nOeV4lySeoZTxAAAATElEQVT7d1Oe6zDLzavVmrUISX0RcVK922ETl4ekzMwsF/cwzMwsF/cwzMws\nFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwzM8vl/wO0uXPFD3zcLgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266a1960e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_history_SVRG, lw=4)\n",
    "plt.grid()\n",
    "#plt.yscale('log')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Number of step')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id='section4'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# SAGA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Pour terminer, SAGA est aussi proposé comme une accélération, le voici :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def SAGA(T, lam, eta):\n",
    "    # on déclenche le timer\n",
    "    t0 = time.time()\n",
    "    \n",
    "    # initialisation\n",
    "    theta = np.zeros(n_features)\n",
    "    \n",
    "    gi_hat = np.zeros([n_samples, n_features])\n",
    "    g_bar = np.zeros(n_features)\n",
    "    \n",
    "    # historique des fonctions de perte\n",
    "    cost_history = np.empty(shape=[T])\n",
    "    time_history = np.empty(shape=[T,2])\n",
    "    \n",
    "    for t in range(0,T): \n",
    "        cost_history[t] = f(theta, lam)\n",
    "        time_history[t] = (time.time()-t0, f(theta, lam))\n",
    "        \n",
    "        # random sample an indice\n",
    "        idx = np.random.randint(0, n_samples) \n",
    "        \n",
    "        # compute partial gradient\n",
    "        gi = partial_grad(theta, idx)\n",
    "        \n",
    "        # estimate gradient\n",
    "        g_hat = gi - gi_hat[idx] + g_bar\n",
    "        \n",
    "        # update average gradient\n",
    "        g_bar = g_bar + (1./n_samples) * (gi - gi_hat[idx])\n",
    "        \n",
    "        # update stored gradient\n",
    "        gi_hat[idx] = gi\n",
    "        \n",
    "        # compute prox\n",
    "        theta = theta - eta * g_hat - eta * lam * np.sign(theta)\n",
    "                          \n",
    "    return theta, cost_history, time_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "theta, cost_history_SAGA, time_history_SAGA = SAGA(T=T, lam=lam, eta=eta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8XVd95/3PV0eSZcvyLbYV3xKb4EBM7jhOoIEIUlIH\nWtIwdBqTQp8U6iczhDJPW54JdJ4ZBqYzUOgFSmhqaBpgKIY+JCQtbkKACAcIxE6wYzuxE8dxfI3v\nN8mydTm/+eNs2+ccXXwse+scHX3fr5deOnvttc75Le2X9NNae++1FRGYmZmdTk25AzAzs+HBCcPM\nzErihGFmZiVxwjAzs5I4YZiZWUmcMMzMrCROGGZmVhInDDMzK4kThpmZlaS23AGcS5MnT47Zs2cP\nqm17ezuNjY3nNqAycV8qT7X0A9yXSjXYvjz99NN7I2JKKXWrKmHMnj2blStXDqpta2srLS0t5zag\nMnFfKk+19APcl0o12L5IeqXUup6SMjOzkjhhmJlZSZwwzMysJE4YZmZWEicMMzMrSWoJQ9J9knZL\nWtvPfkn6oqSNkp6VdHXevoWSNiT77k4rRjMzK12al9XeD3wJ+Ho/+28G5iZf1wJ/B1wrKQPcA7wD\n2AaskPRwRDx3rgPcuPsIn3ggl88OHurgy+uf5JJpTfzpb7yOpoa6c/1xZmbDWmoJIyKWS5o9QJVb\ngK9H7hmxv5A0QdI0YDawMSI2AUhamtQ95wmj/XgPT23ef6rgwH6e2ryfjq4e/uK9V5zrjzMzG9bK\neePeDGBr3va2pKyv8mv7exNJi4HFAM3NzbS2tpYcwKZDPX2WL39+O62tB0p+n0rT1tZ2Rj+HSlYt\nfamWfoD7UqmGoi/D/k7viFgCLAGYP39+nMmdjhO3HoQnf9arPFtTP6zv/vTdq5WnWvoB7kulGoq+\nlDNhbAdm5W3PTMrq+ik/5y6aOpZvL76O9s5u/uD+U0uKtB3vTuPjzMyGtXImjIeBu5JzFNcChyJi\np6Q9wFxJc8glituA96URwNhRtVz7mvPIZgMBkZQf7eyhJxtkapTGx5qZDUupJQxJ3wJagMmStgH/\njdzogYi4F1gGvBPYCBwF7kj2dUu6C3gUyAD3RcS6tOIEqKkRDbXQkTewaDvezfjRvlLKzOyENK+S\nWnSa/QF8uJ99y8gllCEzulZ0dMfJbScMM7NCvtM70VCUOtuO+TyGmVk+J4zE6Ezh+Yq2411lisTM\nrDI5YSRG1xYmjCMeYZiZFXDCSPSakvKltWZmBZwwEsUjDJ/DMDMr5ISRGO0RhpnZgJwwEj6HYWY2\nMCeMREPxlJRHGGZmBZwwEr2mpDzCMDMr4ISR6HXS2yMMM7MCThiJ4hHGk5v2lScQM7MK5YSRKB5h\n7G/v5BtPbi5LLGZmlcgJI1F80hvgf/9iSxkiMTOrTE4YieYxvRPG3rbjZYjEzKwyOWEk6jNiyfvf\nWFB2xCe+zcxOcsLI8455zQVP2evsznK8u6eMEZmZVQ4njDySGDuq8HKp9uNOGGZmkHLCkLRQ0gZJ\nGyXd3cf+iZIelPSspKckXZq3b7OkNZJWSVqZZpz5eicMT0uZmUG6z/TOAPcA7wC2ASskPRwRz+VV\n+wSwKiJulfT6pP6NefvfFhF704qxL8UJw2tKmZnlpDnCWABsjIhNEdEJLAVuKaozD/gxQESsB2ZL\nak4xptMaW/RgDN/xbWaWk9oIA5gBbM3b3gZcW1RnNfAe4AlJC4ALgZnALiCAH0rqAf4+Ipb09SGS\nFgOLAZqbm2ltbR1UsG1tbbS2ttLZfqyg/OcrnuHoK2n+mM69E32pBtXSl2rpB7gvlWoo+lLuv4Sf\nAb4gaRWwBvgVcOIs8/URsV3SVOAxSesjYnnxGySJZAnA/Pnzo6WlZVCBtLa20tLSwj/veIY1e3ee\nLJ9z8SW0XDljUO9ZLif6Ug2qpS/V0g9wXyrVUPQlzYSxHZiVtz0zKTspIg4DdwBIEvAysCnZtz35\nvlvSg+SmuHoljHOtyVdJmZn1Kc1zGCuAuZLmSKoHbgMezq8gaUKyD+BDwPKIOCypUVJTUqcRuAlY\nm2KsJxWf9G473jUUH2tmVvFSG2FERLeku4BHgQxwX0Ssk3Rnsv9e4BLga5ICWAd8MGneDDyYG3RQ\nC/xTRDySVqz5ep309lVSZmZAyucwImIZsKyo7N68108CF/fRbhNwRZqx9afXZbW+SsrMDPCd3r30\nmpLyCMPMDHDC6MX3YZiZ9c0Jo0jvk95OGGZm4ITRS1PRCOOJF/eyv72zTNGYmVUOJ4wiY0fV9Sr7\nxANryhCJmVllccIoMrGxd8L45cv7yhCJmVllccIoMrWpgTdfdF5B2eFj3UREmSIyM6sMThh9uP+O\nBQXbPdnwyW8zG/GcMPpQX1vD+eMaCsoOdXiJEDMb2Zww+jFhTOG5DCcMMxvpnDD6MW60E4aZWT4n\njH6ML0oYh50wzGyEc8LoR3HC8AjDzEY6J4x+OGGYmRVywuhHccI4eNQJw8xGNieMfniEYWZWKNWE\nIWmhpA2SNkq6u4/9EyU9KOlZSU9JurTUtmlzwjAzK5RawpCUAe4BbgbmAYskzSuq9glgVURcDnwA\n+MIZtE2VE4aZWaE0RxgLgI0RsSkiOoGlwC1FdeYBPwaIiPXAbEnNJbZNVfF9GL6s1sxGujQTxgxg\na972tqQs32rgPQCSFgAXAjNLbJsqjzDMzArVnr5Kqj4DfEHSKmAN8Cug50zeQNJiYDFAc3Mzra2t\ngwqkra2toO2h44Wr0+49fHTQ7z3UivsynFVLX6qlH+C+VKqh6EuaCWM7MCtve2ZSdlJEHAbuAJAk\n4GVgEzD6dG3z3mMJsARg/vz50dLSMqhgW1tbyW/b2Z3lo4//28ntjh5xww03kAuzshX3ZTirlr5U\nSz/AfalUQ9GXNKekVgBzJc2RVA/cBjycX0HShGQfwIeA5UkSOW3btNXX1jC6LnNy20ucm9lIl9oI\nIyK6Jd0FPApkgPsiYp2kO5P99wKXAF+TFMA64IMDtU0r1v6MH11HR9epGbJDHV00NfR+Ip+Z2UiQ\n6jmMiFgGLCsquzfv9ZPAxaW2HWrjR9fx6uFjJ7cPdXQxc2IZAzIzKyPf6T0AXyllZnaKE8YAfC+G\nmdkpThgD8AKEZmanOGEMwFNSZmanOGEMwAnDzOwUJ4wBjB9deBGZE4aZjWROGAMYP8YjDDOzE5ww\nBuApKTOzU5wwBlCcMHxZrZmNZE4YAxg/ur5g2yMMMxvJnDAG0Os+DCcMMxvBnDAG0NeUVDYb/dQ2\nM6tuThgDqK+tobH+1BLn2YAjXuLczEYoJ4zTmDCm6DyGlwcxsxHKCeM0ep/H6CxTJGZm5eWEcRoT\nxngBQjMzcMI4rV4Jw1dKmdkIlWrCkLRQ0gZJGyXd3cf+8ZL+RdJqSesk3ZG3b7OkNZJWSVqZZpwD\nKb4X4+BRT0mZ2ciU2iNaJWWAe4B3ANuAFZIejojn8qp9GHguIn5L0hRgg6RvRsSJv8pvi4i9acVY\niomekjIzA9IdYSwANkbEpiQBLAVuKaoTQJMkAWOB/UBFXbfqcxhmZjlpJowZwNa87W1JWb4vAZcA\nO4A1wEcjIpvsC+CHkp6WtDjFOAc0oXhKyldJmdkIldqUVIl+A1gFvB24CHhM0hMRcRi4PiK2S5qa\nlK+PiOXFb5Akk8UAzc3NtLa2DiqQtra2Pttu3VU44Hlp66uD/oyh0l9fhqNq6Uu19APcl0o1FH1J\nM2FsB2blbc9MyvLdAXwmIgLYKOll4PXAUxGxHSAidkt6kNwUV6+EERFLgCUA8+fPj5aWlkEF29ra\nSl9tR2/ax9/+6hcnt2vHjKOl5c2D+oyh0l9fhqNq6Uu19APcl0o1FH1Jc0pqBTBX0hxJ9cBtwMNF\ndbYANwJIagZeB2yS1CipKSlvBG4C1qYYa7+K7/T2VVJmNlKlNsKIiG5JdwGPAhngvohYJ+nOZP+9\nwKeB+yWtAQT854jYK+k1wIO5c+HUAv8UEY+kFetAfNLbzCwn1XMYEbEMWFZUdm/e6x3kRg/F7TYB\nV6QZW6mKE8aBo530ZINMjcoUkZlZefhO79MYVZthXMOpvJqNXNIwMxtpnDBKMKVpVMH2niPHyxSJ\nmVn5OGGUwAnDzMwJoyRTmhoKtp0wzGwkcsIowZSxRSOMNicMMxt5nDBK4CkpMzMnjJI4YZiZlZgw\nJF0kaVTyukXSH0makG5olaM4Ybyy/yg/37iXdTsOkVvVxMys+pU6wvgu0CPpteTWbZoF/FNqUVWY\n4nMYq7ce5H1f/SXv+uJP+YtHN5QpKjOzoVVqwshGRDdwK/C3EfExYFp6YVWW4hFGvvt++jKd3dl+\n95uZVYtSE0aXpEXA7wP/mpTVDVC/qkxqrGfy2Po+9x3vzvqqKTMbEUpNGHcAbwL+PCJeljQH+EZ6\nYVWWTI347+++lHENtUhQvIzU7sPHyhOYmdkQKmnxweQ53H8EIGki0BQRn00zsErzrsun8c7Lzqc7\nGyz++koe37Dn5L7dvmrKzEaAUq+SapU0TtIk4BngK5L+Kt3QKo8k6jI1TC2689sjDDMbCUqdkhqf\nPDb1PcDXI+Ja4NfTC6uyNY8rPAnuEYaZjQSlJoxaSdOAf8+pk94j1pRxhSOMXR5hmNkIUGrC+BS5\nJ+e9FBErkifivZheWJVtapNHGGY28pSUMCLinyPi8oj4D8n2poj4d6drJ2mhpA2SNkq6u4/94yX9\ni6TVktZJuqPUtuXUXDTC2LSnvUyRmJkNnVJPes+U9KCk3cnXdyXNPE2bDHAPcDMwD1gkaV5RtQ8D\nz0XEFUAL8JeS6ktsWzbFI4wt+4/y1Sc2lSkaM7OhUeqU1D8CDwPTk69/ScoGsgDYmIxGOoGlwC1F\ndQJokiRgLLAf6C6xbdn0def353+wgWNdPWWIxsxsaJSaMKZExD9GRHfydT8w5TRtZgBb87a3JWX5\nvgRcAuwA1gAfjYhsiW3Lpi5TwyXTxhWUHevKsuNgR5kiMjNLX0k37gH7JP0e8K1kexGw7xx8/m8A\nq4C3AxcBj0l64kzeQNJiYDFAc3Mzra2tgwqkra3tjNq+98IePr2zsOyR5b9k3nmZQX3+uXSmfalk\n1dKXaukHuC+Vaij6UmrC+APgb4G/JjeN9HPg/zpNm+3kVrU9YWZSlu8O4DORWyN8o6SXgdeX2BaA\niFhCbgVd5s+fHy0tLafvTR9aW1s5k7YtwK+OPsO/Pnsqa0yd/Tpa3jjgqZ0hcaZ9qWTV0pdq6Qe4\nL5VqKPpS6lVSr0TEuyNiSkRMjYjfBk53ldQKYK6kOZLqgdvInQfJtwW4EUBSM/A6YFOJbctu2vjC\nq6VePeQpKTOrXqWOMPryx8Df9LczIrol3UXu/o0McF9ErJN0Z7L/XuDTwP2S1gAC/nNE7AXoq+1Z\nxJqKaeNHF2xv2NXG/1z2PM+8coAxo2r5yNtfyzWzJ5UpOjOzc+tsEoZOVyEilgHLisruzXu9A7ip\n1LaVpniE8S+rdxRsr956kF9+4kYa6sp/XsPM7GydzTO9R/yzSadNGD3g/kMdXby4q22IojEzS9eA\nIwxJR+g7MQgY+K/lCFA8wujLjkMdXDZz/BBEY2aWrgETRkQ0DVUgw9HksaOorRHd2f4HW68e8sKE\nZlYdzmZKasTL1Ih508cNWGeHr5wysyrhhHGWPvnuN3DRlEbG1GeYNan3LJ1HGGZWLc7mKikDrr5g\nIj/6k5aT20+8uIf3/8NTJ7d3HnTCMLPq4BHGOVZ8InznYU9JmVl1cMI4x84vupnv1UPHyA5wUtzM\nbLhwwjjHxo6qpanh1ExfV0+wr72zjBGZmZ0bThgpmF40ytjpK6XMrAo4YaTg/OLzGL5SysyqgBNG\nCqZPKEoYfrCSmVUBJ4wUnD+ueErKIwwzG/6cMFIwrXiE4YRhZlXACSMFve7F8ElvM6sCThgpKH6w\nkkcYZlYNnDBSUDzC2Haggz/5zmrWbj9UpojMzM5eqglD0kJJGyRtlHR3H/s/JmlV8rVWUo+kScm+\nzZLWJPtWphnnudY4qpZxDYXLdH33mW0sWvIL2o93lykqM7Ozk1rCkJQB7gFuBuYBiyTNy68TEZ+L\niCsj4krg48BPImJ/XpW3JfvnpxVnWmZNGtOr7Mjxbp7ZcqAM0ZiZnb00RxgLgI0RsSkiOoGlwC0D\n1F8EfCvFeIbU714zq8/yLfuPDnEkZmbnRpoJYwawNW97W1LWi6QxwELgu3nFAfxQ0tOSFqcWZUo+\n8KbZLF18HZPH1heUO2GY2XBVKc/D+C3gZ0XTUddHxHZJU4HHJK2PiOXFDZNkshigubmZ1tbWQQXQ\n1tY26LYDuXUOfGXNqe2n179C6+hd5/xz8qXVl3Kolr5USz/AfalUQ9GXNBPGdiB/XmZmUtaX2yia\njoqI7cn33ZIeJDfF1SthRMQSYAnA/Pnzo6WlZVDBtra2Mti2A2ncvJ+vrHny5HZHppGWlrec88/J\nl1ZfyqFa+lIt/QD3pVINRV/SnJJaAcyVNEdSPbmk8HBxJUnjgRuAh/LKGiU1nXgN3ASsTTHW1FxY\ndPJ73Y7DvLjrCD1+RoaZDTOpJYyI6AbuAh4Fnge+ExHrJN0p6c68qrcCP4iI9ryyZuCnklYDTwHf\nj4hH0oo1TVOaRjGqtvDH/I6/Xs4f3L/CD1Yys2El1XMYEbEMWFZUdm/R9v3A/UVlm4Ar0oxtqEji\ngkljeHF3W0H5T17YwzNbDjB/9qQyRWZmdmZ8p/cQmD97Yp/l6189MsSRmJkNnhPGEPjTm17Huy6b\n1qv85b3tfdQ2M6tMThhD4Lyxo7jn9qv520VXFZRv2tPWTwszs8rjhDGE5kxuLNj2CMPMhpNKuXFv\nRChOGFsPdPD5RzdQl6nhXZefz2unNpUpMjOz0/MIYwg1jqrl/HGnlj7vyQZfenwjf/3DF7jlSz9j\nb9vxMkZnZjYwJ4whVjzKOKG9s4cfP797iKMxMyudE8YQu/my8/vd98IuX2ZrZpXL5zCG2Puvu5Cm\nhlpWbz3EzkMdPLru1EKEL+z2VVNmVrmcMIaYJG69aia3XjWTDa8eKUgYL3qEYWYVzFNSZTRnciO1\nNTq5vfPQMf6/7631s7/NrCI5YZRRfW0Ns4tOgn/jF6/wni//nC37/KAlM6ssThhl9rrm3vdedPZk\neWTdzjJEY2bWPyeMMrv1qj6fWstzOw4PcSRmZgNzwiizX5/XzNLF1/GWuZMLyp/b6YRhZpXFCaMC\nXPea87jn9qsLyl7Y1caTL+3jWFdPmaIyMyvkhFEhxjXUcUHR41wXfeUXLPyb5Rw82lmmqMzMTkk1\nYUhaKGmDpI2S7u5j/8ckrUq+1krqkTSplLbVaN60cb3KNu87yv//9LYyRGNmVii1hCEpA9wD3AzM\nAxZJmpdfJyI+FxFXRsSVwMeBn0TE/lLaVqO3Xjylz/KVmw8McSRmZr2lOcJYAGyMiE0R0QksBW4Z\noP4i4FuDbFsVfmf+TD7y9tdSl1FB+dNbDhARZYrKzCwnzYQxA9iat70tKetF0hhgIfDdM21bTeoy\nNfzJTa/juU8tZHRd5mT5niPH2Xago4yRmZlVzlpSvwX8LCL2n2lDSYuBxQDNzc20trYOKoC2trZB\nt03DhU3B+ryfxjce+Tlvnl7a4aq0vpyNaulLtfQD3JdKNRR9STNhbAdm5W3PTMr6chunpqPOqG1E\nLAGWAMyfPz9aWloGFWxrayuDbZuGFcfXs/7xl05ud4w5n5aWS0tqW2l9ORvV0pdq6Qe4L5VqKPqS\n5pTUCmCupDmS6sklhYeLK0kaD9wAPHSmbavZGy+cWLD99Cs+8W1m5ZXaCCMiuiXdBTwKZID7ImKd\npDuT/fcmVW8FfhAR7adrm1asleiqWYUJY/2rh2k73s3YUZUyi2hmI02qf30iYhmwrKjs3qLt+4H7\nS2k7kkxsrOeiKY28tCeXR7MBq7ce5NdeO/k0Lc3M0uE7vStY8bTUL18+42sCzMzOGSeMCnbN7EkF\n28tf2FOmSMzMnDAq2g1Fd36v3naQA+1eV8rMysMJo4JNHdfAJXnrS0XAExv3ljEiMxvJnDAqXPEo\n4ycbPC1lZuXhhFHh3npx4VVRy1/cQ1dPls7uLN092TJFZWYjkS/qr3DzL5zEmPoMRztzD1Lac+Q4\nc//s3wDI1Ii3zp3MPbdfzZh6H0ozS5dHGBWuvraGN1/U970XPdng8Q17+MKPXhziqMxsJHLCGAZu\nvvT8Afc/+Mx2erJe/tzM0uWEMQzcetUMbr/2AhrrM0hQU/i4DHYfOc4vN+0rT3BmNmJ44nsYqKkR\nf37rZfz5rZedLPvjb6/igV+dWsD3oVU7eLOXDTGzFHmEMUy9+8rpBdvfXrmVQ0e7yhSNmY0EThjD\n1PWvncx5jfUFZVd86ge8sOtImSIys2rnhDFM1WZq+M3Lp/Uq//S/PleGaMxsJHDCGMZuW3BBr7In\nXtzLjjbf0Gdm554TxjB2ybRxfOUD83uV/2iLz2WY2bnnhDHMvWNeM1+47cqCsh9t6ebx9bvLFJGZ\nVatUE4akhZI2SNoo6e5+6rRIWiVpnaSf5JVvlrQm2bcyzTiHu5svncaUplEFZXfcv4KHVm3vp4WZ\n2ZlLLWFIygD3ADcD84BFkuYV1ZkAfBl4d0S8Afidord5W0RcGRG9513spPraGn7v2gt7lX906SqW\nLH+JX205UIaozKzapDnCWABsjIhNEdEJLAVuKarzPuCBiNgCEBGeRxmk97/pQprHjepV/j+XrefW\nL/+cpU9tKUNUZlZN0kwYM4CtedvbkrJ8FwMTJbVKelrSB/L2BfDDpHxxinFWhUmN9TzwH3+NTPG6\nIYn/8f3n2X3k2BBHZWbVRBHpLFon6b3Awoj4ULL9fuDaiLgrr86XgPnAjcBo4EngXRHxgqQZEbFd\n0lTgMeAjEbG8j89ZDCwGaG5ufuPSpUsHFW9bWxtjx44dVNtKcvh48F9+1s7hzt6JY1w9LL68gelj\nxaSG4XG9Q7Ucl2rpB7gvlWqwfXnb2972dKnT/mmuJbUdmJW3PTMpy7cN2BcR7UC7pOXAFcALEbEd\nctNUkh4kN8XVK2FExBJgCcD8+fOjpaVlUMG2trYy2LaV5ljPj9lZP4sv/vjFglVsD3fC51ceI1Mj\n/tetl/Hvr5k1wLtUhmo5LtXSD3BfKtVQ9CXNfzNXAHMlzZFUD9wGPFxU5yHgekm1ksYA1wLPS2qU\n1AQgqRG4CVibYqxVZeqYGj7663NZ/+mFvGH6uF77e7LB//vdZ7nmz3/oE+JmVrLUEkZEdAN3AY8C\nzwPfiYh1ku6UdGdS53ngEeBZ4CngqxGxFmgGfippdVL+/Yh4JK1Yq1VdpoYvLrqKMfWZPvfvOXKc\nW7/8c97++Vb+09JfsfNQxxBHaGbDSarLm0fEMmBZUdm9RdufAz5XVLaJ3NSUnaWLpozlW394Hf/w\n05d54sU9HOhjRdtNe9vZtLed763awR/dOJemUbXMnDiam95wfr8n0c1s5PHzMEaAK2ZN4IuLrgLg\nW09t4eMPrOm37hfzHvd64+un8vfvfyO1meFxgtzM0uW/BCPMogUX8PL/eicfeFPvG/2K/Wj9bj70\n9ZVs3X+Uo53dQxCdmVUyJ4wRSBKfuuVSfnb32/n24usYO6r/gWbrhj285S8e57JP/oC/a31pCKM0\ns0rjKakRbMaE0cyYMJonP/52vrNyG7sOHyMiWLbmVbYfLDwB3pMNPvvIer7woxf4rcun86G3vIaL\nm8fSdryb53Yc5sDRLt4ydzKNAyQfMxve/NttNDXU8cHr55zc/t1rLuDf/d3POdTR+wT5sa4s//z0\nNv756W3UCPJu82Bq0ygWLbiAyU2jyGaDGsGY+loWzJnErEljSopl+8EOurqzXDBpDDU+4W5WUZww\nrJfXTh3LNz90Lf/j+8/xwq429rd39lkvW7RIwO4jx/lC3knzfG+YPo7zxo4iI2ioyzBv2jgmNNaT\nkagRvHr4GA+v3sGmPe0ATB5bzzWzJ9F24DiHJmznNy+f7iu2zMrMCcP6dOmM8Sxd/CYAVm7ez3/8\n5jPsPnJ80O+3bsfhgu1/W/vqgPX3tnWerPPE0lV8dOkqGuszzJw4hkmN9dTUQI1EjUSmRkweW8+E\nMfVIufITiaimRkm9U68nNdYzZ3LjybbjR9dxYQkjmoNHOznc0c2MiaOdvGxEcsKw05o/exKP/fEN\nfOupLSxZvqlgxFGfqaGzZ2geCdve2cOGXUdSee9xDbW8Ze4UFsyZxKjaGpoa6njd+U2Mqq1Bgsee\n28XnH91Ae2cPE8fUMX1CLmmcTEYSo+pqmH1eI6PrMmRqhPL2bdvaye6xWxnXUMvFzU1MnzA6eW8n\nHhs+nDCsJONH13HnDRdx5w0Xcaijix0HOxhTn2HWxNx/5rsPH+OhVTt4aU8bNTW5//CzEbyw6wgr\nXzlASmtcnjOHj3Xz/TU7+f6anaete+BoV583QAL8bOO+ftt9b+OzBdsSjKqtYXRdhtF1GRqSRHMi\nETWPG8Vrp45lYjJ1l6kRtTUik6kho+R1jajN5L6Pqs3QOCrDmPpaGupqqK0RPdncBQtB5CW4XCI7\ncLSLwx1d1GVqqMuI2kzNyQRXI50crfVkg9XbDrLh1SPUZsSxfV1s/cUrZCR6Ijjc0cV5jfWMqqsp\nGPXVCDI1NUxpGsXougwSKOk36OT2ic9SUnbiZyPpZP0T+w4e7eLJl/ayt62T2oyoy+T6WZt8rznx\nuSf6eTKOXAIvHnmu291NbNhNbY2Sn0MN9ZkaBsrj+ftyEfZR3kedwrK+Ff+aFP/eRFIjv/zE6y2H\nezjW1UNDXd8rO5wLThh2xsaPrmP86LqCsqnjGvjDt76mz/q7Dx/jhV1tdGWzZLPBjkPH2LjrCN3Z\nIBtBNgvZCEbV1fCOeedz6fRxrNh8gP3tnXznp8+x/mBwrGtoRjFDKSJ3EcGxriwH6J2AntsJj2/Y\nU4bISrC+ipZ2e2ZFuSM4Z37tTR1cNCW91XedMCx1U8c1MHVcwxm1WXjp+QBM79jEW956A109WY52\n9rBpTxsPtWd8AAAJwklEQVTHu7NkI3L/OQd09mR59dAxjnX10BO5sp4TySggm7zuiaC7J9i8t52D\nHV30ZHPv8cq+dg4f842JNvylPZJ3wrCKl5umyU3ZTGqcdM7f/8SUy/IX9vDqoWP0ZIPN+9p59fAx\nInK/hNkImhpqef91F3L1hROThEQyQsq93t/eybYDR08lqrx96ze+TN34Kew+fJznXz3M0c4eOrur\nb9Rk1c0Jw0a8TI24+oKJXH3BxNQ+ozWznZaWqwrKerLB8e4eOjp7ONadpaOz5+TIqLM7y+Z97by8\nt52Orh6y2aA7GRF1Z6PXdk821779eA8dXbm57GycOm8BucTXE7n3j8idP2ke10BPNujsydLVk00S\nZG40dmKqMAIa6jO88YKJnDe2nmeef4nmadPJJtdVNzXUsr+9i55slp78EV02ON6dZdfhY8l5lNx7\nB0Bwcjsbubn5E8mZvHqRvy/ZnjGhgWtmT2JMfYaubNDdkz35sziZyPNGlYUjzsI6e/btY8LESfRk\ns3R1n/o55Cs4X1BQ3ve/84X1+zjnUPQexRc+FJ/fKD6f0tc5EYD29nbqU173zQnDrEwyNWJMfS1j\n6vv+Nbxi1oQhjqg0rdpGS8tl5Q7jnMg9dGhBucM4J1pbW7ngvNJukB0sryVlZmYlccIwM7OSOGGY\nmVlJUk0YkhZK2iBpo6S7+6nTImmVpHWSfnImbc3MbOikdtJbUga4B3gHsA1YIenhiHgur84E4MvA\nwojYImlqqW3NzGxopTnCWABsjIhNEdEJLAVuKarzPuCBiNgCEBG7z6CtmZkNoTQTxgxga972tqQs\n38XAREmtkp6W9IEzaGtmZkOo3Pdh1AJvBG4ERgNPSvrFmbyBpMXA4mSzTdKGQcYyGdg7yLaVxn2p\nPNXSD3BfKtVg+3JhqRXTTBjbgVl52zOTsnzbgH0R0Q60S1oOXJGUn64tABGxBFhytsFKWhkR88/2\nfSqB+1J5qqUf4L5UqqHoS5pTUiuAuZLmSKoHbgMeLqrzEHC9pFpJY4BrgedLbGtmZkMotRFGRHRL\nugt4FMgA90XEOkl3JvvvjYjnJT0CPAtkga9GxFqAvtqmFauZmZ1equcwImIZsKyo7N6i7c8Bnyul\nbcrOelqrgrgvlada+gHuS6VKvS/qb8VFMzOzfF4axMzMSjLiE8ZwX4JE0mZJa5LlVVYmZZMkPSbp\nxeR7eg96OAuS7pO0W9LavLJ+Y5f08eQ4bZD0G+WJum/99OWTkrYnx2aVpHfm7avkvsyS9Lik55Il\nez6alA+rYzNAP4bdcZHUIOkpSauTvvz3pHxoj0lEjNgvcifUXwJeA9QDq4F55Y7rDPuwGZhcVPYX\nwN3J67uBz5Y7zn5ifytwNbD2dLED85LjMwqYkxy3TLn7cJq+fBL40z7qVnpfpgFXJ6+bgBeSmIfV\nsRmgH8PuuJB7rtLY5HUd8EvguqE+JiN9hFGtS5DcAnwtef014LfLGEu/ImI5sL+ouL/YbwGWRsTx\niHgZ2Eju+FWEfvrSn0rvy86IeCZ5fYTcpe4zGGbHZoB+9Kci+wEQOW3JZl3yFQzxMRnpCaMaliAJ\n4IfJ0ion7nhvjoidyetXgebyhDYo/cU+XI/VRyQ9m0xZnZguGDZ9kTQbuIrcf7TD9tgU9QOG4XGR\nlJG0CtgNPBYRQ35MRnrCqAbXR8SVwM3AhyW9NX9n5Manw/JSuOEce+LvyE13XgnsBP6yvOGcGUlj\nge8C/ykiDufvG07Hpo9+DMvjEhE9ye/6TGCBpEuL9qd+TEZ6wihl+ZKKFhHbk++7gQfJDTt3SZoG\nkHzf3f87VJz+Yh92xyoidiW/5FngK5yaEqj4vkiqI/dH9psR8UBSPOyOTV/9GM7HBSAiDgKPAwsZ\n4mMy0hPGsF6CRFKjpKYTr4GbgLXk+vD7SbXfJ7cEy3DRX+wPA7dJGiVpDjAXeKoM8ZXsxC9y4lZy\nxwYqvC+SBPwD8HxE/FXermF1bPrrx3A8LpKmKPf8ICSNJvesoPUM9TEp99n/cn8B7yR39cRLwJ+V\nO54zjP015K6EWA2sOxE/cB7wI+BF4IfApHLH2k/83yI3JdBFbo71gwPFDvxZcpw2ADeXO/4S+vIN\nYA25pW8eBqYNk75cT25q41lgVfL1zuF2bAbox7A7LsDlwK+SmNcC/zUpH9Jj4ju9zcysJCN9SsrM\nzErkhGFmZiVxwjAzs5I4YZiZWUmcMMzMrCROGFZVJIWkv8zb/lNJnzxH732/pPeei/c6zef8jqTn\nJT1eYv1PpB2TGThhWPU5DrxH0uRyB5JP0pk83fKDwB9GxNtKrO+EYUPCCcOqTTe5R1X+P8U7ikcI\nktqS7y2SfiLpIUmbJH1G0u3J8wfWSLoo721+XdJKSS9I+s2kfUbS5yStSBa0+7/z3vcJSQ8Dz/UR\nz6Lk/ddK+mxS9l/J3XD2D5I+V1R/mqTlyTMc1kp6i6TPAKOTsm8m9X4viX2VpL+XlDnRX0l/nTxP\n4UeSppzFz9lGICcMq0b3ALdLGn8Gba4A7gQuAd4PXBwRC4CvAh/Jqzeb3NpD7wLuldRAbkRwKCKu\nAa4B/jBZjgFyz8j4aERcnP9hkqYDnwXeTm4RvGsk/XZEfApYCdweER8rivF9wKORW4DuCmBVRNwN\ndETElRFxu6RLgN8Ffi2p1wPcnrRvBFZGxBuAnwD/7Qx+PmacyTDZbFiIiMOSvg78EdBRYrMVkSwT\nLekl4AdJ+Rogf2roO5FbtO5FSZuA15Nbw+vyvNHLeHJr93QCT0XueQTFrgFaI2JP8pnfJPcQpu8N\nFCNwX7Kg3vciYlUfdW4E3gisyC2lxGhOLUiXBb6dvP7fwAO9WpsNwAnDqtXfAM8A/5hX1k0yqpZU\nQ+4piyccz3udzdvOUvh7UryWTpB7GtpHIuLR/B2SWoD2wYXfW0QsT5avfxdwv6S/ioivF1UT8LWI\n+Hgpb3muYrORwVNSVpUiYj/wHXLTRSdsJvffN8C7yT217Ez9jqSa5LzGa8gt7PYo8B+S//yRdHGy\nevBAngJukDQ5OcewiNw0Ub8kXQjsioivkJsquzrZ1XXis8ktRPdeSVOTNpOSdpD7fT8xCnof8NPS\numyW4xGGVbO/BO7K2/4K8JCk1cAjDO6//y3k/tiPA+6MiGOSvkru3MYzyZLaezjNY3EjYqeku8k9\n10DA9yPidMvQtwAfk9QFtAEfSMqXAM9KeiY5j/FfgB8ko6gu4MPAK+T6uyDZv5vcuQ6zknm1WrMR\nQlJbRIwtdxw2fHlKyszMSuIRhpmZlcQjDDMzK4kThpmZlcQJw8zMSuKEYWZmJXHCMDOzkjhhmJlZ\nSf4PCqviFF4Fw14AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266a1a1e2b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_history_SAGA, lw=4)\n",
    "plt.grid()\n",
    "#plt.yscale('log')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Number of step')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id='section5'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Primal Coordinate Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Nous allons commencer avec l'algorithme de coordinate descent dans le cadre primal avant de passer au problème dual (ie. l'algo SDCA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def CD(T, lam, eta):\n",
    "    # on déclenche le timer\n",
    "    t0 = time.time()\n",
    "    \n",
    "    # initialisation\n",
    "    theta = np.zeros(n_features)\n",
    "    \n",
    "    # historique des fonctions de perte\n",
    "    cost_history = np.empty(shape=[T])\n",
    "    time_history = np.empty(shape=[T,2])\n",
    "    \n",
    "    for t in range(0,T):\n",
    "        cost_history[t] = f(theta, lam)\n",
    "        time_history[t] = (time.time()-t0, f(theta, lam))\n",
    "        \n",
    "        # .. pick random coordinate ..\n",
    "        idx = np.random.randint(0, n_features)\n",
    "        \n",
    "        # compute partial gradient\n",
    "        gj = partial_coordinate_grad(theta, idx)\n",
    "        \n",
    "        # compute prox\n",
    "        theta[idx] = theta[idx] - eta * gj - eta * lam * np.sign(theta[idx])\n",
    "               \n",
    "    return theta, cost_history, time_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "theta, cost_history_CD, time_history_CD = CD(T=T,lam=lam,eta=eta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VVX2//H3Sqd3Qm8CAiKoBBCRbsFeZ8Zewd7G33xn\n7OM0R51xdHQs49iwMjr20VERiIgiVXqv0kOHBEgh6/fHPUASkhAIN/fe5PN6nvvknr33uVmbk7By\n9j5nH3N3REREDldcpAMQEZHYpkQiIiLlokQiIiLlokQiIiLlokQiIiLlokQiIiLlokQiIiLlokQi\nIiLlokQiIiLlkhDpACpCw4YNvU2bNoe1b1ZWFjVq1DiyAUWI+hKd1JfopL7A1KlTN7p7o4O1qxKJ\npE2bNkyZMuWw9k1PT2fgwIFHNqAIUV+ik/oSndQXMLMVZWmnoS0RESkXJRIRESkXJRIRESkXJRIR\nESkXJRIRESmXsCUSM3vFzDLMbHYJ9WZmT5vZYjObaWYnFKgbamYLgrp7CpTXN7NRZrYo+FovXPGL\niEjZhPOM5DVgaCn1ZwAdgtcNwPMAZhYPPBvUdwEuNbMuwT73AKPdvQMwOtgOi7v/PZ2Ln/+eRybu\n4ra3pzFj5dZwfSsRkZgWtkTi7uOAzaU0OQ943UN+AOqaWVOgF7DY3Ze6ew4wMmi7d58RwfsRwPnh\niR7mrNnOlBVbWLgln//OXMuVL09kw47scH07EZGYFckbEpsDKwtsrwrKiivvHbxPdfe1wft1QGpJ\nH25mNxA60yE1NZX09PRDCm5H5s5C29t359HzT1/Tp2k8Q1on0r5u/CF9XjTIzMw85H+HaKW+RCf1\nJTqFuy8xe2e7u7uZeSn1LwIvAqSlpfmh3tWZMiUdsrIOKJ+wdg+zNhtf/fIkmtWtdkifGWm6Uzc6\nqS/RSX0pu0hetbUaaFlgu0VQVlI5wPpg+Ivga0a4gnv60uN58/rexdZlZudx0qNjmLpiS7i+vYhI\nzIhkIvkEuCq4eutEYFswbDUZ6GBmbc0sCbgkaLt3n6uD91cDH4cruK7N63Byh4bceUIy8XFWbJuL\nnv+ebxdtYMfu3HCFISIS9cI2tGVm7wADgYZmtgr4LZAI4O4vAJ8DZwKLgZ3AtUFdnpndBnwJxAOv\nuPuc4GMfBd41s+uBFcDPwxX/Xsc3TuCHe/sxc9VWrh9x4MKPV748iTiDi05owWMXdSOuhKQjIlJZ\nhS2RuPulB6l34NYS6j4nlGiKlm8ChhyRAA9Bo1rJDOmcyojrenH1K5MOqM93eG/qKk5s14CLerSo\n6PBERCJKd7YfggEdGzHxviEkJRT/z/bMmEXk7cmv4KhERCJLieQQpdZOYfL9p3Bu92YH1C3ftJO3\nJ/1ETp6SiYhUHUokh6FOtUSevvR4lj96Fpf0bFmo7qGP59Dxgf9x97+nK6GISJWgRFJOtw5qT0Ix\nE+wf/LiaF75ZEoGIREQqlhJJObWsX52fpbUstu4fYxazOGNHBUckIlKxlEiOgAfO6syZxzYhMb7w\nmUnOnnxO+ds4nhy1kNBFaiIilU/MLpESTWokJ/Dc5T0A+HTGGm5/58dC9X8fvYjm9arx8xLOXERE\nYpkSyRF2dremfPTjakbPL7x6y6//M5NnxiwiKT6O+DgjOSGeQZ0ac+eQDiXeOS8iEguUSI4wM+PR\ni7pxwXPfsWrLrkJ1KzcX3p61ehvTV27lX1f1IDkh9lYTFhEBzZGERaNayYz/zWCGHtPkoG3HLdxA\n2h++ZvnGA1caFhGJBUokYfTYRd3o3qLOQdvtyM7jd5/OOWg7EZFopKGtMKpTPZGPbu3Luu272Z2b\nT747+fnOZ7PW8tTXiwq1HbtgAz/+tIXjW+kx9CISW5RIwszMaFqn8AOw7kqtResG1fnlv2cUKr/g\nue85t3szEuKMuDijZ5t6/DytJWaajBeR6KVEEiEXHN+CxrVSuPyliYXKP5mxZt/7/0xdxeasXG4e\neFRFhyciUmaaI4mgvu0bctaxTUtt8+zYxWzdmVNBEYmIHDolkgh7+NxjaFW/eon1mdl5vPTtsgqM\nSETk0GhoK8Ia1Urm09tPZsKSTWRl5wEwZcVm3pm0cl+bV79bxvUnt6VejaRIhSkiUiIlkihQp1oi\nQ7vuv+dkaNcmfDlnPZuzQkNaWTl7eC59Mfef1SVSIYqIlEhDW1GoRnICN/ZvV6js5fHLmLlqa4Qi\nEhEpmRJJlLqyT2ua1knZt53vofW69LAsEYk2SiRRqnpSAo9ccGyhsvnrdvB8uh6WJSLRRXMkUWxQ\np8ZccHxzPvxx9b6yv49eyFsTVwAQZ0aH1Jo8csGxtCzlyi8RkXDSGUmUe+jsLjSsuf9qrXyHjB3Z\nZOzIZt323Xy7aCPXvTaZ7Lw9EYxSRKoyJZIoV69GEr87t2upbRZlZPJC+tIKikhEpDAlkhhw5rFN\nuGnAUSSU8gCsJ79eyIc/rmLVlp0VGJmIiOZIYoKZcc8ZnbhjSHsyd+fhwNaduZz+1LhC7fYuAvnw\nOV24pm/bCEQqIlWRzkhiSPWkBBrXTiG1dgpHN6nFA2d1Lrbdw5/OZfpK3XMiIhVDiSSGXd67NS3q\nVSu27vxnv+OlbzVvIiLhp0QSw6olxfPG9b25tFerYuv/+Nk8Pp2xhmk/bSFjx+4Kjk5EqgrNkcS4\ntg1r8OcLj+Xhc7tw+pPjWL6p8GT77e/8uO/9xR0SGTiwggMUkUpPZySVRHJCPCOu60V8KVd2/WdR\nLp/OWMPcNdvZmJldgdGJSGWmM5JKpHWDGsz/w1CGPjWOJRuyim1T8AzlpgFHcc8ZnSoqPBGppHRG\nUskkxsfx6jW9uPCE5pzQqm6pbV/4ZgnnP/sd67Zp/kREDp8SSSXUqkF1/vbz4/jglr4sfeRMujSt\nXWLb6Su3cvEL37N9d24FRigilUlYE4mZDTWzBWa22MzuKaa+npl9aGYzzWySmXUtUHenmc02szlm\ndleB8uPM7Aczm25mU8ysVzj7EOvi4ox/XZ3GOd2b0bJWHEnxBx7yVVt28ej/5kcgOhGpDMKWSMws\nHngWOAPoAlxqZkUf8XcfMN3duwFXAX8P9u0KDAd6Ad2Bs82sfbDP48Dv3P044KFgW0rRvG41nrn0\neP7QtxoL/3QGvzqt4wFt3p74E/PWbo9AdCIS68J5RtILWOzuS909BxgJnFekTRdgDIC7zwfamFkq\n0BmY6O473T0P+Aa4MNjHgb1jNXWANWHsQ6V02+AOvD2s9wHlT45aGIFoRCTWhTORNAdWFtheFZQV\nNIMgQQRDVK2BFsBsoJ+ZNTCz6sCZQMtgn7uAv5jZSuCvwL1h60EldlL7hrx0VVqhsq/mrmfWqm0R\nikhEYpW5e3g+2OxiYKi7Dwu2rwR6u/ttBdrUJjScdTwwC+gEDHf36WZ2PXALkAXMAbLd/S4zexr4\nxt3fN7OfAze4+ynFfP8bgBsAUlNTe4wcOfKw+pGZmUnNmjUPa99oU7Qv7s7vf9jNsm37H9/brVE8\nd/dIKW73qFKZj0ssU1+i0+H2ZdCgQVPdPe1g7cKZSPoAD7v76cH2vQDu/ucS2huwDOjm7tuL1D0C\nrHL358xsG1DX3T3YZ5u7l3xZEpCWluZTpkw5rH6kp6czsJLcDl5cX9IXZHDNq5MLlfVp14BBnRoB\nUCslkVO7pNKwZnJFhVkmlf24xCr1JTodbl/MrEyJJJw3JE4GOphZW2A1cAlwWcEGZlYX2BnMoQwD\nxu1NImbW2N0zzKwVoeGvE4Pd1gADgHRgMLAojH2o9AZ0bERa63pMWbFlX9mEpZuYsHTTvu1/jFnM\nx7f1jbpkIiLRIWxzJMEk+W3Al8A84F13n2NmN5nZTUGzzsBsM1tA6OquOwt8xPtmNhf4FLjV3feu\niz4ceMLMZgCPEAxfyeExMx46pwulrKzC6q27SPvj17wxYXlFhSUiMSSsS6S4++fA50XKXijwfgJw\n4LWoobp+JZSPB3ocwTCrvG4t6nLroPY8M2Zxqe0e/HgOjWuncPoxTSooMhGJBVprSwC4+9SOHN2k\nFjNXbcPdyd3jvPb98gPaPfjRbE5s14A61RIrPkgRiUpKJAKEhrjO7taMs7s121d2ZZ/WXPjc92zb\ntX/5lIwd2Tz6v3n8+cJukQhTRKKQ1tqSEh3VqCbTHzqVi05oUaj8nUkrmbBkUwl7iUhVo0QipTIz\n/nRBV9o1qlGo/J4PZrI7d0+EohKRaKJEIgeVkhjP4xd1wwpc2bVi005en7A8UiGJSBRRIpEySWtT\nnytPbF2o7IVvlpKVnRehiEQkWiiRSJn98pSO1EiK37e9OSuHEROWRyweEYkOSiRSZvVqJHHdyW0L\nlb04bik79FAskSpNiUQOybCT21ErZf9V41t35vLYF3oolkhVpkQih6RO9USGndyuUNmbP/zE+c9+\nx84czZeIVEVKJHLIrj25zQF3tk9fuZUHPpzN6q27in1lbN9NuFaaFpHI0p3tcshqpyRy04CjDhjS\n+uDH1Xzw4+oS9zs6tRavX9+L1NrR/7wTESk7nZHIYbmhfzuu7tP64A0LWLB+B7e+NY2M7bvZlJmt\nMxSRSkJnJHJY4uOM353XlZb1q/PHz+aVeb8pK7bQ65HRALRvXJPXru1Ji3rVwxWmiFQAJRIpl+v6\ntmVDZjZfzVlPdjFLpjiwdtvuYvddnJHJzW9O483re4NBnEGcGTWS9WMpEkv0GyvlEhdn3HtGZ+49\no3OJbUbPW8/1I4p/1PGs1dvo/vuvCpW1aVCd56/oQeempT5BWUSihOZIJOyGdE7lgbM6065hDRrV\nOvjjepdv2skZf/+Wv41ayAvfLGHsggzNp4hEMZ2RSIUY1q8dw/qF7j+ZuHQTv3jxh4Pu8/ToRfve\n3zGkA3efWuzDNEUkwpRIpML1bteA3593DC+PX8aWrBwcwGFHKQtAPj16EZm782herxpHNapBvs5Q\nRKKGEolExFV92nBVnzaFynbszuWcZ8azfNPOYvd55btl+96f1jqBwYPCGaGIlJUSiUSNWimJ/Ofm\nk/jox9VszMzhvSkr2ZSVU2zbUSvyWLR+Bx1Sa1VwlCJSlCbbJao0rJnMsH7tuOeMTvxw3xC6Ni/+\nyi0Hnvx6YcUGJyLFUiKRqJUYH8fr1/XmgbM6c13ftjSrU3hplc9nrWPs/IwIRScieymRSFSrXyOJ\nYf3a8dA5XRj/m8EHnKFc+9pkZq/eFqHoRASUSCSGxMUZj1xwLHFWuPzsZ8bzxFcLIhOUiGiyXWJL\ntxZ1Gd6vHf8ct7RQ+TNjFvPWxJ84oVU9AJrWSWFYv7a0blAjEmGKVClKJBJz7jqlIx9NWcb6nYXv\nJdmclcPX89bv2x67IINRvxxAtQLPmReRI09DWxJzqiXF8+ueKQddi2vVll28NXFFBUUlUnUpkUhM\nalAtjv/d2Y87h3Qotd3z6UvIKuWOeREpPw1tSUz75akdOfPYpqzYlAXAnnznV+/NICsntKT9pqwc\nXp+wgpsHHhXJMEUqNSUSiXlHN6nF0U323+E+Z812/jF28b7tf45bwhUntqJWSmJxu4tIOWloSyqd\nYf3aUqvAw7G27szlte+WRy4gkUpOiUQqnbrVk7i+X9tCZS9+u5SMHcU/qVFEykeJRCql605uS51q\n+4eyduzO4+FP5kQwIpHKS4lEKqXaKYncPrh9obLPZ63ji9lrIxSRSOUV1kRiZkPNbIGZLTaze4qp\nr2dmH5rZTDObZGZdC9TdaWazzWyOmd1VZL/bzWx+UPd4OPsgseuak9rQvUWdQmUPfjyHbTtzIxSR\nSOUUtqu2zCweeBY4FVgFTDazT9x9boFm9wHT3f0CM+sUtB8SJJThQC8gB/jCzP7r7ovNbBBwHtDd\n3bPNrHG4+iCxLSE+jscu7sbZT48nLz90F/yGHdn0+OMoEuKNODPqVU9iWL+2XNu37UE+TURKEs4z\nkl7AYndf6u45wEhCCaCgLsAYAHefD7Qxs1SgMzDR3Xe6ex7wDXBhsM/NwKPunh3sp3XEpUSdmtTm\nliL3kOTlO7tz89mZs4fVW3fxu0/nMmHJpghFKBL7wplImgMrC2yvCsoKmkGQIMysF9AaaAHMBvqZ\nWQMzqw6cCbQM9ukY1E00s2/MrGcY+yCVwK2D29O+cc1S2/xt1AJcz4EXOSwWrl8eM7sYGOruw4Lt\nK4He7n5bgTa1gb8DxwOzgE7AcHefbmbXA7cAWcAcINvd7zKz2cBY4A6gJ/BvoJ0X6YiZ3QDcAJCa\nmtpj5MiRh9WPzMxMatYs/T+hWFGV+7ImM5/npu9mVWbJP++/Skuma8OKv0e3Kh+XaKa+wKBBg6a6\ne9rB2oUzkfQBHnb304PtewHc/c8ltDdgGdDN3bcXqXsEWOXuz5nZF8Bj7j42qFsCnOjuG0qKJS0t\nzadMmXJY/UhPT2fgwIGHtW+0UV8gd08++e64w01vTiV9wf4fmziDhLg4MKiRFM/P0lpyz9BOxBV9\nAMoRpuMSndQXMLMyJZIyDW2Z2VFmlhy8H2hmd5hZ3YPsNhnoYGZtzSwJuAT4pMjn1g3qAIYB4/Ym\nkb2T6GbWitDw19tBu4+AQUFdRyAJ2FiWfogkxseRnBBPSmI8d5/asVBdvkPOnnxy8vLZsjOXF8ct\n5RcvTmBzVk6EohWJDWWdI3kf2GNm7YEXCc1XvF3aDsEk+W3Al8A84F13n2NmN5nZTUGzzsBsM1sA\nnAHcWfB7mtlc4FPgVnffGpS/ArQLhrhGAlcXHdYSKYtuLepyapfUUttMXr6FoU+NY+22XRUUlUjs\nKeuAcL6755nZBcAz7v6Mmf14sJ3c/XPg8yJlLxR4P4HQ5Hlx+/YroTwHuKKMcYuU6pELjmVLVg5T\nVmwpsU3GjmyeGrWIxy7uVoGRicSOsiaSXDO7FLgaOCco01KqEvMa1UrmPzefRE5ePk5o7uS9qat4\n8KPZhdr9Z9oqbhl0lB7dK1KMsg5tXQv0Af7k7svMrC3wRvjCEqlYSQn7506uPLE1z19+QqH6PfnO\nM2MWl7C3SNVWpkTi7nPd/Q53f8fM6gG13P2xMMcmEjFnHNuUp35xXKGyD6atYsmGzAhFJBK9yjS0\nZWbpwLlB+6lAhpl95+53hzE2kYg6p3sznhmziCUbQk9fzHcY8sQ3HNeyLimJcZzapQnX9W1D6Mp1\nkaqrrENbdYLLci8EXnf33sAp4QtLJPLi44xfnnrgtSDTV27lh6Wb+cN/5/Ly+GURiEwkupQ1kSSY\nWVPg58B/wxiPSFQ569im9OvQsMT6v361gJ827azAiESiT1kTye8J3Q+yxN0nm1k7YFH4whKJDmbG\n4xd3o3WD6sXW787N554PZmqdLqnSyjRH4u7vAe8V2F4KXBSuoESiSdM61fj67gEsWLeD7Lx8Ji3b\nzGNfzN9X//2STYz4fjnXaCl6qaLKukRKi+ABVBnB630zaxHu4ESiRWJ8HF2b16FH63rcNKAdJ7cv\nPNz18KdzOfcf49m6U8upSNVT1qGtVwmtk9UseH0alIlUOWbGIxccS7XE+ELlM1dt4+53Z0QoKpHI\nKWsiaeTur7p7XvB6DWgUxrhEolqrBtX51elHH1A+Zn4GPf4wioztuyMQlUhklDWRbDKzK8wsPnhd\nAeiRclKlXXtSG4b3O3BeZFNWDre/8yNbsnL0fHipEsq61tZ1wDPAk4AD3wPXhCkmkZgQF2fcf1YX\n0trU58Y3phaqm7hsM8f/YRQAHRrX5JVretKyfvFXfonEurIukbLC3c9190bu3tjdz0dXbYkAcPox\nTbj3jE4l1i/KyOTWt6dVYEQiFas8z2zX8igigRsHHMWr1/QssX7mqm20ueczVmzKqsCoRCpGeRKJ\nFhgSKWBQp8bcf2ZnWtavRt3qxT9l4aY3p7EzJ4/duXsqODqR8CnrHElxdCuvSBHD+7djeP92AExd\nsZmLnp9QqH7e2u10eehLANo2rMGzl51wwGeIxJpSz0jMbIeZbS/mtYPQ/SQiUoIerevzh/O7lli/\nbGMWZz79LR8syuH59CWMnree/Hz9fSaxp9QzEnevVVGBiFRGV57YmuNa1OWcf4wvsc0nS3L5ZElo\nyZWbBhzFPaVM3ItEo/LMkYhIGRzbog5/vvBYWtSrdsDd8EX969ulmpCXmFOeORIRKaNLe7Xi0l6t\nAMjKzuOcf4xn6YYDE8aefOfJUQt56pLjKzpEkcOmMxKRClYjOYH3buzDA2d15sYB7aiXXPgCyI+m\nr2Hs/IwIRSdy6HRGIhIBDWomM6xf6Oqu3inreHxGPPPX7dhX/+DHsxnTfiBJCfpbT6KffkpFIizO\njMcu6kZ83P4zk1VbdtHxgf8x5Il0Ji7VsnYS3ZRIRKJA95Z1uSyYQyloyYYsbn5rGlnZeRGISqRs\nlEhEosTtg9uTknjgr+TmrByO+e2XfL94YwSiEjk4JRKRKNG4dgpP/eJ4mtROKbb+5remsW2XlqWX\n6KPJdpEoMrRrE04/JpXNWTn0e3wsO3P2r8m1bVcu3X/3FdUS42laN4X7zujMKV1SIxitSIjOSESi\njJnRoGYyz15e/Dpcu3L3sHRDFre9M01PYpSooEQiEqUGHd2YWQ+fRvWk4u+G352bT69HRjP4r+l8\nNnNtBUcnsp8SiUgUq5WSyItXppU4bwKwdGMWt749jT9/Po/xizaSnacl6qViaY5EJMqd3KEhE+4d\nzK7cPeTk5TPwr+lsLeZZ8P8ct5R/jltK77b1eXNYbxLj9XeiVAz9pInEADOjelICdasn8fLVPWle\nt1qJbScu20yH+//HhCW6kVEqhhKJSIzp0boe438ziFkPn8bgTo1LbHfta5NYuXlnBUYmVZUSiUgM\nMjNqpSTywhU9uO/MTgw8utEBbXbn5tPv8bEMf30K4xfpZkYJn7AmEjMbamYLzGyxmd1TTH09M/vQ\nzGaa2SQz61qg7k4zm21mc8zsrmL2/X9m5mbWMJx9EIlmSQlx3ND/KF67thcjrutVbJtRc9dz3YjJ\nLFq/o9h6kfIKWyIxs3jgWeAMoAtwqZl1KdLsPmC6u3cDrgL+HuzbFRgO9AK6A2ebWfsCn90SOA34\nKVzxi8SaAR0bMf2hU4uty8nL59Qnx3HlyxO57e1pfLNwQwVHJ5VZOM9IegGL3X2pu+cAI4HzirTp\nAowBcPf5QBszSwU6AxPdfae75wHfABcW2O9J4NeAHnAtUkDd6km8d1OfYtfsAvh20Ub+O3Mt1782\nmVmrtlVwdFJZhTORNAdWFtheFZQVNIMgQZhZL6A10AKYDfQzswZmVh04E2gZtDsPWO3uM8IYu0jM\n6tmmPlMeOJX3bupTYpu8fOecf4xnzPz1FRiZVFbmHp4/6s3sYmCouw8Ltq8Eerv7bQXa1CY0nHU8\nMAvoBAx39+lmdj1wC5AFzAGyCQ2FjQVOc/dtZrYcSHP3A2YSzewG4AaA1NTUHiNHjjysfmRmZlKz\nZs3D2jfaqC/RKZx9WbptD49N2k12Kfco3tgtmT7NjswtZTou0elw+zJo0KCp7p52sHbhTCR9gIfd\n/fRg+14Ad/9zCe0NWAZ0c/ftReoeIXRG8y0wGth7TWMLYA3Qy93XlRRLWlqaT5ky5bD6kZ6ezsCB\nAw9r32ijvkSncPdlx+5cZq/eTl5+Ple+PKnYNie3b8hLV6eRklj8cixlpeMSnQ63L2ZWpkQSzqGt\nyUAHM2trZknAJcAnBRuYWd2gDmAYMG5vEjGzxsHXVoSGv95291nu3tjd27h7G0LJ5YTSkohIVVcr\nJZE+RzWgX4dGfH5Hv0JPYtxr/OKNPPTxbFZu3knGDi0EKYcmbEukuHuemd0GfAnEA6+4+xwzuymo\nf4HQpPoIM3NCw1fXF/iI982sAZAL3OruW8MVq0hV0aVZbaY9cCoXv/A9izIyC9W9O2UV705ZBUBa\n63q8em1PaqUkRiJMiTFhXWvL3T8HPi9S9kKB9xOAjiXs268Mn9+mnCGKVDl1qify2R39uO/DWfxn\n6qpi20xZsYWnvl7Eg2cXvWJf5EC6s12kCkpKiOOvP+vOr4ceXWKbN39YoeedSJkokYhUYTf2P4rh\n/drStmENmtetRnLC/v8SsvPy+fvoRazfvpv123ezMTObcF2cI7FNy8iLVGHxccb9Z3Xh/rNCQ1hv\n/LCCBz+ava/+rYk/8dbE/QtIdGpSi1eu6UmzUlYflqpHZyQiss/P01qUukT9/HU7+OW/p5OfrzMT\n2U+JRET2SU6I5/bB7UttM3HZZv4xdnEFRSSxQIlERAr5Rc+W3DG4Pa0bVKdxrWQa10qmVnLhUfC/\njVpI/8fHsmqLnncimiMRkSLMjLtPO5q7T9t/Rdearbs4/clx7MjO21f20+adnPzYWN68vjcnd9DT\nHKoynZGIyEE1q1uNB88p/p6SK16eyHtTVhZbJ1WDzkhEpEx+1qMFG3Zk85cvFxxQ93//mcm0n7aw\nOSOb77Lm0rV5Hc7t3ozQEnpS2SmRiEiZmBm3DmrPhSc055xnxrMxM6dQ/TuTgrOS5csAmL5yK789\n55iKDlMiQENbInJImtapxie3nXzABHxRr363nC4PfcGcNXqAVmWnRCIih6xZ3Wp8d+9g2jWsUWq7\nnTl7OOvp8fztqwWMmrted8ZXUhraEpHDUjslkfdvPomv561n265c3OG/kxcyY8OBT9F6ekzovpPr\n+rbloRIm7SV2KZGIyGGrVyOJn6W13LfdIf8n1lRrx30fziq2/SvfLeOV75ZRIym+0ET83nfN6lbj\nwbO76HLiGKOhLRE5oi7r3Yo5vzuddo1KHvbKytlDZnbevteO4LVg/Q5ufGMKKzfrRsdYokQiIkdc\njeQE3ruxDw+c1Zlebesf0r5ZOXu494NZmk+JIRraEpGwaFAzmWH92jGsXzvenvgTf/58XqE740sz\nfvFGbn5zGn07NMQAMzAMM6hfI4n+HRpRLal8z5eXI0eJRETC7rLerfhFz5bsyg1NxBc823DAHa54\naSKzVu+/VPiLOev4Ys66Yj/vuJZ1GXnDiaQkKplEAw1tiUiFiI8zaiYnUDM5gVopiftetVMSqVMt\nkT+c37WkBVeBAAAQCElEQVTMnzV95VaeHr0ojNHKoVAiEZGocFzLupzbvVmZ2z+XvoQnvlrApGWb\nyd2TH8bI5GA0tCUiUePxi7vRo3U9FmXswH3/sBc47jB6fgYbdmTva//MmMU8M2YxPdvU481hvUlO\n0FBXJCiRiEjUSEmM5+qT2pRYP3Z+Bte+NvmA8snLtzD89alc3ac1EJqcr56UwHEt62oepQIokYhI\nzBjUqTEX92jBf6auOqBu3MINjFu4oVBZx9Sa/PuGPtSrkVRRIVZJSiQiElMeueBY2jSozuj5Gfz4\n09ZS2y5cn8nxfxhF/46NiDOCS4mNaonxDO3ahHMOYU5GSqZEIiIxJSkhjtsGd+C2wR144qsFPDPm\n4M+PL3qmAvDZrLUkJcRx+jFNwhFmlaJEIiIx665TOlI7JZFJyzezJz90b0runny+XbSxTPvf+MZU\nhnRqTHxc6GbHhPg4+ndoyM8LrB8mB6dEIiIxKz7OGN6/HcP7tytUPm/tdi771w9s2Zl70M8YPT+j\n0PZnM9eSs8dRKik7JRIRqXQ6N63NN78exOxV28jZk09wBTH57vzqvRkHTTAPfjSbDnXjeHPFFE5s\nV5/r+rYlLk6PDS6JEomIVEq1UxI5qf2By9H/787+XPDcd6zdtrvU/RdtzWfR1vV8PW89mdl53HVK\nx3CFGvOUSESkSmlSJ4Wv7x7A1BVb2JW7Z98Nj0+OWsSC9TuK3eef3yzlihNb07BmcoXGGiuUSESk\nyqmRnED/jo0KlZ3coRGXvDiB2au3H9B+V+4eXkhfwgNn6+mOxdFaWyIiQM3kBD66pS8f3nISI67r\nxdA2iYXq3/hhBeu3lz4cVlUpkYiIBBLi4zi+VT0GdGzEhR0SaVI7ZV9ddl4+z449+D0rVZESiYhI\nMZLijVsHty9UNnLSSlZv3RWhiKKXEomISAl+kdaS5nWr7dvO2ZPP41/Mj2BE0SmsicTMhprZAjNb\nbGb3FFNfz8w+NLOZZjbJzLoWqLvTzGab2Rwzu6tA+V/MbH6wz4dmVjecfRCRqispIY47hhQ+K/l4\n+hq+nrs+QhFFp7BdtWVm8cCzwKnAKmCymX3i7nMLNLsPmO7uF5hZp6D9kCChDAd6ATnAF2b2X3df\nDIwC7nX3PDN7DLgX+E24+iEiVduFJ7Tg1e+WM3/d/kuDf/P+TE6c3gDYvxDk3vftG9dkWL+2VE+q\nOhfFhrOnvYDF7r4UwMxGAucBBRNJF+BRAHefb2ZtzCwV6AxMdPedwb7fABcCj7v7VwX2/wG4OIx9\nEJEqLjE+jr/+rDvnPfvdvvW8NmXl8NnMtSXu8/W89Xx4S1/iq8jd8Oahu3GO/AebXQwMdfdhwfaV\nQG93v61Am0eAau7+SzPrBXwP9AZ2Ah8DfYBdwGhgirvfXuR7fAr8293fLOb73wDcAJCamtpj5MiR\nh9WPzMxMataseVj7Rhv1JTqpL9GpaF/eX5jDp0sPvnZXQXedkMxxjSN/ZnK4x2XQoEFT3T3tYO0i\n3cNHgb+b2XRgFvAjsMfd5wXDVl8BWcB0YE/BHc3sfiAPeKu4D3b3F4EXAdLS0nzgwIGHFWB6ejqH\nu2+0UV+ik/oSnYr2pc/Je1j30kQmL99S5s94ac4eRp3el6Z1qh28cRiF+7iEM5GshkILaLYIyvZx\n9+3AtQAWGmRcBiwN6l4GXg7qHiE0z0KwfQ1wNjDEw3VKJSJSQHJCPG8O682U5VvYsjNn3zPlAdyd\nnzbt5IlRCwvtk5mdx28/nsOLVx30j/qYFs5EMhnoYGZtCSWQS4DLCjYIrrja6e45wDBgXJBcMLPG\n7p5hZq0IzY+cGJQPBX4NDNg7hyIiUhGSE+LpW8xCkHt1b1mXq16ZVKjsq7nruf2dH3n0wmOpkRzp\nQaDwCNvlv+6eB9wGfAnMA9519zlmdpOZ3RQ06wzMNrMFwBnAnQU+4n0zmwt8Ctzq7nufqfkPoBYw\nysymm9kL4eqDiMih6N+xEUsfOZPjWxW+K+HTGWu49e1pVNYBlLCmR3f/HPi8SNkLBd5PAIpdm9nd\n+5VQ3r64chGRaBAXZ/z69E5c+q8fCpWnL9hA+sINDDq6cYQiCx/d2S4icoT1OaoB5x/X7IDy//fu\nDFZsymLN1l2s3RZ6rdu2m3XbdrN+e+i1K2dPMZ8Y3SrngJ2ISIT95WfdadeoJn8rMAG/OSuHAX9J\nL3W/hDjjhv7t+PXQTmGO8MjRGYmISBgkxsdxx5AOXHh880PaLy/feS59Cf+duSZMkR15SiQiImH0\n0DldaFTr0J+sePe/Z8TMSsNKJCIiYVS3ehIjru1Fr7b1aVonhaZ1UmhSO/RKrZ1Mau1kGtcKvQqu\nqJKzJ5++j47hkc/nRf3VXpojEREJsy7NavPujX0O2u7ZsYv5y5cLCpW9OG4pHVNrcXGPFuEKr9x0\nRiIiEiVu7N+O7i3qHFD+u0/nsHxjFlnZeezMySNvT34EoiuZzkhERKJEQnwc/7wyjWGvT2b26u37\nynfszmPgX9P3bScnxHFpr1b89pwu+5awjySdkYiIRJEmdVL47+39eOjsLiW2yc7L57Xvl/Phj6tL\nbFORlEhERKLQNSe1oVfb+qW2+fvoReRGwTCXEomISBSKizOevewEBnRsRK2UBGokxVM9Kb5QmxWb\ndvL+1FUlfELF0RyJiEiUalQrmRHX9SpUdu8Hs3hn0k/7tp8Zs5gLTmhOckJ80d0rjM5IRERiyO2D\n25MUv/+/7tVbd/HGhBVszMxmx+5De4LjkaJEIiISQ5rVrcZlvVsVKvvjZ/NI++PXHPvwV1z/2mR2\n51bswo9KJCIiMeaWgUeRnFD8f9+j52fw58/nsSkze99rd15474zXHImISIxpXDuFa05qwz/HLS22\nfsSEFYyYsGLf9pltExl6SvjiUSIREYlBd5/Wka07c/lm4QY278whJy9ylwErkYiIxKDkhHgeu7jb\nvu3n05fw2BfzIxKLEomISCVwQ/92rN++my/nrCO7yNlJcrzmSERE5CDi44yHzz2Gh8895oC69PT0\nsH5vXbUlIiLlokQiIiLlokQiIiLlokQiIiLlokQiIiLlokQiIiLlokQiIiLlYu7hvVElGpjZBmDF\nQRsWryGw8QiGE0nqS3RSX6KT+gKt3b3RwRpViURSHmY2xd3TIh3HkaC+RCf1JTqpL2WnoS0RESkX\nJRIRESkXJZKDezHSARxB6kt0Ul+ik/pSRpojERGRctEZiYiIlIsSSSnMbKiZLTCzxWZ2T6TjOVRm\nttzMZpnZdDObEpTVN7NRZrYo+Fov0nEWx8xeMbMMM5tdoKzE2M3s3uA4LTCz0yMT9YFK6MfDZrY6\nOC7TzezMAnVR2Q8AM2tpZmPNbK6ZzTGzO4PyWDwuJfUl5o6NmaWY2SQzmxH05XdBecUdF3fXq5gX\nEA8sAdoBScAMoEuk4zrEPiwHGhYpexy4J3h/D/BYpOMsIfb+wAnA7IPFDnQJjk8y0DY4bvGR7kMp\n/XgY+FUxbaO2H0F8TYETgve1gIVBzLF4XErqS8wdG8CAmsH7RGAicGJFHhedkZSsF7DY3Ze6ew4w\nEjgvwjEdCecBI4L3I4DzIxhLidx9HLC5SHFJsZ8HjHT3bHdfBiwmdPwiroR+lCRq+wHg7mvdfVrw\nfgcwD2hObB6XkvpSkmjui7t7ZrCZGLycCjwuSiQlaw6sLLC9itJ/0KKRA1+b2VQzuyEoS3X3tcH7\ndUBqZEI7LCXFHovH6nYzmxkMfe0dcoiZfphZG+B4Qn/9xvRxKdIXiMFjY2bxZjYdyABGuXuFHhcl\nksrtZHc/DjgDuNXM+hes9NB5bkxethfLsQPPExoyPQ5YCzwR2XAOjZnVBN4H7nL37QXrYu24FNOX\nmDw27r4n+F1vAfQys65F6sN6XJRISrYaaFlgu0VQFjPcfXXwNQP4kNDp63ozawoQfM2IXISHrKTY\nY+pYufv64Bc/H/gX+4cVor4fZpZI6D/et9z9g6A4Jo9LcX2J5WMD4O5bgbHAUCrwuCiRlGwy0MHM\n2ppZEnAJ8EmEYyozM6thZrX2vgdOA2YT6sPVQbOrgY8jE+FhKSn2T4BLzCzZzNoCHYBJEYivTPb+\ncgcuIHRcIMr7YWYGvAzMc/e/FaiKueNSUl9i8diYWSMzqxu8rwacCsynIo9LpK84iOYXcCahqzmW\nAPdHOp5DjL0doSszZgBz9sYPNABGA4uAr4H6kY61hPjfITS0kEtoDPf60mIH7g+O0wLgjEjHf5B+\nvAHMAmYGv9RNo70fQWwnExoemQlMD15nxuhxKakvMXdsgG7Aj0HMs4GHgvIKOy66s11ERMpFQ1si\nIlIuSiQiIlIuSiQiIlIuSiQiIlIuSiQiIlIuSiRSZZiZm9kTBbZ/ZWYPH6HPfs3MLj4Sn3WQ7/Mz\nM5tnZmPL2P6+cMckokQiVUk2cKGZNYx0IAWZWcIhNL8eGO7ug8rYXolEwk6JRKqSPEKPHP1l0Yqi\nZxRmlhl8HWhm35jZx2a21MweNbPLg+c/zDKzowp8zClmNsXMFprZ2cH+8Wb2FzObHCwEeGOBz/3W\nzD4B5hYTz6XB5882s8eCsocI3Uj3spn9pUj7pmY2LniGxmwz62dmjwLVgrK3gnZXBLFPN7N/mln8\n3v6a2ZPB8yxGm1mjcvw7SxWjRCJVzbPA5WZW5xD26Q7cBHQGrgQ6unsv4CXg9gLt2hBam+ks4AUz\nSyF0BrHN3XsCPYHhwbIUEHpOyZ3u3rHgNzOzZsBjwGBCiwf2NLPz3f33wBTgcnf/vyIxXgZ86aGF\n+7oD0939HmCXux/n7pebWWfgF0DfoN0e4PJg/xrAFHc/BvgG+O0h/PtIFXcop9QiMc/dt5vZ68Ad\nwK4y7jbZg+W4zWwJ8FVQPgsoOMT0rocW+1tkZkuBToTWOOtW4GynDqG1jXKASR56HkRRPYF0d98Q\nfM+3CD0g66PSYgReCRYi/MjdpxfTZgjQA5gcWmqKauxfyC8f+Hfw/k3ggwP2FimBEolURU8B04BX\nC5TlEZyhm1kcoadi7pVd4H1+ge18Cv8OFV1vyAk9ve52d/+yYIWZDQSyDi/8A7n7uOAxAWcBr5nZ\n39z99SLNDBjh7veW5SOPVGxS+WloS6ocd98MvEto2Gmv5YT+Wgc4l9BT5g7Vz8wsLpg3aUdoQbwv\ngZuDMwXMrGOwGnNpJgEDzKxhMIdxKaHhphKZWWtgvbv/i9CQ2wlBVe7e701oAb+LzaxxsE/9YD8I\n/V+w96zpMmB82bosojMSqbqeAG4rsP0v4GMzmwF8weGdLfxEKAnUBm5y991m9hKhuZNpwdLlGzjI\n443dfa2Z3UPouRIGfObuB1vufyDwf2aWC2QCVwXlLwIzzWxaME/yAPBVcNaVC9wKrCDU315BfQah\nuRSRMtHqvyKCmWW6e81IxyGxSUNbIiJSLjojERGRctEZiYiIlIsSiYiIlIsSiYiIlIsSiYiIlIsS\niYiIlIsSiYiIlMv/B+l//9suuh1XAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266a1a9ca58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_history_CD, lw=4)\n",
    "plt.grid()\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Number of step')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id='section6'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# SDCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Nous allons maintenant implémenter la version la plus classique de l'algorithme (http://www.jmlr.org/papers/volume14/shalev-shwartz13a/shalev-shwartz13a.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Nous définissons les fonctions de coût (primal et dual) qui permettront de monitorer l'avancée de l'algorithme. \n",
    "Remarque : la fonction P est identique à la fonction f_hinge définie plus haut, mais nous avons décidé de la redéfinir ici pour faire correspondre les notations aux notations de l'article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def P(theta, lam):\n",
    "    cost = 1-np.multiply(b,X.dot(theta))\n",
    "    cost[cost <= 0] = 0\n",
    "    return np.mean(cost) + lam * np.linalg.norm(theta, ord=1)\n",
    "\n",
    "def D(alpha, lam):\n",
    "    cost = np.multiply(b,X.dot(theta))\n",
    "    cost[cost > 0] = np.nan\n",
    "    cost[cost < -1] = np.nan\n",
    "    return cost                    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def SDCA(T, lam, eta):\n",
    "    # on déclenche le timer\n",
    "    t0 = time.time()\n",
    "    \n",
    "    # initialisation\n",
    "    alpha = np.zeros(n_samples)\n",
    "    w = (1/(lam*n_samples))*np.dot(X.T,alpha)\n",
    "    \n",
    "    # historique des fonctions de perte\n",
    "    cost_history = np.empty(shape=[T])\n",
    "    time_history = np.empty(shape=[T,2])\n",
    "    \n",
    "    for t in range(0,T):\n",
    "        cost_history[t] = P(w, lam)\n",
    "        time_history[t] = (time.time()-t0, P(w, lam))\n",
    "        \n",
    "        # .. pick random sample ..\n",
    "        idx = np.random.randint(0, n_samples)\n",
    "        \n",
    "        # on calcule les constantes A et B (utiles pour la résolution du delta alpha i)\n",
    "        A = (1/(lam*n_samples)) * np.linalg.norm(X[idx])**2\n",
    "        B = X[idx].T.dot(w)    \n",
    "        \n",
    "        y_i = b[idx]\n",
    "        delta_alpha_i_tilde = (y_i-B)/A        \n",
    "        delta_alpha_i = y_i * np.clip(y_i*(delta_alpha_i_tilde + alpha[idx]), a_min=0, a_max=1) - alpha[idx]\n",
    "        \n",
    "        alpha[idx] = alpha[idx] + delta_alpha_i\n",
    "    \n",
    "        w = (1/(lam*n_samples))*np.dot(X.T,alpha)\n",
    "        \n",
    "    return w, cost_history, time_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "theta, cost_history_SDCA, time_history_SDCA = SDCA(T=T,lam=lam,eta=eta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH1pJREFUeJzt3X2UXHWd5/H3Jx0CSYghSGgxiSZgkAdX0GkCs8too4JB\nRyOOMwPq6rKOWWaND7NnOBOdOY47ntnF5ejoKk6MDqKzIjOuZoizOUQyWgQ8KgnS5AkCTYOkm0AI\nkUCHkKS7v/tH3ZDqW9XVt5u+VdVVn9c5fdL3sb6/uiQf7u/e+7uKCMzMzEYzpd4FmJnZ5ODAMDOz\nTBwYZmaWiQPDzMwycWCYmVkmDgwzM8vEgWFmZpk4MMzMLBMHhpmZZTK13gVMpFNOOSUWLlw4rm0P\nHDjAzJkzJ7agOnFbGk+ztAPclkY13rbcc889eyNibpZ1myowFi5cyObNm8e1baFQoLOzc2ILqhO3\npfE0SzvAbWlU422LpN9kXdddUmZmlokDw8zMMnFgmJlZJg4MMzPLxIFhZmaZ5BYYkm6UtEfSthGW\nS9L/ltQtaYukN5YsWyppZ7JsZV41mplZdnneVnsT8DXguyMsvxxYnPxcCPw9cKGkNuAG4FKgF9gk\naW1E7JjoAh988jk+e+s2hobgt88cZP2+LfzP975+oj/GzKwp5BYYEbFR0sIqqywDvhvFd8T+UtJJ\nkk4DFgLdEdEDIOmWZN0JD4z+QwP8smffi9Mzdj830R9hZtY06vng3jxgV8l0bzKv0vwLR9qJpOXA\ncoD29nYKhULmAnr2Dw6b3r//2TFt36j6+/uboh3QPG1plnaA29KoatGWSf+kd0SsBlYDdHR0xFie\ndDylbz/84q4Xp2fMPJHOzt+b6BJrzk+vNp5maQe4LY2qFm2pZ2D0AQtKpucn844bYf6Ek4ZPD0Xk\n8TFmZk2hnrfVrgU+lNwtdRGwPyJ2A5uAxZIWSZoGXJmsO+HapgxPDAeGmdnIcjvDkPR9oBM4RVIv\n8NcUzx6IiFXAOuAdQDfwPHB1smxA0gpgPdAG3BgR2/OosS11ijE45MAwMxtJnndJXTXK8gA+NsKy\ndRQDJVdKBYZPMMzMRtbST3qnu6QGnRhmZiNq7cBwl5SZWWYtHRhTUq33CYaZ2chaOzB8hmFmlllL\nB4avYZiZZdfSgZE+wwgHhpnZiFo8MIZPu0vKzGxkLR0YZV1SDgwzsxG1dGBMmeIH98zMsmrtwEjf\nJeXEMDMbUUsHhh/cMzPLrqUDww/umZll19qB4S4pM7PMWjow3CVlZpZdSwdG+i4p8MN7ZmYjaenA\nAD+8Z2aWVcsHhseTMjPLJtfAkLRU0k5J3ZJWVlg+R9IaSVsk3S3pdSXLHpW0VVKXpM151Vg+nlRe\nn2RmNrnl+U7vNuAG4FKgF9gkaW1E7ChZ7TNAV0RcIemsZP23liy/JCL25lUjeIhzM7Os8jzDWAJ0\nR0RPRBwGbgGWpdY5B/gpQEQ8ACyU1J5jTWXcJWVmlk2egTEP2FUy3ZvMK3Uf8F4ASUuAVwPzk2UB\nbJB0j6TleRWZvugdQ3l9kpnZ5JZbl1RG1wFfkdQFbAXuBQaTZRdHRJ+kU4HbJT0QERvTO0jCZDlA\ne3s7hUJhTAUMDg4Mm954113MmlZ+u+1k0t/fP+bvoVE1S1uapR3gtjSqWrQlz8DoAxaUTM9P5r0o\nIp4FrgaQJOARoCdZ1pf8uUfSGopdXGWBERGrgdUAHR0d0dnZOaYiT7jzdg4cOfzi9EW/+++ZO+v4\nMe2j0RQKBcb6PTSqZmlLs7QD3JZGVYu25NkltQlYLGmRpGnAlcDa0hUknZQsA/gTYGNEPCtppqRZ\nyTozgcuAbXkUWT7Eua9hmJlVktsZRkQMSFoBrAfagBsjYruka5Llq4Czge9ICmA78JFk83ZgTfGk\ng6nAzRFxWx51lj2458AwM6so12sYEbEOWJeat6rk918AZ1bYrgc4L8/ajvJ4UmZm2bT8k95+656Z\nWTYODJ9hmJll0vKB4Qf3zMyyafnAKHtwz4FhZlaRA6OsS6pOhZiZNbiWD4yyLilfwzAzq6jlAyN9\nhjHkLikzs4ocGKlvwIFhZlZZyweGH9wzM8um5QMj/eCe88LMrDIHhq9hmJll0vKB4S4pM7NsWj4w\nfNHbzCwbB0a6S8oP7pmZVdTygeGxpMzMsmn5wPBFbzOzbBwYqcEHh3zR28ysolwDQ9JSSTsldUta\nWWH5HElrJG2RdLek12XddqJ4LCkzs2xyCwxJbcANwOXAOcBVks5JrfYZoCsiXg98CPjKGLadEOVd\nUnl8ipnZ5JfnGcYSoDsieiLiMHALsCy1zjnATwEi4gFgoaT2jNtOCF/DMDPLJs/AmAfsKpnuTeaV\nug94L4CkJcCrgfkZt50Q7pIyM8tmap0//zrgK5K6gK3AvcDgWHYgaTmwHKC9vZ1CoTCmAvY+9cKw\n6W3bdzDrtw+OaR+Npr+/f8zfQ6NqlrY0SzvAbWlUtWhLnoHRBywomZ6fzHtRRDwLXA0gScAjQA8w\nfbRtS/axGlgN0NHREZ2dnWMqcs0T98ITj784fdbZZ9H5hvlj2kejKRQKjPV7aFTN0pZmaQe4LY2q\nFm3Js0tqE7BY0iJJ04ArgbWlK0g6KVkG8CfAxiRERt12opSPJZXHp5iZTX65nWFExICkFcB6oA24\nMSK2S7omWb4KOBv4jqQAtgMfqbZtHnWWDW/uaxhmZhXleg0jItYB61LzVpX8/gvgzKzb5qHswT3f\nJWVmVlHLP+ntsaTMzLJp+cAoH63WgWFmVokDw096m5ll0vKB4Qf3zMyyafnA8NAgZmbZODB8l5SZ\nWSYtHxjlXVJ1KsTMrMG1fGCUPbjnMwwzs4ocGH7jnplZJi0fGGVjSfkMw8ysopYPDI8lZWaWjQPD\nD+6ZmWXS8oHhsaTMzLJp+cDwWFJmZtk4MPzgnplZJi0fGH5wz8wsm5YPDI8lZWaWjQPDXVJmZpnk\nGhiSlkraKalb0soKy2dL+rGk+yRtl3R1ybJHJW2V1CVpc141enhzM7Nscnunt6Q24AbgUqAX2CRp\nbUTsKFntY8COiHiXpLnATknfi4jDyfJLImJvXjWCx5IyM8sqzzOMJUB3RPQkAXALsCy1TgCzJAk4\nEdgHDORYU5n00CBDvuhtZlZRbmcYwDxgV8l0L3Bhap2vAWuBx4FZwB9HxNF/sgPYIGkQ+EZErK70\nIZKWA8sB2tvbKRQKYyrywd4jw6b7du+mUNg3pn00mv7+/jF/D42qWdrSLO0At6VR1aIteQZGFm8H\nuoC3AGcAt0u6MyKeBS6OiD5JpybzH4iIjekdJEGyGqCjoyM6OzvHVMDee3ph230vTp96ajudneeP\ntz0NoVAoMNbvoVE1S1uapR3gtjSqWrQlzy6pPmBByfT8ZF6pq4EfRVE38AhwFkBE9CV/7gHWUOzi\nmnBtqW/A1zDMzCrLMzA2AYslLZI0DbiSYvdTqceAtwJIagdeC/RImilpVjJ/JnAZsC2PItPPYQw6\nL8zMKsqtSyoiBiStANYDbcCNEbFd0jXJ8lXA54GbJG0FBPxFROyVdDqwpngtnKnAzRFxWx51eiwp\nM7Nscr2GERHrgHWpeatKfn+c4tlDerse4Lw8azsq/RyGu6TMzCrzk96pJ7394J6ZWWUODI8lZWaW\nScsHRnmXVJ0KMTNrcC0fGGV3STkxzMwqcmD4oreZWSaZAkPSGZKOT37vlPQJSSflW1ptlI0l5cAw\nM6so6xnGD4FBSa+hOAzHAuDm3KqqId8lZWaWTdbAGIqIAeAK4KsRcS1wWn5l1U5Zl5RHqzUzqyhr\nYByRdBXwYeBfk3nH5VNSbfnBPTOzbLIGxtXA7wJ/GxGPSFoE/GN+ZdVOWZeUA8PMrKJMQ4Mkb8n7\nBICkOcCsiPhCnoXVSvq22ucPDdapEjOzxpb1LqmCpJdJOhn4NfBNSV/Kt7TaSHdJ7XzyOe586Kk6\nVWNm1riydknNTl5q9F7guxFxIfC2/MqqnfQZBsDXf/ZwHSoxM2tsWQNjqqTTgD/i2EXvpnDG3BPL\n5j3Vf6gOlZiZNbasgfE3FN9r8XBEbEreV/FQfmXVzvRpbXz+Pa8bNs/PYpiZlct60fsHwA9KpnuA\nP8irqFp70+JThk07MMzMymW96D1f0hpJe5KfH0qan3dxteIBCM3MRpe1S+rbFN/H/crk58fJvKok\nLZW0U1K3pJUVls+W9GNJ90naLunqrNtOpKltwwNjwI97m5mVyRoYcyPi2xExkPzcBMyttoGkNuAG\n4HLgHOAqSeekVvsYsCMizgM6gS9KmpZx2wmTvrV20HlhZlYma2A8LemDktqSnw8CT4+yzRKgOyJ6\nIuIwcAuwLLVOALMkCTgR2AcMZNx2wqRHrB30GYaZWZmsgfGfKd5S+wSwG3gf8J9G2WYesKtkujeZ\nV+prwNnA48BW4JMRMZRx2wkzdcrwr8HXMMzMymW9S+o3wLtL50n6FPDll/j5bwe6gLcAZwC3S7pz\nLDuQtBxYDtDe3k6hUBhzEQcHhgfE4SMD49pPo+jv75/U9ZdqlrY0SzvAbWlUtWhLpsAYwX+jemD0\nUXxvxlHzk3mlrgaui4gAuiU9ApyVcVsAImI1xXd00NHREZ2dnWNoQtHBw4Ow4bZj+9QUxrOfRlEo\nFCZ1/aWapS3N0g5wWxpVLdryUl7RWj6mxnCbgMWSFkmaBlxJ8U6rUo8BbwWQ1A68FujJuO2E8RDn\nZmajeylnGFX/VY2IAUkrKD4h3gbcGBHbJV2TLF8FfB64SdJWigH0FxGxF6DSti+h1qrSgTHgaxhm\nZmWqBoak56gcDAKmj7bziFgHrEvNW1Xy++PAZVm3zUv6nRgRMDQUZW/jMzNrZVUDIyJm1aqQepLE\nFEHpicVgBFNG7XUzM2sdL+UaRlMpe/Oeu6XMzIZxYCQcGGZm1TkwEqnhpHzh28wsxYGRSJ9hDDkw\nzMyGcWAk0oHhMwwzs+EcGIn0AIR+eM/MbDgHRsJnGGZm1TkwEmV3SQ06MMzMSjkwEum7pAbdJWVm\nNowDI6Gy5zD8EiUzs1IOjETZGYbzwsxsGAdGYorSI9Y6MczMSjkwEh4axMysOgdGorxLyoFhZlbK\ngZHwGYaZWXUOjIQDw8ysulwDQ9JSSTsldUtaWWH5tZK6kp9tkgYlnZwse1TS1mTZ5jzrBAeGmdlo\nXso7vauS1AbcAFwK9AKbJK2NiB1H14mI64Hrk/XfBfxZROwr2c0lR9/xnTc/uGdmVl2eZxhLgO6I\n6ImIw8AtwLIq618FfD/HeqpS2W21Dgwzs1J5BsY8YFfJdG8yr4ykGcBS4IclswPYIOkeSctzqzJR\ndobhsaTMzIbJrUtqjN4F/DzVHXVxRPRJOhW4XdIDEbExvWESJssB2tvbKRQK4yogBgeAY6lx39at\nTN1z/7j2VW/9/f3j/h4aTbO0pVnaAW5Lo6pFW/IMjD5gQcn0/GReJVeS6o6KiL7kzz2S1lDs4ioL\njIhYDawG6OjoiM7OznEV+9V7bwMGX5w++5xz6fx3p41rX/VWKBQY7/fQaJqlLc3SDnBbGlUt2pJn\nl9QmYLGkRZKmUQyFtemVJM0G3gzcWjJvpqRZR38HLgO25VirH9wzMxtFbmcYETEgaQWwHmgDboyI\n7ZKuSZavSla9AvhJRBwo2bwdWJNciJ4K3BwRt+VVK/i2WjOz0eR6DSMi1gHrUvNWpaZvAm5KzesB\nzsuztrTywQcdGGZmpfykdyLdJTXkwDAzG8aBkUi/QMlnGGZmwzkwEn7S28ysOgdGouyit1+5Z2Y2\njAMjkQ4Md0mZmQ3nwEikA2PIXVJmZsM4MBJtvq3WzKwqB0ai7AzDgWFmNowDI+FrGGZm1TkwEn5w\nz8ysOgdGwg/umZlV58BIeLRaM7PqHBiJ9OCDDgwzs+EcGAlf9DYzq86BkSi76O0H98zMhnFgJHyG\nYWZWnQMjUT74oAPDzKyUAyPh4c3NzKrLNTAkLZW0U1K3pJUVll8rqSv52SZpUNLJWbadaH6nt5lZ\ndbkFhqQ24AbgcuAc4CpJ55SuExHXR8T5EXE+8GngjojYl2Xbiebbas3MqsvzDGMJ0B0RPRFxGLgF\nWFZl/auA749z25fMZxhmZtVNzXHf84BdJdO9wIWVVpQ0A1gKrBjHtsuB5QDt7e0UCoVxFXv40AvA\nsdR4Ys+ece+r3vr7+ydt7WnN0pZmaQe4LY2qFm3JMzDG4l3AzyNi31g3jIjVwGqAjo6O6OzsHFcB\n9/zTBuDQi9NzTj6Fzs6Oce2r3gqFAuP9HhpNs7SlWdoBbkujqkVb8uyS6gMWlEzPT+ZVciXHuqPG\nuu2EKB9Lyu/0NjMrlWdgbAIWS1okaRrFUFibXknSbODNwK1j3XYilV3D8CUMM7NhcuuSiogBSSuA\n9UAbcGNEbJd0TbJ8VbLqFcBPIuLAaNvmVStUukvKZxhmZqVyvYYREeuAdal5q1LTNwE3Zdk2Tx7e\n3MysOj/pnfBttWZm1TkwEh580MysOgdGwu/0NjOrzoGR8BmGmVl1DoyEr2GYmVXnwEikb6vtPzRQ\np0rMzBqTAyMxrW34dO9vD/LBb/2K3x44XJ+CzMwajAMjMXe6mD9n+rB5d3Xv5dr/u6VOFZmZNRYH\nRkISf/XOs8vmb7j/Sbb17a9DRWZmjcWBUeLt576Cq5a8qmz+73/1Lv51y+N1qMjMrHE4MEpI4n9c\n8TpWXPKasmV/d/uDdajIzKxxODBSJPFfLzmD6ccNvwq+e/8LdarIzKwxODAqmDFtKjd/dPgL/g4e\nGfTT32bW0hwYI3jDq+Zw/NRjX08EvDAwWMeKzMzqy4FRxczjh4/+/vxhB4aZtS4HRhUzUk/zPX/I\ngWFmrcuBUUU6MA4c9nAhZta6cg0MSUsl7ZTULWnlCOt0SuqStF3SHSXzH5W0NVm2Oc86RzJjmruk\nzMyOyu0VrZLagBuAS4FeYJOktRGxo2Sdk4CvA0sj4jFJp6Z2c0lE7M2rxtGUdUn5DMPMWlieZxhL\ngO6I6ImIw8AtwLLUOu8HfhQRjwFExJ4c6xmz9BnGAV/DMLMWlmdgzAN2lUz3JvNKnQnMkVSQdI+k\nD5UsC2BDMn95jnWOaObxw88wDh7xGYaZta7cuqTG8Pm/A7wVmA78QtIvI+JB4OKI6Eu6qW6X9EBE\nbEzvIAmT5QDt7e0UCoVxFdLf31+27TN7Dw2bvnfr/czZ3z2u/ddSpbZMVs3SlmZpB7gtjaoWbckz\nMPqABSXT85N5pXqBpyPiAHBA0kbgPODBiOiDYjeVpDUUu7jKAiMiVgOrATo6OqKzs3NcxRYKBdLb\n3tm/gzt6H3lxesHCM+h80+nj2n8tVWrLZNUsbWmWdoDb0qhq0ZY8u6Q2AYslLZI0DbgSWJta51bg\nYklTJc0ALgTulzRT0iwASTOBy4BtOdZakW+rNTM7JrczjIgYkLQCWA+0ATdGxHZJ1yTLV0XE/ZJu\nA7YAQ8C3ImKbpNOBNSq+NnUqcHNE3JZXrSPxbbVmZsfkeg0jItYB61LzVqWmrweuT83rodg1VVfp\ni96+rdbMWpmf9K4iPcS5hwYxs1bmwKgiPfigr2GYWStzYFRR/qS3zzDMrHU5MKrwRW8zs2Pq/eBe\nQ/MZhlntde16hm/c8TBP9x/O/bOe2X+Qrz/wi9w/pxae2X+Q15z3PPPnzMjtMxwYVaQD4+Gn+nn3\n1+4aNi8qvLU1iNHXqbhdep1sr4RNr9Z/4Hlm3ntH1XWyfl7FCmL0dbLsK8t398LBFzj+Vz8ddbvx\nfH6lfaU/f6TPO/H4qbz5tXP5o44FnDrr+FHrefZw8HT/oVHXawSDQ8H9TzxHz1P9Fdve/egReu46\n9kDrcVOncO4rX8YrZ0+neCf8cBVmjbhw9zMv8J4bfj6uusftt/tq+3k5euHIUK77d2BUkb7ofXhg\niC29++tUzRj199e7golz8GC9Kyiz57lD9Ow9wLd//mj2jX66Ibd6au6BHaOvY03H1zCqmD39uGHv\n9TYza2U+w6jihOPa+NPOM/jyhofqXYpZS/r+Ry9iStU+rZemq6uL888/P78PqKGuri7mnTQ9189w\nYIziU287kysveBVPPPtC2bJK/x2n+3BVYa1K/bwTsZ+j05s2beaCCzoqbpe1j7l8vQyfX3E/yrBO\nej/HZvzyV7/kogsvGtd3NhGfX2mdCPjVI0/zg829PPDEswxluKZy5PBhjps2bfQVG8SMaW288VVz\nOHlmec19fb3Mmzf/xeld+57n/t3PMlDhi6j21VS7FnXuK1/GF/7g9bxi9gljKXvMDj7WxoWnvzzX\nz6iVg4+1MT113XWiOTAyeMXsE3L/D3ciPTFrCme94mX1LmNC9MyYwqtent9dH+O17Px5LDs//XqX\nkTXXqKhP0dl5br3LsDpwB72ZmWXiwDAzs0wcGGZmlokDw8zMMnFgmJlZJg4MMzPLxIFhZmaZKOsA\nd5OBpKeA34xz81OAvRNYTj25LY2nWdoBbkujGm9bXh0Rc7Os2FSB8VJI2hwRHfWuYyK4LY2nWdoB\nbkujqkVb3CVlZmaZODDMzCwTB8Yxq+tdwARyWxpPs7QD3JZGlXtbfA3DzMwy8RmGmZll0vKBIWmp\npJ2SuiWtrHc9YyXpUUlbJXVJ2pzMO1nS7ZIeSv6cU+86K5F0o6Q9kraVzBuxdkmfTo7TTklvr0/V\nlY3Qls9J6kuOTZekd5Qsa+S2LJD0M0k7JG2X9Mlk/qQ6NlXaMemOi6QTJN0t6b6kLf89mV/bYxIR\nLfsDtAEPA6cD04D7gHPqXdcY2/AocEpq3v8CVia/rwS+UO86R6j9TcAbgW2j1Q6ckxyf44FFyXFr\nq3cbRmnL54A/r7Buo7flNOCNye+zgAeTmifVsanSjkl3XCi+++vE5PfjgF8BF9X6mLT6GcYSoDsi\neiLiMHALsKzONU2EZcB3kt+/A7ynjrWMKCI2AvtSs0eqfRlwS0QciohHgG6Kx68hjNCWkTR6W3ZH\nxK+T358D7gfmMcmOTZV2jKQh2wEQRf3J5HHJT1DjY9LqgTEP2FUy3Uv1/6AaUQAbJN0jaXkyrz0i\ndie/PwG016e0cRmp9sl6rD4uaUvSZXW0u2DStEXSQuANFP+PdtIem1Q7YBIeF0ltkrqAPcDtEVHz\nY9LqgdEMLo6I84HLgY9JelPpwiien07KW+Emc+2Jv6fY3Xk+sBv4Yn3LGRtJJwI/BD4VEc+WLptM\nx6ZCOyblcYmIweTv+nxgiaTXpZbnfkxaPTD6gAUl0/OTeZNGRPQlf+4B1lA87XxS0mkAyZ976lfh\nmI1U+6Q7VhHxZPKXfAj4Jse6BBq+LZKOo/iP7Pci4kfJ7El3bCq1YzIfF4CIeAb4GbCUGh+TVg+M\nTcBiSYskTQOuBNbWuabMJM2UNOvo78BlwDaKbfhwstqHgVvrU+G4jFT7WuBKScdLWgQsBu6uQ32Z\nHf2LnLiC4rGBBm+LJAH/ANwfEV8qWTSpjs1I7ZiMx0XSXEknJb9PBy4FHqDWx6TeV//r/QO8g+Ld\nEw8Df1nvesZY++kU74S4D9h+tH7g5cC/AQ8BG4CT613rCPV/n2KXwBGKfawfqVY78JfJcdoJXF7v\n+jO05R+BrcCW5C/waZOkLRdT7NrYAnQlP++YbMemSjsm3XEBXg/cm9S8DfhsMr+mx8RPepuZWSat\n3iVlZmYZOTDMzCwTB4aZmWXiwDAzs0wcGGZmlokDw5qKpJD0xZLpP5f0uQna902S3jcR+xrlc/5Q\n0v2SfpZx/c/kXZMZODCs+RwC3ivplHoXUkrS1DGs/hHgoxFxScb1HRhWEw4MazYDFF9V+WfpBekz\nBEn9yZ+dku6QdKukHknXSfpA8v6BrZLOKNnN2yRtlvSgpN9Ptm+TdL2kTcmAdv+lZL93SloL7KhQ\nz1XJ/rdJ+kIy77MUHzj7B0nXp9Y/TdLG5B0O2yT9nqTrgOnJvO8l630wqb1L0jcktR1tr6S/S96n\n8G+S5r6E79lakAPDmtENwAckzR7DNucB1wBnA/8RODMilgDfAj5est5CimMPvRNYJekEimcE+yPi\nAuAC4KPJcAxQfEfGJyPizNIPk/RK4AvAWygOgneBpPdExN8Am4EPRMS1qRrfD6yP4gB05wFdEbES\nOBgR50fEBySdDfwx8B+S9QaBDyTbzwQ2R8S5wB3AX4/h+zFjLKfJZpNCRDwr6bvAJ4CDGTfbFMkw\n0ZIeBn6SzN8KlHYN/XMUB617SFIPcBbFMbxeX3L2Mpvi2D2Hgbuj+D6CtAuAQkQ8lXzm9yi+hOlf\nqtUI3JgMqPcvEdFVYZ23Ar8DbCoOpcR0jg1INwT8U/L7/wF+VLa1WRUODGtWXwZ+DXy7ZN4AyVm1\npCkU37J41KGS34dKpocY/vckPZZOUHwb2scjYn3pAkmdwIHxlV8uIjYmw9e/E7hJ0pci4rup1QR8\nJyI+nWWXE1WbtQZ3SVlTioh9wD9T7C466lGK//cN8G6Kby0bqz+UNCW5rnE6xYHd1gN/mvyfP5LO\nTEYPruZu4M2STkmuMVxFsZtoRJJeDTwZEd+k2FX2xmTRkaOfTXEguvdJOjXZ5uRkOyj+fT96FvR+\n4K5sTTYr8hmGNbMvAitKpr8J3CrpPuA2xvd//49R/Mf+ZcA1EfGCpG9RvLbx62RI7acY5bW4EbFb\n0kqK7zUQ8P8iYrRh6DuBayUdAfqBDyXzVwNbJP06uY7xV8BPkrOoI8DHgN9QbO+SZPkeitc6zDLz\naLVmLUJSf0ScWO86bPJyl5SZmWXiMwwzM8vEZxhmZpaJA8PMzDJxYJiZWSYODDMzy8SBYWZmmTgw\nzMwsk/8PSdmsMKMBNLEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266a1ae82b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_history_SDCA, lw=4)\n",
    "plt.grid()\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Number of step')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id='section7'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Comparaison des algorithmes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Comparons tous ces algorithmes sous deux critères : \n",
    "- le nombre d'itérations nécessaire\n",
    "- le temps nécessaire\n",
    "\n",
    "Il est important de comparer nos algorithmes sous ces deux angles car le temps d'exécution d'un algorithme est fonction des performances d'une machine donnée. En effet, les résultats sont biaisés par les différentes optimisations de la machine et ces optimisations n'ont pas de lien avec l'algorithme en lui-même.\n",
    "\n",
    "A l'inverse, le nombre d'itérations est une mesure objective de la performance d'un algorithme, qui ne variera pas d'une machine à une autre, ni d'une implémentation à une autre. Son inconvénient est que certaines itérations d'algorithmes sont très longue (batch gradient descent) par rapport à d'autres (coordinate descent). Il est donc aussi difficile de comparer objectivement chaque algorithme.\n",
    "\n",
    "Les graphiques produit ci-dessous permettent néanmoins de se faire une idée (grossière) de la réalité :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAEjCAYAAABw0XjjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8VNX5P/DPMzPZCSGBEEISCYshCwGUCHVBQKsCBRRX\nVECpfi1VW6sWsVpbf1oXWqhFtFW+2K8stlRREQVLsRBxZTUsgSSEQMhCgJCE7Mksz++Peye5985k\nZSbr8351Xpl77rn3njvBzpPnnHsOMTOEEEIIITzJ1NkNEEIIIUTPIwGGEEIIITxOAgwhhBBCeJwE\nGEIIIYTwOAkwhBBCCOFxEmAIIYQQwuMkwBANiOhdIvqDl859LxH9p5n9k4ko3xvXbo4379mbiIiJ\naEQnXXskEaURUQUR/bIz2iCE6PokwOiFiCiViEqJyK+jrsnM7zHzjZo2dNoXpLhoTwHYwczBzPx6\naw8iovuJ6GsvtksI0YVIgNHLEFEsgIkAGMCsDrqmpSOuI9qunb+bIQDSPd0WIUTPIgFG7zMfwPcA\n3gVwX3MViegpIjpNRIVE9KA260BEIUS0hojOEVEuEf2WiEzqvvuJ6Bsieo2IzgN4XvvXKxHtVC9x\ngIgqieguzTWfJKKz6nUXaMrfJaK/EtHn6jHfENEgIvqLmo3JIKLLNPUXE1GBmsbPJKLrm7nVAUS0\nTa37JRENUc/xJhEtM3wmm4jo8SY+LyaihUR0jIjK1ONJ3fc8Ea3T1I1V61vU7VQi+gMRfave36dE\n1J+I3iOiciLaowaHWtOJKIeIionoT87PXz3fT4noqPrZbHXek6adjxDRMQDHmriXWUSUrt5HKhEl\nqOXbAUwB8Ibazjg3x96vtquCiE6o3WMJAN4CcKV6XJla14+IlhLRKSI6Q0RvEVGAum8yEeUT0TPq\nPZ4konvdtVcI0QUxs7x60QtANoCHAYwDYAUQodn3LoA/qO+nAigCkAQgEMA6KFmPEer+NQA+ARAM\nIBZAFoAH1H33A7AB+AUAC4AAtexrzbUazqVuT1aPeQGAD4DpAKoBhGraVqy22x/AdgAnoARMZgB/\ngJK2B4CRAPIADFa3YwEMb+LzeBdABYBrAfgBWO5sJ4DxAAoBmNTtAWqbIpo4FwP4DEA/AJcAOAdg\nqrrveQDrNHVj1foWdTtV/d0MBxAC4Ij6mf5Y/QzXAPg/w7V2AAhTr5UF4EF1383quRLUY38L4FvD\nsdvUYwPc3EccgCoAN6i/i6fU8/lq2vpgE59BEIByACPV7UgASZp/F18b6r8GYJPalmAAnwJ4xfBv\n4s/q72aS2q6Rnf3fkbzkJa+WX5LB6EWI6Boo6e33mXkfgOMA7mmi+p1QvtDSmbkayhek8zxmAHMA\n/IaZK5j5JIBlAOZpji9k5hXMbGPmmlY20QrgBWa2MvMWAJVQggWnj5l5HzPXAvgYQC0zr2FmO4B/\nAXBmMOxQvpASiciHmU8y8/FmrruZmXcycx2AZ6H8lR3DzLsBXADgzH7MAZDKzGeaOderzFzGzKeg\nBABjW3nvgPJ5H2fmCwA+B3Ccmb9gZhuADzT357SEmUvUa/0FwN1q+UIoX9JH1WNfBjBWm8VQ95c0\n8bu5C8pnso2ZrQCWQgkSr2rlfTgAjCKiAGY+zcxuu1PU7M5DAB5X21KhtnWOoepzzFzHzF8C2Azl\n36YQoouTAKN3uQ/Af5i5WN3+B5ruJhkMJQvgpH0/AMpftrmaslwAUU3Ub63z6heiUzWAPppt7Rd7\njZvtPgDAzNkAfgUlKDpLROuJaHAz121oKzNXAiiBcv8AsBrAXPX9XABrW7iHomba35JW3Z+G9jPO\nRWObhwBYrnZvlEG5H0Lrfz+DofndMrNDrR/V5BGNdaugBCgLAZwmos1EFN9E9XAo2bF9mrb+Wy13\nKlXP6aS9TyFEFyYBRi+h9mvfCWASERURURGAxwGMIaIxbg45DSBasx2jeV8MJdug/Yv4EgAFmu1O\nXaaXmf/BzM6MDQNY0kz1hnsjoj5Q0vWFatE6ADern1ECgI3tbFIVlC9Tp0HtPI+W9ndyCRrbnAfg\nZ8zcT/MKYOZvNfWb+/0UQvO7VTMNMdD/fpvEzFuZ+QYo3SMZAP63iWsWQwmckjTtDGFmbSAVSkRB\nTdynEKILkwCj97gFStdBIpS0/VgoX5hfQRnHYPQ+gAVElEBEgQCec+5QuyTeB/ASEQWrqfcnoHwZ\nt9YZAMPacyMtIWWehutIeQy3FsqXmKOZQ6YT0TVE5AvgRQDfM3MeADBzPoA9UDIXH7ahu8coDcC1\nRHQJEYUA+E07z6O1iIhCiSgGwGNQuokAZTDlb4goCWgYkHtHG877PoCfENH1ROQD4EkAdQC+bf4w\ngIgiiOhmNSiog9LN5fzszwCIVj9nZ2bkfwG8RkQD1eOjiOgmw2n/HxH5EtFEADOgdBcJIbo4CTB6\nj/ug9PGfYuYi5wvAGwDuJcPjisz8OYDXoYwjyIby5AmgfGkAygDOKgA5AL6G0t3y9za053kAq9XU\nuKf71P0AvArlL+QiAAPR/Bf6PwD8HkpXwjg0dok4rQaQjJa7R5rEzNugBAAHAeyDMhj0Yn2inisN\nytiEd9RrfQwlY7OeiMoBHAYwrQ1tzYTyGayA8hnOBDCTmetbcbgJSrBZCOXznATg5+q+7VAeby0i\nImc33WKo/77Utn4B/bibIgCl6vneA7CQmTNaey9CiM5DzJ2ayRbdhPqY4WEAfoZxEj0eEV0LJTsz\nhOU/mA5DRJOhPHkT3VJdIUTXIxkM0SQimq3OUxAK5S/iT3thcOEDpfthlQQXQgjRehJgiOb8DMBZ\nKI+z2tGY6u4V1KxNGZTBin/p5OYIIUS3Il0kQgghhPA4yWAIIYQQwuMkwBBCCCGEx0mAIYQQQgiP\nkwBDCCGEEB4nAYYQQgghPE4CDCGEEEJ4nAQYQgghhPA4CTCEEEII4XESYAghhBDC4yTAEEIIIYTH\nSYAhhBBCCI+TAEMIIYQQHicBhhBCCCE8TgIMIYQQQnicpbMb4EkDBgzg2NjYdh1bVVWFoKAgzzao\nk8i9dD095T4AuZeuqr33sm/fvmJmDr+Ya+/bt2+gxWJZBWAU5A/X3sIB4LDNZntw3LhxZ91V6FEB\nRmxsLPbu3duuY1NTUzF58mTPNqiTyL10PT3lPgC5l66qvfdCRLkXe22LxbJq0KBBCeHh4aUmk4kv\n9nyi63M4HHTu3LnEoqKiVQBmuasjkaYQQoiLNSo8PLxcgovew2QycXh4+AUoWSv3dTqwPUIIIXom\nkwQXvY/6O28yjpAAQwghhBAeJwGGEEKIbs9sNo+Lj49PvPTSS5OmTZs2rKKiost/v40fP37kzp07\nA5ur89lnnwVPmTJlREe1yai4uNj86quvtmsQcJf/BQghhBAt8fPzc2RkZBw5duxYuo+PDy9btkz3\npehwOGC3273eDqvV6vVrdKTz58+b33nnnYHtOdZrT5EQ0d8BzABwlpldBoEQEQFYDmA6gGoA9zPz\nfnXfVHWfGcAqZn7VW+0UQgjhGbFPbx7nzfOffPUn+1pT75prrqk8ePBgQGZmpu9NN90Ud9lll1Ue\nOnQoaMuWLcd27NjRZ9myZYOYmX784x+X/e1vfyvIysryveGGG+K+//77jIEDB9rGjx8/8tlnnz19\n6623lmvP+9prrw1Yvnz5oODgYHtSUlK1r68vr1mz5tRtt90W6+fn5zh8+HDg+PHjK++9996Sxx9/\n/JK6ujqTv7+/49133z0xZsyYusrKSpozZ87QI0eOBAwfPry2traW3LV/w4YNfRctWhQTEBDgGD9+\nfKWzvLy83PTAAw9ckpGREWCz2ejZZ58tnDt3btnevXv9FyxYMNRqtZLD4cCHH354PDk5ue6NN97o\n//rrr0cQERISEmo2btx4orCw0LJgwYIhBQUFvgDw5z//+dSNN95Y9cQTTwzOy8vzzc3N9SssLPRd\nuHDhmd/+9rdnn3zyyei8vDy/+Pj4xEmTJpW//fbb+a39fXnzMdV3AbwBYE0T+6cBuFR9TQDwNwAT\niMgM4E0ANwDIB7CHiDYx8xFPN3DjiiU4+e1RgBmAA4fffh+DBl4P/2FDYfK1wGQmmEwEk8UEs5lg\nMptgsqg/zQSTmRrLzQQyqfU1711/orGe87wWE8wWgsnkPD/BbDE1XL/hWDPBRMq2EEIIV1arFVu3\nbu174403lgPAqVOn/N55550T119//cmTJ0/6PP/881H79u07Gh4ebps4cWLc2rVr+82bN6/sscce\nK/rpT396SUpKStXIkSNrjcHFyZMnfZYuXRq5f//+I/369XNcddVVcUlJSTXO/adPn/bdv39/hsVi\nQUlJiWnPnj0ZPj4+2LhxY/BTTz0VvXXr1uNLly4dGBAQ4MjJyUnftWtXwNVXX51obH91dTU9+uij\nsdu2bctMSkqqmzFjxjDnvmeeeSZyypQp5R988MHJ4uJic0pKSsKsWbPKV6xYEf7www+f+fnPf15S\nW1tLNpsNe/fu9V+6dGnkd999lxEZGWk7c+aMGQB+9rOfxTzxxBNnbrrppspjx4753nTTTZfm5OSk\nA0B2drb/t99+m1lWVmZOSEgYtWjRonPLli3LnzFjRkBGRkabv4O9FmAw804iim2mys0A1jAzA/ie\niPoRUSSAWADZzJwDAES0Xq3r8QCj6vx52B3FDdsOkx/O1IUCR8s8fSnPIsCsBjvOnyYTgUgJYGpr\nHSjY8b0amAAgJRAyW7THNAY3zmDJZDapgY9S16wGPkqw0/je+TKZCSAo1yUY3is/iZqu0xA4kSYo\n0wZVZoLdyrBbHQ37hRDCnbq6OlN8fHwiAEyYMKHiscceK87NzfWJjIysv/7666sA4Ouvvw760Y9+\nVDF48GAbANx1110lX375ZZ958+aVPfHEE8UfffRR6Lvvvht+8OBBl++br776KmjChAkVERERdgCY\nPXt2aVZWlr9z/6233lpqsShfqSUlJea77rpr6MmTJ/2JiK1WK6nX7/PLX/7yrNrGmri4uGrjddLS\n0vyjo6PrkpOT6wDg3nvvPb9q1apwAEhNTe27devWfq+//vog9Z4pOzvb98orr6xaunRpZH5+vu+c\nOXNKk5OT67Zu3dp35syZpZGRkTYAcLb7m2++6Xvs2LEA5/UqKyvNFy5cMAHAjTfeWBYQEMABAQG2\nsLAwa35+/kXFCJ050VYUgDzNdr5a5q58QlMnIaKHADwEABEREUhNTW11A2rqjE9VdZOnrBiw2xyw\n2wAr3Pcp1le6/LvttjI+TFXeENQMENQARVNmBkzqTzIDJjMaAholqDH8NJ5P9yLlPJrrNPdye37D\nz+qaamz7fEfDMSazc3/3C5oqKyvb9N9ZVyb30nM4x2AYywMDAx2tOb6iosJUVFTkCwDl5eXm0NDQ\nVh3n1KdPn4b6ixcvjpo0aVLFtm3bjmdmZvped911I9tyrqYwMzZs2JA9ZsyYOm355ZdfXjtx4sSq\njz/+OGTGjBmXrlixosnJ05gZ+/fvPxoYGOjyhefn59dQZjabYbPZLur/oLr9TJ7MvBLASgBISUnh\ntsxkV3X4CC7kHNWUtOnfk+hoDLBdeXn9Qh4X4HpegppF0mRu1C4wMsGwre2S03bTqd1zhIbMj/N4\n57EmbWZIPQe57b5r/Gm26K9j1mS6DhYdxIgRo1uXpWrY1xhMuWSsNFkrZz3te5gM5/MgmcnTs1o7\nRqKzTJw4seqpp56KOX36tCU8PNz2wQcfhD388MNnAeDRRx+Nuv32288PGTKk/v777x+yY8eObO2x\n11xzTdXTTz8dc+7cOXO/fv3sn3zySWhCQkKNu+uUl5ebo6Oj6wHg7bffHqA5R+V7770XNmvWrIo9\ne/b4Z2VluTxBMnbs2NqCggLf9PR0v6SkpLr169eHOfdNmTKlfNmyZRHvvvvuKZPJhG+++Sbg6quv\nrjly5IhvQkJCXVJS0tlTp075pqWlBUyfPr389ttvH/Hss88WDRo0yH7mzBlzRESE/Zprril/5ZVX\nBr744otnAODbb78NuOqqq9zeBwCEhITYq6qq2vVASGcGGAUAYjTb0WqZTxPlHhc6Ohb4qnE7oOYc\nkk++CfYPQNSy12C3O8AOht3GcNgdcNgZdpvyU3mp7x3c8JOd5QywWs4O5eVo+Amww3k+5zn157er\n52E7K+eyN55D9BCaTFR3k5ua1inXdQZJZCKoPXENgY6SmSJN5qjpOs5MVW2tA0Vf7Va6/3xMsPio\nXYA+jV2BusCHAJiUsVDawMcZgGm7DxuO03Rf6n7q3mvqqdcyadrk7L40Bp+gxvf2euX/H0zSldik\nIUOGWH//+98XTJo0Kc45yHPu3Lllmzdv7pOWlhb0zjvvZFgsFnz88cehy5cv7//YY4+ddx47dOhQ\n6+OPP346JSUlISQkxDZixIjakJAQt3/uLF68uOjBBx8cumTJksE33HBDQ5/7r3/967Nz5swZOmzY\nsKQRI0bUJiYmVhmPDQwM5BUrVuTOmDFjREBAgGPChAmVlZWVZgB49dVXCx966KFL4uPjEx0OB8XE\nxNTt2LEje926dWHvv/9+f4vFwuHh4dYXX3zxdEREhP3JJ588PXHixHiTycSjRo2q/vDDD0+uXLky\n78EHH7wkLi4u0W6304QJEyquuuqqU019ZoMGDbKPGzeu8tJLL0267rrrLrRlkCcpQyC8Qx2D8VkT\nT5H8BMCjUJ4imQDgdWYeT0QWAFkArocSWOwBcA8zp7d0vZSUFG7LWiT707/CjheWNGz3qanHtVlK\n70x8+mGQ2dzqc3UUZgYz4LA5lCDE5oDdxkq5GoDs+n4XrrhivBLYaAIdu93hcpwzoFHqOPcpde02\nh66esq15b1fOD2YwlOs0tNEBw/vGtgBQgi11Wxd82fXvrfU2gAkOuwRWQrRkzu/Go//gPm06hoj2\nMXPKxVz3wIEDJ8eMGVPccs3u7cKFC6aQkBCH1WrFTTfdNOL+++8vnj9/fhcftOddBw4cGDBmzJhY\nd/u8+ZjqPwFMBjCAiPIB/B5KdgLM/BaALVCCi2woj6kuUPfZiOhRAFuhPKb699YEF+1hMelvnzWB\nv6OqCua+fb1x2Yvi/AvH5Gtu8pfn15cQFtkzVoh0pn2Z1UyRzU3myO5QBoPaHLBZHbBb7bBZlTpg\nZzDDDcGOMxhjh7KvIXCyO3TvnddyZpOcwZXSDkdjZsuhBkqszVBxQ5nDwaitqYOPj69ynFW5jhdj\ne9FLdccxPd3JokWLBu/cubNvXV0dTZo0qXzu3Lm9OrhoiTefIrm7hf0M4JEm9m2BEoB4ldnSTIBR\nWdklA4zeipxPwpgBJe7sXpRA6RpdmcPhGrjoAiBuDIIasjt2Q9CjljUET85skCGI0h3vaMxasTMA\nszMcbKhn01/HGVCVlpSiX2i/FrNUzI0ZN6g/ne+dwaH2eo333Xic9r5Ey6R7xLtWrlzZ6u4B0QMG\neV4Mi8nHUNL4H6e9shLGvUJ4kslEMJnM6G7/0JRg6fIOv64zeHIGI1D+1xh8aAMaoLGOWq4vUwKX\n77/fhZRxKWoGzA5bvUPtAmTYrfbG7kdndx67noNZM3bK6lAzYZogkQE4nAEc1AALDUGXMyDUtt8Z\neDkzctogzNk1qT0/OxhWq00e5RZdSu8OMMyGDIbmvaPSZeyNEKITkYlgNhHMHvx/Lf8QwoDoYM+d\nsBOlpqYiJDyg5YpCdJBevRaJxaz/01E/BqMSQgghhGifXh5gGP8UaowwHJUSYAghhBDt1asDDLOx\ni0STwbBLgCGEEN2Gc7l25yszM9NXu9R5Xl6eZcqUKSNGjhyZOHz48KRJkyaN2L17d4CzfkhIyNio\nqKjk+Pj4xKuuuirOed4XXnhhoJ+f3+Xnz5/vfqPLO1kvH4Nh6CLRZTBkDIYQQnQX7qYKP3bsmJ/z\n/eLFi6Ouu+668ueee+4sAOzatStg/PjxNc5jbrvtttgZM2ZcWLBgQan2HBs2bAgbNWpU1bp16/pp\nJ94SLevVAYaP2Ve3bXxMVQghRBs8H+LV5drx/IV2T0VeVFTkc+ONN15wbk+YMKHJ6bGd0tPT/aqr\nq83Lly/PffnllyMlwGibXt1F0vxTJBJgCCFEd+FcTTU+Pj7xhhtuGG7c/8gjj5z9xS9+ETthwoS4\nxYsXDzp58mSLD4ivWbMmdPbs2SVTp06tPHHihH9eXl6v/qO8rXp1gGEcgwHNLHh2eYpECCG6DWcX\nSUZGxpFt27YdN+6/7bbbyrOzsw8tWLCgODMzM2DcuHGJhYWFzQYMH330Uf/58+eXmM1mTJ8+vXTt\n2rWh3ruDnqdXBxgmw1ojMg+GEEL0XBEREfaFCxeWbNy48cTo0aOr/vOf/zS5cMvu3bsDcnNz/aZO\nnRoXFRWVvGnTprANGzb078j2dne9Ot1DJn18pR2DUb1rVwe3RgghurmLGCPhbZs2bQqeMmVKVXBw\nsKO0tNSUm5vrN3To0Pqm6q9ZsybsySefLHzllVeKnGVRUVHJWVlZvnFxcU0eJxr16gwGkSHA0E4V\nXlqKkn/8o6ObJIQQwgv27NkTOHbs2IS4uLjE8ePHJ8ybN6940qRJ1U3V37hxY9idd96pW8xs2rRp\npatXrw7zfmt7hl6dwTA1k8EAgLJ/rkfYPfd0YIuEEEK0R3V19Q/GshkzZlTMmDGjAgBefPHFMy++\n+OKZpo7/8MMPT2q38/PzDxnrrFq1ShY7a4PencEwBBgg0o3DsJ2XJ5KEEEKI9ujdAQY1v/KgPKoq\nhBBCtE+vDjAAwEGs22bNNtfXw1EvY3mEEEKItur1AQaMSQz95J5wVMnjqkIIIURbeTXAIKKpRJRJ\nRNlE9LSb/aFE9DERHSSi3UQ0SrPvJBEdIqI0ItrrrTYaB3bCR5/RkABDCCGEaDuvPUVCRGYAbwK4\nAUA+gD1EtImZtYvRPAMgjZlnE1G8Wv96zf4pzFzsrTaqDdVtsjHAkHEYQgghRJt5M4MxHkA2M+cw\ncz2A9QBuNtRJBLAdAJg5A0AsEUV4sU0ujBkMNsxOLwGGEEJ0D4sXLx40YsSIpLi4uMT4+PjEJ598\nMvKRRx6J0tb59ttvA4YNG5YEKBNnxcXFJcbFxSVeccUVI7Oysho6yfPy8iwzZ84cGh0dnZyUlJQw\nduzY+DVr1vTr6Hvqzrw5D0YUgDzNdj6ACYY6BwDcCuArIhoPYAiAaABnoMzc/QUR2QG8zcwr3V2E\niB4C8BAAREREIDU1tU2NZDC0AzEchgAj7bvvUN/NgozKyso2fw5dVU+5l55yH4DcS1fVk+6lPb74\n4ougrVu39jt06NCRgIAAPn36tCUtLc3/f/7nf4a++eabBc5669atC7v11ltLnNtffvllVmRkpO3x\nxx8f/Lvf/S5y/fr1uQ6HAzNnzhxxzz33nP/0009PAEBWVpbvBx98IAFGG3T2RFuvAlhORGkADgH4\nAYBd3XcNMxcQ0UAA24gog5l3Gk+gBh4rASAlJYUnT57cpgZ8//elum2HjwNA4xolSUOHIaSN5+xs\nqampaOvn0FX1lHvpKfcByL10VV3hXpJXJ3t1ufZD9x1qcirygoICn7CwMFtAQAADQGRkpC0yMrIy\nJCTEtn379qDrrruuCgA2bdoU9vnnn2cZj7/66qsrV6xYMRAAPv3002AfHx9+6qmnzjn3x8XF1T/7\n7LNnPX9XPZc3u0gKAMRotqPVsgbMXM7MC5h5LID5AMIB5Kj7CtSfZwF8DKXLxfNcxmDot2WQpxBC\ndH233HJLeWFhoW9sbOyouXPnXrJ58+Y+AHDbbbeVvPfee2EA8N///jeoX79+tuTk5Drj8Vu2bAmZ\nOXNmGQAcOnQoYPTo0U1OIy5ax5sBxh4AlxLRUCLyBTAHwCZtBSLqp+4DgAcB7GTmciIKIqJgtU4Q\ngBsBHPZKKw1jMOwyyFMIIbqdkJAQx+HDh4+88cYbueHh4bb77rtv+Ouvv95//vz5JZs3bw612+14\n7733wm677bYS7XGTJk2KGzhw4Ojt27eHPPDAAyXuzj1v3rxLRo4cmThq1KiEjrmbnsFrAQYz2wA8\nCmArgKMA3mfmdCJaSEQL1WoJAA4TUSaAaQAeU8sjAHxNRAcA7AawmZn/7ZV2GjIYxjEYjioJMIQQ\nojuwWCyYMWNGxWuvvVb4pz/96dTGjRtDR4wYYY2Ojq7bsmVL8JYtW0Lnz5+vCyK+/PLLrIKCgoNJ\nSUnVixYtGgwAycnJNQcPHgx01lm7du2p1NTUrNLS0s4eVtCtePXDYuYtALYYyt7SvP8OQJyb43IA\njPFm2xq0kMGwSwZDCCFapbkxEt524MABP5PJBGf3xw8//BAQHR1dDwB33HFHyaJFi2JiYmLqhg8f\nbjUe6+Pjg7/+9a95Y8eOTXzppZdOz5w5s+K5556jJUuWhC9evPgcAFRWVsrElG0kH5hJH2HYDWMw\nai9UdGRrhBBCtEN5ebl5/vz5Q4cPH54UFxeXmJGREbBkyZJCAJg/f35pdna2/x133OG2CwQAhgwZ\nYp01a1bJ0qVLB5pMJnz66afHv/rqq+CoqKjk5OTkhLlz58Y+//zzsppqG0i6x9BFYrfoMxiVJRc6\nsjVCCCHaYeLEidU//PBDhrt9kZGRNpvNtt9YXlBQoFuSffXq1Q1TKwwZMsT62Wef5Xi+pb2HZDCM\nXSSGkKumrLzj2iKEEEL0EL0+wDAu2W4zDPLsc2gfbKWlHdgiIYQQovvr9QGGcQyGzU2nUdHvft9B\njRFCCCF6BgkwDBkMq59rleo9ezqoMUIIIUTPIAGGIYNh9QcCB+onebNXVIBZP/hTCCGEEE3r9QGG\ncQyGnQmXTDqvr2S3y5ThQgghRBv0+gADJv1HYAeBzIAlwK4rd1yQx1WFEKIrMy7Xvn379iAAOH36\ntMVisVz+xz/+MVxb/8KFC6Z77733kpiYmFGJiYkJSUlJCcuWLRugrfPCCy8M9PPzu/z8+fNmiDbp\n9QGGMYPhUJ9bNfs6dOX2cnlcVQghuirtcu1ZWVlHduzYkTVs2LB6AFizZk3omDFjqj744IMw7TH3\n3ntvbGgLbecjAAAgAElEQVRoqP3kyZOHjxw5cnTbtm3HSkpKdEP9N2zYEDZq1KiqdevWyVLtbdTr\nJ9pyeUyVAZCbAOOCBBhCCNGco/EJXl2uPSHjaJuWa3fu++CDD8KWLl2ad9999w07fvy4z/Dhw63p\n6el+aWlpQZ988kmO2awkJwYPHmx76aWXipzHpaen+1VXV5uXL1+e+/LLL0c+9thj510uLJrU6zMY\nxkGeDvUjMblkMKSLRAghuqqmlmvPzs72OXfunM+UKVOqZ82aVbpmzZowAEhLS/NPSEiodgYX7qxZ\nsyZ09uzZJVOnTq08ceKEf15eXq//o7wten2AQYYxGM6wwuxrWLZdukiEEKLLamq59jVr1oTNmjWr\nFADmzZtX8uGHH4a5O37x4sWD4uPjEwcOHDjaWfbRRx/1nz9/fonZbMb06dNL165dG9pR99MT9Ppo\nzOUpkibGYNSWlHVYm4QQQrSdc7n2GTNmVIwePbpm7dq1/c+cOeNz7tw5n48++igMAM6ePetz6NAh\nvzFjxtQePXo00G63w2w2Y8mSJUVLliwpCgwMvAwAdu/eHZCbm+s3derUOACwWq0UHR1d/8wzz5zr\nzHvsTiTAMJmgzVU4mAACTD76AOP86WIM6timCSFEt9LcGAlvc7dcu91uR1VVlfns2bMHnfUef/zx\nwatXrw5bunTp6dGjR1c99thjUX/5y18KLBYLqquryTnn0Zo1a8KefPLJwldeeaVhTEZUVFRyVlaW\nb1xcXH2H32A3JF0kxi4SNaFhzGCUn5OxPUII0VW5W649JSWlavr06brFpObMmVPqzGasW7fuZElJ\niWXIkCHJo0aNSpg4cWLc7373u3wA2LhxY9idd96pS11PmzatdPXq1W67WISrXp/BcFmunU3qUyT6\nMRjV56WLRAghuqrmlmvXmjBhQk1OTk46AISFhTn+8Y9/5Lqrl5+ff8hYtmrVqvyLb2nv4dUMBhFN\nJaJMIsomoqfd7A8loo+J6CAR7SaiUa091lNMTQ7y1GcwrDLRlhBCCNFqXgswiMgM4E0A0wAkArib\niBIN1Z4BkMbMowHMB7C8Dcd6pp3GAIOVjIbxMVWukKdIhBBCiNbyZgZjPIBsZs5h5noA6wHcbKiT\nCGA7ADBzBoBYIopo5bEe4RJgmH0AuHaR+FZXoc6mnz5cCCGEEO55cwxGFIA8zXY+gAmGOgcA3Arg\nKyIaD2AIgOhWHgsAIKKHADwEABEREUhNTW1TI6uqquCj2bY1jMHQZzCCrNX44PMvER3c9cfFVlZW\ntvlz6Kp6yr30lPsA5F66qp50L6Jn6OxBnq8CWE5EaQAOAfgBQJvSBMy8EsBKAEhJSeHJkye3qQFH\nvt6AGmgGGVv8ADtgNjym2sdai36XjMTkMVFtOn9nSE1NRVs/h66qp9xLT7kPQO6lq+pJ9yJ6Bm8G\nGAUAYjTb0WpZA2YuB7AAAEiZ8eoEgBwAAS0d6ynGLhI7KR8JmQEyO8B2Zb+ZHcjJPQd0gwBDCCGE\n6GzezPfvAXApEQ0lIl8AcwBs0lYgon7qPgB4EMBONeho8VhPMT5FwqbGmMs4DqMw74w3miCEEMID\nTp06ZZkxY8awmJiYUUlJSQmTJk0acfDgQT9/f//LExISEocNG5aUnJyc8Prrr/fv7Lb2Bl7LYDCz\njYgeBbAVgBnA35k5nYgWqvvfApAAYDURMYB0AA80d6w32mnMYDA1Lnxj9nXAVtO4faZQZogVQoiu\nyOFwYNasWSPuueee85999lkOAHz33XcBhYWFPjExMXVHjx49AgBHjhzxvfXWW0cwM2R1VO/y6hgM\nZt4CYIuh7C3N++8AxLX2WG8wmfQr6dnRuG18VLXsbAnqbQ74Wrr+QE8hhOhoby7c7tXl2h9567om\npyL/7LPPgi0WCz/11FMNfwleeeWVNZmZmb7aeomJifV//OMf8xYvXhwjAYZ39fpvSmOAYcxgaAXU\nVSOnuLJD2iWEEKL1Dh48GDBmzJjq1tS96qqrqk+cOOHv7Tb1dhJguMzk2bht9tGPwQiur0bG6YoO\naZcQQgjvcC5oJryr1wcYroudNa5NYsxg9LFWI6NIAgwhhOhqkpOTaw4cOBDYmrrfffdd4LBhw2q8\n3aberrPnweh0Ll0k3BhwGMdg9LHWIKNIpgwXQgh3mhsj4W0zZ86seO6552jp0qUDfv3rXxcDwK5d\nuwJKS0t1/yefmZnp+/TTT0f/7Gc/O9s5Le09JMBweYpEm8EwdpHUSBeJEEJ0QSaTCZs2bTr+8MMP\nxyxfvnyQn58fR0dH161YsSIvLy/PLyEhIbGuro6CgoIcCxcuPPvLX/5SBnh6mQQYLhmM5rpIalBU\nXouy6nr0C9QNTBZCCNHJYmNjrVu2bMkxltfW1u7vjPb0dr1+DIbZbAgwNO9du0iUAcoyDkMIIYRo\nXq8PMMiQwXA0l8GoV8YEZZyWcRhCCCFEc3p9gOEyBoMZsAQAcB2D0ceqBBiZZySDIYQQQjSn1wcY\nZpN+GAo7HEBAP2WfIYMRXK90kRyVgZ5CCCFEsyTAMI7BcDgA/xBln2HJ9iBrDYgdyCyqgN0hE7UI\nIYQQTZEAw10Gw1/JYJAZMFkagwwzGIHWOtRY7ThRXNWh7RRCCCG6EwkwzPoAw8GNXSSA65MkweqT\nJOmFF7zfOCGEEK22ePHiQSNGjEiKi4tLjI+PT9y+fXvQ+PHjR8bGxo6Ki4tLHDp0aNL8+fMvKS4u\nbkhdN7XEu3P/Cy+8MNDPz+/y8+fPm91fVTSl1wcYFrOPblubwQDcTbblDDDkSRIhhOgqvvjii6Ct\nW7f2O3To0JGsrKwjO3bsyBo2bFg9AKxZsyYnKyvryNGjR4/4+fk5pk2bNgJoXOL92muvrcjLyzuc\nnp5+9NVXXy0oLCxs+GLYsGFD2KhRo6rWrVvXr6lrC/d6/URbFmMGw6HPYLgM9FQzGIcLJIMhhBBa\ny+6a4dXl2p/812dNTkVeUFDgExYWZgsICGAAiIyMtBnr+Pv789/+9rf8IUOGJH/33XcB586ds7hb\n4t35Pj093a+6utq8fPny3JdffjlSlndvm16fwTB2kbhkMPzcP0mSXlguK/IJIUQXccstt5QXFhb6\nxsbGjpo7d+4lmzdv7uOunsViQUJCQvXhw4f9W1rifc2aNaGzZ88umTp1auWJEyf88/Lyev0f5W3R\n6wMMH7ddJCEN2009qnqhxor8UlmMTwghuoKQkBDH4cOHj7zxxhu54eHhtvvuu2/466+/3t9d3db+\ncfjRRx/1nz9/fonZbMb06dNL165dG+rRRvdwXo3GiGgqgOUAzABWMfOrhv0hANYBuERty1Jm/j91\n30kAFQDsAGzMnOKNNrodg9FcF0l9Y7CbXngBMWGtWh1YCCGEl1ksFsyYMaNixowZFaNHj65Zu3at\nS4Bhs9mQmZkZOHr06MKioiLbxo0b3QYNu3fvDsjNzfWbOnVqHABYrVaKjo6uf+aZZ865qy9ceS3A\nICIzgDcB3AAgH8AeItrEzEc01R4BcISZZxJROIBMInqPmevV/VOYudhbbQTcP0XCfiFwThju0kVi\n1QYY5Zg6KtKbzRNCiG6juTES3nbgwAE/k8mE5OTkOgD44YcfAqKjo+szMjICnHXq6urosccei4qM\njKyfMGFCjcPhQFNLvH/22WchTz75ZOErr7xS5Dw+KioqOSsryzcuLq7etQXCyJtdJOMBZDNzjhow\nrAdws6EOAwgmIgLQB0AJAJeBOd5kXE2VmGDz79uw3VwGQwZ6CiFE11BeXm6eP3/+0OHDhyfFxcUl\nZmRkBCxZsqQQAObPnz8sLi4uceTIkUlVVVWmzz//PBtoXOJ9+/btfWNiYkaNGDEiafHixVFRUVHW\njRs3ht15551l2mtMmzatdPXq1WGdcX/dkTe7SKIA5Gm28wFMMNR5A8AmAIUAggHcxczOb3QG8AUR\n2QG8zcwr3V2EiB4C8BAAREREIDU1tU2NLD6Wpds2MbDrcAauUbebekwVAPadKG7z9TpCZWVll2xX\ne/SUe+kp9wHIvXRVPele2mPixInVP/zwQ4axfPfu3ZnNHdfUEu/5+fmHjGWrVq3Kv7hW9i6dPSL2\nJgBpAK4DMBzANiL6ipnLAVzDzAVENFAtz2DmncYTqIHHSgBISUnhyZMnt6kB6WRH7o6turLkH00E\n1ESfMYPR19Y4sLO8nhF/2Y8wKMS/Tdf0ttTUVLT1c+iqesq99JT7AOReuqqedC+iZ/BmF0kBgBjN\ndrRaprUAwEesyAZwAkA8ADBzgfrzLICPoXS5eBwZVlMlJlh9GwduGmfy7O+o1W2n5ekyaEIIIYSA\ndwOMPQAuJaKhROQLYA6U7hCtUwCuBwAiigAwEkAOEQURUbBaHgTgRgCHvdFIY4BhYsBm9gFMytMl\nxkGefev1j0wfyJcAQwjR6zkcDge1XE30JOrv3NHUfq8FGMxsA/AogK0AjgJ4n5nTiWghES1Uq70I\n4CoiOgTgvwAWq0+NRAD4mogOANgNYDMz/9sb7TQZAgwwYHXYmlyy3a+6EiZuLDsgGQwhhDh87ty5\nEAkyeg+Hw0Hnzp0LQTN//Ht1DAYzbwGwxVD2luZ9IZTshPG4HABjvNk2J+UBFs02A1aHVZnNs+oc\nTGbA5OOAw2pS9zsQXF+FC37BAICD+RfgcDBMJvnvSgjRO9lstgeLiopWFRUVjYJM4NhbOAActtls\nDzZVobMHeXY6ly4SkBJgaCbbsgTYUW9trDcENTgIJcCorLMhp7gSIwYGd0yDhRCiixk3btxZALM6\nux2ia+n1kSa5zIPhzGA0Thdu8dd3k1zWR7+dlifzYQghhBBavT7AcD8Gw6pb8Mzib9dVSfCz6rZl\nHIYQQgih1+sDDOMYDBMTrHZDF4khgzHUpH9UVZ4kEUIIIfQkwHCZB8M1g2EO0AcYEbYq3fbR0+Wo\nteqzHEIIIURvJgGGsYsEcB3kaegi8blQipiwgMb6dsbBfBmHIYQQQjhJgEHGibbIzRgMfQajPi8P\nM22FGFZWALCyVsne3BLvN1YIIYToJlr1mCoRDQeQz8x1RDQZwGgAa5i52w8+MA7yJIYyBkP3FIk+\ng1F78CBmHjyImQD+del1eDdpOvbnlnZEc4UQQohuobUZjA8B2IloBJSFxWIA/MNrrepATY7B0M2D\n0eRMqLjl+E5YHDbsyy0FMzdZTwghhOhNWhtgONSpv2cDWMHMiwBEeq9ZHcclwIBrF4nZ1wFzgPuZ\nOv0cNoTWVqC02orj56rc1hFCCCF6m9YGGFYiuhvAfQA+U8t8vNOkjkWGKb7Dy/xQV1ujy2CQCRh0\nFcPUty9ABBiCkrDacgDAPhmHIYQQQgBofYCxAMCVAF5i5hNENBTAWu81q+OYDDN5AkDZ1r26DAYA\n9B1chrhd3yP+4AEETbxGty+stgIAsPekjMMQQgghgFYO8mTmIwB+CQBEFAogmJmXeLNhHcW/j+sa\nIvW55wC/YIDMAKsDPK3VILsV8PGFJTxcVz+sVnlEdZ8M9BRCCCEAtDKDQUSpRNSXiMIA7Afwv0T0\nZ+82rWOEDIwAwgJ1ZWy3K10hmidJAAC1ykMzPgMH6oqdGYyc4ioUV9Z5r7FCCCFEN9HaLpIQZi4H\ncCuUx1MnAPix95rVsfxvv0K3zQ71aZAAfTcJapQAw5jB6K+OwQCAXTkyDkMIIYRobYBhIaJIAHei\ncZBnj+Hj46svYPWxVMM4DGcGw6WLpK4xwPg+57zH2yeEEEJ0N60NMF4AsBXAcWbeQ0TDABzzXrM6\nlsWiDzDYoQYYTWUwDF0k0RXnGt5LgCGEEEK0MsBg5g+YeTQz/1zdzmHm21o6joimElEmEWUT0dNu\n9ocQ0adEdICI0oloQWuP9SQfs+GJW7vaRdLKDEZk9Xnckr0TAHDsbKWMwxBCCNHrtXaQZzQRfUxE\nZ9XXh0QU3cIxZgBvApgGIBHA3USUaKj2CIAjzDwGwGQAy4jIt5XHeoyxi6RhRs6mMhgDBric476j\nn8PHbgUg4zCEEEKI1naR/B+ATQAGq69P1bLmjAeQrWY76gGsB3CzoQ4DCCYiAtAHQAkAWyuP9Rgf\nQxcJHM1nMMjHB37x8bpd/nYrBqoBiHSTCCGE6O1aNQ8GgHBm1gYU7xLRr1o4JgpAnmY7H8AEQ503\noAQuhQCCAdzFzA4ias2xAAAiegjAQwAQERGB1NTUFprl6kTpSWjn82S7A6mpqYgpPI/hmvK8Y4dw\nnJXzW26ehf4ZGbrzDKgpQ0GfcPz38Clc36+4ze3wlMrKynZ9Dl1RT7mXnnIfgNxLV9WT7kX0DK0N\nMM4T0VwA/1S37wbgiT/TbwKQBuA6AMMBbCOir9pyAmZeCWUBNqSkpPDkyZPb3IjarCqk4+uGbQIw\nefJkYF8ukLO6oTxmQDBinOefPBkFhw6hfMvnDfvD1QxGYSUjcdyPMDDYv81t8YTU1FS053PoinrK\nvfSU+wDkXrqqnnQvomdobRfJT6E8oloE4DSA2wHc38IxBVBWXXWKVsu0FgD4iBXZAE4AiG/lsR7j\nY9Z3kVAL82A4WSIG6bYH1FxoeP9NdudlMIQQQojO1tqpwnMBzNKWqV0kf2nmsD0ALlXXLSkAMAfA\nPYY6pwBcD+ArIooAMBJADoCyVhzrMb4+fvoC56rrTYzBcPKJ1AcYQ8qL8MDhT5FQkougw31R/f+e\nQuC4cR5urRBCCNH1tbaLxJ0n0EyAwcw2InoUyvwZZgB/Z+Z0Ilqo7n8LwItQxnMcgtIzsZiZiwHA\n3bEX0dZmGQd5EitPklAbMxiTC9IaN0qAvIcfxqU7d8LkZwhghBBCiB7uYgIMaqkCM28BsMVQ9pbm\nfSGAG1t7rLf4mn3hAMOkuSV2OEBtzGAYOS6Uo+5YNgJGJXmsrUIIIUR30NoxGO5wy1W6Bx+TD9jw\nSTjs9jaPwXDHVnT6YpsnhBBCdDvNZjCIqALuAwkCEOCVFnUCi8kCBzHMhgwGfIMBMjWuTWKtAuxW\nQJ350zKgP2CxADZbk+e2Fp3xatuFEEKIrqjZDAYzBzNzXzevYGa+mO6VLsXX7As2dPg4HHbAZHJd\nsl2TxSCzGf6GCbeMagsKPdVMIYQQotu4mC6SHsPP7AcmfaLG4Wh+RVWnQb99Fr7DhoECA+ET7Tp7\netHxUx5tqxBCCNEdSIABJcBwGDIYbLcrb1oYhxEwdiyGb9mM+P37MOKLbfjip7/V7a/I89r0HUII\nIUSXJQEGAD+LmwyGM8BoIYNhNHrspbptOne2cfE0IYQQopeQAAOAv9nfzRgMtYukhQyGUcoV+jEZ\n/arKkHm6/GKbKIQQQnQrEmBAfUzVEGDYbMrS623NYASF9kWtX2DjudmOr/ZmeaKZQgghRLchAQYA\nInIJMGqtNcqbgFD9jhYyGADA4QN124f2H7uY5gkhhBDdjgQYKmOAUWetVd64dJGUtniu4OjBuu3i\nE3k4X1l3Mc0TQgghuhUJMJxIH2E0BBht7CIBgD4xUbrtAdVl+O/RsxfVPCGEEKI7kQDDyfBJ1DV0\nkbRtkCcAWAZF6LYH1JTh88MyZbgQQojeQwIMFRsyGLUXkcHwGRSp2x5QewHfZJ9Hea31otoohBBC\ndBcSYKjIEGDU29QxEx7IYITXlKHe7sCODOkmEUII0TtIgOFk8twYDJ9IQwZDDUo+P1TU/vYJIYQQ\n3YgEGE6k/yguJoPhE6HPYAyqLsUT+9bj1K79qK5veuVVIYQQoqeQAENFpia6SPz6AmRu3GGtAmzN\nP3JqCgqCqW9fXdkNeXvxQupfsSMt1yPtFUIIIboyrwYYRDSViDKJKJuInnazfxERpamvw0RkJ6Iw\ndd9JIjqk7tvrzXaqjdFt1lvrG8sDw/R1q0taPJ1PdJRLWZCtFvs/39nuJgohhBDdhdcCDCIyA3gT\nwDQAiQDuJqJEbR1m/hMzj2XmsQB+A+BLZtZ+e09R96d4q50N7TXpPwqrNksR2F9fufp8i+frd9tt\nbsvPZOTgQrU8TSKEEKJn82YGYzyAbGbOYeZ6AOsB3NxM/bsB/NOL7WkWuYzBqG/cCDBkMGpazmCE\n3XsvLlmzGub++uBkYGUxtsicGEIIIXo4ixfPHQUgT7OdD2CCu4pEFAhgKoBHNcUM4AsisgN4m5lX\nNnHsQwAeAoCIiAikpqa2q7HGBdXz8nMbzpVUbUe4Zl/6nq9wLtfRqvP6z5yBkHdXN2xHVp3H6tR0\nRFbntKudrVFZWdnuz6Gr6Sn30lPuA5B76ap60r2InsGbAUZbzATwjaF75BpmLiCigQC2EVEGM7sM\nYFADj5UAkJKSwpMnT25XA77+9C3ddv8B/dFwrvIPgeJdDfuShg4CrmjddaqDg5GrCTAGVZUgs9SB\nkZdNQGRIQLva2pLU1FS093PoanrKvfSU+wDkXrqqnnQvomfwZhdJAYAYzXa0WubOHBi6R5i5QP15\nFsDHULpcvMZlDIZVMwbD2EXSikGeTj4xMbrtERcKEHOhCB/tPdXmNgohhBDdhTcDjD0ALiWioUTk\nCyWI2GSsREQhACYB+ERTFkREwc73AG4EcNiLbQVpH0UFYLNr5qswDvJsxRgMJ0t4OMjPT1f29val\nGPjKM3DY7W1upxBCCNEdeC3AYGYblDEVWwEcBfA+M6cT0UIiWqipOhvAf5i5SlMWAeBrIjoAYDeA\nzcz8b2+1FQBMLk+RaAZ5tuMxVScigk9MtEv5qPwj2L9FHlkVQgjRM3l1DAYzbwGwxVD2lmH7XQDv\nGspyAIzxZtuMTCZjBkPzKKlLF0nLj6lqBV52Oeqzj7uU/7BzH1JmTmnTuYQQQojuQGbyVJlcukg0\nAcZFdJEAQPjjv0Lw1Kku5SUZ2aiQFVaFEEL0QBJgqIxdJDabNsC4uAyGJSwM0X95DZHLlunKB5Wf\nwca0wjadSwghhOgOJMBQmUz63qJmMxjVpe26hv/QWN12VGUx1n53EszGWTiEEEKI7k0CDJVxDIZd\n+xSJfwgAzVoldRcAe9u7NnyHDNFtD6ouwY92fIC0l5ah7rjrGA0hhBCiu5IAQ2U2Bhg2TYBhMrtZ\ntr3tWQxTUBAsmqXczezA3Vn/hf+6d3DyjjthO9+2rhchhBCiq5IAQ2XsItFlMAA33SRtG+jp5Bsb\n67bcUV2NSpnmVwghRA8hAYbK0twYDOCiH1V1Cr7xhib31R3Lbtc5hRBCiK6mq6xF0unMJh/dtu4p\nEuCiH1V1Cr3nHpiDg/HDf75BdnoOrj7dOEFpddaxdp1TCCGE6Gokg6HyMQYYxgzGRT6q6kRECJk1\nCxP+/DI+uewnun0Xjma265xCCCFEVyMBhspiCDDsdjusjubmwmhfBsMpwNeM636cAhs1/gr8SotR\n+Pz/Q016+kWdWwghhOhsEmCojKupmhiotlY3FnhoDIbW3ImX4nSfcF3ZhfXrkTvnbtTn5V30+YUQ\nQojOIgGGyhhgEBOqrJr114wZjHY8pmoUHuwH+5ChLuVstaLiP9su+vxCCCFEZ5EAw4lIv8lApbWy\nscDlMVXPzFkRN+9Ot+W1GRkeOb8QQgjRGSTAULl2kRgyGC5dJBc3BsNp+M3T8OmC57A/PE5XXnv0\nqEfOL4QQQnQGCTBURMYuEhi6SDzzmKo7t/90Fl6ZME9XVp+djapdu+GorfXYdYQQQoiOIgGGily6\nSMjQReL5QZ5Ow8L74Mcpw3HaEMScuu8+5Nx8M+xlZR67lhBCCNERJMBwavEpklB9/ZoywDid+EX4\n1Y/jcKLfYJdya+4plG3c6LHrCCGEEB3BqwEGEU0lokwiyiaip93sX0REaerrMBHZiSisNcd6vK0u\nT5EAlfWaDIbZxxBksEezGLEDguDzo6vc7qvZt99j1xFCCCE6gtcCDCIyA3gTwDQAiQDuJqJEbR1m\n/hMzj2XmsQB+A+BLZi5pzbFeaK9u22WQJwAEDdRvV57xaBumP/UgPky8EVbSr+xanfYDmNmj1xJC\nCCG8yZsZjPEAspk5h5nrAawHcHMz9e8G8M92HnvR3GUwXAKMPoYAo+qsR9swoF8fhP7iF5g982XU\nmhtnFrWfK4a1oMCj1xJCCCG8yZuLnUUB0E5HmQ9ggruKRBQIYCqAR9tx7EMAHgKAiIgIpLZzyfPa\nunr9eZlw7NQxpFY1ni+hGojQ1Dm6dyfO5Hv2IxzhYPQNsCAz9BKMKT7eUL7vvfdQN8HtR+CisrKy\n3Z9DV9NT7qWn3Acg99JV9aR7ET1DV1lNdSaAb5i5zc9+MvNKACsBICUlhSdPntyuBmw8oV8q3cRA\n3wF9MXmS5nw1/wbOftWwmRDTHwlXt+96zanol49DP8TqAgxTYSlae2+pqamtrtvV9ZR76Sn3Aci9\ndFU96V5Ez+DNLpICADGa7Wi1zJ05aOweaeuxHtHiVOEAYFg3BJWe7SJxuvWyKNgSknRlpbv3oaLW\n2sQRQgghRNfizQBjD4BLiWgoEflCCSI2GSsRUQiASQA+aeuxntSqMRjGQZ5V57zSFpOJMP+BGbqy\n6NICLN90wCvXE0IIITzNawEGM9ugjKnYCuAogPeZOZ2IFhLRQk3V2QD+w8xVLR3rrbYCcFmLxGSc\naAtwHeTppQwGAIyMi0FFRFTDthmMvVu/xv5TF7/ImhBCCOFtXp0Hg5m3MHMcMw9n5pfUsreY+S1N\nnXeZeU5rjvWmFufBAIAgQxeJlzIYTpFXjddtjyrOwRP/SkNVnecm+BJCCCG8QWbyVLlOFQ6U1Rmm\n6O7ADAYABF9xhW573NlMnDxfjRc+PeLV6wohhBAXSwIMJ8NiZyYHodpWjXq75vFV4xiM6mLAYfda\nk4KuuVq3HVeah+D6Kvxr7yl8sT8DqKv06vWFEEKI9uoqj6l2OnddJICSxRgYqAYWFl9l2XbnSqrs\nULIYfSO90iafgQPhFx+PuowMAIAJjFXn/4Tk2FwEbqprHPbafwQQMwEY8WMg/ideaYsQQgjRFpLB\nUOQ7C5IAABpSSURBVLmbKhwASmsNgyr7GhYkqyj0ZrPQ50r9OIyYM2cRSHX6SuezgbT3gA0LgD8n\nIvbEe0BtuVfbJYQQQjRHAgxVcxkMnWBDtqL8tPcadfIbBBW/pyuqLPIDOwC2KwkUF9XFiM19H3gj\nBUj7JyBrmAghhOgE0kXiZAgwQit9EXMmAKU1xgyGIcCo8FKAkfYP4JNHEOjvAFkGgW1K++y1ZmS8\nr2ZRiBE0qA7RV5fCZDEEEpVngI0LgczNwKw3gIB+3mmnEEII4YZkMFR+wX1h8fXTlV2/byDytn2t\nrxhs6CIp90IXye7/BTb+HGAHyAwERdS5r8eEqtP++L+DN6Bw5HzAr69rnaOfAm9fC5zx7jQiQggh\nhJYEGCqTxQeXT5vpUl7+1SHUVWtm9PR2BiPtn8CWX+uKgqNrmz1k5KlTuCl9Gg7P+R64dhEc5KOv\nUJYL/H0qkPOlZ9sqhBBCNEECDI0f3TYH5gHB+kKrHW8suAvnC9TFXb2ZwcjbA3z6S32ZyQchi/6G\nfnPugikwUJlx1NCd07+2HMNOZ2H+uiPISf4V9lzxOhA5Rn+eunJg3W1A+kbPtVcIIYRoggQYGj5+\n/oj+n1mwml1HT25buUJ5460MRvlp4F/3Atp5Nyz+wN3rQaNmI/L55zFy/z4kHD2ChCPpCLl5lu7w\nyfk/oKSqHvPe2Y18GgQ8sA1I+an+Gg4rsOGnwOEPPdNmIYQQogkSYBiE9x+Mj691zUqcyz2pvHHJ\nYHggwGAGPnlEGZipdfObwKU/dntI3xn6xdCm5u5Gn/pqFJTV4JXdtcgrtwM/+TNwwwuGa9mBDx8E\nDm24+HYLIYQQTZAAwyDELwTVAXbsSizRlVvrasHMQGAYYNYMBq2vAGovXNxF9/4dOP5ffdnVvwKS\nb2/ykKArr4Q5LExX9sGW3+GS8iIU1zDueOs75BRXAVc/Bsx+Wz9TKTuAj/5HggwhhBBeIwGGQah/\nKADgaGyFrpwdDtitVmUMRL8Y/UGlue2/4PnjwH9+qy8bcjVw/e+aPYwsFvSdNs2l/KHDyvSeReW1\nuPPt73G44AIwZg4we6VrkPHxz4DsL9rfdiGEEKIJEmAYhAc0rpha56Nf56O+tkZ502+I/qCydgYY\nzMoTI9bqxjLfPsAtfwNM5hYP73fnHS5l485mIbpCWYStuLIOd779HbYdOQOMvgO4bRVAmvM6bMC/\n5gH5e9vXfiGEEKIJEmAYBPoEIsxf6XqwmvWTV1lr1cdFQw0BRnszGBmfAce368umvuJ6/ib4jxyJ\n6L++6VI+M6dx7o7qejseWrsXq77KASfdCty6EoBmWnRrNfDe7cC5zPbcgRBCCOGWBBhuDA5SBnLa\nLPqnSax1aoDhiQxGfTXw79/oy2InApfNa9Npgq+7DoOXLtWVzTrxLa4oOtqwzQz8YfNR/OpfaaiM\nuwWYtkR/kppSYO1soCyvTdcWQgghmiIBhhtRwVEAvJzB+O4N4ILmC91kAab/SRnj0UZ9b7wB5vAB\nurIXvn8H1xX8oCv7JK0Qs1Z8jSMxdwPXPqU/SXmBMk9GtX5wqxBCCNEeXg0wiGgqEWUSUTYRPd1E\nnclElEZE6UT0pab8JBEdUvd16CCBwX2cGQx9gOGxMRg1ZcC3b+jLJiwEBia07Twq8vVF6Jw5LuWL\n9ryHe07sxMiSxvblFFfhlr9+g/81z4Fj3AL9AcWZwPp7AGvzM4cKIYQQLfFagEFEZgBvApgGIBHA\n3USUaKjTD8BfAcxi5iQAxlGLU5h5LDOneKud7kQFKRkMm2HCrcYMRqz+gLJTbVu19Pu/AXWaR1sD\nQoFJi9vR0kah99wDy8CBLuXzDmzCX3auwE0ndzWU1dsceOnzDNxx6lZUDpuuP+DUd8ojrA47hBBC\niPbyZgZjPIBsZs5h5noA6wHcbKhzD4CPmPkUADDzWS+2p9WcGQzXLhI1gxEQCvhqphS3Vrd+Rs+a\nUuD7v+rLrvoF4O9mobI2sISGInb9PwGz+6dPHj76Gf5/e3ceHGdx5nH8+5sZnZZkW/KBL3yBY0PA\nBmxzmCROCNkEthY2RcKVLKkiEBIC2ewmW4TKhmyK3cUhkJOEGMK1gbDeAgIVEgxksSG1BtsQY4yN\nOWyDLRvLl6zDOkYzz/7xvpJGo9HJyNJIz8eeeuftt/t9u9WSptX9vt1jG2s6hL28s5Yz3ryEXWUL\nOkbe8gQ8dYMv9e6cc67fBnK59ilA6l2Du4DT0+LMAfIkrQJKgZ+a2QPhMQOelZQAfm1myzNdRNLV\nwNUAEydOZNWqVf3KbF1dXVva9+PvA51v8ty0cSN7E8E9EqcWHENZc/tcGa8+u4JD5Wkf1BnM2P4Q\nM5raP+jjsVJebJpHop/5Tqf//A/Kb/53YjUdGxP5TQ3c88KPWXbqpbxdPIH94fLtdS0xzqv6Go8V\n/BuzVdmeYO1y3tnfxM5jP5uVfPVXar3ksuFSDvCyDFXDqSxueBjIBkZvr38acA5QBKyR9KKZvQmc\nbWaVkiYAz0h6w8yeTz9B2PBYDrBw4UJbunRpvzKyatUqWtM2J5pZ9uCyTj0YM6cfy8LW81cvhg1v\ntR2bP7kQzujh2g2HYE3Hp0TyPvZPfOQj53WRoH9eaGpiblUV+3/5K0i0D3UU1tdy0wvLSSrCTxdc\nxNPTFwNQQwlfaLqBRwtuYpLab/Kcve1+Zs9fAvMvzmr++iK1XnLZcCkHeFmGquFUFjc8DOQQSSWQ\nOuXl1DAs1S5gpZnVm9l+4HlgPoCZVYbbKuAxgiGXoyI/ms+sMbM6P6bamHLz4/gPdUy0742eT7zm\njmBV01ZF5bD4qg+Q08wS48cz/tprmbvhrxSc0PnG0Ygl+eZfV/C7p37QdgPoHir4UvO/UGNFHeLa\n41/rPFeHc84514OBbGCsA46XNFNSPnAJ8ERanMeBsyXFJBUTDKFskTRKUimApFHAp4BNA5jXTuaW\nz6Ul2sVTJADj0hsYPUxUdeQgvHhnx7Al10NBaeb4WaC8PKb86DZUXJzx+JjGGn7y/M+569llfHv9\nQxw4UsbV8X+mydo7tpRsofmhy6nd/vKA5dM559zwM2ANDDNrAb4OrAS2ACvM7HVJ10i6JoyzBXgK\n2AisBe42s03AROAvkl4Nw580s6cGKq+ZzCuf1/U8GJChB2NL9zdFrrkjWBitVXEFLMp+70W6glkz\nmX7/fZSdfz7RMWMyxplat49P7HqF/1p5M8e9vpMHNn+Smp2FWNiBk584QuN9n2XZwyt5+d1DwaJv\nzjnnXDcG9B4MM/sj8Me0sDvT9m8Fbk0L20Y4VDJY5lXM45FOQyQpPRhjpkOsCFrCsIZDULMbRk/p\nfLIjB+GlX3cMO+t6KCjJcq4zKzrpJKbcFsz2eWjFCt7/3k1dxr186zMAVFJOyeRGpp59EEVgvKq5\nePN1XLLhu+SXT+PCBZO54JQpzB5/dMrgnHMut/hMnl2YWz6XRKzjrJo19dXtO5EITDq5Y6Jd6zKf\nbM0vOvdeDMC9F70x9vOfZ+6WzYy97LIe49btLmTnC+U010VJtogZkb08nH8z8YM7+dn/vs05t63m\n3NtXc8uf3uDldw+SSHrPhnPOucBgP0UyZI3KG8WxFTOB9psy9x1+v2OkqYtgZ/sEVuxaByde2DFO\n/YHOvRdLvgH5o7Kb4T6QxDHf+1cqvnwl8cpKdl7zVZL19Rnj1u8p5J0/FIKM8SfXMmPeXn6XfzOX\nNn+XPVTwVlUdb1XVcefqdygflc+Zsys4Y1YFZ84qZ/b4EtSPqc+dc87lPm9gdOPkyadQS9vs5VTX\nHugYYeqijvs713Y+yf/9DJrr2veLx8GiL2cxl/2XN3kyeZMnc9zqVRx+5BHie6vAjNqVK4nv3t0x\nsol9r5axf1MJZcc28vCcH3Bl0bfYHR/PrMOVlDUf4ZUJc3hyYzNPbgwmHRtXks/pMys45dgxnDRl\nNCdOGU1JgX/LOefcSOC/7buxcOpinktpYDQ01LH98HZmjp4ZBExLe3J2zwaIN0Be+Khn9U54Kf3J\nkcHtvcgkWlJC+RVXtO2P+dxF7Lj0MpKHD3eKa4kIh7cXw3b4uX4C1t5DcaCwjD9NP53qwlIiliSJ\naNxcwPKKWVSNKkeCWeNGcdKU0XzomDKOn1DCnImlTB1bRCQSnCe+ezcWj5M3bRqKHMURvGQiWPCt\nfh801QWNwmQiqKuC0mD21tHTIK/w6OVphNr5+kZe+v3/cKSm8/dfttXV1fHenx4d8OscDXV1dZxy\nwlxGTzhmsLPiHAAaTk8ELFy40Nav79+6aJkmqTm4u5J7v/mVtv2kDCsroKKwoi3Mqt+DZEt7opKJ\n7Q2M+v1tvReGQFEomwxq/eDs+LXPWBcZgzIFtoc1NTVRUFCQ8amWTiGZ4phhLS3BsEkigSWTneIE\nMgx/ZAgyIKEIljJckr4fxYgmE0RaryVBLIaZEcnPQ7G84NwZi57pezj9a5t+OBnUWzIRvMda/3dP\nAiLk5ceYefx0TjznM4yadkLmgqdYu24tixcdtalcPpBkMsH+93ZwaE9lxgej3t2xg+kzZrTtR6NR\nJsycTUl5ReYhsW6GydKPVO99n8d/dHP/Mu740u2/omLKtJ4jppD08tFe78mNDN6D0Y38oo6TTkVM\ncLiZ6sOp647kha/QoRpS79sIJihNsX9flnOZWby+rudIvRGNBK8si9C50ZKIiEQkZS2V1udkm5uD\n14DpQ/nCD9yWxhY2v/YOm1/7RffxU2x++N4+5mvo2vPymsHOgnNuiPOnSLpRVFpKXoF3iTvnnHN9\n5T0Y3YjG8lhy8RdZ/dvfdDNM4JwbCJH8GMeef2YwpCjDTEQaGil6ZzfRuiNYNAKRCBaJhO8VvARI\nwe1BUof3FqFtyMZkVB/cx9iyYrBmoAWsGVkztDQQSTRBvBElGlFLE8rQ65aqy+G1tHEgi0SxWGHw\nihZisQIsWhDuh9tIfrifB0RBkSCdwqFW1DZUqvDCVXurSGZeTNm5QeENjB6cdv4FfPjj5/LAuru5\n9/X+dXFbr57UzHAvRC/S9foOmvRfcr1Ol3YvQy+TpcYbXW9MOWBEkhAxKK+DSQeMaLgvgm1zBDYc\nJ3ZMEHMqjdIGmL/NOOE9oyDRxcl7nYuwOL1Nmp4u5VQmODSqkF1jS6ktKuh9dnJINJlk7JEm8lsS\n3cYzoCE/Rm1hQcbv1+6/Nl3XRlljEydU7id/XQ9T8B812f5VGQ9ftT1F7JO3Fixk/Hlfyuo5nesv\nb2D0QkFxMVd97HpKJ07gh+t+SDwZH+ws5ZSqAqgqTw/t7qPeeHF08O6Z04SSEEtAYRwmH4S8FkNG\nW4MlloTyGshvCfYj4TGZkFl7mEE0AZMOwaiGIDyahInVUNLYTXY6Z4/xtQ2Mr23oOa5zR9Mwumnf\n5T5vYPTBJXMvYcmUJdyz6R6e3vE0Nc01PSdyH5hFRDwC8TzYWgz974fITEnjuD1Bb0lFbdDwOOag\nUV4X9Fy0vo4UwNOnRtg6VUSTQVhrw0UGpUeMCYfb91MbQQVxo6IGxtbD9L1GQRzyu+8ccM65nOYN\njD6aVjqNm868iRsX38i6vevYenAr2w5vY1/DPmobq6mt2UVTohmL5WOxIiz8F/w3JCFERBGE2vZT\nwyGYbTNCpO2xv0zxW4+nh9ccrmHMmDEdrhP8bz9/b6kvH+atj5HG64NHdJtqId4IieawTRAJ5pGI\nFaG8IsgrhvziYE2XTHNemFG9/33GjioI5heJHwm3DcEjppYILqooxPIhVojyR0Hh6OAVzet8zq4c\nC1WnQ1W4+5oZyaSRMEgmjaQZSQseiR0dbtvDIGlGtRkHw/eWssUSWP0BymJxoskmYokmYtZMNNFC\nLJGgoNmItUC0BUgG9RVJwJhqGHtIRFvaGytKKuydCYeYktbWY5MXN/LjwTYWprFIMKRjAKZgqCcM\niMciHCzJR0kRTRqxhAW3L5i1XaO1EdWUF+GtycXUFMcYU9NMDBEJp4Y/UhCltKGlPS+t6ZJGXsIY\nWxcnGsZtOz/Wtg3iW3DHubXmof1bCuu4rSmFyslBgzOSbH8p2bFxhwVPfsmCYyLcWnvj0ExBT1cS\nIongaxtJBFe28BVcuvPPQeqwWYejHcKti7idz94WN22bmrj1R6zTE9vhF6isaOBWZ3aur7yB0U95\n0TzOmnwWZ00+a7Cz0kmmOT1y1XApy3ApB3hZhqpVq1axYJiUxQ0P/piqc84557LOGxjOOeecyzpv\nYDjnnHMu67yB4ZxzzrmsG9AGhqRPS9oq6W1JN3QRZ6mkDZJel7S6L2mdc845NzQN2FMkkqLAHcC5\nwC5gnaQnzGxzSpwxwC+BT5vZe5Im9Datc84554augezBWAy8bWbbzKwZeBi4IC3OZcCjZvYegJlV\n9SGtc84554aogZwHYwqwM2V/F3B6Wpw5QJ6kVUAp8FMze6CXaQGQdDVwdbhbJ6m/ixeMA/b3M+1Q\n42UZeoZLOcDLMlT1tyzTs50R52DwJ9qKAacB5wBFwBpJL/blBGa2HFj+QTMiab2ZLfyg5xkKvCxD\nz3ApB3hZhqrhVBY3PAxkA6MSmJayPzUMS7ULOGBm9UC9pOeB+WF4T2mdc845N0QN5D0Y64DjJc2U\nlA9cAjyRFudx4GxJMUnFBMMgW3qZ1jnnnHND1ID1YJhZi6SvAyuBKHCPmb0u6Zrw+J1mtkXSU8BG\nIAncbWabADKlHai8hj7wMMsQ4mUZeoZLOcDLMlQNp7K4YUBm1nMs55xzzrk+8Jk8nXPOOZd1I76B\nkeszhkraIem1cDbU9WFYuaRnJL0VbscOdj4zkXSPpCpJm1LCusy7pO+E9bRV0t8MTq4z66Is35dU\nGdbNBknnpRwbymWZJuk5SZvDGXa/EYbnVN10U46cqxdJhZLWSno1LMu/heE5VSduhDGzEfsiuL/j\nHWAWkA+8Cpww2PnqYxl2AOPSwn4I3BC+vwFYNtj57CLvHwVOBTb1lHfghLB+CoCZYb1FB7sMPZTl\n+8C3MsQd6mWZBJwavi8F3gzznFN10005cq5eAAEl4fs84CXgjFyrE3+NrNdI78EYrjOGXgDcH76/\nH7hwEPPSJTN7HjiYFtxV3i8AHjazJjPbDrxNUH9DQhdl6cpQL8seM3slfF9L8GTXFHKsbropR1eG\nZDkALFAX7uaFLyPH6sSNLCO9gZFpxtDufgENRQY8K+nlcFZTgIlmtid8/z4wcXCy1i9d5T1X6+o6\nSRvDIZTW7uucKYukGcApBH8x52zdpJUDcrBeJEUlbQCqgGfMLKfrxA1/I72BMRycbWYLgM8A10r6\naOpBMzOCRkjOyeW8h35FMPy2ANgD3Da42ekbSSXAI8A/mllN6rFcqpsM5cjJejGzRPizPhVYLOnD\nacdzpk7cyDDSGxi9mW10SDOzynBbBTxG0A26V9IkgHBb1fUZhpyu8p5zdWVme8MPhSRwF+1d1EO+\nLJLyCD6UHzSzR8PgnKubTOXI5XoBMLNq4Dng0+RgnbiRY6Q3MHJ6xlBJoySVtr4HPgVsIijDFWG0\nKwhmTM0VXeX9CeASSQWSZgLHA2sHIX+91vqLP/T3BHUDQ7wskgT8BthiZrenHMqpuumqHLlYL5LG\nSxoTvi8CzgXeIMfqxI0sg73Y2aCyLmYbHeRs9cVE4LHg9ygx4CEze0rSOmCFpCuBd4HPD2IeuyTp\nd8BSYJykXcBNwC1kyLsFs8CuADYDLcC1ZpYYlIxn0EVZlkpaQNBtvQP4Cgz9sgBLgC8Cr4Vj/gA3\nknt101U5Ls3BepkE3C8pSvCH4Qoz+4OkNeRWnbgRxGfydM4551zWjfQhEuecc84NAG9gOOeccy7r\nvIHhnHPOuazzBoZzzjnnss4bGM4555zLOm9guGFFkkm6LWX/W5K+n6Vz3yfpomycq4frfE7SFknP\n9TL+jQOdJ+ec6ytvYLjhpgn4rKRxg52RVJL6MufMlcBVZvbxXsb3BoZzbsjxBoYbblqA5cA30w+k\n90BIqgu3SyWtlvS4pG2SbpF0uaS1kl6TNDvlNJ+UtF7Sm5L+NkwflXSrpHXhAlpfSTnvC5KeIJjw\nKD0/l4bn3yRpWRj2PeBs4DeSbk2LP0nS85I2hGk+IukWoCgMezCM94Uw7xsk/TqcnAlJdZJ+LOl1\nSX+WNP4DfJ2dc65b3sBww9EdwOWSRvchzXzgGmAeweyPc8xsMXA3cF1KvBkEa1ecD9wpqZCgx+Gw\nmS0CFgFXhdMzA5wKfMPM5qReTNJkYBnwCYJFtxZJutDMfgCsBy43s2+n5fEyYGW44NV8YIOZ3QA0\nmNkCM7tc0jzgYmBJGC8BXB6mHwWsN7MTgdUEs40659yAGNFThbvhycxqJD0AXA809DLZutZlryW9\nAzwdhr8GpA5VrAgXyXpL0jZgLsEaMCen9I6MJlj7oRlYa2bbM1xvEbDKzPaF13wQ+Cjw++7yCNwT\nLuD1ezPbkCHOOcBpwLpwCvki2hfASgL/Hb7/LfBop9TOOZcl3sBww9VPgFeAe1PCWgh77SRFgPyU\nY00p75Mp+0k6/pykz61vgIDrzGxl6gFJS4H6/mW/MzN7XtJHCXpP7pN0u5k9kBZNwP1m9p3enDJb\neXPOuXQ+ROKGJTM7CKwgGL5otYPgr3uAvwPy+nHqz0mKhPdlzAK2EiyW99WwZwFJc8LVbbuzFviY\npHHhPRKXEgxbdEnSdGCvmd1FMHRzango3npt4M/ARZImhGnKw3QQ/Ly39rJcBvyld0V2zrm+8x4M\nN5zdBnw9Zf8u4HFJrwJP0b/ehfcIGgdlwDVm1ijpboJ7M14JlwjfB1zY3UnMbI+kG4DnCHodnjSz\nx7tLQ7Ba67clxYE64B/C8OXARkmvhPdhfBd4OuyliQPXEqy0WQ8sDo9XEdyr4ZxzA8JXU3VuhJBU\nZ2Ylg50P59zI4EMkzjnnnMs678FwzjnnXNZ5D4Zzzjnnss4bGM4555zLOm9gOOeccy7rvIHhnHPO\nuazzBoZzzjnnss4bGM4555zLuv8HaOKnnaXUEGQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266a1acc630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_history_PGD, lw=4, label='Prox grad descent')\n",
    "plt.plot(cost_history_FISTA, lw=4, label='FISTA')\n",
    "plt.plot(cost_history_SVRG, lw=4, label='SVRG')\n",
    "plt.plot(cost_history_SAGA, lw=4, label='SAGA')\n",
    "plt.plot(cost_history_CD, lw=4, label='CD')\n",
    "plt.plot(cost_history_SDCA, lw=4, label='SDCA')\n",
    "\n",
    "plt.grid()\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Number of step')\n",
    "plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)\n",
    "plt.suptitle('Algorithms by number of step')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAEjCAYAAABw0XjjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl4lNXZ+PHvnX0hhARCCAk7BJKwKRFcS9Cq4Au4VlER\nS7WUtlZftYo/rdZX37q02BaXVqltEdDXqiAioIjFuFdwYYeEGAIhAZKQQPZlMuf3x0ySmckkzEAG\nksz9ua5czHOe8zzPOTPhmjtnFWMMSimllFIdKeBMF0AppZRS3Y8GGEoppZTqcBpgKKWUUqrDaYCh\nlFJKqQ6nAYZSSimlOpwGGEoppZTqcBpgKI+IyBIR+V8f3ftmEfmgnfMZInLQF89uj4/rfJGIZPni\n3l6W41ERWX6my+FIRIyIDD/T5VBKnRoNMJQTEckUkTIRCT1dzzTGvGqMucyhDN3uC8a1TsaYT40x\nI89kmZRSypc0wFDNRGQwcBFggJmn6ZlBp+M5SimlTi8NMJSjOcB/gCXAre1lFJH7ReSQiBSKyO2O\nf6GLSLSILBWRYhHZLyK/EZEA+7kfi8jnIvInETkKPGpP+8x+/hP7I7aKSKWI3ODwzHtFpMj+3LkO\n6UtE5C8i8p79ms9FpJ+I/NneGrNHRM5yyL9ARApEpEJEskTkknaq2kdENtjzfiwig+z3eEFEnnF5\nT1aLyN1u3qtWdXLt9hGRPBG5T0S2iUiViPxdROLtdaoQkQ9FJMYh/7ki8oWIHBORrSKS0c5n1V9E\nVtg/j30icmc7ed8UkcMiclxEPhGRNJf3+cU23g+xf6ZFIlIuIttFZLT9XKiILBSRAyJyxH6PcIf7\n3ufwu/STtj8KpVRXogGGcjQHeNX+c7mIxLvLJCJTgXuAHwLDgQyXLM8B0cBQYLL9vnMdzk8CcoF4\n4HeOFxpjfmB/Oc4Y08MY8y/7cT/7PROB24AXHL9wgeuB3wB9gDrgS+Bb+/FbwB/tZR8J3AGcY4yJ\nAi4H8tp+S7gZeNx+ny3Y3huAV4AbHQKnPvb34zXXG7RTJ1fXApcCycAM4D3gQSAO2//VO+3PSgTW\nAv8LxAK/BlaISJzrDe3lexfYiu29uwT4bxG5vI0yvAeMAPpie/9edTnf1vtxGfADe9mjsX0eR+3n\nnrKnj8f2+5IIPGIv31R7+S+1P/eHbZRLKdXFaIChABCRC4FBwBvGmG+A74Gb2sh+PfBPY8xOY0w1\n8KjDfQKBWcD/M8ZUGGPygGeAWxyuLzTGPGeMsRhjajwsYgPwmDGmwRizDqgEHMcwvG2M+cYYUwu8\nDdQaY5YaYxqBfwFNLRiNQCiQKiLBxpg8Y8z37Tx3rTHmE2NMHfAQcJ6IDDDGbAKOY/vCxl7nTGPM\nEQ/r485zxpgjxpgC4FPgK2PMdw51aqrDbGCdMWadMcZqjNkAfA1c4eae5wBxxpjHjDH1xphc4G/2\n8rZijPmH/XOrw/a5jhORaIcsbt8PbJ9PFDAKEGPMbmPMIRERYB5wtzGm1BhTATzh8Pym36Udxpgq\nHH6XlFJdmwYYqsmtwAfGmBL78Wu03U3SH8h3OHZ83QcIBvY7pO3H9leru/yeOmqMsTgcVwM9HI4d\nv9hr3Bz3ADDG5AD/je2LrEhEXheR/u08t7msxphKoBRb/cHWijHb/no2sMzTyrTBozpgCwR/ZO8e\nOSYix4ALgQQ39xwE9HfJ+yC21iMnIhIoIk+JyPciUk5Ly04fh2xu3w9jzEbgeeAFbO/rYhHpia31\nJQL4xuH579vTofXvkuPvjVKqC9MAQ2HvD78emGzvfz8M3I3tr9dxbi45BCQ5HA9weF2C7a/ZQQ5p\nA4ECh+MzuoWvMeY1Y0xTi40Bnm4ne3PdRKQHti6JQnvScuBK+3uUAqzyTYlbyQeWGWN6OfxEGmOe\naiPvPpe8UcYYd60dNwFXYuumiAYG29PFIU+b74cx5lljzAQgFVuXyH3Yfh9qgDSH50cbY5qCpUM4\n//4M9OaNUEp1XhpgKICrsHUdpGLrJx+P7QvzU2zjJ1y9AcwVkRQRiQAebjph75J4A/idiETZBwHe\ng+3L2FNHsI3f6HAiMlJELhbbNNxabF9+1nYuuUJELhSREGxjD/5jjMkHMMYcBDZja7lYcYLuno6s\n03Jghohcbm91CLMPGk1yk3cTUGEf2Bpuzz9aRM5xkzcK2/iVo9haHZ5wk8ft+yEi54jIJBEJBqqw\nvbdWY4wVW5fMn0SkL9jGkDiMAXkD+LGIpNp/l3570u+KUqpT0QBDga0r5J/GmAPGmMNNP9iavG8W\nl6mkxpj3gGeBj4AcbDNPwPblBPArbF8yucBn2Lpb/uFFeR4FXrE3qV9/knVqSyi2QYclwGFsgxn/\nXzv5X8P2pVcKTKClS6TJK8AYTtw98igdVCd7gHMltq6OYmytFPfh5v+zPeCbji1o3Iet3i9ja6Fw\ntRRbF0UBsIuWz9VRW+9HT2yBRJn9HkeBP9jPLcD+e2LvevkQ+/gZ++/Sn4GN9jwbPXoTlFKdnhhz\nRlurVTcgIinADiDUZZxEtyciP8DWojDIdPP/TCKyBDhojPnNmS6LUqrz0xYMdVJE5Gr7+gYx2MYw\nvOuHwUUwcBfwcncPLpRSylsaYKiT9TOgCNt01kbg52e2OKeXvdXmGLaZG38+w8VRSqlOR7tIlFJK\nKdXhtAVDKaWUUh1OAwyllFJKdTgNMJRSSinV4TTAUEoppVSH0wBDKaWUUh1OAwyllFJKdTgNMJRS\nSinV4TTAUEoppVSH0wBDKaWUUh1OAwyllFJKdTgNMJRSSinV4TTAUEoppVSH0wBDKaWUUh1OAwyl\nlFJKdbigM12AjtSnTx8zePDgk7q2qqqKyMjIji3QaaZ16By0Dp1Hd6iHr+vwzTfflBhj4k7xHn2D\ngoJeBkajf7j6Cyuww2Kx3D5hwoQidxm6VYAxePBgvv7665O6NjMzk4yMjI4t0GmmdegctA6dR3eo\nh6/rICL7T/UeQUFBL/fr1y8lLi6uLCAgwHREuVTnZrVapbi4OPXw4cMvAzPd5dFIUyml1KkaHRcX\nV67Bhf8ICAgwcXFxx7G1WrnPcxrLo5RSqnsK0ODC/9g/8zbjCA0wlFJKKdXhNMBQSinV5QUGBk4Y\nNWpU6ogRI9KmTZs2tKKiotN/v02cOHHkJ598EtFenjVr1kRNmTJl+Okqk6uSkpLAp5566qQGAXf6\nD0AppZQ6kdDQUOuePXt27d27d2dwcLB55plnnL4UrVYrjY2NPi9HQ0ODz59xOh09ejTw73//e9+T\nudZns0hE5B/AdKDIGNNqEIiICLAIuAKoBn5sjPnWfm6q/Vwg8LIx5ilfldOJMSByWh6llFLdzeAH\n1k7w5f3znvqvbzzJd+GFF1Zu27YtPCsrK+Tyyy9PPuussyq3b98euW7dur0fffRRj2eeeaafMUZ+\n+MMfHvvrX/9akJ2dHXLppZcm/+c//9nTt29fy8SJE0c+9NBDh6655ppyx/v+6U9/6rNo0aJ+UVFR\njWlpadUhISFm6dKlB6699trBoaGh1h07dkRMnDix8uabby69++67B9bV1QWEhYVZlyxZsm/cuHF1\nlZWVMmvWrCG7du0KHzZsWG1tba3bL5y33nqr53333TcgPDzcOnHixMqm9PLy8oDbbrtt4J49e8It\nFos89NBDhbNnzz729ddfh82dO3dIQ0ODWK1WVqxY8f2YMWPqnn/++d7PPvtsvIiQkpJSs2rVqn2F\nhYVBc+fOHVRQUBAC8Mc//vHAZZddVnXPPff0z8/PD9m/f39oYWFhyPz584/85je/Kbr33nuT8vPz\nQ0eNGpU6efLk8pdeeumgp5+XL6epLgGeB5a2cX4aMML+Mwn4KzBJRAKBF4BLgYPAZhFZbYzZ1dEF\nLPhwGW8u+Q6rqUKMhc0vL6bnObdxw43n0atvu61WSimlOqGGhgbWr1/f87LLLisHOHDgQOjf//73\nfZdcckleXl5e8KOPPpr4zTff7I6Li7NcdNFFycuWLet1yy23HLvrrrsO/+QnPxmYnp5eNXLkyFrX\n4CIvLy944cKFCd9+++2uXr16Wc8///zktLS0mqbzhw4dCvn222/3BAUFUVpaGrB58+Y9wcHBrFq1\nKur+++9PWr9+/fcLFy7sGx4ebs3Nzd351VdfhV9wwQWpruWvrq6WO+64Y/CGDRuy0tLS6qZPnz60\n6dyDDz6YMGXKlPI333wzr6SkJDA9PT1l5syZ5c8991zcL37xiyM///nPS2tra8VisfD111+HLVy4\nMOHLL7/ck5CQYDly5EggwM9+9rMB99xzz5HLL7+8cu/evSGXX375iNzc3J0AOTk5YV988UXWsWPH\nAlNSUkbfd999xc8888zB6dOnh+/Zs8fr72CfBRjGmE9EZHA7Wa4ElhpjDPAfEeklIgnAYCDHGJML\nICKv2/N2eIBRXngEq7UU01iCAQIaoX57Ma/u/k9znokzhtB/RC8CgwMIiwimZ1w4AQHayqGUUp1J\nXV1dwKhRo1IBJk2aVHHXXXeV7N+/PzghIaH+kksuqQL47LPPIs8999yK/v37WwBuuOGG0o8//rjH\nLbfccuyee+4pWblyZcySJUvitm3b1ur75tNPP42cNGlSRXx8fCPA1VdfXZadnR3WdP6aa64pCwqy\nfaWWlpYG3nDDDUPy8vLCRMQ0NDSI/fk97rzzziJ7GWuSk5OrXZ+zZcuWsKSkpLoxY8bUAdx8881H\nX3755TiAzMzMnuvXr+/17LPP9rPXWXJyckLOO++8qoULFyYcPHgwZNasWWVjxoypW79+fc8ZM2aU\nJSQkWACayv3555/33Lt3b3jT8yorKwOPHz8eAHDZZZcdCw8PN+Hh4ZbY2NiGgwcPnlKMcCYX2koE\n8h2OD9rT3KVPausmIjIPmAcQHx9PZmamxwWoKjRA+8HCpnf3uU0P6wURfaHfeEE6ScBRWVnpVf07\nI61D59Ad6gDdox7doQ6nQ9MYDNf0iIgIqyfXV1RUBBw+fDgEoLy8PDAmJsaj65r06NGjOf+CBQsS\nJ0+eXLFhw4bvs7KyQi6++OKR3tyrLcYY3nrrrZxx48bVOaafffbZtRdddFHV22+/HT19+vQRzz33\nXJuLpxlj+Pbbb3dHRES0mlYcGhranBYYGIjFYjmlL7cuv5KnMWYxsBggPT3deLPi3ZcH6qB1oOqR\n2mO2n9JsA9g+k4AA4YIfDWfo+Dh6xIS1fwMf0FULOwetQ+fRHerRlerg6RiJM+Wiiy6quv/++wcc\nOnQoKC4uzvLmm2/G/uIXvygCuOOOOxKvu+66o4MGDar/8Y9/POijjz7Kcbz2wgsvrHrggQcGFBcX\nB/bq1avxnXfeiUlJSalx95zy8vLApKSkeoCXXnqpj8M9Kl999dXYmTNnVmzevDksOzu7VV/8+PHj\nawsKCkJ27twZmpaWVvf666/HNp2bMmVK+TPPPBO/ZMmSAwEBAXz++efhF1xwQc2uXbtCUlJS6tLS\n0ooOHDgQsmXLlvArrrii/Lrrrhv+0EMPHe7Xr1/jkSNHAuPj4xsvvPDC8ieffLLv448/fgTgiy++\nCD///PPd1gMgOjq6saqq6qQmhJzJAKMAGOBwnGRPC24jvcMNm5rCthWFVIYFO6Se/FoxVqvh03/t\n5dN/7W15xllxDBkfR9KoGCKjQ0+htEoppU7FoEGDGn77298WTJ48OblpkOfs2bOPrV27tseWLVsi\n//73v+8JCgri7bffjlm0aFHvu+6662jTtUOGDGm4++67D6Wnp6dER0dbhg8fXhsdHe12WsqCBQsO\n33777UOefvrp/pdeeumxpvRf//rXRbNmzRoydOjQtOHDh9empqZWuV4bERFhnnvuuf3Tp08fHh4e\nbp00aVJlZWVlIMBTTz1VOG/evIGjRo1KtVqtMmDAgLqPPvooZ/ny5bFvvPFG76CgIBMXF9fw+OOP\nH4qPj2+89957D1100UWjAgICzOjRo6tXrFiRt3jx4vzbb799YHJycmpjY6NMmjSp4vzzzz/Q1nvW\nr1+/xgkTJlSOGDEi7eKLLz7uzSBPsQ2B8A37GIw1bcwi+S/gDmyzSCYBzxpjJopIEJANXIItsNgM\n3GSM2Xmi56Wnpxtv9iI5Xnec5TdeR21wyxd/eZ/hPPD073nnz1s4erCynau9Fx4VzISpgxl3yYAT\nZz4JXekvnbZoHTqH7lAH6B71OA17kXxjjEk/lXts3bo1b9y4cSUdVabO6vjx4wHR0dHWhoYGLr/8\n8uE//vGPS+bMmXPsxFd2X1u3bu0zbty4we7O+XKa6v8BGUAfETkI/BZb6wTGmBeBddiCixxs01Tn\n2s9ZROQOYD22aar/8CS4OBkB0rrVp4eljKDwIGb9ZqLba44WVJK3vYT/rMr1+nk1FQ0EBHaO8RpK\nKaW8c9999/X/5JNPetbV1cnkyZPLZ8+e7dfBxYn4chbJjSc4b4BftnFuHbYAxKcCJbBVWh+OcbSy\nnn7R7sdQ9E7sQe/EHkyYOhhjNWz/uIBP/5Xt8TNjErr21tFKKeWvFi9e7HH3gOoGgzxPRWBAoG1x\nLQdxcpziiro2AwxHEiCMnZLE2ClJTum1VQ3s315CXU0jx4uqKcw5RumhKqwWQ0w/XV9DKaVU9+fX\nAYa7LpLecpySyjo3uT0XFhnMyHMTnNKsVkN5SQ0RPUNO6d5KKaVUV+DXe5G46yLpTTlFFbUd/qyA\nAKFX3whElyJXSinlB/w6wAiQgFbLbAVj4ejxjp09opRSSvkbvw4w3LECx48dPWE+pZRSnUfTdu1N\nP1lZWSGOW53n5+cHTZkyZfjIkSNThw0bljZ58uThmzZtCm/KHx0dPT4xMXHMqFGjUs8///zkpvs+\n9thjfUNDQ88+evRo6yZv1S6/HoPhjlWEyuOlZ7oYSimlvOBuqfC9e/c2L3K0YMGCxIsvvrj84Ycf\nLgL46quvwidOnFjTdM211147ePr06cfnzp1b5niPt956K3b06NFVy5cv7+W48JY6MQ0wXBigtqLs\nhPmUUkq5eDTap9u18+jxk16K/PDhw8GXXXbZ8abjSZMmtbk8dpOdO3eGVldXBy5atGj/E088kaAB\nhne0i8RFI1BfpWunKKVUV9K0m+qoUaNSL7300mGu53/5y18W/epXvxo8adKk5AULFvTLy8sLdncf\nR0uXLo25+uqrS6dOnVq5b9++sPz8fP2j3AsaYLgwIliqj2O1+m4JdaWUUh2rqYtkz549uzZs2PC9\n6/lrr722PCcnZ/vcuXNLsrKywidMmJBaWFjYbsCwcuXK3nPmzCkNDAzkiiuuKFu2bFmM72rQ/WiA\n4aIR6EElZdX1Z7ooSimlOlB8fHzj/PnzS1etWrVv7NixVR988EGPtvJu2rQpfP/+/aFTp05NTkxM\nHLN69erYt956q/fpLG9X5/fNPa7TVA0QRQ2Hy2vp3UN3P1VKKY+dwhgJX1u9enXUlClTqqKioqxl\nZWUB+/fvDx0yZEibf0kuXbo09t577y188sknDzelJSYmjsnOzg5JTk7Wv0A9oC0YLhoRoqjmSHnH\nL7allFLqzNi8eXPE+PHjU5KTk1MnTpyYcsstt5RMnjy5uq38q1atir3++uudBuRNmzat7JVXXon1\nfWm7B79vwXBtwzACUVLNoeMaYCilVFdRXV39nWva9OnTK6ZPn14B8Pjjjx95/PHHj7R1/YoVK/Ic\njw8ePLjdNc/LL7+sm515we9bMATnwZxWxNZFogGGUkopddL8PsBwbcFoRFswlFJKqVOlAYYLA/Sk\nSlswlFJKqVOgAYYLK0KU1HDo+AkXeVNKKaVUG3waYIjIVBHJEpEcEXnAzfkYEXlbRLaJyCYRGe1w\nLk9EtovIFhH52mdldDm2gn0WSZ2vHqmUUkp1ez4LMEQkEHgBmAakAjeKSKpLtgeBLcaYscAcYJHL\n+SnGmPHGmHRfldOVrQWjmso6CxW1DafrsUoppVS34ssWjIlAjjEm1xhTD7wOXOmSJxXYCGCM2QMM\nFpF4H5bphKwCPbF1j+g4DKWU6joWLFjQb/jw4WnJycmpo0aNSr333nsTfvnLXyY65vniiy/Chw4d\nmga2hbOSk5NTk5OTU88555yR2dnZIU358vPzg2bMmDEkKSlpTFpaWsr48eNHLV26tNfprlNX5st1\nMBKBfIfjg8AklzxbgWuAT0VkIjAISAKOYBtv+aGINAIvGWMWu3uIiMwD5gHEx8eTmZnpXSmN6zRV\nCJUGQqln/adfUdCn6ywVUllZ6X39OxmtQ+fQHeoA3aMe3aEOp8OHH34YuX79+l7bt2/fFR4ebg4d\nOhS0ZcuWsJ/+9KdDXnjhhYKmfMuXL4+95pprSpuOP/744+yEhATL3Xff3f+RRx5JeP311/dbrVZm\nzJgx/Kabbjr67rvv7gPIzs4OefPNNzXA8MKZ/vZ8ClgkIluA7cB32GaKAlxojCkQkb7ABhHZY4z5\nxPUG9sBjMUB6errJyMjwqgDZi37nfD/7qIwoaug7KJ2McwZ4db8zKTMzE2/r39loHTqH7lAH6B71\n6Ep1GPPKGJ9u17791u1tLkVeUFAQHBsbawkPDzcACQkJloSEhMro6GjLxo0bIy+++OIqgNWrV8e+\n99572a7XX3DBBZXPPfdcX4B33303Kjg42Nx///3FTeeTk5PrH3rooaKOr1X35csukgLA8ds5yZ7W\nzBhTboyZa4wZj20MRhyQaz9XYP+3CHgbW5eLz1nt/+paGEop1XVcddVV5YWFhSGDBw8ePXv27IFr\n167tAXDttdeWvvrqq7EA//73vyN79eplGTNmTKtR/OvWrYueMWPGMYDt27eHjx07ts1lxJVnfBlg\nbAZGiMgQEQkBZgGrHTOISC/7OYDbgU+MMeUiEikiUfY8kcBlwA4flrVZc4BBNYfLdaqqUkp1BdHR\n0dYdO3bsev755/fHxcVZbr311mHPPvts7zlz5pSuXbs2prGxkVdffTX22muvLXW8bvLkycl9+/Yd\nu3Hjxujbbrut1N29b7nlloEjR45MHT16dMrpqU334LMAwxhjAe4A1gO7gTeMMTtFZL6IzLdnSwF2\niEgWttkmd9nT44HPRGQrsAlYa4x531dldWQVexeJtmAopVSXEhQUxPTp0yv+9Kc/Ff7hD384sGrV\nqpjhw4c3JCUl1a1bty5q3bp1MXPmzHEKIj7++OPsgoKCbWlpadX33Xdff4AxY8bUbNu2LaIpz7Jl\nyw5kZmZml5WVnelhBV2KT98sY8w6YJ1L2osOr78Ekt1clwuM82XZmrhbBwNsYzDyNMBQSimPtTdG\nwte2bt0aGhAQQFP3x3fffReelJRUD/CjH/2o9L777hswYMCAumHDhrVafyA4OJi//OUv+ePHj0/9\n3e9+d2jGjBkVDz/8sDz99NNxCxYsKAaorKzUhSm9pG+Yi+ZBntqCoZRSXUZ5eXngnDlzhgwbNiwt\nOTk5dc+ePeFPP/10IcCcOXPKcnJywn70ox+57QIBGDRoUMPMmTNLFy5c2DcgIIB33333+08//TQq\nMTFxzJgxY1Jmz549+NFHH9XdVL2gzT0umlowelLN8ZoGqustRITo26SUUp3ZRRddVP3dd9/tcXcu\nISHBYrFYvnVNLygocNqS/ZVXXmleWmHQoEENa9asye34kvoPbcFw6SRpDjDENoBYF9tSSimlvOf3\nAUbbYzDsAUa5BhhKKaWUt/w+wHDVKsDQFgyllFLKaxpguDDN01Rta2DoQE+llFLKexpguGjamaSp\nBWPXofIzVxillFKqi/L7AMN1DEbTRihR9kGem/eVYlw2RFNKKaVU+/w+wHDVtA5GT3sLRlFFHQdK\ndUl6pZTq7Fy3a9+4cWMkwKFDh4KCgoLO/v3vfx/nmP/48eMBN99888ABAwaMTk1NTUlLS0t55pln\n+jjmeeyxx/qGhoaeffTo0cDTWZfuQAMMF81dJNISVGzOKzszhVFKKeURx+3as7Ozd3300UfZQ4cO\nrQdYunRpzLhx46refPPNWMdrbr755sExMTGNeXl5O3bt2rV7w4YNe0tLS50WPnrrrbdiR48eXbV8\n+XLdqt1Lfr+CVKsukqZBnlRjCzeEzftKuW5C0ukumlJKdSm7R6X4dLv2lD27vdquvencm2++Gbtw\n4cL8W2+9dej3338fPGzYsIadO3eGbtmyJfKdd97JDQy0NU7079/f8rvf/e5w03U7d+4Mra6uDly0\naNH+J554IuGuu+466sPqdTvaguGiqQUjRBoJxbZk/ea8NleXVUop1Qm0tV17Tk5OcHFxcfCUKVOq\nZ86cWbZ06dJYgC1btoSlpKRUNwUX7ixdujTm6quvLp06dWrlvn37wvLz8/3+j3JvaIDhwurwumkc\nRm5JFWVV9WemQEoppU6ore3aly5dGjtz5swygFtuuaV0xYoVse6uX7BgQb9Ro0al9u3bd2xT2sqV\nK3vPmTOnNDAwkCuuuKJs2bJlMaerPt2B30djrl0kxiElSqopNrZut/2l1cREhpzGkimllPJG03bt\n06dPrxg7dmzNsmXLeh85ciS4uLg4eOXKlbEARUVFwdu3bw8dN25c7e7duyMaGxsJDAzk6aefPvz0\n008fjoiIOAtg06ZN4fv37w+dOnVqMkBDQ4MkJSXVP/jgg8Vnso5did8HGG3tRQItLRgA+aXVjB+g\nY3yUUqot7Y2R8DV327U3NjZSVVUVWFRUtK0p3913393/lVdeiV24cOGhsWPHVt11112Jf/7znwuC\ngoKorq6WpmUJli5dGnvvvfcWPvnkk81jMhITE8dkZ2eHJCcna5O2B7SLxIXVpQWjycGymjNRHKWU\nUh5wt117enp61RVXXOE0DXDWrFllTa0Zy5cvzystLQ0aNGjQmNGjR6dcdNFFyY888shBgFWrVsVe\nf/31xxyvnTZtWtkrr7zitotFteb3LRitu0haROEYYOhaGEop1Vm1t127o0mTJtXk5ubuBIiNjbW+\n9tpr+93lO3jw4HbXtJdffvngqZfUf/i0BUNEpopIlojkiMgDbs7HiMjbIrJNRDaJyGhPr+24Qjof\nWh2OI6UPXf1oAAAgAElEQVRlH5J8bcFQSimlPOazAENEAoEXgGlAKnCjiKS6ZHsQ2GKMGQvMARZ5\nca1POK4KHkVLUKEtGEoppZTnfNmCMRHIMcbkGmPqgdeBK13ypAIbAYwxe4DBIhLv4bUdQly2GbFK\nSxNGpEOAUVBWo3uSKKWUUh7y5RiMRCDf4fggMMklz1bgGuBTEZkIDAKSPLwWABGZB8wDiI+PJzMz\n06tCWq1Wp2PHEKJXYE3z7md1FivvrP+IXmGdd1xsZWWl1/XvbLQOnUN3qAN0j3p0hzoo/3SmB3k+\nBSwSkS3AduA7WjY09YgxZjGwGCA9Pd1kZGR4VYD9zz7ldOwYbvQLawSHyUgDUs5iwqDOu85KZmYm\n3ta/s9E6dA7doQ7QPerRHeqg/JMvA4wCYIDDcZI9rZkxphyYCyAiAuwDcoHwE13bYcR5lKc4jPLs\nE+w81flgWXWnDjCUUkqpzsKX7f2bgREiMkREQoBZwGrHDCLSy34O4HbgE3vQccJrO0yA6zSSlpcx\nQXVOp3QtDKWU6rwOHDgQNH369KEDBgwYnZaWljJ58uTh27ZtCw0LCzs7JSUldejQoWljxoxJefbZ\nZ3uf6bL6A5+1YBhjLCJyB7AeCAT+YYzZKSLz7edfBFKAV0TEADuB29q71icFDXCJsayOC23VOp3S\nmSRKKdU5Wa1WZs6cOfymm246umbNmlyAL7/8MrywsDB4wIABdbt3794FsGvXrpBrrrlmuDEG3R3V\nt3w6BsMYsw5Y55L2osPrL4FkT6/1CRHnuakOLRiOs0gA8ko0wFBKqba8MH+jT7dr/+WLF7e5FPma\nNWuigoKCzP3339+8V8h5551Xk5WV5bSJVGpqav3vf//7/AULFgzQAMO3Ou+UiNPFtQXDtLRghBnn\ngCK3pPJ0lEgppZSXtm3bFj5u3DiP/go8//zzq/ft2xfm6zL5Ow0wAlwHeba8DrZUEehw/kh5HZV1\nltNVMqWUUj6gaxqdHn4fYIhLC4bjLBKpq2RATLjT+X3FVaelXEoppTw3ZsyYmq1bt0Z4kvfLL7+M\nGDp0qI7a97EzvQ7GmddqkKfDa0sNwxPDyTva0uqWW1LJmKTo01M2pZTqQtobI+FrM2bMqHj44Ydl\n4cKFfX7961+XAHz11VfhZWVlgY75srKyQh544IGkn/3sZ0VnpqT+QwMMl3UwAoxzwDEqBj50OM7V\nFgyllOp0AgICWL169fe/+MUvBixatKhfaGioSUpKqnvuuefy8/PzQ1NSUlLr6uokMjLSOn/+/KI7\n77xTB3j6mN8HGBLY9iBPgBE9G5yO80t1JolSSnVGgwcPbli3bl2ua3ptbe23Z6I8/s7vx2C4dpEE\nGKfWNIYEOQe5utiWUkopdWIaYLSapup83M962OlYF9tSSimlTszvAwwJcG6xEJe3JLb+kNPx4fJa\nGhqdd2BVSimllDO/DzBadZFYncdgBB3fT1xUaPOx1cDh485LiCullFLKmd8HGOIyi8S4viVleST2\ncl4Lo+CYjsNQSiml2uP3AUaAOL8FxiXgoCyPvg4tGAClVc7buCullFLKmd8HGK4tGFYMBDrsjVNT\nRr9Q5y6RsmoNMJRSqrNZsGBBv+HDh6clJyenjho1KnXjxo2REydOHDl48ODRycnJqUOGDEmbM2fO\nwJKSkubBd21t8d50/rHHHusbGhp69tGjRwPdP1W1RQMM1y4SY6DXIKe0wQElTsfHqp3XxlBKKXVm\nffjhh5Hr16/vtX379l3Z2dm7Pvroo+yhQ4fWAyxdujQ3Ozt71+7du3eFhoZap02bNhxatnj/wQ9+\nUJGfn79j586du5966qmCwsLC4Kb7vvXWW7GjR4+uWr58ea8zVbeuyu8X2gpwibGsGIgZDEf3Nqcl\nUgQkNh8f0xYMpZRq5Zkbpvt0u/Z7/7WmzaXICwoKgmNjYy3h4eEGICEhodXOlGFhYeavf/3rwUGD\nBo358ssvw4uLi4PcbfHe9Hrnzp2h1dXVgYsWLdr/xBNPJOj27t7RFoxWLRhWW4DhIL7ReapqmbZg\nKKVUp3LVVVeVFxYWhgwePHj07NmzB65du7aHu3xBQUGkpKRU79ixI+xEW7wvXbo05uqrry6dOnVq\n5b59+8Ly8/P9/o9yb/h9gBHgOgbDmFYBRu8G58W2dLlwpZTqXKKjo607duzY9fzzz++Pi4uz3Hrr\nrcOeffbZ3u7yerpd+8qVK3vPmTOnNDAwkCuuuKJs2bJlMR1a6G7Op9GYiEwFFgGBwMvGmKdczkcD\ny4GB9rIsNMb8034uD6gAGgGLMSbdJ2V0ibEM7gKMQqfj74srfVEUpZRSpyAoKIjp06dXTJ8+vWLs\n2LE1y5YtaxVgWCwWsrKyIsaOHVt4+PBhy6pVq9wGDZs2bQrfv39/6NSpU5MBGhoaJCkpqf7BBx8s\ndpdfteazAENEAoEXgEuBg8BmEVltjNnlkO2XwC5jzAwRiQOyRORVY0zTIIcpxhjnEZYdX06nY6ub\nLpLwynxCAgOot6/gWVJZz7HqenpFhKCUUsqmvTESvrZ169bQgIAAxowZUwfw3XffhSclJdXv2bOn\neSGjuro6ueuuuxITEhLqJ02aVGO1Wmlri/c1a9ZE33vvvYVPPvlkcxN2YmLimOzs7JDk5GQdiOcB\nX3aRTARyjDG59oDhdeBKlzwGiBLbt3wPoBRoNTDHl1y7SDDQGJ3klCTHDjCsd5hTmrZiKKVU51Fe\nXh44Z86cIcOGDUtLTk5O3bNnT/jTTz9dCDBnzpyhycnJqSNHjkyrqqoKeO+993KgZYv3jRs39hww\nYMDo4cOHpy1YsCAxMTGxYdWqVbHXX3/9McdnTJs2reyVV16JPRP164rE074or28sch0w1Rhzu/34\nFmCSMeYOhzxRwGpgFBAF3GCMWWs/tw84jq2L5CVjzOI2njMPmAcQHx8/4fXXX/eqnAdfW8KR4y2N\nJCLFjJ73OBlf3kZIQ3lz+s97Ps97RS2/V3NHhzA5KZjOpLKykh493I5r6jK0Dp1Dd6gDdI96+LoO\nU6ZM+eZUu6C3bt2aN27cOJ+2NqvOaevWrX3GjRs32N25Mz0i9nJgC3AxMAzYICKfGmPKgQuNMQUi\n0teevscY84nrDeyBx2KA9PR0k5GR4VUBVq9Z6RRgAEy6YBIh2YPh8LbmtAsH9eC9opY8QTGJZGSk\nevUsX8vMzMTb+nc2WofOoTvUAbpHPbpDHZR/8mUXSQEwwOE4yZ7maC6w0tjkAPuwtWZgjCmw/1sE\nvI2ty6XjSeukBmsDhPZ0Shvc03kH1Zwi7SJRSiml2uLLAGMzMEJEhohICDALW3eIowPAJQAiEg+M\nBHJFJNLefYKIRAKXATt8UUhxE2E0NDZAqHOT5IBIlwBDx2AopVQTq9VqdfPnmurO7J+5ta3zPgsw\njDEW4A5gPbAbeMMYs1NE5ovIfHu2x4HzRWQ78G9ggX3WSDzwmYhsBTYBa40x7/uqrK7qrfUQ4hxg\n9Auz4Dge9GBZDbUNjaerSEop1ZntKC4ujtYgw39YrVYpLi6Opp0//n06BsMYsw5Y55L2osPrQmyt\nE67X5QLjfFm29tQ31rdqwQhprCIpJp78UtsqssbYZpKk9Y8+E0VUSqlOw2Kx3H748OGXDx8+PBpd\nwNFfWIEdFovl9rYynOlBnp2SuxYM6ioZHtejOcAA2zgMDTCUUv5uwoQJRcDMM10O1blopOmGbQxG\nlHNifSXD+zoHHd/rQE+llFLKLb8PMFxX8gR7F4m7FgyXAEMHeiqllFLu+X2A4Y5tmqpLgFFf0TrA\n0BYMpZRSyi0NMFqRtlsw4py7TfaVVNFo9c1KqEoppVRX5vcBhrs5VfXWerdjMKIjgukZ1jIutqHR\ncKxa97xRSimlXPl9gOGObZqqS4BRZ+sO6dMj1Cn5aJUGGEoppZQrjwIMERkmIqH21xkicqeI9PJt\n0U6TRufFslLyjW0MRqsukgoAevdw3qK9pLLOp8VTSimluiJPWzBWAI0iMhzbxmIDgNd8VqrTqKGw\nsFWau4W2qLcFGK4tGCWV2oKhlFJKufI0wLDal/6+GnjOGHMfkOC7Yp0+AT2dNzVDmqapetZFkl9a\n7cviKaWUUl2SpwFGg4jcCNwKrLGnBfumSKdXxPjxLinSxjRVW4CRHO+cvvtQuQ9Lp5RSSnVNngYY\nc4HzgN8ZY/aJyBBgme+KdfpIsHOcdKRnuG0WSVAYSGDLicZ6sNST2t+5xWNXoQYYSimllCuP9iIx\nxuwC7gQQkRggyhjztC8LdrqUHSpwOo6oa6C8sQFEbK0YtcdbTtZXMqpfT0Rsm50B7DtaRVWdhchQ\n3dZFKaWUauLpLJJMEekpIrHAt8DfROSPvi3a6dEjtrfTcYAxtjEY4GYcRgWRoUEM6R3ZnGQM7Dms\nrRhKKaWUI0+7SKKNMeXANcBSY8wk4Ie+K9bpE9Ovf6u0eqs9wGhjHEaKSzfJTu0mUUoppZx4GmAE\niUgCcD0tgzy7rZYWjNbLhQOk6TgMpZRSql2eBhiPAeuB740xm0VkKLDXd8U6jVrtpmqfRQJtroWR\nmqAtGEoppVR7PAowjDFvGmPGGmN+bj/ONcZce6LrRGSqiGSJSI6IPODmfLSIvCsiW0Vkp4jM9fTa\njuIaXhiBhkZ7gNFmC0a0U3LWkQoaGq0+KqFSSinV9Xg6yDNJRN4WkSL7zwoRSTrBNYHAC8A0IBW4\nUURSXbL9EthljBkHZADPiEiIh9d2jFYtGI5jMFpveAYQFxVKXFTLglv1FivfF+vW7UoppVQTT7tI\n/gmsBvrbf961p7VnIpBjb+2oB14HrnTJY4AoERGgB1AKWDy81mdONAYDdByGUkop1R5PF2+IM8Y4\nBhRLROS/T3BNIpDvcHwQmOSS53lsgUshEAXcYIyxiogn1wIgIvOAeQDx8fFkZmaeoFjOjuTua512\n9AiZmZkMOVLKIIf03KxtHKi13b9Hg/MeJOu/2klseY5Xz+5olZWVXte/s9E6dA7doQ7QPerRHeqg\n/JOnAcZREZkN/J/9+EbgaAc8/3JgC3AxMAzYICKfenMDY8xibBuwkZ6ebjIyMrwqwOZjxRzc5PzI\nHj17kJGRAQFfw4EVzelD+8cx1H7/qthDrMn9tvlcRVA0GRnnevXsjpaZmYm39e9stA6dQ3eoA3SP\nenSHOij/5GkXyU+wTVE9DBwCrgN+fIJrCrDtutokyZ7maC6w0tjkAPuAUR5e6xMGHGaRuB+DATCs\nb6TTqUPHa31cMqWUUqrr8HQWyX5jzExjTJwxpq8x5irgRLNINgMjRGSIiIQAs7B1hzg6AFwCICLx\nwEgg18NrO4S4DPIMNJ6NwejXM8zp1OHjtZim9cOVUkopP+dpC4Y797R30r69+x3Y1s/YDbxhjNkp\nIvNFZL492+PA+SKyHfg3sMAYU9LWtadQ1na0nkUSVWQPJNpYyRMgOjyYkKCWt6+moZGKOotPSqiU\nUkp1NaeyQ1frb2YXxph1wDqXtBcdXhcCl3l6rU+4qcXl7xXD7bhpwahouUyEfj3DOFBa3ZyWX1rd\nao0MpZRSyh+dSgtGt+wPMEDP4ycegwEwoq9zALIl/5gPS6aUUkp1He0GGCJSISLlbn4qsK2H0eW1\nasAQIajBvipnO2MwAM4eFON0/M3+so4tnFJKKdVFtdtFYoyJau98dyDBwa3S8vrZ4652xmAAnD3Q\nOcD47oC2YCillFJwal0k3UTrQRjHw+29PydowRg3IJrAgJbr95VUUVrlvACXUkop5Y/8PsBw3YrE\nAMH1jbYpp63GYFSAw1TUiJAgUhKc83yr3SRKKaWUBhjuNjsLbbAvthUYDIEtm5phrNBQ7ZTXtZvk\n2wMaYCillFJ+H2CImy6S0AaHxbZcx2G4dJNM0IGeSimlVCt+H2C4E9LgsGW76ziMEwz03HbwOJZG\nqy+Lp5RSSnV6GmC4dJEY7F0kjW2sheGw2BZAUkw4cVEt3Sg1DY3sOeycRymllPI3fh9guBmCQYAx\nHrdgiAhnD+zllKbdJEoppfyd3wcYrdgDjpYWjPbHYEDrcRib80p9UTKllFKqy9AAo9UgT9uxpy0Y\nAOcMjnU63rSvVHdWVUop5df8PsBw3a69SduzSFqPrxidGE14cGDzcVFFHfuPVrfKp5RSSvkLvw8w\n2tIcYIS0v+EZQHBgQKtukk37tJtEKaWU/9IAw81KnuDQReLBGAyAiUOcu0m+0gBDKaWUH9MAw81C\nWwAWq8X2woMxGNA6wNiUd/SUS6aUUkp1VX4fYJxwDEZYT+cTtcfd5h8/oBchgS1vZ35pjW58ppRS\nym/5NMAQkakikiUiOSLygJvz94nIFvvPDhFpFJFY+7k8EdluP/e1L8vpXCjbPy2DPF0CjLpyt5eF\nBQcyNC7SKW1fifvWDqWUUqq781mAISKBwAvANCAVuFFEUh3zGGP+YIwZb4wZD/w/4GNjjOPghSn2\n8+m+KucJx2CERTtnqHUfYACtAozc4qpTLJxSSinVNfmyBWMikGOMyTXG1AOvA1e2k/9G4P98WB63\n3G12Bt63YAAM7eM8XmNL/rFTKptSSinVVQX58N6JQL7D8UFgkruMIhIBTAXucEg2wIci0gi8ZIxZ\n3Ma184B5APHx8WRmZnpVyJKsPW7Td2XtIvNQJhFVB5jokF5dephNbTwjpNzidPz+1nx+2KukzXEe\nHa2ystLr+nc2WofOoTvUAbpHPbpDHZR/8mWA4Y0ZwOcu3SMXGmMKRKQvsEFE9hhjPnG90B54LAZI\nT083GRkZXj14Bxb2f7TeIcUWDAweOpiM0RlwvAA2t5yNCLTQ1jPObWjkha0fUGex7aZ6tNaQlJrO\niPgot/k7WmZmZptl6yq0Dp1Dd6gDdI96dIc6KP/kyy6SAmCAw3GSPc2dWbh0jxhjCuz/FgFvg1ND\nQodpcxaJta1ZJG13kYQFB3Lu0N5OaZlZxadUPqWUUqor8mWAsRkYISJDRCQEWxCx2jWTiEQDk4F3\nHNIiRSSq6TVwGbDDh2V1kprvuJJnDxCHt8lSA00bobkxZWSc0/FHWUW+KKJSSinVqfkswDDGWLCN\nqVgP7AbeMMbsFJH5IjLfIevVwAfGGMcpF/HAZyKyFdgErDXGvO+rsjqVu+nfcntLhQiEunRxtNOK\nkTGyr9Px5rxSahsaO7CESimlVOfn0zEYxph1wDqXtBddjpcAS1zScoFxvixbk1ZdJPZDOeawhkVo\ntPMCW3XHIdK5K6TJ4D6R9IoI5li1rZWjodFQUWshzGEzNKWUUqq78/uVPGljDEaD1aEbxIsWDICg\ngNMza0QppZTqrDTAaEPzXiTQeqBnO2thKKWUUkoDjFbLbDWNwXBuwXANMCp8WSSllFKqy/P7AMOj\nLhIvpqq6c6xaNz1TSinlXzTAaCPAsDS214LRfoCREB3udPz42t0YY9rIrZRSSnU/GmC0IarQYdaI\nly0YN08a6HT8SXYxH+7W9TCUUkr5D78PMFq1X9hbNMoiHVocWrVgHKc916cPYNKQWKc0XXBLKaWU\nP/H7AMO1i6R5oa0Kh3UwvGzBCAgQrpuQ5HyJLrallFLKj2iA0QapcFhYNDTa+aQH01QDXAKXLfnH\nNMhQSinlN/w+wGhrszOprGk5OIlZJOMH9nI6zi2u4vfvZ3ldPqWUUqor8vsAw80oDACuzKyh0Wpv\ncfByFgnAsLgeXHu2czfJ8q/2U1VnaeMKpZRSqvvw+wCjjQYMetZAZYN9HMZJroPx6MxUYiKCm4/r\nLVYKj9W0c4VSSinVPfh9gOGqaZDnjoHSEmCcRAsGQFRYML17hDqlZWYVn2IJlVJKqc5PA4w2mjBy\nE6Ci3r4k+Cms5BkbEeJ0/MyGLHYUtD/NVSmllOrq/D7AENcxGA6HzQFGSJTziYYqaPRsLMVdPxxB\noMPuqrUNVhas2KYreyqllOrW/D7AaMsPtxjK6+0tFQEBrbds97Cb5ILhffifmWlOaTsLy/n2wLGO\nKKZSSinVKWmA0XopTwAi6hxaMADCXNbCqPW8m2P2uYP4YUpfp7RXv9rvRSGVUkqprsWnAYaITBWR\nLBHJEZEH3Jy/T0S22H92iEijiMR6cm0HFtLp0LHjorLecTVP53UtqPWuBeLmcwc5Ha/Zdkh3WVVK\nKdVt+SzAEJFA4AVgGpAK3CgiqY55jDF/MMaMN8aMB/4f8LExptSTazusnG2sg2EJcGnBCHcJMGrK\nvHrOD0bEkRTTsstqvcXKW98c9OoeSimlVFfhyxaMiUCOMSbXGFMPvA5c2U7+G4H/O8lrfaJ5DAZA\neIzzSS8DjMAA4caJzrusvvbVAR3sqZRSqlsK8uG9E4F8h+ODwCR3GUUkApgK3HES184D5gHEx8eT\nmZnpVSGP7ctxmx5ghb0H9pJZbbtf8rFa+jucz966icKS3l49K6neECjQaI8pckuqeHHlRlJ6B3p1\nn7ZUVlZ6Xf/ORuvQOXSHOkD3qEd3qIPyT74MMLwxA/jcGFPq7YXGmMXAYoD09HSTkZHh1fXfR4Xz\n/furWqUHAJG9Imi+X8NHcOiD5vPJA+JI/oF3zwL44Oi3rN12qPl4Z10MP8842+v7uJOZmYm39e9s\ntA6dQ3eoA3SPenSHOij/5MsukgJggMNxkj3NnVm0dI94e22HMg5DMqrrOm4MRpObJzl3k2RmFdPQ\naD2peymllFKdlS8DjM3ACBEZIiIh2IKI1a6ZRCQamAy84+21HaONzUiA6lrHAMN1DMbJrWNx3tDe\n9HFYPryyzsJ2XdlTKaVUN+OzAMMYY8E2pmI9sBt4wxizU0Tmi8h8h6xXAx8YY6pOdK0vytnWdu0A\nlbUdN8jT8XljEp2XHv8q1+ueIaWUUqpT8+kYDGPMOmCdS9qLLsdLgCWeXOsTbccXlNeUYYyxBSEd\nFGAAhAQ5x3ULP8ji0tR4hvftcdL3VEoppToTXcmzlZaIo76+hpKaEttBBwYYFw7v43TcaDU8sW73\nSd9PKaWU6mz8PsBwXWjLcVWKQCvklefZDlxX8jyFAOPmSYO4eJTz0uEb9xRR29DY9kUNtbZxHw01\nYNVBoUoppTq3zjJN9cxpZwxGgBX2l+/nnH7nQKhL90VD9Uk/MiBAeGR6Khv3FDml3/qPTbz203Nt\nu69a6mHn23DgC/hmifsbDboAxt0I42ZBYPBJl0cppZTqaBpguHKINwLtAQYAwZHO+eqrwJh2A5T2\nDIyNYGR8FFlHWmaqfLWvlHe2FHDNUAPLr4WSrPZvsv9z28/qO+C/niG4vk/7+ZVSSqnTRLtI2jkX\n3OjQRRIYDOKw4qZphMaGk35uQICw5CfnOO1PAvDdZ+9jFmecOLhwtfZeLvjiVvh80UmXSSmllOoo\nfh9guLZAWB2OI2odWjBEIMSlFaOhilOREB3On24Y33x8dcCn/OboAqS65ORvuuER+OuFkPPvUyqb\nUkopdSr8voskODTM6bi0R0uLwjl7rbwxMB+L1UJQQBAER0Cdw9oYDTWtZ5d4KX1QDBcN7cm5Bxbz\nyyA3a4nFj4ZLHrH927O/LdA5+j1sfR0+/zM0utny/ch2WH4NzMuE/medUvmUUkqpk+H3LRg9YmNb\npTXN0WgIAovVwqFK+94hwc7dGdSf/EBP24MakW+W8M+Kn7kPLiYvgJ99CsmXQ3RiS2tL72Fw8UPw\ncDH86tu27784w9aicZKrjiqllFIny+8DjJ5x8a3SrAG2L/IQ+xCL5nEYrbpITjHAeOcOWPPfBFU4\nb7NSZ4L5Vf0dZKf+CgJO8BH1HgYPH4Vpf3B//vNF8NwEyN90amVVSimlvOD3AYa7pcKPRNsCiRCL\n7fhAxQHbC9cWjFMJMOqrYOtrrZKLTTSz6n/Du9bzeWLdbqxW4+ZiF4FBMGkeX5y3BPqMbH2+ugSW\nXgWHtp18eZVSSikv+H2A4U5NsG1oSmhTC8bxPNuL4AjnjCcbYBgD6x9qlZwz5Gam1T3Fd2YEYNtp\n9aVPcj2+bX1oDPz8C5jxLEQluJS1Cl66CGp1YzWllFK+pwEG0GfgYKfjpjaNphaM5pkkIS6LbZ3s\nl/Xu1fDNP53TovozaPbzDBg4yCn59+v3sH7nYc/vHRgEE261jc0YdGHr889NgANfnUShlVJKKc9p\ngAEMOSvdbXpTC0ZzF0kP5+W9qXReidNjX7sEFyFRcMMyggMDeP6ms+kV0bIqpzFwz7+2sK/Eyymx\nIRFw3d9bp1cVw7Kr4PuPTqLgSimllGc0wGhHsL0Fo6i6CGMMRPVzzlBxyPubFn4HuS5f7tf+DZJs\nQU5ir3AWzTrLtly4XVV9I7949Vtq6tvZq8SdqH4w6zWIdAmMGqrhteth9xrvy6+UUkp5QAOMdoQ2\n2AZYNlgbKK8vhx4uM04qjnh3w4YaWDnPOS1mCCRPdUqanBzHI9NTndJ2HypnwYpttkDHG6P+C+78\nDs6/0zm9sR7emANb/+Xd/ZRSSikPaIDRjoi6ltfF1cWtB05WejE2AuDD/4GSbOe0y/7X7X4mc84b\nxH+NcX7e6q2F/HFDdqu8JxTaAy57HC5/0jndNMLb8+A/L9r6YpRSSqkOogFGOwYVt7wurimGKNcW\nDA8DDGPgk4Xw1V+d08+aDSnT3V4iIjx17RhG9HUeWPrcxhye37jXs+e6Ou8XMPN5EJeP/f0F8NoN\nntdHKaWUOgGfBhgiMlVEskQkR0QeaCNPhohsEZGdIvKxQ3qeiGy3n/val+Vsl/0v+5KaEujhOgbD\ngy9kY+D9B2Dj487pvQbB1KfavTQqLJi/zUknOtx5K/aFH2Sz+JPvT/xsd86+Ba77BwS4bO++dz38\n7RLbMuRKKaXUKfJZgCEigcALwDQgFbhRRFJd8vQC/gLMNMakAT9yuc0UY8x4Y4z7aR6nQZh9q4+S\nmhKIjHP+67+mFCx17i9skv0+fPWic1pgKFyzGEKjTvj8wX0ieeUnE+kR6rxtzBPr9vD8xr3ej8kA\nSCnTcYQAABP0SURBVLsabvy/1ut6lB+Ef06Dsjzv76mUUko58GULxkQgxxiTa4ypB14HrnTJcxOw\n0hhzAMAYc5LzPn0nzr63WXFNsW2NCdcZGeWF7d/gyI7Wabe8DQPP9bgM4wf0Ysncc4gICXRKX/hB\nNv/z7i7PVvt0NeJSmP8ZDHApR+URWHaNdpcopZQ6Jb7cTTURyHc4PghMcsmTDASLSCYQBSwyxiy1\nnzPAhyLSCLxkjFns7iEiMg+YBxAfH09mZqbXBT144ECb5wLtM0O3520nsyqTswKiiably3fLx+9y\nLGZsm9eP3r6ePg7HRXEXsCuvAfK8L+evxgXzp28aqbe2pC35Io9dufncNiaUhpoq7+s/9H5GNrxA\nwmGH7d1Lv6f6L1PYMv5/qQ/t7XU5T0VlZeVJfYadidah8+gO9egOdVD+6Uxv1x4ETAAuAcKBL0Xk\nP8aYbOBCY0yBiPQFNojIHmPMJ643sAceiwHS09NNRkaG14X4pDCPI9+53wws2B5gFEkRGRkZUDIG\ndmQ1nx8/KAbObuOZB/4DH3/jlNR34rX0Pcf7MgJkABPOLuW2VzZTUWtpTt90uJEKCWLuiAimn0T9\nmTwFVt4OO1Y0J0XUFHL+nsdh7jrbNvGnSWZmJifzGXYmWofOozvUozvUQfknX3aRFAADHI6T7GmO\nDgLrjTFVxpgS4BNgHIAxpsD+bxHwNrYul9MuyB5gFFYV2qaq9hronOHYfvcXVpfCW7eBcWhuCI+F\n1KtOqTwTh8Tyxs/OIy4q1Cl996Fy/ufLGt7fccj7cRkBAXDVXyF5mnN62T5Y8VOwernAl1JKKb/n\nywBjMzBCRIaISAgwC1jtkucd4EIRCRKRCGxdKLtFJFJEogBEJBK4DHAzmMH3Iupavqy3Fm+FGOe9\nQjjmpnvFGFj9K9ugSUdXvwiRp97lkJLQk5U/P59hcc7bx1c1wPzl33LbK1+TX+rlRmxBoXD9Uhjl\nMm12/2ew8qdgqT/FUiullPInPgswjDEW4A5gPbAbeMMYs1NE5ovIfHue3cD7wDZgE/CyMWYHEA98\nJiJb7elrjTHv+6qs7bnx45YWiC1FW9y0YLgJMHL+DXtcluE+7w5IvrzDyjUgNoJ37riQaaP7tTq3\ncU8RP/zjxzz3771U11vcXN2GoBD40RIYMtk5fccKeP0mqD+F7emVUkr5FZ+ug2GMWWeMSTbGDDPG\n/M6e9qIx5kWHPH8wxqQaY0YbY/5sT8s1xoyz/6Q1XXsmDHRYbGtL8Rbb+hWO3K0bsS/T+ThmCFzy\n2w4vW4/QIP5y89k8MG2U094lAHUWK89syOYHv8/kH5/to7bBw26OwGBbS0uUy7iLnA22lgxd8VMp\npZQHdCVPD0TW2L5Ud5bspDoyDgJDWk5WFdnGWzgqyXE+HnuDrXXAB0SE+ZOHsfbOC0mOaf1xllTW\n8diaXWT8IZOlX+ZRWedBi0bP/vCT922BkaM9a2wtGVUlHVN4pZRS3ZYGGB4IsX8nW4yFb0q2Qp9k\n5wxFu1teH94B2e85n49O9G0BgVH9evLAxDD+cN1YYiNbBzOHy2t55J2dnPO/H/LrN7eyaV9p+4NB\nYwbBT9ZDXIpzetY6+NuU1kGUUkqp/9/evQfHVd0HHP/+7r600kp+SbKNbR42tokN5WGDcUpAoUBc\nhg6ZTEtCJ7gJDZQmmZROEqYJTJJOmE4S0kzTNp2EaTOFQimU4kB5paHYxjXYxnZs+W2DwSAsLEvG\nkqzHvu6vf9wra7VaWSt5V9qVfp+Z692995y757dH0h6fe+45JoM1MEZoc/NmqM/60m3Z6z32nIQn\nPz/wWLBi0GqpxeKI8EfL5/Hq16/jyw0LiIYCg9L0JNM8va2J237xBp/88Tr+5sV9vPF2G8m0O/iE\n1TPh9n+H6LSB+0++B4/8AaRHML7DGGPMpDLe82CUhV/8Y5qv3hOgvQpefvdl7q2/aeAHd2wPuC6N\nf/cl9hyoJule3n9sylz4/oODznn9n9zN3CUXF6W8UyvD3LfqIu685gJ+vu5t/m3TEeKpwQ2Id9u6\nefi1wzz82mGqK4Jcu6iO6xbVccW505hfW4XjCEyfD3/6ijf+4uj2/sydR+Gnl8I9G6ByelHiKBhV\n6DoOve3Q2wHxdoh3epe6wjGIToWKqd5U8KGK8S6tMcZMCNbAyOHtuqkcmVGDiuBdRBBu3eBNLRoP\nC/8Q2Iib/ATnVn3EJ+sPk9izjcbd32DPjl5gysCTtZwETg56j3hP8e/IqI1FeOCWJdx17Xwe23SE\np7c10dzemzNtZ2+KFxqbeaGxGYCaiiCXzpvKknNquLAuxoKbnuSyZ67H6ciYnLWjCX50Adx/rLS+\nmDua4Z310HoQ9qyBE4dHlv+iW7x1Ws5bCRfeMPjOIWOMMcOyBkYOqWCAFIMvLwAEXHD9iafe65rG\nI+8s848cHKPSjdzMmgq+ftNi7r1hERsOHec/tzbxP3s/JJkeegxGR2+KDYda2XCof0DnVfIFnop8\nf1DaxN8upXXpF+la9mWqKqNURYLEIsFBd7YUTaIbju+HI6/DK98F9ywv3fTdYrzrqf59l30eZv8O\nLL4Zps7Lna9UpeJej02iC5Ld3ueV9Bu4TtC7c8gJelskBpW1dreQMeasWQMDqJoybfhEE0DAERoW\n19OwuJ72niQbDh3n1f0trD9wnLau4SfS2qIf4zPx7/FM5HsD9od7Wzln20Ow7SFuT9zPm+5iUgSp\nCDnEIiFikQBVkeDphkdVJEgs7FAf6GS600VNMI1bM4dQbAaHW1JUHG4jGgoQDQeIhgJEQo73OhQg\nGPCHDfWchPU/gk0/81a41RxjSAppx2OwA3jpPm813MWrIDYLrv0GxOqHzV5UyR5vkbq3/tdb4ber\n1Vus7tQxaG6EZNeIT/kJJwy/rYOOD2D6Api5FJZ9wbvkVzPHa4iUomQvtL8PPR9BsofpbVthb7vX\nyEr2eI+pHm922mDE3yq8LVwFlTP8bbp32UzGqJFszAQko1ruu0QtX75ct27dOuJ83R3t/Ou3/pKe\n1sIt5jp/4RwWrbwRCQVx5s3193qftaoSq6sjXFmJMvDz76uPvv2akSfzdf9D//7NmzezYsWKIc8x\nKI+fLu26HDzWyZZ3T7CvuYMDH3bQleibNyP750O5IbCNrwR+RUjSGUdlQOod7nxedFfQqjVEJc4i\n+YCwJKijnXo5wTxppVq6B+RR/xytVPNq+nL26Pmc1CqCpIlJD9cFdlJBkoVOE9PozMqb/SiZxTq9\n/0TFPFKBStKBCA5pQukepnQNvIQymt+IRLSedLiGzlkraP6omxnzLiQdqvbGeYggmsJJxQkkOtFA\nCAcX1EUURNOImwAUNzINdQIgQXAC3nMngJPqRdwU4Y4jBHpaCZ84SKjzfZx0fBSlLYzk9EVopAa3\nYhpudDq4KVJTzoNQJeoEUcfrGVEnBAz8XHXgP6d7TCTzN0L7a9NJnAJcpOckoZOH0WAFTvIUEu8g\nenQzbigGbgInXcAZZyUAkWp/q/EaIJGYt4VjEK7uf57ZC5S5ifhx+LFEYl4DdQR27drFJZdckvPY\nlg+3EE/HWXnOSoIS5MpZV1IZqhxZmCLbVHX5iDIZkwdrYPjWrl3LskuW4qbSiCOI4yDikDrWwpHb\nbkP09FcWh2ZOo7W68vQfwt5QkHTAwXFdquJJ6jq7WdR8Agc4Ugff/JJ1FBljiu/ZTz/L/CnzR5TH\nGhimWOybzyci1NTm6OqeUcu03bs5+sADtD/trTa65Ggb0JbXeQNF7rk3xpg+Ack9dsyY8WDzYOTp\nnAcfZOa3vzXifNbAMMaMhdpoLXNjc4dPaMwYsR6MEZi+ejXTV6/Gjcc59vijtDz1BOF3mwelO1YX\nIu0oKQdOTAkwo2Iq4g8WEwY+kjGG7PSxIdJK1oCz7P29Pb1Eo9Gc5zidJ99zZ7weKm3Ocye6kUQX\ndB5F/GvvkvLGCUggBNWzvYF1oSgEIv3nUoXEKRLtLURSnUgo6t0N4qYg1Yso3uA7J4gGIqQr63Gd\nMGlVVMFVcF1vfEnfa/AeVb3xJop4j/5xpe8YuP5+9fe7fp6s4SuDBDVBjdtBUFMESHqPboKwpAmS\nIqJxBKWXCK44BDVNmAQdUnP6FugabSdFkB6JIl4pvU37nzu4hEnSLjWkcYhonONOLS4OH8kUEuSe\nij57/E3//lxp+w+4rouL4LpK0BFvThRfmCRBUoRIESJJlDgztZVuoji4OH55+8rtoETwxkbEs8qZ\n+3OVnPujxOmkiriEqdQeWmUGCQmRIkiKAD0SJZXxJ21hfQxJnGLGjFGuYOy6kE54m5uEdN+W8H4u\n0wnvtZv0f3Bcf8t4nhETAsy8GKpqR1SMtra2YWOojdZyx8fuIOBYD4YpHdbAGAUnEmH2nXcx+867\nch7PmueT24tfJADWrVtHQ0PDGL1bcVgMpWEixAATI46JEIOZnOwSiTHGGGMKzhoYxhhjjCk4a2AY\nY4wxpuCsgWGMMcaYgitqA0NEVonIARF5S0T+aog0DSKyQ0T2iMj6keQ1xhhjTGkq2l0kIhIAfgbc\nCDQBb4rIc6q6NyPNVOCfgFWq+p6I1Oeb1xhjjDGlq5g9GFcBb6nqYVVNAP8B3JqV5o+BZ1T1PQBV\nbRlBXmOMMcaUqGLOgzEHeD/jdROwIivNIiAkIuuAauCnqvponnkBEJG7gbv9l6dE5MAoy1sLtA6b\nqrRZDKXBYigdEyGOYsdwXhHPbSax8Z5oKwgsA34PiAJviMimkZxAVR8GHj7bgojI1nJf8MdiKA0W\nQ+mYCHFMhBjM5FTMBsYHwLyM13P9fZmagDZV7QK6ROQ14FJ//3B5jTHGGFOiijkG401goYhcICJh\n4HPAc1lpngWuEZGgiFTiXQbZl2deY4wxxpSoovVgqGpKRL4K/BoIAL9U1T0ico9//Oequk9EXgYa\nARf4Z1XdDZArb7HK6jvryywlwGIoDRZD6ZgIcUyEGMwkJKpDrRNpjDHGGDM6NpOnMcYYYwpuUjUw\nhpsdVDx/7x9vFJErxqOcw8kjjgYRafdnSN0hIt8Zj3IORUR+KSItIrJ7iOMlXw95xFDSdQAgIvNE\nZK2I7PVn0v2LHGlKui7yjKEc6qJCRLaIyE4/jr/Okaak68KYQVR1Umx4YzneBuYDYWAnsCQrzc3A\nS4AAVwObx7vco4yjAXh+vMt6hhiuBa4Adg9xvBzqYbgYSroO/DLOBq7wn1cDB8vtdyLPGMqhLgSI\n+c9DwGbg6nKqC9tsy94mUw9GPrOD3go8qp5NwFQRmT3WBR1G2c9yqqqvASfOkKTk6yGPGEqeqjar\n6nb/eSfeHVxzspKVdF3kGUPJ8z/fU/7LkL9lD5Ar6bowJttkamDkmh00+w9RPmnGW75l/LjfjfqS\niCwdm6IVTDnUQz7Kpg5E5Hzgcrz/OWcqm7o4QwxQBnUhIgER2QG0AL9R1bKtC2Ng/GfyNMWxHThX\nVU+JyM3Ar4CF41ymyaZs6kBEYsB/Afeqasd4l2c0homhLOpCVdPAZf4ikGtE5GL1b9s3phxNph6M\nfGYWzSfNeBu2jKra0dfdqqov4q33Ujt2RTxr5VAPZ1QudSAiIbwv5sdV9ZkcSUq+LoaLoVzqoo+q\nngTWAquyDpV8XRiTaTI1MPKZHfQ5YLU/WvtqoF1Vm8e6oMMYNg4RmSUi4j+/Cq+e28a8pKNXDvVw\nRuVQB375/gXYp6o/GSJZSddFPjGUSV3U+T0XiEgUuBHYn5WspOvCmGyT5hKJ5jGzKPAi3kjtt4Bu\n4IvjVd6h5BnHHwJ/LiIpoAf4nKqWzIxqIvIE3sj+WhFpAr6LN6itbOohjxhKug58vwvcAezyr/0D\nfBs4F8qmLvKJoRzqYjbwiIgE8BpAT6nq8+X298mYTDaTpzHGGGMKbjJdIjHGGGPMGLEGhjHGGGMK\nzhoYxhhjjCk4a2AYY4wxpuCsgWGMMcaYgrMGhjFDEJH7/ZUtG/1VOFeM8fs3iMjzY/mexhhTKJNm\nHgxjRkJEVgK34K3UGfdnfgyPc7GMMaZsWA+GMbnNBlpVNQ6gqq2qelRElonIehHZJiK/7lvNUkQu\nFJFXRGSniGwXkQX+jIsPichuEdklIp/10zaIyDoReVpE9ovI4xkzTa7y920HPtNXGBG5zu9F2SEi\nvxWR6rH/SIwxJn820ZYxOfiLZ/0fUAm8AjwJvA6sB25V1eN+g+FTqnqniGwGfqCqa0SkAq/x/vvA\nPXhrStTiTfO+AlgMPAssBY4CG4FvAluBQ8D1eLM1PglUquotIvLf/vk3+mXrVdXUWHwWxhgzGtaD\nYUwO/uJYy4C7geN4X/Z/BlwM/MaflvoBYK7fmzBHVdf4eXtVtRu4BnhCVdOqegyvcXKl/xZbVLVJ\nVV1gB3A+cBHwjqoe8qeyfiyjSBuBn4jI14Cp1rgwxpQ6G4NhzBD85bPXAetEZBfwFWCPqq7MTDfK\nyxXxjOdphvldVNUfiMgLeGtRbBSRT6lq9mJYxhhTMqwHw5gcRGSxiCzM2HUZsA+o8weAIiIhEVmq\nqp1Ak4h82t8fEZFKYAPwWREJiEgdcC2w5Qxvux84X0QW+K9vzyjPAlXdpao/xLvUclGBQjXGmKKw\nBoYxucXwVrfcKyKNwBLgO3grc/5QRHbiXdr4uJ/+DuBrftrXgVnAGqAR2Am8Ctynqh8O9Yaq2ot3\nSeYFf5BnS8bhe/3Boo1AEnipcKEaY0zh2SBPY4wxxhSc9WAYY4wxpuCsgWGMMcaYgrMGhjHGGGMK\nzhoYxhhjjCk4a2AYY4wxpuCsgWGMMcaYgrMGhjHGGGMKzhoYxhhjjCm4/wcoCLU31wx//AAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266a18efe10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(time_history_PGD[:,0], time_history_PGD[:,1], lw=4, label='Prox grad descent')\n",
    "plt.plot(time_history_FISTA[:,0], time_history_FISTA[:,1], lw=4, label='FISTA')\n",
    "plt.plot(time_history_SVRG[:,0], time_history_SVRG[:,1], lw=4, label='SVRG')\n",
    "plt.plot(time_history_SAGA[:,0], time_history_SAGA[:,1], lw=4, label='SAGA')\n",
    "plt.plot(time_history_CD[:,0], time_history_CD[:,1], lw=4, label='CD')\n",
    "plt.plot(time_history_SDCA[:,0], time_history_SDCA[:,1], lw=4, label='SDCA')\n",
    "\n",
    "plt.grid()\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Seconds')\n",
    "plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)\n",
    "plt.suptitle('Algorithms by time elapsed')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Enfin, voici un récapitulatif des performances des algorithmes dans les quatres conditions considérées (ce sont les images des graphiques précédents enregistrés à l'avance).\n",
    "\n",
    "Commençons avec la comparaison en fonction du nombre d'itérations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<table width=\"900\" border=\"1\" cellpadding=\"5\">\n",
    "\n",
    "<tr>\n",
    "\n",
    "<td align=\"center\" valign=\"center\">\n",
    "<img src=\"resultats/step_petit_petit.png\" alt=\"peu de samples et features\" />\n",
    "<br />\n",
    "peu de samples et features\n",
    "</td>\n",
    "\n",
    "<td align=\"center\" valign=\"center\">\n",
    "<img src=\"resultats/step_grand_petit.png\" alt=\"beaucoup de samples, peu de features\" />\n",
    "<br />\n",
    "beaucoup de samples, peu de features\n",
    "</td>\n",
    "\n",
    "<td align=\"center\" valign=\"center\">\n",
    "<img src=\"resultats/step_petit_grand.png\" alt=\"peu de samples, beaucoup de features\" />\n",
    "<br />\n",
    "peu de samples, beaucoup de features\n",
    "</td>\n",
    "\n",
    "<td align=\"center\" valign=\"center\">\n",
    "<img src=\"resultats/step_grand_grand.png\" alt=\"beaucoup de samples, beaucoup de features\" />\n",
    "<br />\n",
    "beaucoup de samples, beaucoup de features\n",
    "</td>\n",
    "\n",
    "</tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Puis la même comparaison mais cette fois en fonction du temps écoulé (en secondes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<table width=\"900\" border=\"1\" cellpadding=\"5\">\n",
    "\n",
    "<tr>\n",
    "\n",
    "<td align=\"center\" valign=\"center\">\n",
    "<img src=\"resultats/time_petit_petit.png\" alt=\"peu de samples et features\" />\n",
    "<br />\n",
    "peu de samples et features\n",
    "</td>\n",
    "\n",
    "<td align=\"center\" valign=\"center\">\n",
    "<img src=\"resultats/time_grand_petit.png\" alt=\"beaucoup de samples, peu de features\" />\n",
    "<br />\n",
    "beaucoup de samples, peu de features\n",
    "</td>\n",
    "\n",
    "<td align=\"center\" valign=\"center\">\n",
    "<img src=\"resultats/time_petit_grand.png\" alt=\"peu de samples, beaucoup de features\" />\n",
    "<br />\n",
    "peu de samples, beaucoup de features\n",
    "</td>\n",
    "\n",
    "<td align=\"center\" valign=\"center\">\n",
    "<img src=\"resultats/time_grand_grand.png\" alt=\"beaucoup de samples, beaucoup de features\" />\n",
    "<br />\n",
    "beaucoup de samples, beaucoup de features\n",
    "</td>\n",
    "\n",
    "</tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Quelques remarques :\n",
    "- SDCA semble être très performant lorsque le nombre de paramètres est très grand comparativement au nombre d'exemples. C'est cohérent avec l'algorithme : il optimise non pas sur l'espace des paramètres mais sur l'espace des individus.\n",
    "- A l'inverse c'est le pire dans le cas contraire.\n",
    "\n",
    "Concentrons-nous sur la comparaison par nombre d'itérations. C'est globalement un algorithme qui nécéssite plus d'itérations que les autres (mis à part le 3ème cas). En effet, l'algorithme favorise un nombre important d'itérations, chacunes rapides à calculer, à l'inverse de SVRG ou de FISTA qui préférent affecter plus de ressources à chaque pas afin d'espérer en faire moins in fine.\n",
    "\n",
    "A propos du temps de calcul, SDCA est globalement mauvais comparativement aux autres lorsque le problème à résoudre est simple, et semble être de plus en intéressant lorsque le problème se complexifie. On ne fera pas d'autres commentaires sur ces mesures car elles sont trop dépendantes des optimisations d'implémentations possibles.\n",
    "\n",
    "Enfin, et pour conclure, comparons SDCA à sa version conjuguée, le Coordinate Descent. Comme on pouvait s'y attendre, CD est meilleur dans le cas où la dimension du paramètre est petite, et inversement. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}