NB4_PredictiveModelUsingSVM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictive model using Support Vector Machine (SVM)\n",
    "\n",
    "Support vector machines (SVMs) learning algorithm will be used to build the predictive model.  SVMs are one of the most popular classification algorithms, and have an elegant way of transforming nonlinear data so that one can use a linear algorithm to fit a linear model to the data (Cortes and Vapnik 1995)\n",
    "\n",
    "Kernelized support vector machines are powerful models and perform well on a variety of datasets. \n",
    "1. SVMs allow for complex decision boundaries, even if the data has only a few features. \n",
    "2. They work well on low-dimensional and high-dimensional data (i.e., few and many features), but don’t scale very well with the number of samples.\n",
    "> **Running an SVM on data with up to 10,000 samples might work well, but working with datasets of size 100,000 or more can become challenging in terms of runtime and memory usage.**\n",
    "\n",
    "3. SVMs requires careful preprocessing of the data and tuning of the parameters. This is why, these days, most people instead use tree-based models such as random forests or gradient boosting (which require little or no preprocessing) in many applications. \n",
    "4.  SVM models are hard to inspect; it can be difficult to understand why a particular prediction was made, and it might be tricky to explain the model to a nonexpert.\n",
    "\n",
    "### Important Parameters\n",
    "The important parameters in kernel SVMs are the\n",
    "* Regularization parameter C, \n",
    "* The choice of the kernel,(linear, radial basis function(RBF) or polynomial)\n",
    "* Kernel-specific parameters. \n",
    "\n",
    "gamma and C both control the complexity of the model, with large values in either resulting in a more complex model. Therefore, good settings for the two parameters are usually strongly correlated, and C and gamma should be adjusted together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "#Load libraries for data processing\n",
    "import pandas as pd #data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import numpy as np\n",
    "from scipy.stats import norm\n",
    "\n",
    "## Supervised learning.\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn import metrics, preprocessing\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# visualization\n",
    "import seaborn as sns \n",
    "plt.style.use('fivethirtyeight')\n",
    "sns.set_style(\"white\")\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (8,4) \n",
    "#plt.rcParams['axes.titlesize'] = 'large'\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('data/clean-data.csv', index_col=False)\n",
    "data.drop('Unnamed: 0',axis=1, inplace=True)\n",
    "#data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Applications/Programming_Tools/anaconda/envs/jean-env/lib/python3.5/site-packages/sklearn/utils/validation.py:429: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.\n",
      "  warnings.warn(msg, _DataConversionWarning)\n"
     ]
    }
   ],
   "source": [
    "#Assign predictors to a variable of ndarray (matrix) type\n",
    "array = data.values\n",
    "X = array[:,1:31] # features\n",
    "y = array[:,0]\n",
    "\n",
    "#transform the class labels from their original string representation (M and B) into integers\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)\n",
    "\n",
    "# Normalize the  data (center around 0 and scale to remove the variance).\n",
    "scaler =StandardScaler()\n",
    "Xs = scaler.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification with cross-validation\n",
    "\n",
    "As discussed in notebook [NB3](https://github.com/ShiroJean/Breast-cancer-risk-prediction/blob/master/NB3_DataPreprocesing.ipynb) splitting the data into test and training sets is crucial to avoid overfitting. This allows generalization of real, previously-unseen data. Cross-validation extends this idea further. Instead of having a single train/test split, we specify **so-called folds** so that the data is divided into similarly-sized folds. \n",
    "\n",
    "* Training occurs by taking all folds except one – referred to as the holdout sample. \n",
    "* On the completion of the training, you test the performance of your fitted model using the holdout sample. \n",
    "\n",
    "* The holdout sample is then thrown back with the rest of the other folds, and a different fold is pulled out as the new holdout sample. \n",
    "\n",
    "* Training is repeated again with the remaining folds and we measure performance using the holdout sample. This process is repeated until each fold has had a chance to be a test or holdout sample. \n",
    "\n",
    "* The expected performance of the classifier, called cross-validation error, is then simply an average of error rates computed on each holdout sample. \n",
    "\n",
    "This process is demonstrated by first performing a standard train/test split, and then computing cross-validation error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-5-7577dbe278d1>, line 10)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-5-7577dbe278d1>\"\u001b[0;36m, line \u001b[0;32m10\u001b[0m\n\u001b[0;31m    print '\\nThe classifier accuracy score is {:03.2f}\\n'.format(classifier_score)\u001b[0m\n\u001b[0m                                                        ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# 5. Divide records in training and testing sets.\n",
    "X_train, X_test, y_train, y_test = train_test_split(Xs, y, test_size=0.3, random_state=2, stratify=y)\n",
    "\n",
    "# 6. Create an SVM classifier and train it on 70% of the data set.\n",
    "clf = SVC(probability=True)\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    " #7. Analyze accuracy of predictions on 30% of the holdout test sample.\n",
    "classifier_score = clf.score(X_test, y_test)\n",
    "print '\\nThe classifier accuracy score is {:03.2f}\\n'.format(classifier_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a better measure of prediction accuracy (which you can use as a proxy for “goodness of fit” of the model), you can successively split the data into folds that you will use for training and testing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-14-d457d49644cb>, line 4)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-14-d457d49644cb>\"\u001b[0;36m, line \u001b[0;32m4\u001b[0m\n\u001b[0;31m    print '\\nThe {}-fold cross-validation accuracy score for this classifier is {:.2f}\\n'.format(n_folds, cv_error)\u001b[0m\n\u001b[0m                                                                                        ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# Get average of 3-fold cross-validation score using an SVC estimator.\n",
    "n_folds = 3\n",
    "cv_error = np.average(cross_val_score(SVC(), Xs, y, cv=n_folds))\n",
    "print '\\nThe {}-fold cross-validation accuracy score for this classifier is {:.2f}\\n'.format(n_folds, cv_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above evaluations were based on using the entire set of features. You will now employ the correlation-based feature selection strategy to assess the effect of using 3 features which have the best correlation with the class labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The 3-fold cross-validation accuracy score for this classifier is 0.97\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import SelectKBest, f_regression\n",
    "clf2 = make_pipeline(SelectKBest(f_regression, k=3),SVC(probability=True))\n",
    "\n",
    "scores = cross_val_score(clf2, Xs, y, cv=3)\n",
    "\n",
    "# Get average of 3-fold cross-validation score using an SVC estimator.\n",
    "n_folds = 3\n",
    "cv_error = np.average(cross_val_score(SVC(), Xs, y, cv=n_folds))\n",
    "print '\\nThe {}-fold cross-validation accuracy score for this classifier is {:.2f}\\n'.format(n_folds, cv_error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.93157895  0.95263158  0.94179894]\n",
      "Average score and uncertainty: (94.20 +- 0.496)%\n"
     ]
    }
   ],
   "source": [
    "print scores\n",
    "avg = (100*np.mean(scores), 100*np.std(scores)/np.sqrt(scores.shape[0]))\n",
    "print \"Average score and uncertainty: (%.2f +- %.3f)%%\"%avg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above results, you can see that only a fraction of the features are required to build a model that performs similarly to models based on using the entire set of features.\n",
    "Feature selection is an important part of the model-building process that you must always pay particular attention to. The details are beyond the scope of this notebook. In the rest of the analysis, you will continue using the entire set of features.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Model Accuracy: Receiver Operating Characteristic (ROC) curve\n",
    "\n",
    "In statistical modeling and machine learning, a commonly-reported performance measure of model accuracy for binary classification problems is Area Under the Curve (AUC).\n",
    "\n",
    "To understand what information the ROC curve conveys, consider the so-called confusion matrix that essentially is a two-dimensional table where the classifier model is on one axis (vertical), and ground truth is on the other (horizontal) axis, as shown below. Either of these axes can take two values (as depicted)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    " Model says \"+\" |Model says  \"-\" \n",
    "--- | --- | ---\n",
    "`True positive` | `False negative` | ** Actual: \"+\" **\n",
    " `False positive`  | `True negative` |  Actual: \"-\"\n",
    " In an ROC curve, you plot “True Positive Rate” on the Y-axis and “False Positive Rate” on the X-axis, where the values “true positive”, “false negative”, “false positive”, and “true negative” are events (or their probabilities) as described above. The rates are defined according to the following:\n",
    "> * True positive rate (or sensitivity)}: tpr = tp / (tp + fn)\n",
    "> * False positive rate:       fpr = fp / (fp + tn)\n",
    "> * True negative rate (or specificity): tnr = tn / (fp + tn)\n",
    "\n",
    "In all definitions, the denominator is a row margin in the above confusion matrix. Thus,one can  express\n",
    "* the true positive rate (tpr) as the probability that the model says \"+\" when the real value is indeed \"+\" (i.e., a conditional probability). However, this does not tell you how likely you are to be correct when calling \"+\" (i.e., the probability of a true positive, conditioned on the test result being \"+\").\n",
    "          "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The confusion matrix helps visualize the performance of the algorithm.\n",
    "y_pred = clf.fit(X_train, y_train).predict(X_test)\n",
    "cm = metrics.confusion_matrix(y_test, y_pred)\n",
    "#print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAE8CAYAAABU7q18AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEsVJREFUeJzt3Xm41XWdwPH35ZLACIg7q6O5fMx61DZFi6KFnDSFh6ym\ntFzLVpupLFudrGwsbcqScSW1mRYzlMzGTG0xyx7NUtw+oSNWBCO2gJJC4J0/fge7EVwO3fO755yv\n79fz+HCWe36/D1ee9/Pbzjk9fX19SFKJhrV7AEmqi4GTVCwDJ6lYBk5SsQycpGIZOEnFGt7uAfQX\nETEMmAPsA6wCjs/Me9s7lTpNROwPnJ6Z09s9S6dzC66zzAJGZuYBwMnAmW2eRx0mIt4LXACMbPcs\n3cDAdZbnA1cDZOZNwHPaO4460H3A7HYP0S0MXGcZCyzvd39tRHgYQU/IzG8Af273HN3CwHWWFcCY\nfveHZeaadg0jdTsD11luBA4GiIipwIL2jiN1N3d/OsvlwIyI+DHQAxzT5nmkrtbjp4lIKpW7qJKK\nZeAkFcvASSqWgZNULAMnqVgGTlKxDJykYnXEhb4RMQJ4LrAEWNvmcSR1l15gAnBzZq7q/0RHBI4q\nbje0ewhJXW0a8KP+D3RK4JYAnPq2d7DtuHHtnkUdaM/pz2/3COpQS5cu5YjXHwWNjvTXKYFbC7Dt\nuHHssM227Z5FHWjypEntHkGd728Ob3mSQVKxDJykYhk4ScUycJKKZeAkFcvASSqWgZNULAMnqVgG\nTlKxDJykYhk4ScUycJKKZeAkFcvASSqWgZNULAMnqVgGTlKxDJykYhk4ScUycJKKZeAkFcvASSqW\ngZNULAMnqVgGTlKxDJykYhk4ScUycJKKZeAkFcvASSqWgZNULAMnqVgGTlKxDJykYhk4ScUycJKK\nZeAkFcvASSqWgZNULAMnqVgGTlKxDJykYhk4ScUycJKKZeAkFcvASSqWgZNULAMnqVgGTlKxDJyk\nYhk4ScUycJKKZeAkFcvASSqWgZNULAMnqVgGTlKxDJykYhk4ScUycJKKZeAkFWt4uwd4Mlv4wCL+\n66or+ehb38GSh5Zx9le+TE9PD1PGj+f42YczbNgwfn73XXz9mu/QRx9PnTyF42cfTk9PT7tHVxvd\ndtvtnPGZz/Cliy9q9ygdr7bARcQwYA6wD7AKOD4z761rfd1m/vXX8YOf3cLILbYA4OL5V/Dalx/M\n03fbnfMuu5Sb77yDvXffgy9965v821veztjRo5l//XWsWLmSrUaPbvP0apfzL5zLN795JaNGjWr3\nKF2hzl3UWcDIzDwAOBk4s8Z1dZ0dt9uOk44+9on7//ub37DXrrsB8Mw9n8aCX/6SXHQ/O42fwCVX\nzufDXziLrcaMMW5PcjtNmcLnP/fZdo/RNeoM3POBqwEy8ybgOTWuq+tM3Xsfenv7//r7ntj1HDli\nBH967FEeXrmSO+67lyMOOZQPvPEErvrhD/jtsgfbM7A6wkEvm8Hwp3hkqVl1Bm4ssLzf/bUR4f+Z\njeh/XO2xVav4h1GjGL3lluw2ZSe2HjuWUSNG8LRdd2XR4sVtnFLqLnUGbgUwpv+6MnNNjevrartM\nmsyd9y4E4Of33M3TdnkqT500mV8tXcKKRx5h7dq1LHxgEZN3HN/mSaXuUecW1Y3AocClETEVWFDj\nurreGw6byTmXfo013/4Wk3fYkan77EvvsGEccfAr+Pj55wBw4D77stOECW2eVOoePX19fbUsuN9Z\n1L2BHuCYzLxnIz+7M3D/2R/8MDtss20t86i77f3yl7Z7BHWo3yxezEtmHASwS2Yu6v9cbVtwmfk4\n8Oa6li9Jm+I7GSQVy8BJKpaBk1QsAyepWAZOUrEMnKRiGThJxTJwkopl4CQVy8BJKpaBk1QsAyep\nWAZOUrEMnKRiGThJxTJwkopl4CQVy8BJKpaBk1QsAyepWAZOUrEMnKRiGThJxTJwkopl4CQVy8BJ\nKpaBk1SszQ5cRDyljkEkqdWGb+oHIuL5wHTgU8BNwJ4RcUxmfq3m2SRpUJrZgvs0VdhmAUuBvYB3\n1zmUJLVCM4HrzcxrgRnAFZm5COitdSpJaoGmAhcR+wGHANdExDMAj8NJ6njNBO4TwJeBCxtbb1cC\nH6pzKElqhU2eZMjMecC8fg/tlplr6xtJklqjmbOo44ELgd2BacAlEXF0Zi6pezhJGoxmdlHnAFcA\njwK/B34BXFDnUJLUCs0EbufMPB94PDP/nJnvA3aqeS5JGrRmAvd4RDzxcxExpsnXSVJbNROqecB/\nA1tFxAnA9cCltU4lSS2wycBl5mnA/wA3U13sex5was1zSdKgbfIsKkBmXgJcUvMsktRSzVwmsgDo\nW//xzNy7lokkqUWa2YJ7e7/bW1C96f639YwjSa3TzDsZftD/fkRcC/yY6i1cktSx/p7LPbYFJrZ6\nEElqtc09BtdDdZHvuXUOJUmtsLnH4PqAZZl5d03zSFLLbDRwEfGsxs2H13tqVEQ8KzNvrW8sSRq8\ngbbgvjHAc33AU1s8iyS11EYDl5m7DOUgktRqzZxk2A54PTCa6iRDL9WHXh5R82ySNCjNnGS4lOqz\n4J4OfJfq/ag31DmUJLVCM9fB/WNmHgJ8G/gC8Dxgt1qnkqQWaCZwSxt/LgSekZmLafJN+pLUTs2E\n6sGIOAn4CfDRiFgBbFXvWJI0eM1swZ0ArMrMHwG3UH0W3PtqnUqSWmCgC33/heq7UB8EzgJofB+D\ncZPUFQbagnsx8OuIOLfxbfaS1FU2GrjMPIzq0pDFwLci4ocR8eqI6B2y6SRpEAY8BpeZizPzVGAX\n4HTgtcB9EfGRoRhOkgajqc+Dy8y+zLwKOAm4Enh/rVNJUgs081atUcCrgOOo3mB/Pr7RXlIXGOgs\n6v7AscCrgVuBzwOXZ+baIZpNkgZloC24a4CLgQOH6gMuY9pUJk+aNBSrUpdZe/st7R5BHerxB5dt\n9LmBAjcxM1e2fhxJGhoDXSZi3CR1tb/nW7UkqSsYOEnFGugs6lkDvTAzT2z9OJLUOgOdZPjdkE0h\nSTUY6EtnPrqx5yJiy3rGkaTWaeadDDOpPgOu/5fObAOMqXc0SRqcZk4ynAGcBvwKeCtwNXBOnUNJ\nUis0E7iVmfk14CbgMeAtwEtqnUqSWqCZwK2KiBHAvcC+mfk4MKLesSRp8Jr50pn5wFXA0cCPI2Ia\nnmGV1AU2uQWXmacBx2bmb4CZwA+Bw+seTJIGq5mzqM9q/Lld46EbgMnAgzXOJUmD1swu6jf63d4C\nGA/8DNivlokkqUU2GbjM3KX//YiYSvXpvpLU0Tb7zfaZeRPw7BpmkaSWavoYXEMP8BxgVG0TSVKL\nbO4xuD6qkwtvqWccSWqdZgI3rXGJyBMiYq+a5pGklhno8+C2ady8KiKmU+2e9lGdSZ0P7F77dJI0\nCANtwX0FmNG43f+dC2uBebVNJEktMtDnwR0EEBFzM/PYoRtJklqjmctEPhIRcwCickVE7FjzXJI0\naM0E7iLgnsbtB4DvA1+saR5JaplmArddZp4FkJmPZeZngQn1jiVJg9dM4IZHxMR1dxq7pz31jSRJ\nrdHMdXCfAX4REVc37r8EOKm+kSSpNZr5PLi5VJeL/By4GTgXeGfNc0nSoDWzBQfVF86MpPrSmdHA\ngF8KLUmdYMDARUQA/wocCSyiepP9zpm5vP7RJGlwNrqLGhHfpvp48tXA9Mx8BvCwcZPULQY6Brcv\ncCtwB7Cw8Vhf7RNJUosMFLidgLnAa4ElEfF1/Bw4SV1ko4HLzDWZ+fXMfBHVh1wuAUZFxMKIePOQ\nTShJf6emPrI8M+/KzBOBicCngTfVOpUktUCzl4kAkJl/As5r/CdJHW2zv3RGkrqFgZNULAMnqVgG\nTlKxDJykYhk4ScUycJKKZeAkFcvASSqWgZNULAMnqVgGTlKxDJykYhk4ScUycJKKZeAkFcvASSqW\ngZNULAMnqVgGTlKxDJykYhk4ScUycJKKZeAkFcvASSqWgZNUrOHtHkB/Me+K+Vw+/5sArF69mrvv\nSX70vWsZO3ZsmydTO73y5FMYPWokAJN22J5Vq1fz0B+XA7B42UPss/uunPnOt7ZzxI5Va+AiYn/g\n9MycXud6SjF71kxmz5oJwKkfP43Zs2Yatye5VatX00cfF5/y/r95bvkjKzn6Y//OyW94XRsm6w61\n7aJGxHuBC4CRda2jVAvuvJOF993Ha151eLtHUZvd88CveWzVao7/xKc55mOnc9vCe5947guXXc6R\nB72U7bce18YJO1udx+DuA2bXuPxinXf+hbztLW9u9xjqAKNGbMExr/gnzv/AezjluKN47+fPZc3a\ntfxu+QpuuuMuZk2f1u4RO1ptgcvMbwB/rmv5pVqxYgX3L1rE1P2e2+5R1AF2njCeQ6cdSE9PDztP\nHM+4MaNZ9oc/cs1Pb+aQ502ld5jnCQfib6fD3PKzW5m6//7tHkMdYt73buBTX/oqAA/+/g888uij\nbL/1OH6y4C5esO/ebZ6u83kWtcPcv2gRUyZPavcY6hCzX/wCPjjnAo485RNADx8/4TiG9/Zy/5Il\nTN5h+3aP1/EMXIc57pij2z2COsgWw4fz6RP/9njslWec1oZpuk+tgcvMRcDUOtchSRvjMThJxTJw\nkopl4CQVy8BJKpaBk1QsAyepWAZOUrEMnKRiGThJxTJwkopl4CQVy8BJKpaBk1QsAyepWAZOUrEM\nnKRiGThJxTJwkopl4CQVy8BJKpaBk1QsAyepWAZOUrEMnKRiGThJxTJwkopl4CQVy8BJKpaBk1Qs\nAyepWAZOUrEMnKRiGThJxTJwkopl4CQVy8BJKpaBk1QsAyepWAZOUrEMnKRiGThJxTJwkopl4CQV\ny8BJKpaBk1QsAyepWAZOUrEMnKRiGThJxTJwkopl4CQVy8BJKpaBk1QsAyepWAZOUrEMnKRiGThJ\nxTJwkopl4CQVy8BJKpaBk1QsAyepWMPbPUBDL8DSpf/X7jnUofoeXNbuEdShlv7u9+tu9q7/XKcE\nbgLAkUcf2+45JHWvCcB9/R/olMDdDEwDlgBr2zyLpO7SSxW3m9d/oqevr2/ox5GkIeBJBknF6pRd\nVA2RiNiZ6jjFgn4P9wCfy8y5g1z2t4DLMvOiiPgFMD0z/7iRn90KuDwzX7yZ6zgceHtmTl/v8e8B\n12TmJ9d7/N3ACzPzsAGWeRFwR2aesTmzqPMZuCenRzNz33V3ImIScEdE3JKZt7diBf2XvxFbA/u1\nYl0NZwOnAZ9c7/E3Aie2cD3qIgZOZObiiFgI7BERzwKOA7YElmfmiyLiOOCtVIc0fke1BXVPREwE\nLgYmAg8AO6xbZkT0Adtn5kMR8X7gKGANsBA4GvgiMKqxpfdsYA/gc8C2VAeNz1q3RRkRpwJHNNa9\ncCN/jSuAz0XEtMy8ofG6F1JtnX43IoYB/wFMBcY0Hj8+M2/sv5D+c2/g73Eo8CFgC+BPwHsy8ycR\nsSdwITCysdwLMnNOs79/1cdjcCIiDgB2A37aeOjpVLuXL2pE4ihgWmY+E/gUMK/xc2cDN2Xm06m2\nkvbcwLIPowraAZn5DOB+4O3AMfxlS7IHuAw4OTOfDbwQeE9ETI2ImcArgX2BA4GtNvR3yMw1wHlU\ncV7nTcCczOwD9qcK8QGZuRdVmE/ejN/R7lRbiAc3fg9vAuZFxJbAScCVjdkPBl7QCKrazC24J6d1\nW05Q/Rt4CDgiM38dEQC3Z+aKxvOHUMXvx43nALaJiG2AlwLvAcjMeyPi+g2s66XA1zPzD42fexc8\ncSxwnT2AXYG5/dYxCngmsBcwLzMfbrxuLhvf5TwPuCsixgBPAQ6i2vKksaX1IeCEiNgVmA48PMDv\naH0zqC5FuK7fjI9T/W4uBy6JiP2Aa4ETM/PxzVi2amLgnpz+6hjcBjzS73Yv8KXMfB9AY8tkIvAH\noI9q62udNRtY1prGz9F4/Thg3Ho/0wv8cb3jgjsCy6m2GDe1DgAyc0lEfBf4Z6pd7Msyc3ljeYdQ\n7QKfCcwH7gGO3Miiehqv2WK9Ga/LzNf0m3EK8NvMvK2xhTcDeAlwSkQcmJl/ddGphp6b0dqUa4DX\nRsSExv03A9c1bl9NtatGROwEvGgDr78WmB0RYxv3/w14F1WoeiOiB0jgsYg4srGsKcAdVMfmrgZe\nFRHjGnF9/SbmnUN1vO4oql3odWZQ7Ub+J9UFobPYwFt7gGXAcxq3Z/d7/HrgZY3jbUTEwcDtwMiI\n+DLwmsz8KtUW4wpgyibm1BAwcBpQZn4HOJ3qQP3twOuA2Y3jWm8D9oqIu6kOsv9iA6//NtUJhRsj\nYgEwHvgg1btWbgXupjroPxM4vrGOa4APZ+aNjdfPBW6hOka4fBPzfp/qRMWKzOx/Kcw5wAsby/8J\n1aUyu2zgWNmJwNkRcSvVLvKSxnLvpIr5VyPiNuBjwGGZubJx+4jG4z+l2mX9wUBzamj4TgZJxXIL\nTlKxDJykYhk4ScUycJKKZeAkFcvASSqWgZNULAMnqVj/D1R9W0mE6/VRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111d4cf50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.94      0.99      0.96       107\n",
      "          1       0.98      0.89      0.93        64\n",
      "\n",
      "avg / total       0.95      0.95      0.95       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from IPython.display import Image, display\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "ax.matshow(cm, cmap=plt.cm.Reds, alpha=0.3)\n",
    "for i in range(cm.shape[0]):\n",
    "     for j in range(cm.shape[1]):\n",
    "         ax.text(x=j, y=i,\n",
    "                s=cm[i, j], \n",
    "                va='center', ha='center')\n",
    "plt.xlabel('Predicted Values', )\n",
    "plt.ylabel('Actual Values')\n",
    "plt.show()\n",
    "print(classification_report(y_test, y_pred ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Observation \n",
    "There are two possible predicted classes: \"1\" and \"0\". Malignant = 1 (indicates prescence of cancer cells) and Benign\n",
    "= 0 (indicates abscence).\n",
    "\n",
    "* The classifier made a total of 174 predictions (i.e 174 patients were being tested for the presence breast cancer).\n",
    "* Out of those 174 cases, the classifier predicted \"yes\" 58 times, and \"no\" 113 times.\n",
    "* In reality, 64 patients in the sample have the disease, and 107 patients do not.\n",
    "\n",
    "#### Rates as computed from the confusion matrix\n",
    "1. **Accuracy**: Overall, how often is the classifier correct?\n",
    "    * (TP+TN)/total = (57+106)/171 = 0.95\n",
    "\n",
    "2. **Misclassification Rate**: Overall, how often is it wrong?\n",
    "    * (FP+FN)/total = (1+7)/171 = 0.05 equivalent to 1 minus Accuracy also known as ***\"Error Rate\"***\n",
    "\n",
    "3. **True Positive Rate:** When it's actually yes, how often does it predict 1?\n",
    "   * TP/actual yes = 57/64 = 0.89 also known as \"Sensitivity\" or ***\"Recall\"***\n",
    "\n",
    "4. **False Positive Rate**: When it's actually 0, how often does it predict 1?\n",
    "   * FP/actual no = 1/107 = 0.01\n",
    "\n",
    "5. **Specificity**: When it's actually 0, how often does it predict 0? also know as **true positive rate**\n",
    "   * TN/actual no = 106/107 = 0.99 equivalent to 1 minus False Positive Rate\n",
    "\n",
    "6. **Precision**: When it predicts 1, how often is it correct?\n",
    "   * TP/predicted yes = 57/58 = 0.98\n",
    "\n",
    "7. **Prevalence**: How often does the yes condition actually occur in our sample?\n",
    "   * actual yes/total = 64/171 = 0.34\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAHwCAYAAAB36Rx0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8XXV97//XPknIQBJCCElAhkCAT5FRgkyKDIoUQlqr\n10vB9larIsVZW0sHh9t67SDWq/1VWqfaQYGWliqEQQREZkEtIMLHu5kEIQkQSJgSMpzfH2vtsHNy\nhp2Ts8+eXs/HI4+ctdYePnudffZ7f7/ru76r0t/fjyRJaq2+VhcgSZIMZEmS2oKBLElSGzCQJUlq\nAwayJEltwECWJKkNTGx1AWqdiHgI2HPA6ueAO4E/yMxbxqGGTwGnZebhzX6udhQRFeBdwL9k5pqI\neDtwXmbOaUEt/cCSzLxsvJ97MBFxMLBTZl43yvsfD1wHzMjM50a47QLgkMz8drn8EMXv4f8bzXN3\ngq3ZPxoftpD1x8Au5b9dgROBNcBlETFjHJ7/PODkcXiedvU64Mu8/OX4IuCVrSunrXwbOGAb7n8z\nxfv6+QZu+4/AsXXLrwa+vg3PLW01W8h6NjOX1S0/XrbSHgFOAL7TzCcvv5n38rfzSv1CZr4IvNii\nWtpNZeSbDC0zXwKWjXjDQZ4rM5/YlueWRsNA1mDWlv9vqK2IiN8H3g/sBPw38PuZeWu5rQ/4E+Dd\nwGzgh8B7M/Pecvv/Av4U2A24D/h4Zi4tt30KOI2iRfIQ8LnM/GLd814KPJqZvxcR84EvAqdQhPhS\n4KOZuarscnwQ+DjwEeDWzDx14AuLiMOAz5bP9zzwz8CfZua6sgvvv4APAX8BTAUuBj6QmS+U99+v\nrOF1wBMULdqPZ+ba8v4XU7S2zgIuyMyzI+LDwO8BC8q6LyuXd6boMgR4NiLeUf58XmbOqXtNbwU+\nXd7/DuCdmZllPYcAfwcsArJ8PR/IzAUDX3t5+4OBvwGOAp4GvpSZf1F3k8Mj4hPAwUAVOCczf1De\nd1/gc+Vrnwr8P+DcWhd32eX96fK1LwMOA94A/G/gEKAfuBU4OzN/Plw9EfF9isMpfxsR/yMzj9/a\n3z/w19R1yUbEWcDHgN3L234mM/85Ir4BHAccVz7Xgvou65He34Ps40Hf7xGxHfAToJqZv17e9g8o\neqkOysxHI+JtwB8CQfF3eB1wVmYuL99fF1K8P88DZgHfAP4v8DXgCOBu4LczMxt5Pw+oe8j9O9jr\n1Nizy1qbiYg5FH/gy4DaB/F7KML4bOBVwOXAteWHIMAngQ9Q/OG/CngcWBoREyLiZOALFB+UBwH/\nAFwcEUfXP29m9gMXAP+zrpZZwBuBb5Wr/rP8/2hgCbCQ4gOq3qnAkcAfDPLa9i1f070UH17vBn4b\n+EzdzaaX930L8GsUXfhfKu8/BbiKIoheVd73V8vXV7MTsA9FGH0uIs4EPkUREvsCbwfeRBFaj5TP\nQ/laLhpYc+kTZa2vBuZQBA0RsQPw3bKewyjC8s+GeIza7/Za4LHy9Z8FnBsRv1t3s98r6z0IeAD4\nVkRUymPdlwLPUuzfQyk+/P+xDJqaMyh6Vn6HIvi+A/w7RTf8iRSB9tkG6nkz8ChFWL25fOxR//4j\n4lXA+cC5wH4Uv7NvlO+JDwK3lNtfPciuG/L9PfCGw73fyxb77wKnRcSS8svdnwHvL8P4GIovc58t\na3xT+Xx/XPcUO1GMOTi53F/vBb4HfL583ZMpwrdmyPfzIBrZv2oiW8j6XET8ZflzHzABuAF4Q2Y+\nW67/Y+CPMvOKcvkz5bfv90bExyg+xP88M/8TICLeS/GBtGN5389mZi1s7o+IRRQB9dYBtXwL+FhE\nvCIzfwn8BsUXgxsj4gSKVtvx5QcbZWvilxFxAC8fJ/xCZv6/IV7reyhaRu8vvwDcV7Ze/6VsqVO+\n/rMz8+byOT4C/FtEfIDiQ+2lzHx/eduMiLOBG8r9UPOXmXl/ef/dgLfXDZR6OCKuBw7MzA0RsbJc\nvyIzX4yIwer+P5l5Q/l4X6IIFYDTKVqd7yn3yb3lvvjNIV7/6cA64F3l7X8WEeeU62r+KjMvL5/r\nryneCztT7N+vAV/LzJXl9vPK55pH8eUC4CuZ+bNy+74UPSl/W257MCL+BThnpHoyc2VEbKA4pLJy\nNL//8j1as6DcV49k5sPA+RFRBZ4oW9gvAS8M7Kouv4gM9/5+csA+Hvb9npm3RcT/pQjQ5cDlmfmv\n5W3XAO/OzH8plx+OiG8DB9Y9/kTgY5l5D3BPRHwe+K/M/K+ytn+lCOya4d7P9a9z2P1bPp+azEDW\nXwD/SvHN+hyK1sinan+AETEd2AP4akR8ue5+kym61OZQfGDfXtuQmc8AHy3vfwBwZETUf8ufBPx8\nYCGZeVdE3EMR1P+X4gP7wszsLx9nGrBykND6FeBH5c8PDPNaX0nRlV1/RZUby3r2LZc3ULSWan5Y\nbg+KAUYLI6L+mHeF4ovMvnXrNtWQmddFxGER8edlnQeU///zMHUOVP8FY3VZDxQfoHfWPkBLtzB0\nIL8SuKv+9pn5zQG3ub/u52fK/6dm5oqI+DvgzIg4nKIFd1i5vb6lWP/a/19EPF9+WTmQYh8eStHC\nbLSemm39/V8J3AbcGhH3UXTHfqN8rw5n2Pf3EHWO9H7/OMWXu4MoWsG1x/1xRDwbER+n2Df7U+y3\nGwc8R/1rfHGQ5cl1y8O9nwfWPdz+NZDHgYGsJzOzWv783ojYGbg0Ig7OzEd4+T3yOxTHv+q9CNQ+\nTIcagDMR+COK7s566wa5LRSt5P9ZtqReT3E8rfY4DwMnDXKf5RRdebWahrJmkHW1wza1UNlY/mPA\n+g1lDbcA72BLv6ToMtyshigGyJ1P0RV5JfB/KLqEt8ZLA5Zr+3odW3fY6SVGHii1YZB1lfKL2c0U\nX8IuoTgO/hwvHwOvqX/tBwE3UXSpXk/Rwj6Sl1vIjdRTs02//7L34bUU3bGLKbpk3xcRSzLz6mGe\nd6T392B1jvR+3xWYSxGMr6Y4BEREvIFiv15EcWjlC8CZFF+86q0fsLyRoQ33fh5Y93D7V+PAY8ga\n6ByKP/jzYVNrYBnwisys1v5RHE87uRzwsYKXW0tExNSIWBYRR1Acr91zwH3PKP8N5lsUxxPfCfw8\nM+8s199L8UH2bN3jrKPo+pvb4Gu7Fziq7IasOaZ8nFrLcBJFy6XmSIogz/L++1IMMqvVMIfimF/9\ncdR6H6XowjwnM79GcY73vrz8Ab8t1z/9KXDQgGO4gx0Drfk5cGBE1FrYRMQnIuLfGniukylaVcdm\n5mfKLvjafh8qrM4C/jsz35yZX8jM6ym6jmu3H6me+n2zTb//cszCpzLz5sz8k8w8mKJVXTs+Pejv\noYH390DDvt/L995XKcYi/AXwDxExs7zvB4F/y8zfyczzsxg0uQ/bNtp8uPfzwLq39e9L28gWsjaT\nmU9GxLkUXdS/nsVECX8NfCIiHqcY5ftbFMdjjyvv9nngTyPiQYru1Y9TdK3+d3nfC8puwqspvoF/\nkuKb/2DP/3BE3Fo+xl/WbbqaotvswihGfG+gGF08g2J09u4NvLy/oxic88Wy+3VvioFQ/5SZz9R1\n1X05In4P2IFiNOtXM/P58vjcJ4B/KrugZ1K0+h4oj0MO9py/BE6IiFdSfLB+mKI78u5ye637e1FE\n/GiQ+w/nAooW9/kR8VmKltQHgKeGuP03KUY8f6k8/ruwrOfDDTzXLym+dJwexQjowygOK8DmXaQD\n73N6RBxb/vzrFIPTavWNVM9zwP4RMZdt//0/D/xxRDxFMdBsP4rfw1frnmufuvEL9YZ7fw800vv9\nbIoR8a+kGKV/Znmfs8t9dHwUZwI8R9ErdQpFV/u2GOr9XH+bkfavxoEtZA3m6xTdk1+IiGkUXWfn\nUXxw3EPRqnhLbaBIue1rFB9uP6Y45rY4M1/KzEsoRmh/BPgZxYftWZk5XKvsmxSjQ2ujq8nMjRSj\nRFdSdJN+n6LlfmpmDtbNuoXMfIxiVPQiipbqV4B/ohipWu9CihbMv1OcJvKR8v7PU4z63pHiWNy3\nKbqw3zbM036QovV1B0XXbW0UbK3FdTdFN+V3KVqUDStPXVlM0QK6k2Kw11fZsou7dvvVFB/wv1Le\n/nyKwUrfaOC5bqU49ecvKX6PnwB+n+JUpUVD3O2LFK/5UorW6GkUwTM3InZroJ4vUHz5u2pbf/+Z\neRfF7+ksilORvkZxit0/ljc5H3gNcGcUpznVG/L9PcjzDPl+j4jdgb+iaKk/kplrKN57Z5WDqj5J\n0VPzA4qu/gMp9vEroxjhP1qDvp8H1L3Nf1/adpX+/m3pMZO6R3TYVIIRsRewe5bnCZfr/gA4JTNP\nbF1laged9n6WXdZSJ5sJXFMOHLuR4hjvhym6gSV1GLuspQ5VDng7i6L7OCkmofg8xdzYkjqMXdaS\nJLUBW8iSJLWBjjiGHBGTKc6vfJzBJy6QJKndTaC4JOjtmbl24MaOCGSKML6h1UVIkjQGjmXLKVE7\nJpAfB/jmN7/J/PnzW12LJElbbdmyZbztbW+Dl+dz30ynBPIGgPnz57Pbbru1uhZJkrbFoIdeHdQl\nSVIbMJAlSWoDBrIkSW3AQJYkqQ0YyJIktQEDWZKkNmAgS5LUBgxkSZLagIEsSVIbMJAlSWoDBrIk\nSW3AQJYkqQ0YyJIktYGmBnJEHBkR3x9k/ZKIuD0ibomIdzezBkmSOkHTAjkiPgZ8FZgyYP0k4PPA\nG4HjgLMiYl6z6pAkqRM0s4V8P/DmQdbvD1Qz8+nMfAm4EXhdE+uQJKntNS2QM/M/gHWDbJoJrKpb\nfhbYoVl1SJLUCSa24DlXAzPqlmcAz7SgDvWA2effx9NrN7a6DEkd6oCdJvPT3144Ls/VikC+F9g3\nImYDz1F0V5/XgjrUA55eu5H+D72y1WVI6gDVapXHH3+c1772tVQqlXF//nEL5Ig4E5iemV+OiI8A\nV1F0mX89M385XnVIkjRQtVrluuuuo7+/n/7+fo499thxD+WmBnJmPgQcVf78rbr1lwKXNvO5JUlq\nRH0YA9x3331UKhWOPfbYca3DiUEkST1rYBgDVCoVdtlll3GvxUCWJPWkocL4hBNOYJ999hn3elox\nqEuj4Gjh0dlxst85JW2p3cIYDOSO4WhhSRob7RjGYJe1JKmHtGsYg4EsSeohU6ZMoa/v5ehrlzAG\nA1mS1EN22203Tj75ZCZMmNBWYQweQ5Yk9ZhaKK9Zs6ZtwhgM5DHXrNHQjhaWpK330EMPMWfOHKZP\nn77Z+t12261FFQ3NQB5jjoaWpPZQG8A1ffp0lixZskUotxubXZKkrlM/mvrZZ5/l0ksv5bnnnmt1\nWcMykCVJXWWwU5uee+45nnzyyRZWNTIDWZLUNYY7z3jBggWtK6wBBrIkqSu086QfjXBQ1yC2ZaS0\no6Elafx1ehiDgTwoR0pLUufohjAGu6wlSR1s48aN/PSnP+34MAYDWZLUwfr6+jjllFOYM2cO0Llh\nDAayJKnDTZ48mcWLFzN37tyODWPwGLIkqcM899xzW8y6NXnyZH7t135tsys5dZrOrVyS1HOq1SoX\nXnghP//5z7fY1slhDLaQJUkdon409fe//30A9ttvv9YWNYY6++uEJKknDHZq0w9+8IO2n596axjI\nkqS2NtR5xscff3zbX8FpaxjIkqS21S2TfjTCQJYktaVeCmMwkCVJbajXwhgcZb1J/QUlvECEJLXO\nypUrey6MwUDexAtKSFJ7mD17Nocddhg/+tGPgN4IYzCQJUltaNGiRQD8+Mc/7okwBgNZktQG+vv7\nqVQqm61btGgRe+21F7Nnz25RVePLg6WSpJaqVqt897vfZcOGDVts65UwBgNZktRCtdHUDz/8MFdf\nffWgodwrDGRJUksMPLXpF7/4Bd/73vc2G13dSwxkSdK4G+o844ULF25xLLlXGMiSpHHVi5N+NMJA\nliSNG8N4aAayJGlcGMbDM5AlSU33wAMPGMYj6NmJQernrgbnr5akZpo5cybbbbcda9euBQzjwfRs\nIDt3tSSNnzlz5rB48WKWLl3KSy+9ZBgPomcDWZI0vmqhvHr1avbee+9Wl9N27KeVJI25Bx54gKee\nemqL9XPmzDGMh2AgS5LGVLVa5ZprrmHp0qWDhrIGZyBLksZM/alNa9asMZS3goEsSRoTg51nvHbt\nWlatWtXCqjqHgSxJ2mbDTfrhMePGGMiSpG3iDFxjw0CWJI2aYTx2DGRJ0qgYxmPLQJYkbbX+/n6q\n1aphPIYMZEnSVqtUKrzhDW9g991337RsGG8bA1mSNCoTJ07kpJNOYo899jCMx4BzWUuSGvL000+z\n4447brZu4sSJnHzyyVQqlRZV1T1sIUuSRlStVrn44ov5yU9+ssU2w3hsGMiSpGHVj6a+/fbbBw1l\nbTsDWZI0pMFObbrjjjt4+umnW1hVdzKQJUmDGu4844HHkrXtDGRJ0hac9GP8GciSpM0Yxq1hIEuS\nNjGMW8dAliQB8Oyzz/L973/fMG4RA1mSBMCMGTM49thjNy0bxuPLmbokSZtEBAA33HADxx9/vGE8\njgxkSeph/f39W8y0FRHsuuuuzJgxo0VV9Sa7rCWpR1WrVb7zne+wdu3aLbYZxuPPQJakHlQbTb18\n+XIuv/zyQUNZ48tAlqQeM/DUpieeeIIrr7xys9HVGn8GsiT1kKHOMz7ggAO8alOLGciS1COc9KO9\nGciS1AMM4/ZnIEtSlzOMO0PTzkOOiD7gS8AhwFrgXZlZrdv+NuCjwAbg65l5frNqkaRe9eCDDxrG\nHaKZLeQ3AVMy82jgXOBzA7afB7wBeA3w0Yjw4pqSNMbmzJnD9ttvv2nZMG5fzQzk1wJXAmTmrcDh\nA7bfBewATAEqgOPtJWmMzZgxgyVLljB9+nTDuM01c+rMmcCquuUNETExM9eXyz8FfgQ8D/xnZj7T\nxFokqWfVQvnJJ59kr732anU5GkIzW8irgfq51/pqYRwRBwOLgb2ABcDciHhrE2uRpJ5w//3389hj\nj22xfsaMGYZxm2tmIN8EnAoQEUcBd9dtWwW8CLyYmRuAFYDHkCVpG1SrVa699lquuOKKQUNZ7a2Z\ngXwJsCYibgY+D3w4Is6MiLMy82HgH4AbI+JGYBbwjSbWIkldrf7Upg0bNhjKHahpx5AzcyNw9oDV\n99Vt/3vg75v1/JLUKwY7z3jjxo28+OKLLaxKW8uJQSSpgw036cfChQtbWJm2loEsSR3KGbi6i4Es\nSR3IMO4+BrIkdRjDuDsZyJLUYR577DHDuAsZyJLUYY499lj2228/wDDuJs2cOlOS1ASVSoXjjjuO\nvr4+dt11V8O4SxjIktTmVqxYwc4770ylUtm0rlKp8LrXva6FVWms2WUtSW2sWq3y7W9/m5tuummz\n48bqPgayJLWp+tHUP/vZzwzlLmcgS1IbGuzUpnvvvZcnnniihVWpmQxkSWozw51nPHfu3BZWpmYy\nkCWpjTjpR+8ykCWpTRjGvc1AlqQ2YBjLQJakFnvxxRf5wQ9+YBj3OANZklps6tSpnHTSSUyYMAEw\njHuVgSxJbWD33XfnjW98IxMnTjSMe5RTZ0pSC2zYsGFTi7hm991354wzzmDq1KktqkqtZAtZksZZ\ntVrl4osv5rnnnttim2HcuwxkSRpHtdHUq1at4rLLLhs0lNWbDGRJGicDT21avXo1S5cuZcOGDS2u\nTO3AQJakcTDUecaLFi3a4liyepOBLElN5qQfaoSBLElNZBirUQayJDWJYaytYSBLUhM8/PDDhrG2\nioEsSU0wb948Zs+evWnZMNZIDGRJaoIpU6awePFidtppJ8NYDXHqTElqklooL1++nD333LPV5ajN\n2UKWpDFQrVapVqtbrJ8yZYphrIbYQpakbVQbTV1j17RGw0CWpG0w8NSmWjAbytpadllL0igNdp6x\nNFoGsiSNgpN+aKwZyJK0lQxjNYOBLElbwTBWsxjIktQgw1jNZCBLUoOeeeYZw1hNYyBLUoMWLVrE\nYYcdBhjGGnuehyxJDapUKixatAiAWbNmGcYaUwayJA3hscceY968eUyYMGHTukqlwuGHH97CqtSt\n7LKWpEFUq1WWLl3KNddcw4YNG1pdjnqAgSxJA9SPpn7ooYcMZY0LA1mS6gx2atPDDz/M8uXLW1iV\neoGBLEml4c4z3nXXXVtYmXqBgSxJOOmHWs9AltTzDGO1AwNZUk8zjNUuDGRJPeull17i5ptvNozV\nFgxkST1ru+2245RTTmG77bYDDGO1loEsqaftvPPOLF68mClTphjGaimnzpTUU9atW8ekSZM2W7fz\nzjvzm7/5m5taylIr2EKW1DOq1SoXXXQRK1eu3GKbYaxWM5Al9YTaaOoXXniByy67bNBQllrJQJbU\n9Qae2rRmzRouv/xy1q1b1+LKpJcZyJK62lDnGR911FFbHEuWWslAltS1nPRDncRAltSVDGN1GgNZ\nUtcxjNWJDGRJXeXRRx81jNWRDGRJXWX+/PmbXbvYMFanMJAldZWJEydy8skn84pXvMIwVkdx6kxJ\nXacWysuWLWO33XZrdTlSQ2whS+po1WqVu+66a4v1EydONIzVURpqIUfEbsDBwFXArpn5SFOrkqQG\n1I+m7u/v55BDDml1SdKojdhCjojFwM3A3wFzgXsj4tebXZgkDWfgqU233XbboC1lqVM00mX9SeBI\n4JnMfBx4LfBnTa1KkoYx1HnG06ZNa2FV0rZpJJD7yiAGIDP/G+gf5vaS1DRO+qFu1UggvxARe1CG\ncEQcC6xpalWSNAjDWN2skUFd5wLfBXaJiFuAfYG3NLUqSRrAMFa3GzGQM/PmiDgKOBqYANyamU+O\ndL+I6AO+BBwCrAXelZnVuu2vBv4GqADLgN/KTFvekrZgGKsXNDLK+orMfCYzr8jMyzLzyYi4tYHH\nfhMwJTOPpmhlf67uMSvAV4B3ZOZrgSuBPUf3EiR1u3Xr1hnG6npDtpAj4mJgP2BhRNSfSzAJ2NjA\nY9eClsy8NSIOr9u2H/AU8OGIOBBYmpm5tcVL6g37778/ADfccINhrK41XJf17wMLKFqy769bvx64\np4HHngmsqlveEBETM3M9MAc4BngfUAUui4g7MvParahdUg+phfKkSZMMY3WlIQM5Mx8CHoqIyMzN\nWsQRsX0Dj70amFG33FeGMRSt42pm3ls+3pXA4YCBLIlHHnmEefPmsd122222vhbKUjdqZJT1koj4\nM2A6xQCsCcBsNg/bwdwELAH+rRwUdnfdtgeA6RGxTznQ61jga1tbvKTuUxvANXfuXE455ZQtQlnq\nVo2ch3we8BngF8A5FMeF/76B+10CrImIm4HPUxwvPjMizsrMl4B3At+KiNuBRzJz6ahegaSuUT+a\nevny5VxxxRW89NJLrS5LGheNtJCfz8yLIuJQiglBfg+4Y6Q7ld3cZw9YfV/d9muBI7aiVkldbLBT\nm1asWMHy5cvZfffdW1iZND4aaSGvjYjJFIOvDi2DdnJzy5LUS4Y7z9gwVq9opIX8bWAp8Hbg5nLq\nzKeaWZSk3uGkH1JhxBZyZn4G+N3MfJRiso8f4NSZksaAYSy9bNhAjoj9ImKXzPwFQGb+GPh34Avj\nUZyk7mUYS5sbbqauP6C4FnJ/RCwGbgA+Uq4bcVCXJA1l/fr13H777YaxVGe4FvJ7gP2Bk4EPUbSM\nfx84OzNPHIfaJHWpiRMnsnjxYrbfvphjyDCWhg/k5zPzkcy8mWLiju2A/TPzW+NTmqRuNnPmTJYs\nWcKMGTMMY4nhR1lvqPt5FXB6Zr7Y5Hokdak1a9YwZcqUzdbNnDmTt771rUyc2MgJH1J3a+Q8ZIDV\nhrGk0apWq1xwwQU8/vjjW2wzjKXCcH8JcyPiI4P8DEBm/k3zypLULepHU19xxRWccsop7LLLLq0u\nS2o7w7WQrwYOKv/V/3wQcGDzS5PU6Qae2rR+/XquvPJK1qxZ0+LKpPYz3OUX3zGehUjqLkOdZ3zs\nscducSxZUuPHkCWpYU76IW09A1nSmDKMpdExkCWNGcNYGr2GzjeIiCOAVwH/CCzKzFuaWpWkjrNs\n2TLDWNoGI7aQI+LtFEH8MWAW8O2IeHeT65LUYebOncvChQs3LRvG0tZppMv6A8DRFJODrAAWUcxt\nLUmb9PX1cfzxx7PPPvsYxtIoNBLIGzJzdW0hMx8B1jevJEmdqhbKS5YsMYylrdRIIK+MiEOBfoCI\neBuwsqlVSWp71WqVm2++ebNjxlCE8vz581tUldS5GhnUVbv04sKIeAxYA/x6U6uS1NbqR1P39/dz\nzDHHUKlUWl2W1NEaCeT7gEOA/YAJQGbmuqZWJaltDTy16Z577qFSqXDMMce0uDKpszUSyI8AXwO+\nnpkPN7keSW1sqPOM586d28KqpO7QyDHk1wOTgZsi4qqI+B8R4fXSpB7jpB9Sc40YyFk4F9gD+ALw\n+8Avm12YpPZhGEvN1+hMXXOB3wJ+B6gAn25mUZLah2EsjY8RAzkiLgVeA1wMnJWZtzW9KkltwTCW\nxk8jLeTvAGdk5nPNLkZSe5k0aRKVSmVTIBvGUvMMGcgR8VuZ+a/ATOCsiNhse2b+TZNrk9Rie+65\nJyeddBJXX301/f39hrHURMO1kPct/z9wkG39g6yT1IVqobxu3TrDWGqiIQM5Mz9Z/vhfmfnt+m0R\n8dtNrUpSSzz00EPMnTuXadOmbbZ+zz33bFFFUu8Yrst6CTAJ+GxE9FGMrqZc9xngX5pfnqTxUhvA\nNXPmTJYsWbJFKEtqruG6rA8FTgTmUlyCsWY98NlmFiVpfNWPpl61ahWXXnqpoSyNs+G6rP8c+POI\nOCczvzSONUkaR4Od2rR69WpWrFjBggULWleY1GMaGWU9NSI+MnC7o6ylzjfcecaGsTS+RjvKWlKH\nc9IPqb2MOMo6M99RWxcRM4AdM/MX41CbpCYxjKX2M+LFJSLiNyLib8swvhu4MyI+2PzSJDWDYSy1\np0Yuv/hHwJeBtwC3AHtSXGhCUofZuHEjd955p2EstaFGArmSmXcDbwCuyMzVDd5PUpvp6+vj1FNP\nZfbs2YCyA2sVAAAZUElEQVRhLLWTRoJ1Y0T8T+BXge9GxKk4dabUsaZOncrixYvZaaedDGOpjTQS\nyB8FzgL+KDOXAX/C5hOFSGpjzz///Bbrpk6dym/8xm8YxlIbGfHyi5l5I/CGiNgzIvbJzNeMQ12S\nxkC1WuX666/nhBNOYO+9995sW1+fR56kdjJiIEfEvsB/AbsCfRHxJLA4M+9rdnGSRq9+NPU111wD\nsEUoS2ofjXxF/lvgrzNzx8zcAfg04FSaUhsbeGpTf38/11133aDd15LaQyOBPC8z/6m2kJn/COzc\nvJIkbYuhzjM+7rjj2H777VtYmaThNBLIEyNidm0hIubgKGupLTnph9S5RjyGTNFlfWtEXFQunw58\nvnklSRoNw1jqbCO2kDPzy8B7gO2AKcA5mXl+swuT1DjDWOp8w7aQy0lAfgW4PjP/cHxKkrQ1nnrq\nKcNY6gJDtpAj4lyK7uojgcsi4sxxq0pSw2bPns0hhxyyadkwljrTcF3WZwKHZubpwPHA+8alIklb\npVKp8OpXv5pDDz3UMJY62HBd1usz81mAzMyImD5ONUkaQX9/P5VKZdNyLZQXLlzITjvt1MLKJI3W\n1sydt75pVUhqWLVa5ZprrmHjxo2bra9UKoax1MGGayFPiIgdgcpgy5m5stnFSdpc/Wjq/v5+Xv/6\n1zsntdQlhvtLPgh4su7fQcBT5c9PNL80SfUGntr04IMPcu211242ulpS5xqyhZyZfu2W2sRQ5xkv\nWLBgs2PJkjqXoSu1OSf9kHqDgSy1McNY6h0GstSmDGOptzRycQkiYiqwD/BTYEpmvtjUqqQed//9\n9xvGUo8ZsYUcEUcB9wNLgVcAj0bEMc0uTOpl06dPZ+LEl78vG8ZS92uky/qzwBuApzLzUeC3gS80\ntSqpx82bN49TTz2VSZMmGcZSj2gkkKdl5s9qC5l5OQ12dUsavVoon3jiiYax1AMaCeR15Qxd/QAR\nEc0tSeo9DzzwAM8888wW6+fNm8fChQtbUJGk8dZIIH8auB7YLSIuAG4u10kaA7W5qS+99NJBQ1lS\nbxgxkDPzMuDNwCeBm4BjM/M/ml2Y1AvqT2168cUXDWWphzUyyno2sBK4CPgWsKxcJ2kbDHae8Zo1\na1i50uu2SL2okcFZT1IeP67zOLDb2Jcj9YbhJv3Ye++9W1iZpFYZMZDrLzIREZOAtwCHNLMoqZs5\nA5ekwWzV1JmZuS4zLwROalI9UlczjCUNZcQW8oDjxRXgcGDHBu7XB3yJojW9FnhXZlYHud2XgZWZ\neW6jRUudyDCWNJxGWshPAk/U/f9PwB81cL83Ucx7fTRwLvC5gTeIiPcABzVcrdSh+vv7yUzDWNKQ\nGhnU9erM/NEoHvu1wJUAmXlrRBxev7GcD/tI4B+AXxnF40sdo1Kp8MY3vpGrrrqKxx57zDCWtIVG\nWsj/OsrHngmsqlveEBETASJiF4rzmt83yseWOs6kSZM4+eSTecUrXmEYS9pCIy3kuyLiTOBG4Lna\nyswc6WTJ1cCMuuW+zFxf/vxWYA5wOTAfmBYR92XmNxotXGp3q1atYocddths3aRJkzj11FOpVCot\nqkpSu2okkH+dIkDr9QMTRrjfTcAS4N/KSzjeXduQmV8EvggQEW8HfsUwVjepDeA66qijOOigzYdJ\nGMaSBjNkIEfE5Mxcm5lTRvnYlwAnRcTNFKOz31G2tKdn5pdH+ZhS26sfTX3LLbcAbBHKkjTQcC3k\nW4DDRvvAmbkROHvA6vsGud03RvscUrsZ7NSmW2+9lT322GOL7mtJqjfcoC771aStMNx5xoaxpJEM\n10KeEhGvYohgzswfN6ckqfM46YekbTVcIO8N/AeDB3J/uV3qeYaxpLEwXCD/LDNfNW6VSB3IMJY0\nVrbq4hKSXrZ69WrDWNKYGS6QfzBuVUgdaObMmRxzzDGblg1jSdtiyC7rzPzgeBYidaIDDjgAgJtv\nvtkwlrRNGpmpS1Kpv79/i5m2DjjgAHbffXdmzpzZoqokdQOPIUsNqlarXHbZZaxbt26LbYaxpG1l\nIEsNqI2mfvzxx7niiisGDWVJ2hYGsjSCgac2LVu2jCuvvHKz0dWStK0MZGkYQ51nvP/++3vVJklj\nykCWhuCkH5LGk4EsDcIwljTeDGRpAMNYUisYyFKdBx54wDCW1BIGslRn9uzZTJ06ddOyYSxpvBjI\nUp1Zs2Zx2mmnMW3aNMNY0rhy6kxpgFoor1y5kr339rLfksaHgayedv/997P99tszf/78zdbPmjWL\nWbNmtagqSb3ILmv1rGq1yrXXXssVV1zBsmXLWl2OpB5nIKsn1Z/atG7dOkNZUssZyOo5g51nvH79\nep5//vkWViWp1xnI6inDTfqxcOHCFlYmqdcZyOoZzsAlqZ0ZyOoJhrGkdmcgq+sZxpI6gYGsrveL\nX/zCMJbU9gxkdb3jjz9+04Atw1hSu3KmLnW9vr4+TjjhBCqVCnvssYdhLKktGcjqOk8++SQ77bQT\nlUpl07q+vj5OPPHEFlYlScOzy1pdpVqtcskll3DbbbdtdtxYktqdgayuUT+a+q677jKUJXUUA1ld\nYbBTm+6++26eeuqpFlYlSY0zkNXxhjvPeM6cOS2sTJIaZyCroznph6RuYSCrYxnGkrqJgayOZBhL\n6jYGsjrOCy+8wPXXX28YS+oqBrI6zrRp03j961+/aeIPw1hSNzCQ1ZEWLFjASSedxIQJEwxjSV3B\nqTPVETZu3Ehf3+bfHxcsWMAZZ5zBtGnTWlSVJI0dW8hqe9Vqlf/8z//khRde2GKbYSypWxjIamu1\n0dQrV67ksssuGzSUJakbGMhqWwNPbXrmmWe4/PLL2bhxY4srk6SxZyCrLQ11nvGhhx66xbFkSeoG\nfrKp7Tjph6ReZCCrrRjGknqVgay2YRhL6mUGstrCQw89ZBhL6mkGstrC3Llz2WGHHTYtG8aSeo2B\nrLYwbdo0TjvtNGbNmmUYS+pJTp2ptlEL5RUrVrBgwYJWlyNJ48oWslqiWq3y4IMPbrF+2rRphrGk\nnmQga9zVRlN/73vfGzSUJakXGcgaV/WnNvX39xvKklQykDVuBjvPGGDDhg0tqkiS2oeBrHHhpB+S\nNDwDWU1nGEvSyAxkNZVhLEmNMZDVNIaxJDXOQFbTPPXUU4axJDXIQFbTHHHEERx88MGAYSxJI3Hq\nTDVNpVLhyCOPpFKpsNNOOxnGkjQMA1ljZtmyZcydO5e+vpc7XmqhLEkanl3WGhPVapVLL72U6667\njo0bN7a6HEnqOAaytln9aOr777/fUJakUTCQtU0GO7XpgQceYMWKFS2sSpI6j4GsURvuPOP58+e3\nsDJJ6jwGskbFST8kaWwZyNpqhrEkjb2mnfYUEX3Al4BDgLXAuzKzWrf9DOBDwHrgbuCczHQkUJsz\njCWpOZrZQn4TMCUzjwbOBT5X2xARU4FPAydk5muAHYDTmliLxsDatWu58cYbDWNJaoJmBvJrgSsB\nMvNW4PC6bWuBYzLzhXJ5IrCmibVoDEyePJlf/dVfZdKkSYBhLEljqZmBPBNYVbe8ISImAmTmxsxc\nDhAR7wemA1c3sRaNkfnz53PKKaew3XbbGcaSNIaaOXXmamBG3XJfZq6vLZTHmP8a2A94S2b2o7az\nfv16Jk7c/G0yf/58zjjjDCZPntyiqiSp+zSzhXwTcCpARBxFMXCr3j8AU4A31XVdq41Uq1Uuuugi\nnnnmmS22GcaSNLaa2UK+BDgpIm4GKsA7IuJMiu7pO4B3AjcA10YEwBcy85Im1qOtUD+a+rLLLuO0\n005j1qxZrS5LkrpW0wK5PIXp7AGr76v72XOg29TAU5teeOEFli5dyumnn75F97UkaWwYitrMUOcZ\nH3nkkYaxJDWRgaxNnPRDklrHQBZgGEtSqxnIMowlqQ0YyD3ukUceMYwlqQ0YyD1u/vz5zJs3b9Oy\nYSxJrWEg97hJkyZxyimnMH/+fMNYklrI81i0KZSXLVvG7rvv3upyJKkn2ULuMdVqlXvuuWeL9ZMm\nTTKMJamFbCH3kIGjqQ844IAWVyRJqrGF3CMGhvFNN900aEtZktQaBnIPGOo8Y6/YJEntw0Duck76\nIUmdwUDuYoaxJHUOA7lLGcaS1FkM5C5kGEtS5zGQu9CaNWsMY0nqMAZyFzrwwAM5+uijAcNYkjqF\nE4N0qYMOOohKpcKUKVMMY0nqAAZyF3j00UeZN28ekyZN2mz9gQce2KKKJElbyy7rDletVrniiiu4\n6qqrWLduXavLkSSNkoHcwepHUz/22GOGsiR1MAO5Qw12atPjjz/O8uXLW1iVJGm0DOQONNx5xrvt\ntlsLK5MkjZaB3GGc9EOSupOB3EEMY0nqXgZyhzCMJam7GcgdYP369dx2222GsSR1MQO5A0ycOJHF\nixczdepUwDCWpG5kIHeIWbNmsWTJErbffnvDWJK6kFNntqm1a9cyefLkzdbNmjWL008/nYkT/bVJ\nUrexhdyGqtUqF1544aCTfBjGktSdDOQ2UxtNvXbtWi6//HJn3pKkHmEgt5GBpzatW7eOK6+8krVr\n17a4MklSsxnIbWKo84xf85rXbHEsWZLUfQzkNuCkH5IkA7nFDGNJEhjILWUYS5JqDOQWefzxxw1j\nSdImBnKLzJs3j7322mvTsmEsSb3NWSZapK+vjxNPPBGABx980DCWpB5nILdQLZSXL1/OLrvs0upy\nJEktZJf1OKlWq/zwhz/c7JgxFKFsGEuSbCGPg/rR1P39/RxxxBFUKpVWlyVJaiO2kJts4KlNd955\nJ7fffnuLq5IktRsDuYmGOs949uzZLaxKktSODOQmcdIPSdLWMJCbwDCWJG0tA3mMGcaSpNEwkMeQ\nYSxJGi0DeQz19W2+Ow1jSVKjPA95DO29994AXHPNNQCGsSSpYQbyGKuF8saNGw1jSVLDDORt8PDD\nDzN37lymTp262fpaKEuS1CiPIY9StVrlu9/9LkuXLuXFF19sdTmSpA5nII9C/WjqlStXGsqSpG1m\nIG+lwU5tevrpp1mxYkULq5IkdToDeSsMd57xnnvu2cLKJEmdzkBukJN+SJKayUBugGEsSWo2A3kE\nhrEkaTwYyMPYuHEjP/nJTwxjSVLTGcjD6OvrY/Hixeywww6AYSxJah4DeQTTpk1jyZIl7Ljjjoax\nJKlpnDpzgBdeeIFp06Zttm7atGm85S1v2eJqTpIkjRUTpk61WuWCCy7g4Ycf3mKbYSxJaiZTplQb\nTb1hwwauvvrqQUNZkqRmMZDZ8tSmjRs38r3vfY8XXnihxZVJknpFzwbyATtNBoY+z/i4447b4liy\nJEnN0rOB/NPfXuikH5KkttGzgWwYS5LaSdNOe4qIPuBLwCHAWuBdmVmt274E+ASwHvh6Zn6lWbUM\nZBhLktpNM1vIbwKmZObRwLnA52obImIS8HngjcBxwFkRMa+JtWzyxBNPGMaSpLbTzEB+LXAlQGbe\nChxet21/oJqZT2fmS8CNwOuaWMsmc+bM4YADDti0bBhLktpBMwN5JrCqbnlDREwcYtuzwA5NrGWT\nSqXC0UcfzYEHHmgYS5LaRjOnzlwNzKhb7svM9UNsmwE808RaNlML5X333Zedd955vJ5WkqQhNbOF\nfBNwKkBEHAXcXbftXmDfiJgdEdtRdFff0sRatlCpVAxjSVLbaGYL+RLgpIi4GagA74iIM4Hpmfnl\niPgIcBXFl4KvZ+Yvm1iLJEltrWmBnJkbgbMHrL6vbvulwKXNen5JkjpJz04MIklSOzGQJUlqAway\nJEltwECWJKkNGMiSJLUBA1mSpDZgIEuS1AYMZEmS2oCBLElSGzCQJUlqAwayJEltwECWJKkNGMiS\nJLUBA1mSpDbQzOshj6UJAMuWLWt1HZIkjUpdhk0YbHunBPIuAG9729taXYckSdtqF+D+gSs7JZBv\nB44FHgc2tLgWSZJGYwJFGN8+2MZKf3//+JYjSZK24KAuSZLagIEsSVIbMJAlSWoDBrIkSW2gU0ZZ\nj0pE9AFfAg4B1gLvysxq3fYlwCeA9cDXM/MrLSm0DTWw784APkSx7+4GzsnMja2otd2MtO/qbvdl\nYGVmnjvOJba1Bt57rwb+BqgAy4Dfysw1rai13TSw794GfJTibJWvZ+b5LSm0TUXEkcBfZebxA9aP\nS1Z0ewv5TcCUzDwaOBf4XG1DREwCPg+8ETgOOCsi5rWkyvY03L6bCnwaOCEzXwPsAJzWkirb05D7\nriYi3gMcNN6FdYjh3nsV4CvAOzLztcCVwJ4tqbI9jfTeOw94A/Aa4KMRseM419e2IuJjwFeBKQPW\nj1tWdHsg1/5gycxbgcPrtu0PVDPz6cx8CbgReN34l9i2htt3a4FjMvOFcnkiYAvlZcPtOyLiGOBI\n4B/Gv7SOMNz+2w94CvhwRFwPzM7MHP8S29aw7z3gLoov0FMoehg87/Vl9wNvHmT9uGVFtwfyTGBV\n3fKGiJg4xLZnKd6oKgy57zJzY2YuB4iI9wPTgavHv8S2NeS+i4hdgE8C72tFYR1iuL/bOcAxwP9H\n0dJ7fUScOM71tbPh9h3AT4EfAfcAl2XmM+NZXDvLzP8A1g2yadyyotsDeTUwo265LzPXD7FtBuCb\n82XD7Tsioi8izgNOAt6SmX7Tftlw++6tFKFyOUWX4pkR8fbxLa/tDbf/nqJordybmesoWoMDW4G9\nbMh9FxEHA4uBvYAFwNyIeOu4V9h5xi0ruj2QbwJOBYiIoygGH9XcC+wbEbMjYjuKLohbxr/EtjXc\nvoOiu3UK8Ka6rmsVhtx3mfnFzFxUDhr5S+BbmfmNVhTZxoZ77z0ATI+IfcrlYylaeyoMt+9WAS8C\nL2bmBmAF4DHkkY1bVnT11Jl1Iw4Ppjhe8g7gMGB6Zn65buRcH8XIub9rWbFtZrh9B9xR/ruBl49B\nfSEzL2lBqW1npPdd3e3eDvyKo6w318Df7YkUX2YqwM2Z+cGWFdtmGth3ZwO/C7xEccz03eVxUQER\nsQC4MDOPiogzGees6OpAliSpU3R7l7UkSR3BQJYkqQ0YyJIktQEDWZKkNmAgS5LUBrr64hLSeIiI\nfooZkDbUrb4jM981zH3eDvyPzNzmOcAj4lPAe4FfUpyGNoHiHNNzMvPno3i8XYGLM/OYiNgLOC8z\n31K/fgxqXkBx2k39ebLTgUeB383MB0a4/yeAOzPz29tai9QuDGRpbJyQmU+28PkvysxN03GWU5p+\ni1HMYpWZj1FMTwnFhRtikPVj4cXMPLS2UF444ovA/wHOGOG+JwI/G8NapJYzkKUmiojfBd4DbAfM\nBv5y4CXvIuLNwJ8CGyla2X+QmT+IiB2AL1BcFWoScE25bT0juwb4i/LxdwPOp5gusQL8U2Z+tpzj\n+G8pLkjwEsUsWO+gmNrzpxTz9X4VeEVEXFW+jp9SzO37MPAbmXlH+RwXAtdn5vkR8SfAWygOiT1E\n0VJ/rIGapwC7ALV50vcD/o6i5bwr8N/A6cA7Kb5ofDYiNgBLgb+iuBLPBOAnwAcyc3UDzym1DY8h\nS2Pjuoj477p/cyNiOvBu4NTMfBVFmPz1IPf9LEVoHQ58HDi+XP954EeZuQh4FUVQfmSkQsqgfSdw\nXbnqm8B1mXkQxWX3fisifhM4unyug8vneIBihicAyukV3wXcn5kn163fCHwdeHv5fDtSzGn+rYj4\nXxRfII4oW7+XU4T6YKaW++quiFgO/BhI4A/L7e+m+PJwNLAPxRzMi8tZku6g+HJyCcWc4OuBRZl5\nCPAYxUxeUkexhSyNjUG7rCPiNGBxROwLHErR2hvoQuCSiFhKcdWsWmifBhwREe8sl6cO8/ynR8Rr\ny5+3o7iiz7sjYnuKEH4jQGauiohvAKcAH6Rokd9WtoD/IzN/WB7fHcnXgdsj4iMU3cuXlo99GnAE\ncEdEQNFinTbEY2zqso6Ik4F/Bb6bmc+V2/8QOKm8Tu1+FK3kwfbfacCs8ra117+igdcgtRUDWWqS\nsqv4FuDLFNdQvZgiPDaTmX8SEV+jCM23A+dGxCKKMHtrZt5bPt4shr5+7WbHkOtqmEHRTV2vD5iU\nmc9ExCEUgX0icFFEfBEYcU7yzHw4In5cvp53AB8qN00A/qrWLR8Rk2ngAgaZeVVE/A1wQUTsn5mr\ngAsoPqP+jaJbeo9BXkvtOT+YmVeUzzmdAReZlzqBXdZS8xwOPAF8OjOvogzjiJhQu0FETIyIh4Dt\nM/PvgXMoLog+CbgK+HBEVMpg+w5beR3lzHwWuJViFDblcen/BVxdtmavobhAw6eAfwYOGfAQ68ta\nBvMVilbstMy8qVx3FfCuiJhZLv8Z8C8NlnsexWXt/ne5fDLwZ5l5EcUXkSMpwndgXVcB74uI7cqL\nK3yF8vi51EkMZKl5vktxGk9GxE8oWnhPUBwPBaAcoPUhiuOvPwb+neK0n7XAB4DtKU4Nuqv8f7Bj\n0CN5G/D6iLgb+CHwH8A3gCsoLl3404i4g2IE9acG3Pceiovc/5AtW6ffoRgo9rW6dV8FLgNujYh7\nKI5Jv72RIsvrG78PeG9EHAj8MUVX/h3A3wPX8/K+uxQ4LyJ+B/hzisFjP6EYeV0BPtrIc0rtxKs9\nSZLUBmwhS5LUBgxkSZLagIEsSVIbMJAlSWoDBrIkSW3AQJYkqQ0YyJIktQEDWZKkNvD/A1aqVBTM\nZlmKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1126c2d90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve, auc\n",
    "# Plot the receiver operating characteristic curve (ROC).\n",
    "plt.figure(figsize=(10,8))\n",
    "probas_ = clf.predict_proba(X_test)\n",
    "fpr, tpr, thresholds = roc_curve(y_test, probas_[:, 1])\n",
    "roc_auc = auc(fpr, tpr)\n",
    "plt.plot(fpr, tpr, lw=1, label='ROC fold (area = %0.2f)' % (roc_auc))\n",
    "plt.plot([0, 1], [0, 1], '--', color=(0.6, 0.6, 0.6), label='Random')\n",
    "plt.xlim([-0.05, 1.05])\n",
    "plt.ylim([-0.05, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic example')\n",
    "plt.axes().set_aspect(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* To interpret the ROC correctly, consider what the points that lie along the diagonal represent. For these situations, there is an equal chance of \"+\" and \"-\" happening. Therefore, this is not that different from making a prediction by tossing of an unbiased coin. Put simply, the classification model is random.\n",
    "\n",
    "* For the points above the diagonal, tpr > fpr, and the model says that you are in a zone where you are performing better than random. For example, assume tpr = 0.99 and fpr = 0.01, Then, the probability of being in the true positive group is $(0.99 / (0.99 + 0.01)) = 99\\%$. Furthermore, holding fpr constant, it is easy to see that the more vertically above the diagonal you are positioned, the better the classification model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next I will look into optimizing the clas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Cortes, Corinna, and Vladimir Vapnik. 1995. “Support-Vector Networks.” Machine Learning 20: 273. Accessed September 3, 2016. doi: 10.1023/A:1022627411411.\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}