traditional-machine-learning/005-grid-search-vs-random-search-vs-bayes-search/gridsearch-vs-randomsearch-vs-bayessearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Practical Introduction to Grid Search, Random Search, and Bayes Search for Hyperparamter tuning\n",
    "\n",
    "This is a notebook for the medium article [A Practical Introduction to Grid Search, Random Search, and Bayes Search)](https://bindichen.medium.com/a-practical-introduction-to-grid-search-random-search-and-bayes-search-d5580b1d941d)\n",
    "\n",
    "Please check out article for instructions\n",
    "\n",
    "**License**: [BSD 2-Clause](https://opensource.org/licenses/BSD-2-Clause)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting and preparing data\n",
    "We'll use the built in breast cancer dataset from Scikit Learn. We can get with the load function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "cancer = load_breast_cancer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst radius</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>0.09744</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>0.05883</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
       "0        17.99         10.38          122.80     1001.0          0.11840   \n",
       "1        20.57         17.77          132.90     1326.0          0.08474   \n",
       "2        19.69         21.25          130.00     1203.0          0.10960   \n",
       "3        11.42         20.38           77.58      386.1          0.14250   \n",
       "4        20.29         14.34          135.10     1297.0          0.10030   \n",
       "\n",
       "   mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
       "0           0.27760          0.3001              0.14710         0.2419   \n",
       "1           0.07864          0.0869              0.07017         0.1812   \n",
       "2           0.15990          0.1974              0.12790         0.2069   \n",
       "3           0.28390          0.2414              0.10520         0.2597   \n",
       "4           0.13280          0.1980              0.10430         0.1809   \n",
       "\n",
       "   mean fractal dimension  ...  worst radius  worst texture  worst perimeter  \\\n",
       "0                 0.07871  ...         25.38          17.33           184.60   \n",
       "1                 0.05667  ...         24.99          23.41           158.80   \n",
       "2                 0.05999  ...         23.57          25.53           152.50   \n",
       "3                 0.09744  ...         14.91          26.50            98.87   \n",
       "4                 0.05883  ...         22.54          16.67           152.20   \n",
       "\n",
       "   worst area  worst smoothness  worst compactness  worst concavity  \\\n",
       "0      2019.0            0.1622             0.6656           0.7119   \n",
       "1      1956.0            0.1238             0.1866           0.2416   \n",
       "2      1709.0            0.1444             0.4245           0.4504   \n",
       "3       567.7            0.2098             0.8663           0.6869   \n",
       "4      1575.0            0.1374             0.2050           0.4000   \n",
       "\n",
       "   worst concave points  worst symmetry  worst fractal dimension  \n",
       "0                0.2654          0.4601                  0.11890  \n",
       "1                0.1860          0.2750                  0.08902  \n",
       "2                0.2430          0.3613                  0.08758  \n",
       "3                0.2575          0.6638                  0.17300  \n",
       "4                0.1625          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_X = pd.DataFrame(cancer['data'], columns=cancer['feature_names'])\n",
    "df_X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cancer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Cancer\n",
       "0       0\n",
       "1       0\n",
       "2       0\n",
       "3       0\n",
       "4       0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_y = pd.DataFrame(cancer['target'], columns=['Cancer'])\n",
    "df_y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_X, np.ravel(df_y), test_size=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = { \n",
    "    'C': [0.1, 1, 10, 100, 1000], \n",
    "    'gamma': [1, 0.1, 0.01, 0.001, 0.0001]\n",
    "} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "grid = GridSearchCV(\n",
    "    SVC(), \n",
    "    param_grid, \n",
    "    refit=True, \n",
    "    verbose=3\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 25 candidates, totalling 125 fits\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.625, total=   0.0s\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.625, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.625, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.620, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.620, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.613, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.613, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .................. C=0.1, gamma=0.001, score=0.625, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.620, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.620, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.950, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.900, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.938, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.911, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.886, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.625, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.625, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.625, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.620, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.620, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.950, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.900, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.963, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.899, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.848, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.963, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.925, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.950, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.911, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.899, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.625, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.625, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.625, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.633, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.633, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.938, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.900, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.938, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.911, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.873, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.950, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.938, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.963, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.899, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.886, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.625, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.625, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.625, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.633, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.633, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.950, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.900, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.938, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.911, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.873, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.963, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.863, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.950, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.924, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.899, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.613, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.625, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.620, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.613, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.625, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.620, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.625, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.633, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.633, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.950, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.900, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.938, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.911, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.873, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.963, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.875, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.900, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.911, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.873, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 125 out of 125 | elapsed:    1.2s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(estimator=SVC(),\n",
       "             param_grid={'C': [0.1, 1, 10, 100, 1000],\n",
       "                         'gamma': [1, 0.1, 0.01, 0.001, 0.0001]},\n",
       "             verbose=3)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Best params and estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 1, 'gamma': 0.0001}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the best parameter\n",
    "grid.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1, gamma=0.0001)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the best estimator\n",
    "grid.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Test and create report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_predictions = grid.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 55   5]\n",
      " [  1 110]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "# Confusion matrics\n",
    "print(confusion_matrix(y_test, grid_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.92      0.95        60\n",
      "           1       0.96      0.99      0.97       111\n",
      "\n",
      "    accuracy                           0.96       171\n",
      "   macro avg       0.97      0.95      0.96       171\n",
      "weighted avg       0.97      0.96      0.96       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classification report\n",
    "print(classification_report(y_test, grid_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Random Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.stats as stats\n",
    "from sklearn.utils.fixes import loguniform\n",
    "\n",
    "# Specify parameters and distributions to sample from\n",
    "param_dist = {\n",
    "  'C': stats.uniform(0.1, 1e4),\n",
    "  'gamma': loguniform(1e-6, 1e+1),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
      "[CV] C=2844.232407458716, gamma=2.329456664154018 ....................\n",
      "[CV]  C=2844.232407458716, gamma=2.329456664154018, score=0.613, total=   0.0s\n",
      "[CV] C=2844.232407458716, gamma=2.329456664154018 ....................\n",
      "[CV]  C=2844.232407458716, gamma=2.329456664154018, score=0.613, total=   0.0s\n",
      "[CV] C=2844.232407458716, gamma=2.329456664154018 ....................\n",
      "[CV]  C=2844.232407458716, gamma=2.329456664154018, score=0.625, total=   0.0s\n",
      "[CV] C=2844.232407458716, gamma=2.329456664154018 ....................\n",
      "[CV]  C=2844.232407458716, gamma=2.329456664154018, score=0.620, total=   0.0s\n",
      "[CV] C=2844.232407458716, gamma=2.329456664154018 ....................\n",
      "[CV]  C=2844.232407458716, gamma=2.329456664154018, score=0.620, total=   0.0s\n",
      "[CV] C=9425.826706842483, gamma=2.963586771644764e-06 ................\n",
      "[CV]  C=9425.826706842483, gamma=2.963586771644764e-06, score=0.938, total=   0.0s\n",
      "[CV] C=9425.826706842483, gamma=2.963586771644764e-06 ................\n",
      "[CV]  C=9425.826706842483, gamma=2.963586771644764e-06, score=0.938, total=   0.0s\n",
      "[CV] C=9425.826706842483, gamma=2.963586771644764e-06 ................\n",
      "[CV]  C=9425.826706842483, gamma=2.963586771644764e-06, score=0.950, total=   0.0s\n",
      "[CV] C=9425.826706842483, gamma=2.963586771644764e-06 ................\n",
      "[CV]  C=9425.826706842483, gamma=2.963586771644764e-06, score=0.949, total=   0.0s\n",
      "[CV] C=9425.826706842483, gamma=2.963586771644764e-06 ................\n",
      "[CV]  C=9425.826706842483, gamma=2.963586771644764e-06, score=0.899, total=   0.0s\n",
      "[CV] C=2601.2632729742504, gamma=0.012000811626776364 ................\n",
      "[CV]  C=2601.2632729742504, gamma=0.012000811626776364, score=0.613, total=   0.0s\n",
      "[CV] C=2601.2632729742504, gamma=0.012000811626776364 ................\n",
      "[CV]  C=2601.2632729742504, gamma=0.012000811626776364, score=0.613, total=   0.0s\n",
      "[CV] C=2601.2632729742504, gamma=0.012000811626776364 ................\n",
      "[CV]  C=2601.2632729742504, gamma=0.012000811626776364, score=0.625, total=   0.0s\n",
      "[CV] C=2601.2632729742504, gamma=0.012000811626776364 ................\n",
      "[CV]  C=2601.2632729742504, gamma=0.012000811626776364, score=0.620, total=   0.0s\n",
      "[CV] C=2601.2632729742504, gamma=0.012000811626776364 ................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2601.2632729742504, gamma=0.012000811626776364, score=0.620, total=   0.0s\n",
      "[CV] C=7813.8426807624055, gamma=2.1690583207515505e-05 ..............\n",
      "[CV]  C=7813.8426807624055, gamma=2.1690583207515505e-05, score=0.938, total=   0.0s\n",
      "[CV] C=7813.8426807624055, gamma=2.1690583207515505e-05 ..............\n",
      "[CV]  C=7813.8426807624055, gamma=2.1690583207515505e-05, score=0.863, total=   0.0s\n",
      "[CV] C=7813.8426807624055, gamma=2.1690583207515505e-05 ..............\n",
      "[CV]  C=7813.8426807624055, gamma=2.1690583207515505e-05, score=0.900, total=   0.0s\n",
      "[CV] C=7813.8426807624055, gamma=2.1690583207515505e-05 ..............\n",
      "[CV]  C=7813.8426807624055, gamma=2.1690583207515505e-05, score=0.924, total=   0.0s\n",
      "[CV] C=7813.8426807624055, gamma=2.1690583207515505e-05 ..............\n",
      "[CV]  C=7813.8426807624055, gamma=2.1690583207515505e-05, score=0.899, total=   0.0s\n",
      "[CV] C=86.82954008208, gamma=1.5236290252691447 ......................\n",
      "[CV]  C=86.82954008208, gamma=1.5236290252691447, score=0.613, total=   0.0s\n",
      "[CV] C=86.82954008208, gamma=1.5236290252691447 ......................\n",
      "[CV]  C=86.82954008208, gamma=1.5236290252691447, score=0.613, total=   0.0s\n",
      "[CV] C=86.82954008208, gamma=1.5236290252691447 ......................\n",
      "[CV]  C=86.82954008208, gamma=1.5236290252691447, score=0.625, total=   0.0s\n",
      "[CV] C=86.82954008208, gamma=1.5236290252691447 ......................\n",
      "[CV]  C=86.82954008208, gamma=1.5236290252691447, score=0.620, total=   0.0s\n",
      "[CV] C=86.82954008208, gamma=1.5236290252691447 ......................\n",
      "[CV]  C=86.82954008208, gamma=1.5236290252691447, score=0.620, total=   0.0s\n",
      "[CV] C=462.1158690481735, gamma=0.2147380513691475 ...................\n",
      "[CV]  C=462.1158690481735, gamma=0.2147380513691475, score=0.613, total=   0.0s\n",
      "[CV] C=462.1158690481735, gamma=0.2147380513691475 ...................\n",
      "[CV]  C=462.1158690481735, gamma=0.2147380513691475, score=0.613, total=   0.0s\n",
      "[CV] C=462.1158690481735, gamma=0.2147380513691475 ...................\n",
      "[CV]  C=462.1158690481735, gamma=0.2147380513691475, score=0.625, total=   0.0s\n",
      "[CV] C=462.1158690481735, gamma=0.2147380513691475 ...................\n",
      "[CV]  C=462.1158690481735, gamma=0.2147380513691475, score=0.620, total=   0.0s\n",
      "[CV] C=462.1158690481735, gamma=0.2147380513691475 ...................\n",
      "[CV]  C=462.1158690481735, gamma=0.2147380513691475, score=0.620, total=   0.0s\n",
      "[CV] C=2473.176696330547, gamma=0.3347875653171696 ...................\n",
      "[CV]  C=2473.176696330547, gamma=0.3347875653171696, score=0.613, total=   0.0s\n",
      "[CV] C=2473.176696330547, gamma=0.3347875653171696 ...................\n",
      "[CV]  C=2473.176696330547, gamma=0.3347875653171696, score=0.613, total=   0.0s\n",
      "[CV] C=2473.176696330547, gamma=0.3347875653171696 ...................\n",
      "[CV]  C=2473.176696330547, gamma=0.3347875653171696, score=0.625, total=   0.0s\n",
      "[CV] C=2473.176696330547, gamma=0.3347875653171696 ...................\n",
      "[CV]  C=2473.176696330547, gamma=0.3347875653171696, score=0.620, total=   0.0s\n",
      "[CV] C=2473.176696330547, gamma=0.3347875653171696 ...................\n",
      "[CV]  C=2473.176696330547, gamma=0.3347875653171696, score=0.620, total=   0.0s\n",
      "[CV] C=5017.3223308334, gamma=9.274546963131837e-05 ..................\n",
      "[CV]  C=5017.3223308334, gamma=9.274546963131837e-05, score=0.963, total=   0.0s\n",
      "[CV] C=5017.3223308334, gamma=9.274546963131837e-05 ..................\n",
      "[CV]  C=5017.3223308334, gamma=9.274546963131837e-05, score=0.875, total=   0.0s\n",
      "[CV] C=5017.3223308334, gamma=9.274546963131837e-05 ..................\n",
      "[CV]  C=5017.3223308334, gamma=9.274546963131837e-05, score=0.900, total=   0.0s\n",
      "[CV] C=5017.3223308334, gamma=9.274546963131837e-05 ..................\n",
      "[CV]  C=5017.3223308334, gamma=9.274546963131837e-05, score=0.899, total=   0.0s\n",
      "[CV] C=5017.3223308334, gamma=9.274546963131837e-05 ..................\n",
      "[CV]  C=5017.3223308334, gamma=9.274546963131837e-05, score=0.873, total=   0.0s\n",
      "[CV] C=3350.4492649306585, gamma=0.37493729824048494 .................\n",
      "[CV]  C=3350.4492649306585, gamma=0.37493729824048494, score=0.613, total=   0.0s\n",
      "[CV] C=3350.4492649306585, gamma=0.37493729824048494 .................\n",
      "[CV]  C=3350.4492649306585, gamma=0.37493729824048494, score=0.613, total=   0.0s\n",
      "[CV] C=3350.4492649306585, gamma=0.37493729824048494 .................\n",
      "[CV]  C=3350.4492649306585, gamma=0.37493729824048494, score=0.625, total=   0.0s\n",
      "[CV] C=3350.4492649306585, gamma=0.37493729824048494 .................\n",
      "[CV]  C=3350.4492649306585, gamma=0.37493729824048494, score=0.620, total=   0.0s\n",
      "[CV] C=3350.4492649306585, gamma=0.37493729824048494 .................\n",
      "[CV]  C=3350.4492649306585, gamma=0.37493729824048494, score=0.620, total=   0.0s\n",
      "[CV] C=4622.501498273593, gamma=0.00010187498059458797 ...............\n",
      "[CV]  C=4622.501498273593, gamma=0.00010187498059458797, score=0.963, total=   0.0s\n",
      "[CV] C=4622.501498273593, gamma=0.00010187498059458797 ...............\n",
      "[CV]  C=4622.501498273593, gamma=0.00010187498059458797, score=0.875, total=   0.0s\n",
      "[CV] C=4622.501498273593, gamma=0.00010187498059458797 ...............\n",
      "[CV]  C=4622.501498273593, gamma=0.00010187498059458797, score=0.900, total=   0.0s\n",
      "[CV] C=4622.501498273593, gamma=0.00010187498059458797 ...............\n",
      "[CV]  C=4622.501498273593, gamma=0.00010187498059458797, score=0.911, total=   0.0s\n",
      "[CV] C=4622.501498273593, gamma=0.00010187498059458797 ...............\n",
      "[CV]  C=4622.501498273593, gamma=0.00010187498059458797, score=0.873, total=   0.0s\n",
      "[CV] C=1889.804112751185, gamma=0.003660096837486428 .................\n",
      "[CV]  C=1889.804112751185, gamma=0.003660096837486428, score=0.938, total=   0.0s\n",
      "[CV] C=1889.804112751185, gamma=0.003660096837486428 .................\n",
      "[CV]  C=1889.804112751185, gamma=0.003660096837486428, score=0.863, total=   0.0s\n",
      "[CV] C=1889.804112751185, gamma=0.003660096837486428 .................\n",
      "[CV]  C=1889.804112751185, gamma=0.003660096837486428, score=0.925, total=   0.0s\n",
      "[CV] C=1889.804112751185, gamma=0.003660096837486428 .................\n",
      "[CV]  C=1889.804112751185, gamma=0.003660096837486428, score=0.899, total=   0.0s\n",
      "[CV] C=1889.804112751185, gamma=0.003660096837486428 .................\n",
      "[CV]  C=1889.804112751185, gamma=0.003660096837486428, score=0.823, total=   0.0s\n",
      "[CV] C=5853.290938502737, gamma=0.000273800991121007 .................\n",
      "[CV]  C=5853.290938502737, gamma=0.000273800991121007, score=0.938, total=   0.0s\n",
      "[CV] C=5853.290938502737, gamma=0.000273800991121007 .................\n",
      "[CV]  C=5853.290938502737, gamma=0.000273800991121007, score=0.900, total=   0.0s\n",
      "[CV] C=5853.290938502737, gamma=0.000273800991121007 .................\n",
      "[CV]  C=5853.290938502737, gamma=0.000273800991121007, score=0.963, total=   0.0s\n",
      "[CV] C=5853.290938502737, gamma=0.000273800991121007 .................\n",
      "[CV]  C=5853.290938502737, gamma=0.000273800991121007, score=0.911, total=   0.0s\n",
      "[CV] C=5853.290938502737, gamma=0.000273800991121007 .................\n",
      "[CV]  C=5853.290938502737, gamma=0.000273800991121007, score=0.899, total=   0.0s\n",
      "[CV] C=615.0724645788446, gamma=0.0025902320924878434 ................\n",
      "[CV]  C=615.0724645788446, gamma=0.0025902320924878434, score=0.938, total=   0.0s\n",
      "[CV] C=615.0724645788446, gamma=0.0025902320924878434 ................\n",
      "[CV]  C=615.0724645788446, gamma=0.0025902320924878434, score=0.863, total=   0.0s\n",
      "[CV] C=615.0724645788446, gamma=0.0025902320924878434 ................\n",
      "[CV]  C=615.0724645788446, gamma=0.0025902320924878434, score=0.925, total=   0.0s\n",
      "[CV] C=615.0724645788446, gamma=0.0025902320924878434 ................\n",
      "[CV]  C=615.0724645788446, gamma=0.0025902320924878434, score=0.899, total=   0.0s\n",
      "[CV] C=615.0724645788446, gamma=0.0025902320924878434 ................\n",
      "[CV]  C=615.0724645788446, gamma=0.0025902320924878434, score=0.835, total=   0.0s\n",
      "[CV] C=1556.5991770182488, gamma=1.54669493143796e-05 ................\n",
      "[CV]  C=1556.5991770182488, gamma=1.54669493143796e-05, score=0.975, total=   0.0s\n",
      "[CV] C=1556.5991770182488, gamma=1.54669493143796e-05 ................\n",
      "[CV]  C=1556.5991770182488, gamma=1.54669493143796e-05, score=0.887, total=   0.0s\n",
      "[CV] C=1556.5991770182488, gamma=1.54669493143796e-05 ................\n",
      "[CV]  C=1556.5991770182488, gamma=1.54669493143796e-05, score=0.950, total=   0.0s\n",
      "[CV] C=1556.5991770182488, gamma=1.54669493143796e-05 ................\n",
      "[CV]  C=1556.5991770182488, gamma=1.54669493143796e-05, score=0.949, total=   0.0s\n",
      "[CV] C=1556.5991770182488, gamma=1.54669493143796e-05 ................\n",
      "[CV]  C=1556.5991770182488, gamma=1.54669493143796e-05, score=0.886, total=   0.0s\n",
      "[CV] C=4209.3339198181075, gamma=0.0030528542779024986 ...............\n",
      "[CV]  C=4209.3339198181075, gamma=0.0030528542779024986, score=0.938, total=   0.0s\n",
      "[CV] C=4209.3339198181075, gamma=0.0030528542779024986 ...............\n",
      "[CV]  C=4209.3339198181075, gamma=0.0030528542779024986, score=0.863, total=   0.0s\n",
      "[CV] C=4209.3339198181075, gamma=0.0030528542779024986 ...............\n",
      "[CV]  C=4209.3339198181075, gamma=0.0030528542779024986, score=0.925, total=   0.0s\n",
      "[CV] C=4209.3339198181075, gamma=0.0030528542779024986 ...............\n",
      "[CV]  C=4209.3339198181075, gamma=0.0030528542779024986, score=0.899, total=   0.0s\n",
      "[CV] C=4209.3339198181075, gamma=0.0030528542779024986 ...............\n",
      "[CV]  C=4209.3339198181075, gamma=0.0030528542779024986, score=0.823, total=   0.0s\n",
      "[CV] C=2126.1922248455994, gamma=6.042304124354614e-05 ...............\n",
      "[CV]  C=2126.1922248455994, gamma=6.042304124354614e-05, score=0.938, total=   0.0s\n",
      "[CV] C=2126.1922248455994, gamma=6.042304124354614e-05 ...............\n",
      "[CV]  C=2126.1922248455994, gamma=6.042304124354614e-05, score=0.863, total=   0.0s\n",
      "[CV] C=2126.1922248455994, gamma=6.042304124354614e-05 ...............\n",
      "[CV]  C=2126.1922248455994, gamma=6.042304124354614e-05, score=0.912, total=   0.0s\n",
      "[CV] C=2126.1922248455994, gamma=6.042304124354614e-05 ...............\n",
      "[CV]  C=2126.1922248455994, gamma=6.042304124354614e-05, score=0.899, total=   0.0s\n",
      "[CV] C=2126.1922248455994, gamma=6.042304124354614e-05 ...............\n",
      "[CV]  C=2126.1922248455994, gamma=6.042304124354614e-05, score=0.873, total=   0.0s\n",
      "[CV] C=4584.779563499073, gamma=0.11665194819713727 ..................\n",
      "[CV]  C=4584.779563499073, gamma=0.11665194819713727, score=0.613, total=   0.0s\n",
      "[CV] C=4584.779563499073, gamma=0.11665194819713727 ..................\n",
      "[CV]  C=4584.779563499073, gamma=0.11665194819713727, score=0.613, total=   0.0s\n",
      "[CV] C=4584.779563499073, gamma=0.11665194819713727 ..................\n",
      "[CV]  C=4584.779563499073, gamma=0.11665194819713727, score=0.625, total=   0.0s\n",
      "[CV] C=4584.779563499073, gamma=0.11665194819713727 ..................\n",
      "[CV]  C=4584.779563499073, gamma=0.11665194819713727, score=0.620, total=   0.0s\n",
      "[CV] C=4584.779563499073, gamma=0.11665194819713727 ..................\n",
      "[CV]  C=4584.779563499073, gamma=0.11665194819713727, score=0.620, total=   0.0s\n",
      "[CV] C=913.5121221473623, gamma=3.31154848298912e-05 .................\n",
      "[CV]  C=913.5121221473623, gamma=3.31154848298912e-05, score=0.950, total=   0.0s\n",
      "[CV] C=913.5121221473623, gamma=3.31154848298912e-05 .................\n",
      "[CV]  C=913.5121221473623, gamma=3.31154848298912e-05, score=0.863, total=   0.0s\n",
      "[CV] C=913.5121221473623, gamma=3.31154848298912e-05 .................\n",
      "[CV]  C=913.5121221473623, gamma=3.31154848298912e-05, score=0.963, total=   0.0s\n",
      "[CV] C=913.5121221473623, gamma=3.31154848298912e-05 .................\n",
      "[CV]  C=913.5121221473623, gamma=3.31154848298912e-05, score=0.937, total=   0.0s\n",
      "[CV] C=913.5121221473623, gamma=3.31154848298912e-05 .................\n",
      "[CV]  C=913.5121221473623, gamma=3.31154848298912e-05, score=0.873, total=   0.0s\n",
      "[CV] C=3351.617000917777, gamma=8.982634021707856 ....................\n",
      "[CV]  C=3351.617000917777, gamma=8.982634021707856, score=0.613, total=   0.0s\n",
      "[CV] C=3351.617000917777, gamma=8.982634021707856 ....................\n",
      "[CV]  C=3351.617000917777, gamma=8.982634021707856, score=0.613, total=   0.0s\n",
      "[CV] C=3351.617000917777, gamma=8.982634021707856 ....................\n",
      "[CV]  C=3351.617000917777, gamma=8.982634021707856, score=0.625, total=   0.0s\n",
      "[CV] C=3351.617000917777, gamma=8.982634021707856 ....................\n",
      "[CV]  C=3351.617000917777, gamma=8.982634021707856, score=0.620, total=   0.0s\n",
      "[CV] C=3351.617000917777, gamma=8.982634021707856 ....................\n",
      "[CV]  C=3351.617000917777, gamma=8.982634021707856, score=0.620, total=   0.0s\n",
      "[CV] C=9113.847255925142, gamma=2.285576877000574e-06 ................\n",
      "[CV]  C=9113.847255925142, gamma=2.285576877000574e-06, score=0.938, total=   0.0s\n",
      "[CV] C=9113.847255925142, gamma=2.285576877000574e-06 ................\n",
      "[CV]  C=9113.847255925142, gamma=2.285576877000574e-06, score=0.938, total=   0.0s\n",
      "[CV] C=9113.847255925142, gamma=2.285576877000574e-06 ................\n",
      "[CV]  C=9113.847255925142, gamma=2.285576877000574e-06, score=0.950, total=   0.0s\n",
      "[CV] C=9113.847255925142, gamma=2.285576877000574e-06 ................\n",
      "[CV]  C=9113.847255925142, gamma=2.285576877000574e-06, score=0.962, total=   0.0s\n",
      "[CV] C=9113.847255925142, gamma=2.285576877000574e-06 ................\n",
      "[CV]  C=9113.847255925142, gamma=2.285576877000574e-06, score=0.911, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:    1.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(estimator=SVC(), n_iter=20,\n",
       "                   param_distributions={'C': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7ff869e158d0>,\n",
       "                                        'gamma': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7ff869e15950>},\n",
       "                   verbose=3)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "n_iter_search = 20\n",
    "random_search = RandomizedSearchCV(\n",
    "    SVC(), \n",
    "    param_distributions=param_dist,\n",
    "    n_iter=n_iter_search, \n",
    "    refit=True,\n",
    "    verbose=3\n",
    ")\n",
    "random_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Best params and estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 9113.847255925142, 'gamma': 2.285576877000574e-06}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=9113.847255925142, gamma=2.285576877000574e-06)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test and create report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run prediction using the best estimator\n",
    "random_predictions = random_search.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 57   3]\n",
      " [  3 108]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "# Confusion matrics\n",
    "print(confusion_matrix(y_test, random_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.95      0.95        60\n",
      "           1       0.97      0.97      0.97       111\n",
      "\n",
      "    accuracy                           0.96       171\n",
      "   macro avg       0.96      0.96      0.96       171\n",
      "weighted avg       0.96      0.96      0.96       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classification report\n",
    "print(classification_report(y_test, random_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Bayes Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt import BayesSearchCV\n",
    "# parameter ranges are specified by one of below\n",
    "from skopt.space import Real, Categorical, Integer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_spaces = {\n",
    "  'C': Real(0.1, 1e+4),\n",
    "  'gamma': Real(1e-6, 1e+1, 'log-uniform'),\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=2961.787223792435, gamma=4.219094765283724e-05 ................\n",
      "[CV]  C=2961.787223792435, gamma=4.219094765283724e-05, score=0.950, total=   0.0s\n",
      "[CV] C=2961.787223792435, gamma=4.219094765283724e-05 ................\n",
      "[CV]  C=2961.787223792435, gamma=4.219094765283724e-05, score=0.863, total=   0.0s\n",
      "[CV] C=2961.787223792435, gamma=4.219094765283724e-05 ................\n",
      "[CV]  C=2961.787223792435, gamma=4.219094765283724e-05, score=0.900, total=   0.0s\n",
      "[CV] C=2961.787223792435, gamma=4.219094765283724e-05 ................\n",
      "[CV]  C=2961.787223792435, gamma=4.219094765283724e-05, score=0.886, total=   0.0s\n",
      "[CV] C=2961.787223792435, gamma=4.219094765283724e-05 ................\n",
      "[CV]  C=2961.787223792435, gamma=4.219094765283724e-05, score=0.886, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=7614.614699639945, gamma=0.14892219712504987 ..................\n",
      "[CV]  C=7614.614699639945, gamma=0.14892219712504987, score=0.613, total=   0.0s\n",
      "[CV] C=7614.614699639945, gamma=0.14892219712504987 ..................\n",
      "[CV]  C=7614.614699639945, gamma=0.14892219712504987, score=0.613, total=   0.0s\n",
      "[CV] C=7614.614699639945, gamma=0.14892219712504987 ..................\n",
      "[CV]  C=7614.614699639945, gamma=0.14892219712504987, score=0.625, total=   0.0s\n",
      "[CV] C=7614.614699639945, gamma=0.14892219712504987 ..................\n",
      "[CV]  C=7614.614699639945, gamma=0.14892219712504987, score=0.620, total=   0.0s\n",
      "[CV] C=7614.614699639945, gamma=0.14892219712504987 ..................\n",
      "[CV]  C=7614.614699639945, gamma=0.14892219712504987, score=0.620, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=6207.337436516946, gamma=0.0011848013744296202 ................\n",
      "[CV]  C=6207.337436516946, gamma=0.0011848013744296202, score=0.950, total=   0.0s\n",
      "[CV] C=6207.337436516946, gamma=0.0011848013744296202 ................\n",
      "[CV]  C=6207.337436516946, gamma=0.0011848013744296202, score=0.900, total=   0.0s\n",
      "[CV] C=6207.337436516946, gamma=0.0011848013744296202 ................\n",
      "[CV]  C=6207.337436516946, gamma=0.0011848013744296202, score=0.938, total=   0.0s\n",
      "[CV] C=6207.337436516946, gamma=0.0011848013744296202 ................\n",
      "[CV]  C=6207.337436516946, gamma=0.0011848013744296202, score=0.911, total=   0.0s\n",
      "[CV] C=6207.337436516946, gamma=0.0011848013744296202 ................\n",
      "[CV]  C=6207.337436516946, gamma=0.0011848013744296202, score=0.873, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=590.7347374753837, gamma=0.014086215628392828 .................\n",
      "[CV]  C=590.7347374753837, gamma=0.014086215628392828, score=0.613, total=   0.0s\n",
      "[CV] C=590.7347374753837, gamma=0.014086215628392828 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=590.7347374753837, gamma=0.014086215628392828, score=0.613, total=   0.0s\n",
      "[CV] C=590.7347374753837, gamma=0.014086215628392828 .................\n",
      "[CV]  C=590.7347374753837, gamma=0.014086215628392828, score=0.625, total=   0.0s\n",
      "[CV] C=590.7347374753837, gamma=0.014086215628392828 .................\n",
      "[CV]  C=590.7347374753837, gamma=0.014086215628392828, score=0.620, total=   0.0s\n",
      "[CV] C=590.7347374753837, gamma=0.014086215628392828 .................\n",
      "[CV]  C=590.7347374753837, gamma=0.014086215628392828, score=0.620, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=7061.176758794584, gamma=0.0018389740767854191 ................\n",
      "[CV]  C=7061.176758794584, gamma=0.0018389740767854191, score=0.938, total=   0.0s\n",
      "[CV] C=7061.176758794584, gamma=0.0018389740767854191 ................\n",
      "[CV]  C=7061.176758794584, gamma=0.0018389740767854191, score=0.875, total=   0.0s\n",
      "[CV] C=7061.176758794584, gamma=0.0018389740767854191 ................\n",
      "[CV]  C=7061.176758794584, gamma=0.0018389740767854191, score=0.925, total=   0.0s\n",
      "[CV] C=7061.176758794584, gamma=0.0018389740767854191 ................\n",
      "[CV]  C=7061.176758794584, gamma=0.0018389740767854191, score=0.899, total=   0.0s\n",
      "[CV] C=7061.176758794584, gamma=0.0018389740767854191 ................\n",
      "[CV]  C=7061.176758794584, gamma=0.0018389740767854191, score=0.861, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=9615.655689014848, gamma=0.5173800814717748 ...................\n",
      "[CV]  C=9615.655689014848, gamma=0.5173800814717748, score=0.613, total=   0.0s\n",
      "[CV] C=9615.655689014848, gamma=0.5173800814717748 ...................\n",
      "[CV]  C=9615.655689014848, gamma=0.5173800814717748, score=0.613, total=   0.0s\n",
      "[CV] C=9615.655689014848, gamma=0.5173800814717748 ...................\n",
      "[CV]  C=9615.655689014848, gamma=0.5173800814717748, score=0.625, total=   0.0s\n",
      "[CV] C=9615.655689014848, gamma=0.5173800814717748 ...................\n",
      "[CV]  C=9615.655689014848, gamma=0.5173800814717748, score=0.620, total=   0.0s\n",
      "[CV] C=9615.655689014848, gamma=0.5173800814717748 ...................\n",
      "[CV]  C=9615.655689014848, gamma=0.5173800814717748, score=0.620, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=2852.4471357687744, gamma=0.14160729473538394 .................\n",
      "[CV]  C=2852.4471357687744, gamma=0.14160729473538394, score=0.613, total=   0.0s\n",
      "[CV] C=2852.4471357687744, gamma=0.14160729473538394 .................\n",
      "[CV]  C=2852.4471357687744, gamma=0.14160729473538394, score=0.613, total=   0.0s\n",
      "[CV] C=2852.4471357687744, gamma=0.14160729473538394 .................\n",
      "[CV]  C=2852.4471357687744, gamma=0.14160729473538394, score=0.625, total=   0.0s\n",
      "[CV] C=2852.4471357687744, gamma=0.14160729473538394 .................\n",
      "[CV]  C=2852.4471357687744, gamma=0.14160729473538394, score=0.620, total=   0.0s\n",
      "[CV] C=2852.4471357687744, gamma=0.14160729473538394 .................\n",
      "[CV]  C=2852.4471357687744, gamma=0.14160729473538394, score=0.620, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=3363.2073846219805, gamma=0.0044661140900466995 ...............\n",
      "[CV]  C=3363.2073846219805, gamma=0.0044661140900466995, score=0.938, total=   0.0s\n",
      "[CV] C=3363.2073846219805, gamma=0.0044661140900466995 ...............\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=3363.2073846219805, gamma=0.0044661140900466995, score=0.863, total=   0.0s\n",
      "[CV] C=3363.2073846219805, gamma=0.0044661140900466995 ...............\n",
      "[CV]  C=3363.2073846219805, gamma=0.0044661140900466995, score=0.925, total=   0.0s\n",
      "[CV] C=3363.2073846219805, gamma=0.0044661140900466995 ...............\n",
      "[CV]  C=3363.2073846219805, gamma=0.0044661140900466995, score=0.899, total=   0.0s\n",
      "[CV] C=3363.2073846219805, gamma=0.0044661140900466995 ...............\n",
      "[CV]  C=3363.2073846219805, gamma=0.0044661140900466995, score=0.823, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=5634.912052702958, gamma=0.9284752168418909 ...................\n",
      "[CV]  C=5634.912052702958, gamma=0.9284752168418909, score=0.613, total=   0.0s\n",
      "[CV] C=5634.912052702958, gamma=0.9284752168418909 ...................\n",
      "[CV]  C=5634.912052702958, gamma=0.9284752168418909, score=0.613, total=   0.0s\n",
      "[CV] C=5634.912052702958, gamma=0.9284752168418909 ...................\n",
      "[CV]  C=5634.912052702958, gamma=0.9284752168418909, score=0.625, total=   0.0s\n",
      "[CV] C=5634.912052702958, gamma=0.9284752168418909 ...................\n",
      "[CV]  C=5634.912052702958, gamma=0.9284752168418909, score=0.620, total=   0.0s\n",
      "[CV] C=5634.912052702958, gamma=0.9284752168418909 ...................\n",
      "[CV]  C=5634.912052702958, gamma=0.9284752168418909, score=0.620, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=1158.779615044406, gamma=3.2916413676774035 ...................\n",
      "[CV]  C=1158.779615044406, gamma=3.2916413676774035, score=0.613, total=   0.0s\n",
      "[CV] C=1158.779615044406, gamma=3.2916413676774035 ...................\n",
      "[CV]  C=1158.779615044406, gamma=3.2916413676774035, score=0.613, total=   0.0s\n",
      "[CV] C=1158.779615044406, gamma=3.2916413676774035 ...................\n",
      "[CV]  C=1158.779615044406, gamma=3.2916413676774035, score=0.625, total=   0.0s\n",
      "[CV] C=1158.779615044406, gamma=3.2916413676774035 ...................\n",
      "[CV]  C=1158.779615044406, gamma=3.2916413676774035, score=0.620, total=   0.0s\n",
      "[CV] C=1158.779615044406, gamma=3.2916413676774035 ...................\n",
      "[CV]  C=1158.779615044406, gamma=3.2916413676774035, score=0.620, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=1187.9940551892985, gamma=0.0002573324296647919 ...............\n",
      "[CV]  C=1187.9940551892985, gamma=0.0002573324296647919, score=0.938, total=   0.0s\n",
      "[CV] C=1187.9940551892985, gamma=0.0002573324296647919 ...............\n",
      "[CV]  C=1187.9940551892985, gamma=0.0002573324296647919, score=0.900, total=   0.0s\n",
      "[CV] C=1187.9940551892985, gamma=0.0002573324296647919 ...............\n",
      "[CV]  C=1187.9940551892985, gamma=0.0002573324296647919, score=0.963, total=   0.0s\n",
      "[CV] C=1187.9940551892985, gamma=0.0002573324296647919 ...............\n",
      "[CV]  C=1187.9940551892985, gamma=0.0002573324296647919, score=0.911, total=   0.0s\n",
      "[CV] C=1187.9940551892985, gamma=0.0002573324296647919 ...............\n",
      "[CV]  C=1187.9940551892985, gamma=0.0002573324296647919, score=0.899, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=605.6350523919251, gamma=0.0005302318251012832 ................\n",
      "[CV]  C=605.6350523919251, gamma=0.0005302318251012832, score=0.925, total=   0.0s\n",
      "[CV] C=605.6350523919251, gamma=0.0005302318251012832 ................\n",
      "[CV]  C=605.6350523919251, gamma=0.0005302318251012832, score=0.925, total=   0.0s\n",
      "[CV] C=605.6350523919251, gamma=0.0005302318251012832 ................\n",
      "[CV]  C=605.6350523919251, gamma=0.0005302318251012832, score=0.950, total=   0.0s\n",
      "[CV] C=605.6350523919251, gamma=0.0005302318251012832 ................\n",
      "[CV]  C=605.6350523919251, gamma=0.0005302318251012832, score=0.911, total=   0.0s\n",
      "[CV] C=605.6350523919251, gamma=0.0005302318251012832 ................\n",
      "[CV]  C=605.6350523919251, gamma=0.0005302318251012832, score=0.899, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=9467.426591698848, gamma=0.00012293800139494897 ...............\n",
      "[CV]  C=9467.426591698848, gamma=0.00012293800139494897, score=0.963, total=   0.0s\n",
      "[CV] C=9467.426591698848, gamma=0.00012293800139494897 ...............\n",
      "[CV]  C=9467.426591698848, gamma=0.00012293800139494897, score=0.863, total=   0.0s\n",
      "[CV] C=9467.426591698848, gamma=0.00012293800139494897 ...............\n",
      "[CV]  C=9467.426591698848, gamma=0.00012293800139494897, score=0.950, total=   0.0s\n",
      "[CV] C=9467.426591698848, gamma=0.00012293800139494897 ...............\n",
      "[CV]  C=9467.426591698848, gamma=0.00012293800139494897, score=0.911, total=   0.0s\n",
      "[CV] C=9467.426591698848, gamma=0.00012293800139494897 ...............\n",
      "[CV]  C=9467.426591698848, gamma=0.00012293800139494897, score=0.886, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=7226.100510239308, gamma=2.7208283587572996e-06 ...............\n",
      "[CV]  C=7226.100510239308, gamma=2.7208283587572996e-06, score=0.938, total=   0.0s\n",
      "[CV] C=7226.100510239308, gamma=2.7208283587572996e-06 ...............\n",
      "[CV]  C=7226.100510239308, gamma=2.7208283587572996e-06, score=0.938, total=   0.0s\n",
      "[CV] C=7226.100510239308, gamma=2.7208283587572996e-06 ...............\n",
      "[CV]  C=7226.100510239308, gamma=2.7208283587572996e-06, score=0.950, total=   0.0s\n",
      "[CV] C=7226.100510239308, gamma=2.7208283587572996e-06 ...............\n",
      "[CV]  C=7226.100510239308, gamma=2.7208283587572996e-06, score=0.962, total=   0.0s\n",
      "[CV] C=7226.100510239308, gamma=2.7208283587572996e-06 ...............\n",
      "[CV]  C=7226.100510239308, gamma=2.7208283587572996e-06, score=0.911, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.1s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=9148.532393397518, gamma=4.671581589832688e-06 ................\n",
      "[CV]  C=9148.532393397518, gamma=4.671581589832688e-06, score=0.938, total=   0.0s\n",
      "[CV] C=9148.532393397518, gamma=4.671581589832688e-06 ................\n",
      "[CV]  C=9148.532393397518, gamma=4.671581589832688e-06, score=0.925, total=   0.0s\n",
      "[CV] C=9148.532393397518, gamma=4.671581589832688e-06 ................\n",
      "[CV]  C=9148.532393397518, gamma=4.671581589832688e-06, score=0.950, total=   0.0s\n",
      "[CV] C=9148.532393397518, gamma=4.671581589832688e-06 ................\n",
      "[CV]  C=9148.532393397518, gamma=4.671581589832688e-06, score=0.924, total=   0.0s\n",
      "[CV] C=9148.532393397518, gamma=4.671581589832688e-06 ................\n",
      "[CV]  C=9148.532393397518, gamma=4.671581589832688e-06, score=0.899, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.1s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=9084.204811701062, gamma=1.3943028672080794e-06 ...............\n",
      "[CV]  C=9084.204811701062, gamma=1.3943028672080794e-06, score=0.950, total=   0.0s\n",
      "[CV] C=9084.204811701062, gamma=1.3943028672080794e-06 ...............\n",
      "[CV]  C=9084.204811701062, gamma=1.3943028672080794e-06, score=0.938, total=   0.0s\n",
      "[CV] C=9084.204811701062, gamma=1.3943028672080794e-06 ...............\n",
      "[CV]  C=9084.204811701062, gamma=1.3943028672080794e-06, score=0.950, total=   0.0s\n",
      "[CV] C=9084.204811701062, gamma=1.3943028672080794e-06 ...............\n",
      "[CV]  C=9084.204811701062, gamma=1.3943028672080794e-06, score=0.962, total=   0.0s\n",
      "[CV] C=9084.204811701062, gamma=1.3943028672080794e-06 ...............\n",
      "[CV]  C=9084.204811701062, gamma=1.3943028672080794e-06, score=0.937, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=0.1, gamma=1e-06 ..............................................\n",
      "[CV] .................. C=0.1, gamma=1e-06, score=0.900, total=   0.0s\n",
      "[CV] C=0.1, gamma=1e-06 ..............................................\n",
      "[CV] .................. C=0.1, gamma=1e-06, score=0.850, total=   0.0s\n",
      "[CV] C=0.1, gamma=1e-06 ..............................................\n",
      "[CV] .................. C=0.1, gamma=1e-06, score=0.912, total=   0.0s\n",
      "[CV] C=0.1, gamma=1e-06 ..............................................\n",
      "[CV] .................. C=0.1, gamma=1e-06, score=0.937, total=   0.0s\n",
      "[CV] C=0.1, gamma=1e-06 ..............................................\n",
      "[CV] .................. C=0.1, gamma=1e-06, score=0.861, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=10000.0, gamma=1.8170250072185041e-06 .........................\n",
      "[CV]  C=10000.0, gamma=1.8170250072185041e-06, score=0.950, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.8170250072185041e-06 .........................\n",
      "[CV]  C=10000.0, gamma=1.8170250072185041e-06, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.8170250072185041e-06 .........................\n",
      "[CV]  C=10000.0, gamma=1.8170250072185041e-06, score=0.950, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.8170250072185041e-06 .........................\n",
      "[CV]  C=10000.0, gamma=1.8170250072185041e-06, score=0.962, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.8170250072185041e-06 .........................\n",
      "[CV]  C=10000.0, gamma=1.8170250072185041e-06, score=0.924, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.1s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=10000.0, gamma=1.3857512426997007e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3857512426997007e-05, score=0.963, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.3857512426997007e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3857512426997007e-05, score=0.863, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.3857512426997007e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3857512426997007e-05, score=0.963, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.3857512426997007e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3857512426997007e-05, score=0.949, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.3857512426997007e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3857512426997007e-05, score=0.873, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.1s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=10000.0, gamma=0.00038244414061085387 .........................\n",
      "[CV]  C=10000.0, gamma=0.00038244414061085387, score=0.925, total=   0.0s\n",
      "[CV] C=10000.0, gamma=0.00038244414061085387 .........................\n",
      "[CV]  C=10000.0, gamma=0.00038244414061085387, score=0.912, total=   0.0s\n",
      "[CV] C=10000.0, gamma=0.00038244414061085387 .........................\n",
      "[CV]  C=10000.0, gamma=0.00038244414061085387, score=0.963, total=   0.0s\n",
      "[CV] C=10000.0, gamma=0.00038244414061085387 .........................\n",
      "[CV]  C=10000.0, gamma=0.00038244414061085387, score=0.911, total=   0.0s\n",
      "[CV] C=10000.0, gamma=0.00038244414061085387 .........................\n",
      "[CV]  C=10000.0, gamma=0.00038244414061085387, score=0.899, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "BayesSearchCV(cv=5, estimator=SVC(), n_iter=20,\n",
       "              search_spaces={'C': Real(low=0.1, high=10000.0, prior='uniform', transform='identity'),\n",
       "                             'gamma': Real(low=1e-06, high=10.0, prior='log-uniform', transform='identity')},\n",
       "              verbose=3)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_iter_search = 20\n",
    "bayes_search = BayesSearchCV(\n",
    "    SVC(), \n",
    "    search_spaces, \n",
    "    n_iter=n_iter_search, \n",
    "    cv=5,\n",
    "    verbose=3\n",
    ")\n",
    "bayes_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Best params and estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('C', 0.25624177419852506), ('gamma', 0.00016576008531229226)])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=0.25624177419852506, gamma=0.00016576008531229226)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Test and create report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "bayes_predictions = bayes_search.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 51   9]\n",
      " [  1 110]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report,confusion_matrix\n",
    "\n",
    "# Confusion matrics\n",
    "print(confusion_matrix(y_test, bayes_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.85      0.91        60\n",
      "           1       0.92      0.99      0.96       111\n",
      "\n",
      "    accuracy                           0.94       171\n",
      "   macro avg       0.95      0.92      0.93       171\n",
      "weighted avg       0.94      0.94      0.94       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classification report\n",
    "print(classification_report(y_test, bayes_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thanks for reading\n",
    "\n",
    "This is a notebook for the medium article [A Practical Introduction to Grid Search, Random Search, and Bayes Search)](https://bindichen.medium.com/a-practical-introduction-to-grid-search-random-search-and-bayes-search-d5580b1d941d)\n",
    "\n",
    "Please check out article for instructions\n",
    "\n",
    "**License**: [BSD 2-Clause](https://opensource.org/licenses/BSD-2-Clause)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf-tutorial",
   "language": "python",
   "name": "tf-tutorial"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}