docs/automl_gs_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# automl-gs Hello World\n",
    "\n",
    "#### by Max Woolf (@mimaxir)\n",
    "\n",
    "This notebook will give you an example on how automl-gs works with very little effort!\n",
    "\n",
    "(Note: this notebook assumes you have installed automs-gs, TensorFlow and xgboost on the system)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from automl_gs import automl_grid_search\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this Hello World, we'll use the [Titanic dataset](http://web.stanford.edu/class/archive/cs/cs109/cs109.1166/problem12.html), which is small and good to make sure everything works. We'll try to predict `Survival`, i.e. if a person in the given row survived, and select the model that has the best `accuracy`. (a typical model with substantial preprocessing gets about 80% accuracy on this problem)\n",
    "\n",
    "We'll download the dataset; the CSV must be on the local system for automl-gs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>Siblings/Spouses Aboard</th>\n",
       "      <th>Parents/Children Aboard</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Owen Harris Braund</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mrs. John Bradley (Florence Briggs Thayer) Cum...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Laina Heikkinen</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mrs. Jacques Heath (Lily May Peel) Futrelle</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. William Henry Allen</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. James Moran</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.4583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. Timothy J McCarthy</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>51.8625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Master. Gosta Leonard Palsson</td>\n",
       "      <td>male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>21.0750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Mrs. Oscar W (Elisabeth Vilhelmina Berg) Johnson</td>\n",
       "      <td>female</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>11.1333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Mrs. Nicholas (Adele Achem) Nasser</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0708</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Pclass                                               Name  \\\n",
       "0         0       3                             Mr. Owen Harris Braund   \n",
       "1         1       1  Mrs. John Bradley (Florence Briggs Thayer) Cum...   \n",
       "2         1       3                              Miss. Laina Heikkinen   \n",
       "3         1       1        Mrs. Jacques Heath (Lily May Peel) Futrelle   \n",
       "4         0       3                            Mr. William Henry Allen   \n",
       "5         0       3                                    Mr. James Moran   \n",
       "6         0       1                             Mr. Timothy J McCarthy   \n",
       "7         0       3                      Master. Gosta Leonard Palsson   \n",
       "8         1       3   Mrs. Oscar W (Elisabeth Vilhelmina Berg) Johnson   \n",
       "9         1       2                 Mrs. Nicholas (Adele Achem) Nasser   \n",
       "\n",
       "      Sex   Age  Siblings/Spouses Aboard  Parents/Children Aboard     Fare  \n",
       "0    male  22.0                        1                        0   7.2500  \n",
       "1  female  38.0                        1                        0  71.2833  \n",
       "2  female  26.0                        0                        0   7.9250  \n",
       "3  female  35.0                        1                        0  53.1000  \n",
       "4    male  35.0                        0                        0   8.0500  \n",
       "5    male  27.0                        0                        0   8.4583  \n",
       "6    male  54.0                        0                        0  51.8625  \n",
       "7    male   2.0                        3                        1  21.0750  \n",
       "8  female  27.0                        0                        2  11.1333  \n",
       "9  female  14.0                        1                        0  30.0708  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('http://web.stanford.edu/class/archive/cs/cs109/cs109.1166/stuff/titanic.csv')\n",
    "df.to_csv('titanic.csv', index=False)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run automl-gs with just one quick command! By default, automl-gs will use TensorFlow.\n",
    "\n",
    "automl-gs tries to automatically infer the problem type and the data types of the columns. During training, a progress bar will appear to tell you both how far the experiment is progressing, how far a given trial is progressing, times elapsed for both, and ETAs until completion for both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Solving a binary_classification problem, maximizing accuracy using tensorflow.\n",
      "\n",
      "Modeling with field specifications:\n",
      "Pclass: categorical\n",
      "Name: ignore\n",
      "Sex: categorical\n",
      "Age: numeric\n",
      "Siblings/Spouses Aboard: categorical\n",
      "Parents/Children Aboard: categorical\n",
      "Fare: numeric\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b827ef928fda451fb2fff89f6cc53ec6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "60bea55a06ea4c72ba0d922093833b6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=20), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Metrics:\n",
      "trial_id: 269b2f5a-5759-48eb-908a-91301498f173\n",
      "epoch: 20\n",
      "time_completed: 2019-03-25 14:37:26\n",
      "log_loss: 0.6891973887043499\n",
      "accuracy: 0.6142322097378277\n",
      "auc: 0.5012431920435709\n",
      "precision: 0.30711610486891383\n",
      "recall: 0.5\n",
      "f1: 0.3805104408352668\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: 261256e8-1708-43d8-98de-f73ca3d5ad4f\n",
      "epoch: 20\n",
      "time_completed: 2019-03-25 14:37:37\n",
      "log_loss: 0.6832971124166853\n",
      "accuracy: 0.6404494382022472\n",
      "auc: 0.7806950035519773\n",
      "precision: 0.7603359173126615\n",
      "recall: 0.5357861709685058\n",
      "f1: 0.4576844955991876\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: 3a693105-3ff9-42be-802d-6a9147b6638b\n",
      "epoch: 20\n",
      "time_completed: 2019-03-25 14:38:14\n",
      "log_loss: 0.5836427935611889\n",
      "accuracy: 0.700374531835206\n",
      "auc: 0.7412384560738811\n",
      "precision: 0.6830808080808081\n",
      "recall: 0.6802628463177836\n",
      "f1: 0.6814982703089585\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: f3f24805-79f0-4feb-bdff-1050d912df71\n",
      "epoch: 20\n",
      "time_completed: 2019-03-25 14:39:11\n",
      "log_loss: 0.578730115506533\n",
      "accuracy: 0.704119850187266\n",
      "auc: 0.7311449206725077\n",
      "precision: 0.6875487900078064\n",
      "recall: 0.6706725076959508\n",
      "f1: 0.6749372004499992\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: 6c530a43-6413-47e9-8bc3-d96ae9782148\n",
      "epoch: 20\n",
      "time_completed: 2019-03-25 14:39:19\n",
      "log_loss: 0.6386199218503544\n",
      "accuracy: 0.7116104868913857\n",
      "auc: 0.7266457494672033\n",
      "precision: 0.7087053571428572\n",
      "recall: 0.6605197726734549\n",
      "f1: 0.6645564456917228\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: 355ec086-3033-477c-be62-1817b01278a7\n",
      "epoch: 20\n",
      "time_completed: 2019-03-25 14:40:01\n",
      "log_loss: nan\n",
      "accuracy: 0.7640449438202247\n",
      "auc: 0.7292209329860291\n",
      "precision: 0.7659641506791766\n",
      "recall: 0.7248697608335306\n",
      "f1: 0.7337986041874377\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: 998e2eca-9f44-4726-b08e-61e5e381f43d\n",
      "epoch: 20\n",
      "time_completed: 2019-03-25 14:41:07\n",
      "log_loss: 0.4609095337434431\n",
      "accuracy: 0.7940074906367042\n",
      "auc: 0.8314882784750177\n",
      "precision: 0.7872881355932204\n",
      "recall: 0.7709270660667772\n",
      "f1: 0.7768677920775531\n"
     ]
    }
   ],
   "source": [
    "automl_grid_search('titanic.csv', 'Survived')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About 79.4% accuracy: not bad.\n",
    "\n",
    "The model files are saved in a time-stamped folder in the same directory, and `automl_results.csv` has the results from all the training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use another framework like `xgboost`, or change the number of trials/epochs by passing it to `automl_grid_search`.\n",
    "\n",
    "xgboost runs substantially faster than TensorFlow, but may not be as robust on more complicated datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Solving a binary_classification problem, maximizing accuracy using xgboost.\n",
      "\n",
      "Modeling with field specifications:\n",
      "Pclass: categorical\n",
      "Name: ignore\n",
      "Sex: categorical\n",
      "Age: numeric\n",
      "Siblings/Spouses Aboard: categorical\n",
      "Parents/Children Aboard: categorical\n",
      "Fare: numeric\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3cd87fa88929461ebc858cf2f81fb762",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0b38cf3aade46519aaffb403021d33a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=50), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Metrics:\n",
      "trial_id: 0bf1bc3a-e215-4685-9c1e-2f924c4dda15\n",
      "epoch: 50\n",
      "time_completed: 2019-03-25 14:47:29\n",
      "log_loss: 0.6861143221569418\n",
      "accuracy: 0.6891385767790262\n",
      "auc: 0.6915699739521668\n",
      "precision: 0.6718104160925875\n",
      "recall: 0.6476438550793275\n",
      "f1: 0.6512550161303012\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: 6ad4efef-f02a-461e-a3f6-febb49b7eb39\n",
      "epoch: 50\n",
      "time_completed: 2019-03-25 14:47:31\n",
      "log_loss: 0.5730137539546142\n",
      "accuracy: 0.7303370786516854\n",
      "auc: 0.7495560028415817\n",
      "precision: 0.7193349263241736\n",
      "recall: 0.6956251479990527\n",
      "f1: 0.7014906832298138\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: 758c4a9e-ce70-4a1c-95f9-bf936da4b55a\n",
      "epoch: 50\n",
      "time_completed: 2019-03-25 14:47:32\n",
      "log_loss: 0.5655809819475095\n",
      "accuracy: 0.7940074906367042\n",
      "auc: 0.8300378877575183\n",
      "precision: 0.8316425120772947\n",
      "recall: 0.7438432394032678\n",
      "f1: 0.7571643543399533\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: 5e83896a-001d-4372-bf4a-5e4c77addea7\n",
      "epoch: 50\n",
      "time_completed: 2019-03-25 14:47:34\n",
      "log_loss: 0.44416837770952267\n",
      "accuracy: 0.8202247191011236\n",
      "auc: 0.8358690504380772\n",
      "precision: 0.8283059520603285\n",
      "recall: 0.7886573525929435\n",
      "f1: 0.7999500499500499\n",
      "\n",
      "\n",
      "Metrics:\n",
      "trial_id: f6fef923-6598-4266-852c-e0cbd2269158\n",
      "epoch: 50\n",
      "time_completed: 2019-03-25 14:48:12\n",
      "log_loss: 0.4294823957851317\n",
      "accuracy: 0.8352059925093633\n",
      "auc: 0.8462881837556239\n",
      "precision: 0.8433890880127439\n",
      "recall: 0.8062692398768648\n",
      "f1: 0.8175776397515528\n"
     ]
    }
   ],
   "source": [
    "automl_grid_search('titanic.csv', 'Survived', framework='xgboost', num_epochs=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Meanwhile, xgboost got 83.5% accuracy on the same dataset in about 1/4th the time, even after massively increasing the number of epochs.\n",
    "\n",
    "Although automl-gs automates a lot of the option selection, it's encourages to still try multiple options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MIT License\n",
    "\n",
    "Copyright (c) 2019 Max Woolf\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
    "of this software and associated documentation files (the \"Software\"), to deal\n",
    "in the Software without restriction, including without limitation the rights\n",
    "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
    "copies of the Software, and to permit persons to whom the Software is\n",
    "furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all\n",
    "copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
    "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
    "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
    "SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}