Predicting Hard Drive Failure -  A Data Science Case Study.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Hard Drive Failures Using SMART Metrics\n",
    "-----\n",
    "\n",
    "\n",

   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### About this case study :\n",
    "This case study relies on a given data stream provided for this purpose. The goal of this case study is to try and analyze given data and find out meaningful information that can help determine drives failure trends and different factors that may idicate if a drive would fail, and attempt to propose a more data driven answer to future failures based on SMART metrics.\n",
    "\n",
    "The study concludes with discussing possible opportunities and challenges with existing model and features that can help design a better predictive model for future. \n",
    "\n",
    "\n",
    "### What are SMART features ?\n",
    "SMART features or *S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology)* is a software monitoring system for hard drives. It is a widely used industry practice around data center management and disk heavy resources. SMART generates a collection different metrics related to help evaluate the overall health of a Hard Drive. These metrics can be specific to a certain number of manufacturers or be more general, sometimes. \n",
    "\n",
    "A single metrics may not always determine the exact failure prediction but are commonly accepted to help identify any imminent failure and help handle the backup and restore, in time. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "--------\n",
    "\n",
    "Here's a quick look of how this problem has been approached: \n",
    "\n",
    "### Extraction and Load\n",
    "1. Connect to the postgres server.\n",
    "2. Download the dataset offline\n",
    "\n",
    "### Transform\n",
    "3. Wrangle and explore\n",
    "4. Change Dimentions, clean and slice and dice\n",
    "\n",
    "### Analyze\n",
    "5. Analyze dataset, plot most significant trends\n",
    "\n",
    "### Predict:\n",
    "6. Feature Selection\n",
    "7. Model and predict\n",
    "\n",
    "### Conclusion and Improvement Ideas:\n",
    "8. Conclude\n",
    "9. Challenges with the current dataset and ways to improve it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "## Extraction and Load \n",
    "\n",
    "1. Connect to the postgres server\n",
    "-----------------\n",
    "* I'll begin by importing libraries to connect to postgres and download the dataset offline. \n",
    "* I will create a few database utility funtion get the table data and columns\n",
    "* next up, I will use pandas to join columns and dataset and transform incoming data into a pandas dataframe. \n",
    "* Lastly I will save the data locally in a csv format.\n",
    "\n",
    "\n",
    "Next up, I will beging wrangling and exploring the data to understand different attributes that will be used later on in analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "import psycopg2\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "from sklearn import ensemble, metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connecting to postgresql server...\n",
      "Successfully connected to the host\n",
      "\n",
      "Extracting list of tables:\n",
      "\n",
      "Reading table: hard_drive_stats...\n",
      "\n",
      "Reading table: hard_drive_stats...\n",
      "Fetching columns in:  hard_drive_stats\n",
      "Saving data from table: hard_drive_stats, to file: out_data_from_tablename_hard_drive_stats.csv\n"
     ]
    }
   ],
   "source": [
    "#### postgres database utility functions\n",
    "####\n",
    "\n",
    "# db connection object generator\n",
    "def postgres_db_connection(): \n",
    "    # postgresql://35.230.114.237\", \"postgres\", \"luuj\"\n",
    "    conn = psycopg2.connect(host=\"35.230.114.237\", dbname=\"postgres\", \n",
    "    user=\"candidate\", password=\"luuj\")\n",
    "    print('Connecting to postgresql server...')\n",
    "    cur = conn.cursor()\n",
    "    print('Successfully connected to the host\\n')\n",
    "    return cur\n",
    "\n",
    "\n",
    "def get_all_tables(cur):\n",
    "    print('Extracting list of tables:')\n",
    "    cur.execute(\"SELECT * FROM pg_catalog.pg_tables where schemaname NOT IN ('pg_catalog', 'information_schema')\")\n",
    "    tables = cur.fetchall()\n",
    "    t = [i[1] for i in tables]\n",
    "\treturn t\n",
    "\n",
    "\n",
    "def lookup_a_table(cur, tablename):\n",
    "\t# get data from a given table: tablename\n",
    "\tprint(\"\\nReading table: \"+tablename+\"...\")\n",
    "    # cur.execute('SELECT * from '+tablename+' limit 10')\n",
    "\n",
    "\t# get table_data\n",
    "\tcur.execute(\"SELECT * from \"+tablename+' limit 10')\n",
    "\ttable_data = cur.fetchall()\n",
    "\treturn table_data\n",
    "\n",
    "\n",
    "def get_table_columns(cur, tablename):\n",
    "\n",
    "\t# get column_names\n",
    "\tprint('Fetching columns in: ', tablename)\n",
    "\ttry: \n",
    "\t\tcur.execute(\"SELECT table_name, column_name from information_schema.columns where table_name = '\"+tablename+\"'\")\n",
    "\t\tcolumn_names = cur.fetchall()\n",
    "\t\tcolumn_names = [j[1] for j in column_names]\n",
    "\texcept:\n",
    "\t\tprint('Column fetch failed')\n",
    "\n",
    "\treturn column_names\n",
    "\n",
    "\n",
    "# transform data in pandas and save table locally for offline analysis\n",
    "def clean_response(table, data, column_names):\n",
    "\t# inp: table data and column_names\n",
    "\t# out: pandas dataframe\n",
    "\n",
    "\tdata = pd.DataFrame(data)\n",
    "\tdata.columns = [column_names]\n",
    "\tout_file = 'out_data_from_tablename_'+table+'.csv'\n",
    "\tprint('Saving data from table: {}, to file: {}'.format(table, out_file))\n",
    "# \tdata.to_csv(out_file, index=False, encoding='utf-8')\n",
    "    \n",
    "\n",
    "# Etracting all tables at the host in a list and finally,\n",
    "# extracting the table we want i.e. 'hard_drive_stats\n",
    "db_conn_obj = postgres_db_connection()\n",
    "tables = get_all_tables(db_conn_obj)\n",
    "\n",
    "table = 'hard_drive_stats'\n",
    "data = lookup_a_table(db_conn_obj, table)\n",
    "table_data = lookup_a_table(db_conn_obj, table)\n",
    "table_column_names = get_table_columns(db_conn_obj, table)\n",
    "\n",
    "# transform data in pandas\n",
    "clean_response(table, table_data, table_column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of the above `code` snippet, data is downloaded and saved locally to current directory. \n",
    "\n",
    "Name: `out_data_from_tablename_hard_drive_stats.csv`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------\n",
    "## Transform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Now, the dataset is downloaded. \n",
    "2. Filename is: out_data_from_tablename_hard_drive_stats.csv\n",
    "3. We shall be be using this file going forward, in order to avoid calling the postgres again and again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row.names</th>\n",
       "      <th>date</th>\n",
       "      <th>serial_number</th>\n",
       "      <th>model</th>\n",
       "      <th>capacity_bytes</th>\n",
       "      <th>failure</th>\n",
       "      <th>read_error_rate</th>\n",
       "      <th>throughput_performance</th>\n",
       "      <th>spin_up_time</th>\n",
       "      <th>start_stop_count</th>\n",
       "      <th>reallocated_sector</th>\n",
       "      <th>seek_time_performance</th>\n",
       "      <th>power_on_hours</th>\n",
       "      <th>power_cycle_count</th>\n",
       "      <th>reported_uncorrect</th>\n",
       "      <th>command_timeout</th>\n",
       "      <th>high_fly_writes</th>\n",
       "      <th>airflow_temprature</th>\n",
       "      <th>load_cycle_count</th>\n",
       "      <th>total_lbas_written</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1865121</td>\n",
       "      <td>2018-01-20</td>\n",
       "      <td>ZA11RRZY</td>\n",
       "      <td>ST8000DM002</td>\n",
       "      <td>8001563222016</td>\n",
       "      <td>0</td>\n",
       "      <td>209151808.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13874.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>3888.0</td>\n",
       "      <td>5.189666e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1865122</td>\n",
       "      <td>2018-01-20</td>\n",
       "      <td>PL1331LAHD3Y7H</td>\n",
       "      <td>HGST HMS5C4040BLE640</td>\n",
       "      <td>4000787030016</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>14041.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>143.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1865123</td>\n",
       "      <td>2018-01-20</td>\n",
       "      <td>ZA174A42</td>\n",
       "      <td>ST8000NM0055</td>\n",
       "      <td>8001563222016</td>\n",
       "      <td>0</td>\n",
       "      <td>28504744.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4726.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>716.0</td>\n",
       "      <td>2.152110e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1865124</td>\n",
       "      <td>2018-01-20</td>\n",
       "      <td>PL1331LAHGB3VH</td>\n",
       "      <td>HGST HMS5C4040ALE640</td>\n",
       "      <td>4000787030016</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>431.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>8475.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1865125</td>\n",
       "      <td>2018-01-20</td>\n",
       "      <td>ZA14ELXG</td>\n",
       "      <td>ST8000NM0055</td>\n",
       "      <td>8001563222016</td>\n",
       "      <td>0</td>\n",
       "      <td>77116864.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7309.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>2380.0</td>\n",
       "      <td>3.561304e+10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   row.names        date   serial_number                 model  \\\n",
       "0    1865121  2018-01-20        ZA11RRZY           ST8000DM002   \n",
       "1    1865122  2018-01-20  PL1331LAHD3Y7H  HGST HMS5C4040BLE640   \n",
       "2    1865123  2018-01-20        ZA174A42          ST8000NM0055   \n",
       "3    1865124  2018-01-20  PL1331LAHGB3VH  HGST HMS5C4040ALE640   \n",
       "4    1865125  2018-01-20        ZA14ELXG          ST8000NM0055   \n",
       "\n",
       "   capacity_bytes  failure  read_error_rate  throughput_performance  \\\n",
       "0   8001563222016        0      209151808.0                     NaN   \n",
       "1   4000787030016        0              0.0                   106.0   \n",
       "2   8001563222016        0       28504744.0                     NaN   \n",
       "3   4000787030016        0              0.0                   104.0   \n",
       "4   8001563222016        0       77116864.0                     NaN   \n",
       "\n",
       "   spin_up_time  start_stop_count  reallocated_sector  seek_time_performance  \\\n",
       "0           0.0               3.0                 0.0                    NaN   \n",
       "1           0.0               4.0                 0.0                   42.0   \n",
       "2           0.0               4.0                 0.0                    NaN   \n",
       "3         431.0               5.0                 0.0                   42.0   \n",
       "4           0.0               3.0                 0.0                    NaN   \n",
       "\n",
       "   power_on_hours  power_cycle_count  reported_uncorrect  command_timeout  \\\n",
       "0         13874.0                3.0                 0.0              0.0   \n",
       "1         14041.0                4.0                 NaN              NaN   \n",
       "2          4726.0                4.0                 0.0              0.0   \n",
       "3          8475.0                5.0                 NaN              NaN   \n",
       "4          7309.0                3.0                 0.0              0.0   \n",
       "\n",
       "   high_fly_writes  airflow_temprature  load_cycle_count  total_lbas_written  \n",
       "0              0.0                29.0            3888.0        5.189666e+10  \n",
       "1              NaN                 NaN             143.0                 NaN  \n",
       "2              0.0                35.0             716.0        2.152110e+10  \n",
       "3              NaN                 NaN              11.0                 NaN  \n",
       "4              0.0                34.0            2380.0        3.561304e+10  "
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# loading dataset from local machine\n",
    "df = pd.read_csv('out_data_from_tablename_hard_drive_stats.csv')\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get basic look of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows are: 8949492 and number of columns: 20\n",
      "\n",
      "row.names                   int64\n",
      "date                       object\n",
      "serial_number              object\n",
      "model                      object\n",
      "capacity_bytes              int64\n",
      "failure                     int64\n",
      "read_error_rate           float64\n",
      "throughput_performance    float64\n",
      "spin_up_time              float64\n",
      "start_stop_count          float64\n",
      "reallocated_sector        float64\n",
      "seek_time_performance     float64\n",
      "power_on_hours            float64\n",
      "power_cycle_count         float64\n",
      "reported_uncorrect        float64\n",
      "command_timeout           float64\n",
      "high_fly_writes           float64\n",
      "airflow_temprature        float64\n",
      "load_cycle_count          float64\n",
      "total_lbas_written        float64\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "# number of rows\n",
    "rows = df.shape[0]\n",
    "columns = df.shape[1]\n",
    "\n",
    "print('Number of rows are: {} and number of columns: {}\\n'.format(rows, columns))\n",
    "print(df.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More wrangling of  the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First up, I get rid of some irrelevant columns and then indentify the top 10 hard drives.\n",
    "\n",
    "I will apply some cleaning on the columns, changing dtypes and more. Next, I discard/drop columns based on:\n",
    "1. high number of Nan\n",
    "2. irrelevance\n",
    "3. top 10 models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop everything where 10 or more rows are Nan\n",
    "df = df.dropna(thresh=10, axis=0)\n",
    "\n",
    "\n",
    "# drop column row.names\n",
    "df = df.drop(['row.names'], axis=1)\n",
    "\n",
    "\n",
    "# change data type\n",
    "new_date = pd.to_datetime(df['date'])\n",
    "df['date'] = new_date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data types now look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8949141, 19)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 10 most common hard drives\n",
    "* After getting rid of some of the duplicates.\n",
    "* Now based the dataset, I am making the following assumptions:\n",
    "  1. Hard drives with number of datapoints are the most common hard drives.\n",
    "  2. Since there are multiple serial_numbers that belong to the same Hard Drive model, I am taking a unique count only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting a list of the msot common drives\n",
    "most_common_models = df.groupby(['model'], as_index=True)['model', 'date']. size()\n",
    "most_common_models = most_common_models.sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most Common Models in descending order are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 53 models:  53\n"
     ]
    }
   ],
   "source": [
    "# number of different models \n",
    "print(\"There are 53 models: \", len(most_common_models))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Top 10 Models based on most number of Hard Drives are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "model\n",
       "ST4000DM000                2822270\n",
       "HGST HMS5C4040BLE640       1363173\n",
       "ST12000NM0007              1296241\n",
       "ST8000NM0055               1293502\n",
       "ST8000DM002                 888733\n",
       "HGST HMS5C4040ALE640        505026\n",
       "ST6000DX000                 169017\n",
       "Hitachi HDS5C4040ALE630     115984\n",
       "ST10000NM0086               109738\n",
       "HGST HUH728080ALE600         94024\n",
       "dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_models = most_common_models[:10]\n",
    "top10_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 10 common models and the number of hard Drives in each:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAGHCAYAAAB1W7/sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm8XEWZ//HPFwIIyE5UJGAQgiyCLBFwcBQBIYAKjiLgjIQRxVEQGFfwNzOAK4y44QAOI5jgAiIoRNkFhNEBTFgkhChkACHCsIZFUTHk+f1R1eTcpvveJCdd58T7fb9e/crpOt1dT/pW99PnVJ0qRQRmZmZ1LNd0AGZmtuxzMjEzs9qcTMzMrDYnEzMzq83JxMzManMyMTOz2pxMzMysNicTMzOrzcnEzMxqG9N0AKWsu+66MX78+KbDMDNbptx0002PRsTYkR43apLJ+PHjmTFjRtNhmJktUyT9dlEe59NcZmZWm5OJmZnV5mRiZma1OZmYmVltTiZmZlabk4mZmdXmZGJmZrU5mZiZWW2j5qLFRTH+mItrPf/eE/dZSpGYmS1bfGRiZma1OZmYmVltTiZmZlabk4mZmdXmZGJmZrU5mZiZWW1OJmZmVpuTiZmZ1eZkYmZmtTmZmJlZbU4mZmZWm5OJmZnV5mRiZma1OZmYmVltTiZmZlabk4mZmdXmZGJmZrU5mZiZWW1OJmZmVpuTiZmZ1TZiMpG0gaRrJM2WNEvSUbn8eEm/k3Rrvu1dec6xkuZI+o2kPSvlk3LZHEnHVMo3knSjpLskfV/Sirl8pXx/Tt4/fqQ6zMysvEU5MpkPfDQiNgd2Ag6XtEXe95WI2CbfLgHI+w4EtgQmAadJWl7S8sCpwF7AFsBBldc5Kb/WBGAecGguPxSYFxGbAF/Jj+tbxxK/C2ZmVsuIySQiHoyIm/P208BsYP1hnrIvcG5E/Dki7gHmADvk25yIuDsingXOBfaVJGBX4Pz8/KnAfpXXmpq3zwd2y4/vV4eZmTVgsfpM8mmmbYEbc9ERkm6TdJaktXLZ+sD9lafNzWX9ytcBnoiI+V3lQ14r738yP77fa3XHe5ikGZJmPPLII4vzXzUzs8WwyMlE0ouBC4CjI+Ip4HRgY2Ab4EHgS52H9nh6LEH5krzW0IKIMyJiYkRMHDt2bI+nmJnZ0rBIyUTSCqRE8t2I+CFARDwUEc9FxALgv1h4mmkusEHl6eOAB4YpfxRYU9KYrvIhr5X3rwE8PsxrmZlZAxZlNJeAM4HZEfHlSvl6lYe9Hbg9b08DDswjsTYCJgC/BKYDE/LIrRVJHejTIiKAa4B35udPBi6qvNbkvP1O4Or8+H51mJlZA8aM/BB2Bt4DzJR0ay77FGk01jak00v3Ah8AiIhZks4D7iCNBDs8Ip4DkHQEcDmwPHBWRMzKr/dJ4FxJnwVuISUv8r/fljSHdERy4Eh1mJlZeUo/9P/6TZw4MWbMmDHsY8Yfc3GtOu49cZ9azzczaxtJN0XExJEe5yvgzcysNicTMzOrzcnEzMxqczIxM7PanEzMzKw2JxMzM6vNycTMzGpzMjEzs9qcTMzMrDYnEzMzq83JxMzManMyMTOz2pxMzMysNicTMzOrzcnEzMxqczIxM7PanEzMzKw2JxMzM6vNycTMzGpzMjEzs9qcTMzMrDYnEzMzq83JxMzManMyMTOz2pxMzMysNicTMzOrzcnEzMxqGzGZSNpA0jWSZkuaJemoXL62pCsl3ZX/XSuXS9IpkuZIuk3SdpXXmpwff5ekyZXy7SXNzM85RZKWtA4zMytvUY5M5gMfjYjNgZ2AwyVtARwDXBURE4Cr8n2AvYAJ+XYYcDqkxAAcB+wI7AAc10kO+TGHVZ43KZcvVh1mZtaMEZNJRDwYETfn7aeB2cD6wL7A1PywqcB+eXtf4OxIbgDWlLQesCdwZUQ8HhHzgCuBSXnf6hFxfUQEcHbXay1OHWZm1oDF6jORNB7YFrgReGlEPAgp4QAvyQ9bH7i/8rS5uWy48rk9ylmCOszMrAGLnEwkvRi4ADg6Ip4a7qE9ymIJyocNZ1GeI+kwSTMkzXjkkUdGeEkzM1tSi5RMJK1ASiTfjYgf5uKHOqeW8r8P5/K5wAaVp48DHhihfFyP8iWpY4iIOCMiJkbExLFjxy7Kf9XMzJbAoozmEnAmMDsivlzZNQ3ojMiaDFxUKT84j7jaCXgyn6K6HNhD0lq5430P4PK872lJO+W6Du56rcWpw8zMGjBmER6zM/AeYKakW3PZp4ATgfMkHQrcB+yf910C7A3MAZ4B/hEgIh6X9Blgen7cpyPi8bz9QWAKsDJwab6xuHWYmVkzRkwmEfFzevdRAOzW4/EBHN7ntc4CzupRPgN4dY/yxxa3DjMzK89XwJuZWW1OJmZmVpuTiZmZ1eZkYmZmtTmZmJlZbU4mZmZWm5OJmZnV5mRiZma1OZmYmVltTiZmZlabk4mZmdXmZGJmZrU5mZiZWW1OJmZmVpuTiZmZ1eZkYmZmtTmZmJlZbU4mZmZWm5OJmZnV5mRiZma1OZmYmVltTiZmZlabk4mZmdXmZGJmZrU5mZiZWW1OJmZmVpuTiZmZ1eZkYmZmtY2YTCSdJelhSbdXyo6X9DtJt+bb3pV9x0qaI+k3kvaslE/KZXMkHVMp30jSjZLukvR9SSvm8pXy/Tl5//iR6jAzs2YsypHJFGBSj/KvRMQ2+XYJgKQtgAOBLfNzTpO0vKTlgVOBvYAtgIPyYwFOyq81AZgHHJrLDwXmRcQmwFfy4/rWsXj/bTMzW5pGTCYRcR3w+CK+3r7AuRHx54i4B5gD7JBvcyLi7oh4FjgX2FeSgF2B8/PzpwL7VV5rat4+H9gtP75fHWZm1pA6fSZHSLotnwZbK5etD9xfeczcXNavfB3giYiY31U+5LXy/ifz4/u91gtIOkzSDEkzHnnkkSX7X5qZ2YiWNJmcDmwMbAM8CHwpl6vHY2MJypfktV5YGHFGREyMiIljx47t9RAzM1sKliiZRMRDEfFcRCwA/ouFp5nmAhtUHjoOeGCY8keBNSWN6Sof8lp5/xqk0239XsvMzBqyRMlE0nqVu28HOiO9pgEH5pFYGwETgF8C04EJeeTWiqQO9GkREcA1wDvz8ycDF1Vea3LefidwdX58vzrMzKwhY0Z6gKRzgF2AdSXNBY4DdpG0Den00r3ABwAiYpak84A7gPnA4RHxXH6dI4DLgeWBsyJiVq7ik8C5kj4L3AKcmcvPBL4taQ7piOTAkeowM7NmKP3Y/+s3ceLEmDFjxrCPGX/MxbXquPfEfWo938ysbSTdFBETR3qcr4A3M7PanEzMzKw2JxMzM6vNycTMzGobcTSXFXb8GkvhNZ6s/xpmZovBRyZmZlabk4mZmdXmZGJmZrU5mZiZWW3ugLeetpq6Va3nz5w8s3YMszfbvPZrbP7r2bVfw8xG5iMTMzOrzcnEzMxqczIxM7PanEzMzKw2JxMzM6vNycTMzGpzMjEzs9qcTMzMrDYnEzMzq83JxMzManMyMTOz2pxMzMysNicTMzOrzcnEzMxqczIxM7PanEzMzKw2JxMzM6vNycTMzGobMZlIOkvSw5Jur5StLelKSXflf9fK5ZJ0iqQ5km6TtF3lOZPz4++SNLlSvr2kmfk5p0jSktZhZmbNWJQjkynApK6yY4CrImICcFW+D7AXMCHfDgNOh5QYgOOAHYEdgOM6ySE/5rDK8yYtSR1mZtacEZNJRFwHPN5VvC8wNW9PBfarlJ8dyQ3AmpLWA/YEroyIxyNiHnAlMCnvWz0iro+IAM7ueq3FqcPMzBqypH0mL42IBwHyvy/J5esD91ceNzeXDVc+t0f5ktTxApIOkzRD0oxHHnlksf6DZma26JZ2B7x6lMUSlC9JHS8sjDgjIiZGxMSxY8eO8LJmZrakljSZPNQ5tZT/fTiXzwU2qDxuHPDACOXjepQvSR1mZtaQJU0m04DOiKzJwEWV8oPziKudgCfzKarLgT0krZU73vcALs/7npa0Ux7FdXDXay1OHWZm1pAxIz1A0jnALsC6kuaSRmWdCJwn6VDgPmD//PBLgL2BOcAzwD8CRMTjkj4DTM+P+3REdDr1P0gaMbYycGm+sbh1mJlZc0ZMJhFxUJ9du/V4bACH93mds4CzepTPAF7do/yxxa3DzMya4SvgzcysNicTMzOrbcTTXGaj2an/dHXt1zj8G7suhUjM2s1HJmZmVpuTiZmZ1eZkYmZmtTmZmJlZbU4mZmZWm5OJmZnV5mRiZma1OZmYmVltTiZmZlabk4mZmdXmZGJmZrU5mZiZWW1OJmZmVpuTiZmZ1eZkYmZmtTmZmJlZbU4mZmZWm5OJmZnV5mRiZma1OZmYmVltTiZmZlabk4mZmdXmZGJmZrU5mZiZWW21komkeyXNlHSrpBm5bG1JV0q6K/+7Vi6XpFMkzZF0m6TtKq8zOT/+LkmTK+Xb59efk5+r4eowM7NmLI0jkzdFxDYRMTHfPwa4KiImAFfl+wB7ARPy7TDgdEiJATgO2BHYATiukhxOz4/tPG/SCHWYmVkDBnGaa19gat6eCuxXKT87khuANSWtB+wJXBkRj0fEPOBKYFLet3pEXB8RAZzd9Vq96jAzswaMqfn8AK6QFMB/RsQZwEsj4kGAiHhQ0kvyY9cH7q88d24uG658bo9yhqnD7K/Slw54S+3X+Oj3f7IUIjHrrW4y2TkiHshf5ldK+vUwj1WPsliC8kUm6TDSaTI23HDDxXmqmZkthlqnuSLigfzvw8CPSH0eD+VTVOR/H84PnwtsUHn6OOCBEcrH9ShnmDq64zsjIiZGxMSxY8cu6X/TzMxGsMTJRNKqklbrbAN7ALcD04DOiKzJwEV5expwcB7VtRPwZD5VdTmwh6S1csf7HsDled/TknbKo7gO7nqtXnWYmVkD6pzmeinwozxadwzwvYi4TNJ04DxJhwL3Afvnx18C7A3MAZ4B/hEgIh6X9Blgen7cpyPi8bz9QWAKsDJwab4BnNinDjMza8ASJ5OIuBt4TY/yx4DdepQHcHif1zoLOKtH+Qzg1Ytah5mZNcNXwJuZWW1OJmZmVpuTiZmZ1eZkYmZmtTmZmJlZbU4mZmZWm5OJmZnV5mRiZma1OZmYmVltTiZmZlabk4mZmdXmZGJmZrU5mZiZWW1OJmZmVpuTiZmZ1eZkYmZmtTmZmJlZbU4mZmZWm5OJmZnVtsRrwJvZ6DL3mP+u/RrjTvzbpRCJtZGPTMzMrDYnEzMzq83JxMzManOfiZktU44//vjGX+OqqzeuHcNuu/5v7ddoEycTM7Nl1MuuubX2a/zfm7ZZCpH4NJeZmS0FTiZmZlabk4mZmdW2TCcTSZMk/UbSHEnHNB2PmdlotcwmE0nLA6cCewFbAAdJ2qLZqMzMRqdlNpkAOwBzIuLuiHgWOBfYt+GYzMxGJUVE0zEsEUnvBCZFxPvy/fcAO0bEEZXHHAYclu++CvhNzWrXBR6t+Rp1tSEGaEccbYgB2hFHG2KAdsTRhhigHXEsjRheERFjR3rQsnydiXqUDcmMEXEGcMZSq1CaERETl9brLasxtCWONsTQljjaEENb4mhDDG2Jo2QMy/JprrnABpX744AHGorFzGxUW5aTyXRggqSNJK0IHAhMazgmM7NRaZk9zRUR8yUdAVwOLA+cFRGzBlztUjtlVkMbYoB2xNGGGKAdcbQhBmhHHG2IAdoRR7EYltkOeDMza49l+TSXmZm1hJOJmZnV5mRiZma1LbMd8KOJpLWBiIh5Tcdi7eF2YVWSNiPNArI+6Zq7B4BpETG7RP0+MmkpSRtKOlfSI8CNwHRJD+ey8YVjWbNkfdZfm9pFJaa1Ja3VQL1jJH1A0mWSbpP0K0mXSvonSSuUjqcS1+qSti/5nkj6JGlKKQG/JF06IeCcUpPgejTXMCTtCezH0Ex/UURcVqDu64GvAudHxHO5bHlgf+DoiNhp0DFUYpkP/Aw4B7ggIp4oVXclhq0j4ra8vQLwSdL8bLcDn42IZwrFsSfpAtmrIuLeSvl7I+KsAvW3ol1I2hD4d2A34AnSF9fqwNXAMdX3ZoAxnJPrnkq6iBnS32YysHZEHDDoGHIc3yG994/m9vFN0tRNE4CPRcQPCsRwJ7BlRPylq3xFYFZETBh0DESEbz1upA/sJaSLIV+fbwfmsq8VqP+uJdk3oFhmAm8Bvgs8BlyU34uVC8Zwc2X7S8AU4I3AV4CzC8XweeC63Db+F/hwr/hGQ7sArgcOAJavlC2f28UNhWL4zTD77iz4XsysbP8PMD5vrwv8qlAMvybNodVd/orh3qeleXOfSX97R8Sm3YWSvg/cCRw14PpvknQa6VfX/blsA9KvrlsGXHe3v0TET4CfSFoZeCvpS+NUSZdHxLsLxFCdi2034LUR8RdJ1wG/KlA/pP/3tpEumD0e+J6kV0bEP3fFN0htaRfrRsT3qwWRjpTOlfSZQjHMk7Q/6Wh5AYCk5UhHaSX7kZaTtHpEPAUsAO4DiHSkUuo79mjgKkl3sbBdbAhsAhzR91lLkZNJf3+StENE/LKr/LXAnwrUfzBwKHAC6TSbSIfy04AzC9Rf9fwXZUT8ETgPOE/SGqTTgCWsIentpH6+lSIfzkdESCp1rnZMRMzP9T4h6a3AGZJ+AKxYKIa2tIs2JLUDgZOA0yTNI70Xa5JOtR1YKAZIf4trJJ0K/AL4gaSLgF2BgZ8SB4iIyyRtSjr1W20X03OSHzj3mfQhaTvgdGA1Fp6P3QB4CvhQRNzUVGylSfpYRJzccAzf6io6JiIekvQy4LsRsVuBGH4CfDEiru0q/yzwqYgYNQNa8rn4Q1k4emhIUouIPxeOZx3S91kjU75LmgC8D9iU9CN9LnBhRFxeMAaxMJl0+nh/GYW+5J1MRpC/rJ7/sETE/zUYy529Tr1ZGfkUX+forHvf+hHxu/JRje52IWkH0gHq9LzS6iRgdkRc2nBoRUnaAzgNuAvotMNxpNNcH4qIKwYeg5NJf/k0ziSGZvrLo8BoJklP5zqr5+JXAZ4hfXhWH3QMlVjWAI4lndLqLJLzMKkj/sQS70ePmDYCtgXuiIhfF6y3sTaR629Fu8h9AYfSY7Qj6cjkL8M8fWnFcBxp2e4xwJXAjqRRh7uT/iafG3QMOY5XAv9C+v+fSBoU8jpgNvDxKDOybTawV3dd+XNySURsPugYRs1h+eKSdDBwM7AL6cO6KvAm0rnigwuEMAW4ENgkIlaLiNWA+/J2sUSSnUfq0NwlItaJiHVI78U8YODDHgEkXVjZ3pd0XvytwEWSDikUQ9NtAtrTLr4NbEPqL9gb2Cdvvwb4TqEY3gnsDLwBOBzYLyI+DexJGmlWyhTSdR2/B24gDQvei9RfMvDh4lnn1Fq33wFlrrkpMWRsWbyRGsSaPcrXotCwQ2B70pfmkaTEf3dT78WS7FvKMdxS2f4fYKO8XXL4ZeNtoi3tYoQ2UerzcUuv7Xz/1oLvRTWO+/rtG3AMx5IGPnwSeHe+HZPLji0Rg49M+hNdywBnCyg0DDRSJ//u+e61wItK1NvDbyV9QtJLOwWSXpqvur1/mOctTdW/xZiIuAfS8EvS36SExtsEtKZdzJO0fx6KC6RhuZIOoNyw3GclrZK3t6/EsQbl2gTAAkmbSnotsIqkiTmOTUjX3gxcRHyBlEBEOsX2N3nX3+d9A+ehwf19DrhZ0hUMHbf9ZqDUOHoijZ8/JQ8/3bZUvV0OIP3KubaSUP6PNHLnXYVieI2kp0gflpUkvSwi/i+PKirygaUlbQJa0S7aMCz3DZFHjeX3o2MF0hDlUj4B/JiUwPYDjpX0GtKMAO8vFUSkObien4dL0joR8Vip+t0BP4w8t86eDB36eHkUmFgvT1fxcET8KQ/5OwTYDrgD+K/I1zuMdkrzhm0eEdcXqq+xNpHrb127aHpYbiWOTUh9NrMj4o6GY1kXmBelrvGQTgROjnSh5PakvsznSNc/HRxdw9kHEoOTyfDyL/HnR6tExEOF6r0d2CEinpF0ErAxqeN1V4CIeG+JOCrxNDZPWVccjfw92hJDm9qFes9Se1EUGl0n6Rpg//wF+h7gX0nT3ewInBERXy8RR46l6fdiZkRslbevAT4Rabj0psD3ImLioGPwaa4+JG0DfANYg/TrU8A4SU+Qxm3fPOAQlouFkxfuTpo+ZAHwHUmlpg8BQNJXSRdjnc3QCfWOlLRXRAx6ahkkbUu6iHQNKuPo89/jgxEx8KuuW9AmoCXtIveXHUSaqbYzS8Q40nQq50bEiQXCGFs5GjoSeF1EPJb7UW4AiiSTlrwXK0jqzNCwckRMB4iIOyWtVKB+j+bqdwNuBXbsUb4TBUYPAZcDu+btC8iTuAHrlKi/K5aeo3NIX6ZFJhds+u/Rohha0S5I89Ot0KN8xYJt4hZg/bx9DfCivL08aabc0fRefBi4gnSEejxpMtI3kIZrf7tEDD4y6W/ViLixuzAibpC0aoH63wecrTSh4JPArZJuIQ1D/WiB+quanqcMmv97tCWGtrSLBcDLgd92la9HuZFU/wxcIekCYBZwtaTLgL8FuqffGaTG34uI+LqkmcAHWTily6akU6CfLRGD+0z6kHQK6Xz02QydyO5g4J6IKDITp6TNGTrfz/QYOnKlRAyNz1PWhr9HG2KoxNJou5A0CfgP0vQd3bPUfjgKTWeShwG/m6HvRbG+ihzDcO/FEVG4X7EpTibDkLQ38Da6JrKLiEsK1L1Z5wMhaaWoTJwnaaeIuGHQMfSIqdF5yiTtRY+JBUv8PSoxNNYmcv2taRf5GpPGZqkdTqX/oFR9jb4Xkr4aEUfn7aMi4muVfVMi4pCBx+Bk0k6Sbo6I7bq3e90vFE+jc1JZ0pZ2Iem1kTt5e+x7T0R8u0AMP4+I1+ftb0fEeyr7in5G8jDt5mbsbUG7cJ9JH5KmDbc/It426BD6bPe6P9hA0rxTx5E6+Dojqd4EfF7SCRFxdoEYOpNN7gu8JBcXnWyyBW0C2tMuzpT0C9JUHU8ASHo1aebax0lzdw1atZ/q1V37ir0XGmbGXklFZuxl+HZRhJNJf68jnf88B7iR8n+g6LPd6/6g/T9g++4v7HwB342kPoRBO490dfWbOqfX8mm3Q0gXaL25QAxNtwloT7vYDvg4cIvSyopbkSZ7/EikVTlLaMt78TVg9+gzYy8w8Bl7Sas9rkWaq62z3WmfRWaIcDLp72WkL6iDSB18FwPnRMSsQvWPyx2+qmyT769fKIaONsxJNT4iTqoW5KRyoqR/LBRD020CWtIucn/EFyTNB75JOq2zQ0Q8UCoGYE0tXH1zTUl/l8tFuhaolOZn7E3/35tY+HmsXvNUJLE6mfSRO84uAy7LF/0cBPxM0qejzJW1H69sz+ja131/0NowJ9VvJX0CmBr5ivN8JfohFJpssgVtAlrSLiRtTDq18xzpl/dewHWSPhcRpYblXksaDNHZfmtl33WFYoA0zfx0SecydJTfgRRaSjkixvfbJ6nIjwx3wA8jf2HsQ/rSGE+a2PCsaGhFvUpcJ0fExwrXuTawB83NSbUWabLJfYGXkn5tPUT6m5wUEY8XiqOVbQLKtgtJc0hLJ59fKXs58GVgg4jYuUQc/Uh6R0RcULC+Leg9yq/ROcIAJN0XERsOvB4nk94kTSV16l0KnBsRtzcc0vNKNQ4bqs1tAsq2C0kvjojf99m3e0T8tEQc/fgzspCk+yNig4HX42TSm6QFwB8qRZ03ShReNrdbqcZRqa+zVOwLdlHovZD0keH2R8SXC8TQ2jYBjbSLMaTTW5vlotnAZSWv7+in5Hsh6bZ+u0jtYusScfRTKrG6z6SPiGh04bB8WqnnLsqPIrqK1Pn8Q+D7EdE9bUQJJ5PmxroU+DMNjKRquk1Ae9pFPqV1DfAgaY4sAW8BviTpTYU74nsp+St5Qa7ve6R1Tf5YsG4AJH2d/j/41iwSg49MhidpKxb+8rqj1MgdSfeQGkevL4iIiFeWiKMSzxrA35E6FV8EfJ90qqdUX8U2ue5JpFEr5wBXlboorCuWRtpErnu4dkFEbFQojimkpXG/2lV+JGkY+cAXp8pzUfX7At00IsrMlpti2YzUj/ZW0toy3wOuKHWUJmnY9zsipg48BieT3vKX50WkUUu/IjXQrYD7gH0j4qkGw2tMnjbiANL03p8vcXqpRwx/Q/rg7g58MiKGvZhwKdbrNpFJ+nVEbNZn328i4lUFYnjFcPsbOoJGaeniU0kDQ77YRAyVWF4EvDUifjDounyaq7/PkIZa7tqZQC9/kZ5IGir74UEHkM9JPxcRIWkD0qI/cyLi1kHX3SOWzhf43wI/B94eEf/dQBxjScvUbkUaMfNwweobbxO95GG6BwIHRUT3leCDMtypnGeG2bfU9EsWknYmXQd0eIk4cp3rk/4GbwfmkWY0/lGp+rtiWZ408vIg0qqg/026sHegnEz62x3YOiozsUbEAkmfAmYOunJJ7yetsf37fIXxx0kXIm0r6azuC/gGHMu9wBOkxX8OA+bn8u0AosCiUPnCxANIp9jOB94VESUTCTTcJqokrUd6P94NbA18gfTlUcoalYsEh4RGWvu8qHwa9N3Au4B7SP17peq+ljSj9nmk6546p35XlLR2wVPBbyC9B/uQFunaGdgoFi6mNtj6fZqrN0m3RsQ2i7tvKdY/C3g9qZHOJi2C9KjSKnLTI2LLQdbfFcvP6N+hGRGxa4EYFpC+sO/r1NsVxMDnxWq6TeR63k9KGuNIX17nkaZcL9JXUolj2AsTI2LgsxIoLUl7IOn9eIzUj/exiBj29NcA4riXhe2x2i47o7kG3r8paS7ps3E6cGFEPC3pnpLtwkcm/b1IaanYXpPplejYezZfEDhP0pzIy5NGWvv72QL1Py8idilZXx9vajoAmm8TkM7FXw+8OyJmAEgq/ouwRLJYBL8mncJ5a0TMAZD0z6WDGO7q84IuAPYjHa0+J+kiCs/h52TS34Okq3l7KbGOx8r5i2s50uFy50tMpFM9xeTD536iRN9JRFw76DoWQdNtAtKKfvsDX87TyZxHufmfhpC0J+kLrDrt+oURcXmhEN7o16+QAAAY/UlEQVRBOjK5RmmFxXNpYMi4pGGv4YiI+4bbvzRExFGSjib96DoI+CKwuqR3AZf0u8B0afJprpaSdM1w+yOi2C91ST/uFQLwGmBcRAx8VtL8fgx3qm23QcfQNnlQxgGkL49VgB9FxKcK1f1V0uqGZ7NwksNxpFUn74qIo0rEkWNZlZTUDiKtgT6V9F6UmPq9OkS5msgCGAu8pMTno0dMK5AuKD0Q2CMi1h14nU4m/eWx452V/Tq/vIouCdpGkl5PmpZ+LeBzEdEr2SztOrfvUbwT8Ang4Yh47aBjyHG0sk1IehVwYEScUKi+OyNi0x7lAu6MiAkl4uhR/9qkI7cDSvTl9YlhPPBJ0oCNU6LcJKCd+tcm/cCal++vHBEDv5DSyaQPSZ8k/dI5l6G/vA4kXax3YoEYWvXFJWk34F9zLJ+PiCsbiuONOY6Vchyl1htvvE3kOHq1i2kRMbtE/TmG24D3RcQvu8p3AM6MiK0KxdHoCoddsUwg/cjaEfgSaYbrvxSqe0Pg34HdSCMvO6PqriZdizXwa26cTPqQdCewZXdjkLQiMGvQv7za8sWVY9mH9CF5EvhsRPyiVN1dcexJSiJ/Ih0RDXsqcAD1N9omcl2taBd5WPjppNGGnTg2AJ4CPhQRNxWIoe8KhzmGUqe5Xk36fGxJ+kI/Jwqt/V6J4Xrgq8D5nbrz9Sb7A0dHxE4Dj8HJpDdJvwb27M7o+arbKwZ9hW8bvrgqdS4gfWH8ih79FoWG5U4nnYP+Imk0U3cMJa51abRN5Lpa0y5yvS+jMu165FUwC9U9G9gr+qxwGBElVjhE0nOkdUwuJq3vMkREHFkghrv6/e2H27c0eTRXf0cDV0m6i6ELQm0CHFGg/gWkkTvdh6fr5X0ltWFY7h+A3wPvzLeqIHW8DlrTbQJa1C7y9DJvpHKKSdLl0bW88wC1YYVDgPcWrKufmySdRhp8UF2gazJpIs6B85HJMPJUGZ3zsZ0Fb6aXOISVNAn4D9Ih/Au+uCLiskHH0CeuIZ17o02TbSLX34p2Ielg4DjgCoaeYnozcEJEnF0ghmNJV7xXVzjckDTC7byI+MKgY2iLfGR6KAv70p5foIvUh/XngcfgZNJf0517TX9xVeLodO7tSuo3qXbuHdN9mmGAcTQ+IKHpNpFjaLxdSPoNsGP3UYjSipg39hrpNaA4NqfHF2gUXOEwH6EdSxqePDYXP0yaFPTEgkdqjfJprj6G69yTVKRzL9IcUDdUYvpQRNwwzFMG5fukzr2/79G5dy5piO5AdXU8d0YQjQPOlVRqdF3jbQJa0y5E7+t+FlDwwsE8gu35UWyStiuZSLLzSD+sdun0GeW+pENIEyy+edABSHol8C+kdnkS8BXgdaT35uMlfvD5yKSPpjv31HtlwU8Bn4cyKwtWYmm8c68NHc9Nt4lcVyvahdL6Gf9GOs1VPcX0ZuAzETGlQAzb9SieRlpTRCUGZeQ4+k65P9y+pRzDdaQ1ftYA/gGYQvoRuAfpR+DA+xR9ZNJf0517JwCXALNY+EtvedJQzNIa79yjHR3PTbcJaEm7iIipkqaRpjjvnGL6GXBswf60GaQjtGp/wDqkKW9KDcoA+K2kT5CuK3kIIE91cwgLPy+DtlpEnJ7r/lBEnJzLz5RUZHCIk0l/ZwHTJVU79zYgjec/s0D9W5I+FKuSOjSfkTS51BXOXQ4mde6dQI/OvUIxtGEkVdNtAlrULiJiXp7m5vn+o8IDM95FWkPmixFxCYDSTLmlRx8eABwDXJuTCKS52qblGEtYoDSL8hrAKpImRsQMSZuQfmwMnE9zDUPSFsDbaLZzb1/SlCFfAf49Ci/X2yYt6XhuvE3kOBptF0rrh3yD9OU1l/RejCNdff2hgqeYXkxatGwc8FHgZ6PxM5JnpziNdJT+ftLiXK8hDZR5f0RcNPAYnExG1vRwWKU1TE4gjZ4ZbgbfQdU/hnRk0j1D7EWkYYelpoxofCRVJZbGh0g32S4k3Qp8ICJu7CrfCfjPiHhN4Xi2JR2xvToixo70+AHU32sG5YuaGsKfY1oXmFds2LqTSW89hsNC+hVWdDhsjqXpZHYO6RfnVIZO4TEZWDsiDigQQ+NTZ7SpTeR4GmsXIwzKmBMRmzQQk0h9B08VrrcVMyg3PXTeyaSPpue60fATt5VOZsONVuk5e+wAYmjDSKrm5z9qSbuQdAqwMekLtNp/dDBwT0QMvB+rcsT8dtLgjKaOmBufQVktmLPNyaSPpofDtuGLqxLLDaRZUC/I1zh0+i/2Bz4SETsWiOEuYPOImN9VviJwR4lfwk23iVxPm9rFXvS+YPCSQvU3fsSc42h8BuVWDJ13Muktj9h5nN7DYdeNiIGO0mjDF1elvvGkC6F2BeaRvjjWZOGv4XsKxNBr6ozOSKoiU2c03SZyDK1pF01rwxFzrqsNMyg3Pwmpk0lvaniumzZ8cfWJax1Su3m0gbobHUnVdJvIMbSiXeRrF86NiEclbQx8C9gKuJP0K31mgRgaP2LuiqfJGZQbn7PNyaSl2vDF1RVP4/NiVWJpfCRVU9rSLiTNiogt8/bFwDcj4keSdiGtNbNzgRjGM/SIGdLqn8WOmCuxrAFMYujno+QMyo0PnXcyWQwlD53bpBWdey0YSaUWzH/UFtVTTJKmR2XZZEm3RcTWheNp8oi58RmUcxwbAk9FxBM50U4EZkfErCL1O5n0Julp0i+M6qR1qwDPkH4Vr95ATI0ks1Z07rWg41ktmP+oT1zF24Wkz5F+AX+a9KPiGeCHpFFm74iIt5SMpxLX2RFxcOE6G59BWdIxwAdIU8ucDHwM+AVpEtYzo8CcbU4mfUj6OulL4+OxcL6deyJio0L1tyaZtaJzrwUdz5JuiYht8/Z9EbFhr30DjqFN7eIQ4IOkIcIrkc7VXwicFBFPDvPUpVX/tO4i0kJuV0OZFUBzHHcCr+3+P+dTXzMKtc1ZpCORVYB7gVdGxCOSViUltFcPOgbPzdVHRHxY0vbAOZIuJHVulcy8U2gwmXVpw7xYrZhsUg3Pf0RL2oWknSLNDDylZL1dxgF3AN9kYYKdSOqUL+lzwM2Ses6gXCiG5yLij5KeBf4IPAYQEX9Il7sMno9MRpA7tY4gnU7ZOCJeXrDu7UlrnneS2ZxoaN6hxjv3WtDxLGlX0hDQxuY/ynE03i4k3RwRvaaALxnDcsBRwN6k5HqrpLub+IzkU1rVGZTnkjrgiwwSkTQFWJE0AegzwHzgMlIf42pFhq07mfSWf3lVFyBaD9i21AVZlXobS2Y2VHebyGVF5z+q1Ntou2hDMumQNI40GOIh4G3V04+F43gpQ2dQfqhg3WNIbSGA80k//N4N3AecGhF/GHgMTia9tenDAs0ls1z31sAZpA/KpcAnO7+4JP0yInYoEEPjI6na1iag0R85TwDX9dtfqr+iStI+wM4R8anC9bZiBuWmuc+kpST9AynZfxsgIh4EHpT0fuAPEfG9guGcBhxPWojofcDPJb0tIv6XcotCTWHhSKob8v1Pk0ZSnUW5hZAa1aJ28Qjl+yaGkPRa0oWalwJExMXAxZLeBvwuClx5nk2h/wzK3yKdCh0oSauT1qEfB1xabQeSTouIDw08Bh+Z9Nb0Ly9JtwBviIinu8pXI63ZsP0g6++q89aI2KZy/02kI5X3AKeV+LXekpFUjf8ab0u7KPWejxDDz4BDuo9K84CIM0oN1R5hpGGRGZQlXUC6+v0G4L3AX4B3R8SfSx1R+8ikv6Z/eS3f/YUBEBFPSyp1NNAhSWt0hj5GxDWS3gFcAKxdKIY2jKRquk1Ae9rFPEkvizxlSL5w7x2kZZWPj4jHC8SwTq/TmxExJ1/EWMqlSrMA9JpBudR6JhtHxDvy9oWS/h9wdT5KK8LJpL/fR8S1Dda/gqRVuzvO8i/QFQvHchKwOelXDwARcZvS6m7/WiiGjwM/Jo2k2g84VtLzI6kKxdB0m4D2tIs1gWdz3W8ATiQtobsN6aj1nQViWHmYfasWqB+AiDhSvWdQPrVgX9ZKkpaLPEdZRHxO0lzSkfSLi0QQEb71uJEufHpZ5f7BpHUSTiFNbz3o+j9G6uweXykbD1xM6nBu/D0q/PfYqUfZuqRf6qOiTbSpXQC3VrZPJR2NvGDfgGP4BukaD3WVn0A6zVXkvWjDjTTV0O49yieRFugaeAzLLaWc9Neo1y+vs0nzQp0x6Moj4mTSF9W1kh6T9ChwLfCTiPjioOtfVJIG/l5kp3UXRMSjUXZIbqNtAlrVLsbk4aiQplC5urqvUAwfBV4JzJF0Qb7NAV4FfKRQDMMq9fmIiE9ExE97lF8WhZYlcAd8H9VOZ0mnAo9ExPHd+wrF8mLS3+oF58oL1d+vX0TAryJiXIEYGh+W26Y2ketsrF3kc/J7A4+SrvbeLiIi92FNjQKzBldieSWwZb47KyLuLlV3rr8Nn48jgR9GxNwRHzyoGJxMepN0O7BNRMxXmpvqsIi4rrMvCsx10xXP60kXIt0eBdY776r7OVLHanVehs70FetHxMDP1bdkJFWr2kSut8l2sROwHml+tj/ksk2BF0cD11bk5LopcHeUnfq9DZ+PJ4E/AP9LGkL/g4h4ZND1VrkDvr9zSKcSHiXNdfPf8PywwxKT2D1/MWC+huBw4EfAcZK2iwLTvlfcDewWEff1iPP+Ho8fhDaMpGq0TeS6WtMuoms2gFx2Z6n6q9dP5KT6PdKX6SaSPhDlOr/b8Pm4G9ge2B04ADhB0k2kNvvDIkevTXcctflGmr757cCqlbJNSYf0g677lsr2dGBs3l4VmFn4fTgceE2ffR8uFMMtJeppc5toW7to+gbcXNm+pvM3IPWjzCgYRxs+Hzd33V+BtCrpOaTTsQOPwUcmw4hmf3ktpzR53HKk05GP5Pr/IGl+oRg6boqIX/XaERFfLxRDG65raLpNQLvaRZusHvnUWkTcrbTWTSlt+HwMmRo40tpD04BpkoYbQr3UeDRXe60B3ATMANZWWl/6+U7XwrG8YCRVAxofSdUSbWoXTdtM0m2SZgKb5iTbmQSz5AWcbfh8HNBvR0T8sUQAPjJpqYgY32fXc6TTLKPNcpWjjwNI1xFcAFwg6dYG4yrK7WKIzbvudy7kXBv4t8KxNO3RYUaVUeLI3cmkpSStAvwlH64i6VWkoZi/jYgfFg7nlXrhqnbPizIzxI6RNCYi5pOuazisuq9A/a3QsnbRqHjhyp8rSNqWNMljyfeiDZ+Pm1g4gmw94AEWHqkGqR9poEbNh3AZdBlpMai78mih64HvAm+R9NqIOLZgLB5J1R5taheNkvQN4OsRMUtpidzrSUdoa0v6WEScUyiUxj8fUVlpUw1NwunrTFpK0syI2Cpvf4Y0XcfhSisO3tTZVyiWxmeIzXG06rqGJrSpXTRN0qyI2DJvHw3sEhH75X6kS0u12bZ8PjqausDXHfDtVc3yuwJXAkTEs6TJDkua1+nohTSSStJFkk4Z7jzt0hYRN0TEj6IyyWFE3DlaEknWpnbRtGcr228mLWNMZ8RfQa34fDTNp7na6zZJJ5NWFtwEuAJA0poNxNKGGWItaVO7aNoTkt5Cei92Jp3+6yxhW2Q4bNb450NSdS6yl3TdJyK+POgYnEza6/3AUaQZYfeIiGdy+RbAyYVj8Uiq9mhTu2jaB0gzNr8MOLpyRLIbaRblUtrw+Vitsv1fXfeLcJ+JjaiNc1KZtYU/H4mPTFpK0m3D7Y+IrUvFgkdStUbL2kXjlJaQPgLYLBfNBv4jIn5WMIzGPx+STukqCtKMztdExM+LxOAjk3bKh8dBmrzux6RG+rzuMfYF4hn1I6naoG3tokmS9gH+A/g0cDPpuortgH8BjohyEz02/vmQNLlH8drAu4DvR8RXBx6Dk0l7SdoMOAh4K3AH6Qvkinzhno1SbheJpJ8BR3XPiyVpa9L1J29sJLAWyfNy/U+JoctOJssISQeQlkc9KVq00qI1azS3C0m/jojNFnffaFPqOhj3mbSYpPWBA0lzLs0D/pm0doWNYm4Xz/vDEu4bFfIQ6fcARVZf9JFJS0m6ljS87zzgfGDIRG2lply3dnG7WEj9V98U8PqIWKtwSI2R9DRDL2iF1J92LWnY9AMDj8HJpJ0k3cvCxlH9IwmIiBj4xG3WPm4XC0katk8kIq4tFYs5mZiZLfMkDTsXV5ERZU4m7dSGxmHt43axUF4Uq+8X2Gi65kbSNZW725OmpO+IiNh14DE4mbRTGxqHtY/bxUKSXtHZJE2fsnd1/2i65qbKU9BbX22b4trawe1ioaamXW8jT0Fvw3HGt17cLqw1fJ2JmS2TuvqPVs5L9naWqh1t/UdfZ+GPi3Hdc3VFxJEDj8Gnudqpq3EcCJxb3V+icVj7uF0s1NV/1G209R/1mpvreRExddAx+MikvWZUtm/q+ygbbdwusoh4U9MxtEWJZDESH5mYmVlt7oA3M7PanEzMzJZxko5oOgYnk5ZqQ+Ow9nG7WCiv64Kk7Xrdmo6vsPc2HYD7TFrKF2FZL24XC0k6IyIO6zOqa7SN5mq8XTiZtFQbGoe1j9uF9SJpPvBMr12kxLr6wGNwMmmnNjQOax+3i94k/Q0wnsrlDhFxdmMBFdaGqXV8nUl7zWy6cVgruV10kfRtYGPgVuC5XBzAqEkmbeBkYmbLuonAFjG6T7P8oOkAPJqrvRpvHNZKbhcvdDvwsqaDaNgjkiYAKPmWpKck3VZqZJuTSXs13jisldwuMkk/ljQNWBe4Q9LlkqZ1bk3HV9hRwL15+yBga2Aj4CPA10oE4NNc7XUUMCVvVxvHtqTG8bfNhGUNc7tY6OSmA2iR+RHxl7z9FuDsiHgM+Kmkfy8RgI9M2qtn44iInwKrNhiXNcvtIouIayPiWuA+4MbK/V8Co22VxQWS1pP0ImA34KeVfSuXCMDJpL0abxzWSm4XL/QDYEHl/nOMvr6lfyPNKH0vMC0iZgFIeiNwd4kAfJqrvTqNY3kaahzWSm4XLzQmIp7t3ImIZyWt2GRApUXETyS9AlgtIuZVds0ADigRgy9abDFJY+hqHJJWJf3dft9cZNYkt4uhJF0JfD0ipuX7+wJHRsRuzUZWjqTVgZdGxF35/v4sPFK9PCIeGngMTibt1IbGYe3jdvFCkjYGvgu8nDQTwP3AwRExp9HACpJ0BvA/ETEl358DXEpqG/Mj4p8GHoOTSTu1oXFY+7hd9CfpxaTvtKebjqU0SbcA23Uu3KxOryLp5xHx+kHH4D6T9not8IHK/acj4sOQGkczIVkLuF30IGkfYEvgRZIAiIhPNxpUWWO6ZgB4T2V7zRIBeDRXezXeOKyV3C66SPoGqZP5w6TTXPsDr2g0qPIWSHp+FoCIuB1A0voMHek2ME4m7dV447BWcrt4ob+JiIOBeRFxAvA6YIOGYyrti8CPJb1B0mr59kbgwrxv4Hyaq706jeOjwC25bDvSVb9FGoe1ktvFC/0x//uMpJcDj5FmBRg1IuI7kh4FPks63QdpzrJ/i4hLS8TgZNJSbWgc1j5uFz39RNKapGR6M2n6+W82G1J5EXEZcFlT9Xs0l5kt0yStFBF/7mwDLwL+1CkbDSR9nZREe4qIIwcdg49MWqoNjcPax+2ip+tJp/rICeTPkm7ulI0SMyrbJwDHlQ7AyaS9Gm8c1kpuF1keiLA+sLKkbUkjuQBWB1ZpLLAGRMTUzrako6v3S/FprmVAG9Z3tvYZ7e1C0mTgENJKi9Uk+zQwJSJ+2ERcTZN0c0QUPyrzkcmywRnfehnV7SL/+p4q6R0RcUHT8Yx2TiZmtkyS9A8R8R1gvKSPdO+PiC83EFYjJD3Nwh8Xq0h6qrMLiIhYfdAxOJm0VBsah7WP28UQncXAXtxoFC0QEas1HYP7TMzMrDYfmZjZMknSKcPtH6XDpBvjZGJmy6qbKtujeph0G/g0l5kt80b7MOk28KzBZvbXwL+KG+ZkYmZmtfk0l5ktk7qHSQPPdHYx+oZJN87JxMzMavNpLjMzq83JxMzManMyMTOz2pxMzFpI0r2S1q37GLNSnEzMzKw2JxOzpUTSeEm/lvRNSbdL+q6k3SX9QtJdknaQtLakCyXdJukGSVvn564j6QpJt0j6TxauGoikf5D0S0m3SvpPScs39p8068PJxGzp2gT4GrA1sBnwbuD1wMeAT5HmkLolIrbO98/OzzsO+HmeEmQasCGApM2BA4CdI2Ib4Dng74v9b8wWkSd6NFu67omImQCSZgFXRURImgmMB14BvAMgIq7ORyRrAG8A/i6XXyxpXn693YDtgemSAFYGHi74/zFbJE4mZkvXnyvbCyr3F5A+b/N7PCe6/q0SMDUijl1qEZoNgE9zmZV1Hfk0laRdgEcj4qmu8r2AtfLjrwLeKekled/akl5ROmizkfjIxKys44FvSbqNNJfU5Fx+AnCOpJuBa4H7ACLiDkn/AlwhaTngL8DhwG9LB242HM/NZWZmtfk0l5mZ1eZkYmZmtTmZmJlZbU4mZmZWm5OJmZnV5mRiZma1OZmYmVlt/x+9RPRyXr9NBQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "top10_models.plot(kind='bar', legend= False)\n",
    "print('Top 10 common models and the number of hard Drives in each:')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering - Limiting the dataset by only top 10 models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8657708, 19)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top 10 models\n",
    "list_top10_models = [i for i in top10_models.index]\n",
    "\n",
    "# this is the new dataframe based on the top 10 models\n",
    "new_df = df[df['model'].isin(list_top10_models)]\n",
    "\n",
    "new_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------\n",
    "#### Using the new dataframe from here on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "# Changing datatypes\n",
    "new_df['failure'] = new_df['failure'].astype('bool')\n",
    "\n",
    "# saving top10 models data to csv\n",
    "# new_df.to_csv('top10_models.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "## Analysis\n",
    "\n",
    "Keeping in mind that the resources available *do not* accurately describe this particular dataset. It is crucial to proceed with caution. I researched online and read a number of articles. I settled one the ones I found most relevant. I have used this information to help me understand the schema and it's various attributes. \n",
    "\n",
    "### Resources: \n",
    "These are some of the resources I found helpful. \n",
    "\n",
    "1. Understanding differet SMART stats: https://www.backblaze.com/blog/what-smart-stats-indicate-hard-drive-failures/ \n",
    "2. SMART schema on WIKI: https://en.wikipedia.org/wiki/S.M.A.R.T.#ATA_S.M.A.R.T._attributes \n",
    "3. Research Paper: http://cs229.stanford.edu/proj2017/final-reports/5242080.pdf\n",
    "\n",
    "\n",
    "### Tools: \n",
    "1. I have utilized scikit library for prediction.\n",
    "2. Partly used pandas and Google Big Query for faster analysis in SQL, and \n",
    "3. matplotlib + Google Data Studio for plotting charts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's check the cardinality of each columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "date has: 90 unique values\n",
      "serial_number has: 101310 unique values\n",
      "model has: 10 unique values\n",
      "capacity_bytes has: 5 unique values\n",
      "failure has: 2 unique values\n",
      "read_error_rate has: 5938125 unique values\n",
      "throughput_performance has: 67 unique values\n",
      "spin_up_time has: 288 unique values\n",
      "start_stop_count has: 187 unique values\n",
      "reallocated_sector has: 1023 unique values\n",
      "seek_time_performance has: 11 unique values\n",
      "power_on_hours has: 43903 unique values\n",
      "power_cycle_count has: 123 unique values\n",
      "reported_uncorrect has: 70 unique values\n",
      "command_timeout has: 161 unique values\n",
      "high_fly_writes has: 1175 unique values\n",
      "airflow_temprature has: 45 unique values\n",
      "load_cycle_count has: 190691 unique values\n",
      "total_lbas_written has: 6562030 unique values\n"
     ]
    }
   ],
   "source": [
    "# find unique values per columns\n",
    "for cols in new_df.columns:\n",
    "    print(\"{} has: {} unique values\".format(cols, len(new_df[cols].unique())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting graphs to get a visual look and analyze\n",
    "-------\n",
    "#### Using Google Data Studio and Big Query:\n",
    "\n",
    "* Google Data Studio provides for a much more robust and interactive reporting system. \n",
    "* I loaded the dataset into Big Query and used Google Data Studio because of it's SQL support, interactive platform and robustness with doing exploratory analysis on a large dataset. \n",
    "* There are some key charts provided below. \n",
    "\n",
    "#### Click Here for a full report: https://datastudio.google.com/open/1vzmbcHsLQ-OMZZsfXUnECJbIteK_kdF7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Looking at the trends displayed below, we can derive the following (refer to the Studio Report below) :\n",
    "     \n",
    "1. Number of positive Hard drives failure trend are going down. This trend is proportional to the power cycle of the hard drives. This means that as the hard drives get old over time, they are more likely to fail. This is also verified by external sources, a typical life of a hard drive is around 5 years. This can help find out the likelyhood of a drive failing. \n",
    "\n",
    "\n",
    "2. Reported uncorrectable errors tend to go down as the failure count goes down over time. On the other hand, the reallocated sectors a going up. Both of these features should ideally be of a lower value for a healthy hard drive. There are higher chances of failure if both of these factors go up in the future.\n",
    "\n",
    "\n",
    "3. Hard Drives have to reallocated sectors at a much higher rate in the event of a failure. This happens becase hard drives need to remap the data to a different sector in order to avoid data loss. Frequent remapping like this is not a good sign of a healthy hard drive.\n",
    "\n",
    "4. High fly rates decrease with decrease in failure. This may indicate that a lower high fly rate is a potential sign of healthier hard drives.\n",
    "\n",
    "There are more exploratory analysis performed on google data studio report, link is provided below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Data Studio Report \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"900\"\n",
       "            height=\"675\"\n",
       "            src=\"https://datastudio.google.com/embed/reporting/1vzmbcHsLQ-OMZZsfXUnECJbIteK_kdF7/page/xJMf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x10e977e48>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rendering google data studio report\n",
    "from IPython.display import IFrame\n",
    "IFrame('https://datastudio.google.com/embed/reporting/1vzmbcHsLQ-OMZZsfXUnECJbIteK_kdF7/page/xJMf', width=900, height=675)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumptions made and references drawn, in performing the analysis: \n",
    "--------\n",
    "\n",
    "Since there isn't enough detail about the dataset in this case study, some external researching is required to get an understanding. \n",
    "\n",
    "\n",
    "There are exponetial values in some of SMART metrics. The provided data stream is raw and there isn't much information available online about different expoential values. I couldn't find a meaningful method to normalize the raw data to a 100 point scale in order to make a better correlation. \n",
    "\n",
    "### Analysis Conclusion\n",
    "--------------\n",
    "\n",
    "In conclusion, the metrics in SMART systems are most often high uncorrelated. It wouldn't be recommended to rely on one of them to make a decision about a possible drive failure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (Optional)\n",
    "*In case the above embeded code for Data Studio report failed, I am including local PNG import of some of the charts.*\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "<td> 1. Number Hard Drives per model\n",
    "<img src=\"graphs/hard-drives-per-model.png\" width=\"600\"> </td>\n",
    "    <td>2. Number of positive failures by model\n",
    "        <img src=\"graphs/failures-by-model.png\" width=\"600\"> </td></tr></table>\n",
    "        \n",
    "        \n",
    "<table>\n",
    "    <tr>\n",
    "        <td>3. Failure Trend over time\n",
    "            <img src=\"graphs/failure-trend-timeseries.png\" width=\"600\"></td>\n",
    "<td>4. Daily Failure Trend to determine missing failure data pattern\n",
    "    <img src=\"graphs/daily-trend-of-fails.png\" width=\"600\"></td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "------\n",
    "## Machine learning to Predict Possible Failures based"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature selection:\n",
    "\n",
    "Based on my findings and research on SMART attributes, I have found the following variables to be the most significant out of the total available dataset. The variables are highly non correlated, I made the selection based on what works as a industry standard for SMART predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date                            0\n",
       "serial_number                   0\n",
       "model                           0\n",
       "capacity_bytes                  0\n",
       "failure                         0\n",
       "read_error_rate                 0\n",
       "throughput_performance    6579501\n",
       "spin_up_time                    0\n",
       "start_stop_count                0\n",
       "reallocated_sector              0\n",
       "seek_time_performance     6579501\n",
       "power_on_hours                  0\n",
       "power_cycle_count               0\n",
       "reported_uncorrect        2078207\n",
       "command_timeout           2078207\n",
       "high_fly_writes           3374448\n",
       "airflow_temprature        2078207\n",
       "load_cycle_count                0\n",
       "total_lbas_written        2078207\n",
       "dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all columns and the number of NaN in them\n",
    "new_df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8657708, 19)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new_df.groupby(['model'], as_index=True)['failure'].head()\n",
    "# failure_by_model = new_df.groupby('failure').agg('model').head()\n",
    "new_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date                  0\n",
       "serial_number         0\n",
       "model                 0\n",
       "capacity_bytes        0\n",
       "failure               0\n",
       "read_error_rate       0\n",
       "spin_up_time          0\n",
       "start_stop_count      0\n",
       "reallocated_sector    0\n",
       "power_on_hours        0\n",
       "power_cycle_count     0\n",
       "reported_uncorrect    0\n",
       "command_timeout       0\n",
       "airflow_temprature    0\n",
       "load_cycle_count      0\n",
       "total_lbas_written    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# featured selection\n",
    "# -----------------\n",
    "\n",
    "# selecting dataframe slice with no nan values.\n",
    "# first doing a row wise check to see if dropping rows will solve this\n",
    "featured_df = new_df.dropna(axis=0, how='any', thresh=15)\n",
    "featured_df.isna().sum()\n",
    "\n",
    "# there are still three metrics with very high number of nan\n",
    "# dropping more columns\n",
    "featured_df = featured_df.drop(['throughput_performance', 'seek_time_performance', 'high_fly_writes'], axis=1)\n",
    "\n",
    "# final dataframe is ready for any predictive usage\n",
    "# verify nan in featured_df\n",
    "featured_df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# quick look at the featured_df\n",
    "featured_df.shape\n",
    "\n",
    "# find unique models per columns\n",
    "len(featured_df['model'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "90"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of dates in featured_df\n",
    "len(featured_df['date'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "76355"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of dates in featured_df\n",
    "len(featured_df['serial_number'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving feature_df to csv, this is optional. \n",
    "# ----------------\n",
    "# using this to ocassionally push data to Big Query\n",
    "\n",
    "# this is optional for re-runs\n",
    "# featured_df.to_csv('featured_hard_drive_dataset.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "from sklearn import ensemble, metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load featured hard drive dataset\n",
    "hdd = pd.read_csv('featured_hard_drive_dataset.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6579501, 16)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdd.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(76355,)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique hard drives\n",
    "hdd['serial_number'].value_counts().shape\n",
    "\n",
    "# since hard drives serial number is unique across, we use this as the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6,)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# there are 6 models now left in the featured dataset\n",
    "hdd['model'].value_counts().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've used Big Query in parts where I found it easy to do analysis using SQL. Below is the sql script to get % of failure per model\n",
    "#### SQL to get % of failure per model:\n",
    "```\n",
    "SELECT\n",
    "  model,\n",
    "  COUNT(DISTINCT serial_number) number_of_hdd,\n",
    "  SUM(IF(failure IS TRUE,\n",
    "      1,\n",
    "      0)) fails,\n",
    "  ROUND(SUM(IF(failure IS TRUE,\n",
    "        1,\n",
    "        0))/COUNT(DISTINCT serial_number),3) percentage_of_fails\n",
    "FROM\n",
    "\n",
    "  `orbital-linker-226700.pandey.hard_drive_stats_top10_models`\n",
    "GROUP BY\n",
    "  model order by number_of_hdd desc\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>number_of_hdd</th>\n",
       "      <th>fails</th>\n",
       "      <th>percentage_of_fails</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>32091</td>\n",
       "      <td>178</td>\n",
       "      <td>0.006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ST12000NM0007</td>\n",
       "      <td>16833</td>\n",
       "      <td>32</td>\n",
       "      <td>0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HGST HMS5C4040BLE640</td>\n",
       "      <td>15374</td>\n",
       "      <td>16</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ST8000NM0055</td>\n",
       "      <td>14418</td>\n",
       "      <td>28</td>\n",
       "      <td>0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ST8000DM002</td>\n",
       "      <td>9912</td>\n",
       "      <td>21</td>\n",
       "      <td>0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>HGST HMS5C4040ALE640</td>\n",
       "      <td>6237</td>\n",
       "      <td>8</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Hitachi HDS5C4040ALE630</td>\n",
       "      <td>2296</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ST6000DX000</td>\n",
       "      <td>1882</td>\n",
       "      <td>1</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ST10000NM0086</td>\n",
       "      <td>1220</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>HGST HUH728080ALE600</td>\n",
       "      <td>1048</td>\n",
       "      <td>3</td>\n",
       "      <td>0.003</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     model  number_of_hdd  fails  percentage_of_fails\n",
       "0              ST4000DM000          32091    178                0.006\n",
       "1            ST12000NM0007          16833     32                0.002\n",
       "2     HGST HMS5C4040BLE640          15374     16                0.001\n",
       "3             ST8000NM0055          14418     28                0.002\n",
       "4              ST8000DM002           9912     21                0.002\n",
       "5     HGST HMS5C4040ALE640           6237      8                0.001\n",
       "6  Hitachi HDS5C4040ALE630           2296      0                0.000\n",
       "7              ST6000DX000           1882      1                0.001\n",
       "8            ST10000NM0086           1220      0                0.000\n",
       "9     HGST HUH728080ALE600           1048      3                0.003"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exported sql output and reading in pandas\n",
    "sql_output = pd.read_csv('reports/failure_percentage_by_model.csv')\n",
    "sql_output.head(10)\n",
    "\n",
    "# This shows that the data is highly imbalanced and the model with most fails is only about .006 or .06% of total data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above: Full list of model their % of failure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    2822092\n",
       "True         178\n",
       "Name: failure, dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using ST4000DM000\t model\n",
    "hdd_st4000 = hdd.query('model == \"ST4000DM000\"')\n",
    "hdd_st4000.shape\n",
    "\n",
    "hdd_st4000['serial_number'].value_counts().shape\n",
    "\n",
    "# number of failures in this hard drive model\n",
    "hdd_st4000['failure'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing training and testing datasets using dataframe 'hdd'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using data from all models\n",
    "# -----------------\n",
    "date = pd.to_datetime(hdd['date'])\n",
    "hdd['date'] = date\n",
    "\n",
    "# add day of year using date column\n",
    "hdd['day_of_year'] = hdd['date'].dt.dayofyear\n",
    "\n",
    "# grouping by getting all unique hard drives\n",
    "# indexing by serial number as every hard drive will have a unique serial number\n",
    "\n",
    "hdd_group = hdd.groupby('serial_number')\n",
    "\n",
    "# take the last row from each group\n",
    "hdd_last_day = hdd_group.nth(-1)\n",
    "\n",
    "\n",
    "len(hdd_last_day['date'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(76355,)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total failure per model for one day\n",
    "hdd_last_day['failure'].value_counts()\n",
    "\n",
    "# number of drives in the dataset\n",
    "uniq_serial_numbers = pd.Series(hdd_last_day.index.unique())\n",
    "uniq_serial_numbers.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    76271\n",
       "True        84\n",
       "Name: failure, dtype: int64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdd_last_day['failure'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(19089, 16)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# slicing a copy of 25% of all unique hard drives for test\n",
    "test_ids = uniq_serial_numbers.sample(frac=0.25)\n",
    "train = hdd_last_day.query('index not in @test_ids')\n",
    "test = hdd_last_day.query('index in @test_ids')\n",
    "\n",
    "# test data has now looks like this\n",
    "test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    19066\n",
       "True        23\n",
       "Name: failure, dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test data\n",
    "test['failure'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(57266, 16)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    57205\n",
       "True        61\n",
       "Name: failure, dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train data has remaining 24029 data points\n",
    "train['failure'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training and testing labels\n",
    "train_labels = train['failure']\n",
    "# failure is the final label we would like to predict \n",
    "test_labels = test['failure']\n",
    "\n",
    "# drop labels from train and test\n",
    "train = train.drop('failure', axis=1)\n",
    "test = test.drop('failure', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(57266, 15)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "#drop date related features from tree model\n",
    "train = train.drop(['day_of_year', 'date'], axis=1)\n",
    "test = test.drop(['day_of_year', 'date'] , axis=1)\n",
    "\n",
    "# removing other irrelevant or constant columns\n",
    "\n",
    "# this is out final training and test dataset with all the right features\n",
    "train = train.drop(['model', 'capacity_bytes', 'power_on_hours', 'total_lbas_written'], axis=1)\n",
    "test = test.drop(['model', 'capacity_bytes', 'power_on_hours', 'total_lbas_written'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(57266, 9)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['airflow_temprature', 'command_timeout', 'load_cycle_count',\n",
       "       'power_cycle_count', 'read_error_rate', 'reallocated_sector',\n",
       "       'reported_uncorrect', 'spin_up_time', 'start_stop_count'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# these are the training features\n",
    "train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "serial_number\n",
       "S3000A9T    False\n",
       "S3000FZ5    False\n",
       "S3000NSV    False\n",
       "S3000QAP    False\n",
       "S30015PW    False\n",
       "S3001FPA    False\n",
       "S3001HBH    False\n",
       "S30034E6    False\n",
       "S3003A6V    False\n",
       "S3003GAB    False\n",
       "Name: failure, dtype: bool"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check first 10 training labels\n",
    "train_labels[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction using Random Forest Ensemble\n",
    "For prediction, I tried a couple of options from logistic regression to Naive Bayes but finally settled on random forest classifier tree for following reasons:\n",
    "1. Data is full of poorly correlated SMART features.\n",
    "3. For regression, normalization of some of the attributes would be required. \n",
    "4. Since there isn't much information available on how different large float values can be normalized, it's a better idea to stick with the absolute numbers only\n",
    "5. Use raw values instead of normalization since normalization has no impact on performance of a tree.\n",
    "6. Random forest classifiers are designed to reduce the overall error rate and work over raw data.\n",
    "\n",
    "#### Overall\n",
    "There doesn't seem to be a lot of correlation between various SMART attributes, and this varies greatly over different models of hard drive. A decision tree model (random forest) that looks at more than one attribute in order to make a better guess at detecting any future failures. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=3, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_clf = ensemble.RandomForestClassifier(n_jobs=1, max_depth=3)\n",
    "\n",
    "# here's how the random forest object looks\n",
    "rf_clf.fit(train, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, ..., False, False, False])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply the Classifier we trained to the test data\n",
    "rf_clf.predict(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.98842063e-01, 1.15793746e-03],\n",
       "       [9.98980480e-01, 1.01951991e-03],\n",
       "       [9.99389243e-01, 6.10756587e-04],\n",
       "       [9.98842063e-01, 1.15793746e-03],\n",
       "       [9.98842063e-01, 1.15793746e-03],\n",
       "       [9.98842063e-01, 1.15793746e-03],\n",
       "       [9.98980480e-01, 1.01951991e-03],\n",
       "       [9.99256057e-01, 7.43943123e-04],\n",
       "       [9.98712833e-01, 1.28716695e-03],\n",
       "       [9.98842063e-01, 1.15793746e-03]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generating the predicted values of possible of features for test data using trained rf_clf, generated above\n",
    "preds = rf_clf.predict_proba(test)\n",
    "\n",
    "# check predicted values of the first 10 observations\n",
    "preds[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC Area Under Curve 0.7667108305702389\n"
     ]
    }
   ],
   "source": [
    "# performing quick ROC and log loss functions to see how the the data looks\n",
    "\n",
    "print('ROC Area Under Curve', metrics.roc_auc_score(y_true=test_labels, y_score=preds[:,1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7667108305702389"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# performing quick ROC and log loss functions to see how the the data\n",
    "\n",
    "metrics.roc_auc_score(y_true=test_labels, y_score=preds[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since, above tell us that the area under curve is about 0.75, so that's a good enough"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenges: \n",
    " 1. Highly critical features like throughput_performance', 'seek_time_performance', 'high_fly_writes and 'command_timeout' have a lot of missing data. This makes the training dataset unreliable.\n",
    " 2. A sound relationship between these uncorrelated metrics/features is needed to better understand things like:\n",
    "    1. How command_timeout affects retry count or, \n",
    "    2. How reallocated sector changes over time as the drive gets old.\n",
    "    3. Different models are manufactured by different companies, and not all manufacturers have all SMART metrics,\n",
    "    among other factors like usage, data-center wear and tear, climatic conditions. This makes it difficult to design a general training that would work across the board.\n",
    "    4. Data is not normalized and there isn't much information on how to normalize them: \n",
    "    is the reallocated sector by bits or bytes?\n",
    "    are all these drives magnetic tapes, hybrid or SSDs?\n",
    "\n",
    "Normalization would make a much a better regression design, or at least present one such option to do so.\n",
    "\n",
    "\n",
    "#### Our model accurately predicted about 77% of the time that a drive is likely to fail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Conclusion\n",
    "\n",
    "\n",
    "----\n",
    "\n",
    "SMART systems widely used industry practice around data center management and disk heavy resources. \n",
    "\n",
    "The above case study attempted to analyze and predict future hard drive failures based on the data that was provided. was to predict hard drive failures using available dataset. \n",
    "\n",
    "We identified a few key metrics such as 'throughput_performance', 'reallocated sectors', age of the hard drive using 'power on hours' and a few more. We analyzed their effect over time as the failure rate goes down. \n",
    "\n",
    "There was also some extensive researching done leading up to identifying other highly critical metrics but there seems to be missing data about those metrics. Since there isn't enough detail about the dataset in this case study, some external researching was required to get an understanding.\n",
    "\n",
    "Using provided data, we predicted over 77% of possible failure, but this can be improved further. \n",
    "\n",
    "----------\n",
    "\n",
    "### Recommended actions and improvements:\n",
    "1.\tRealistically, since not all Hard Drives are manufactured and used under the same roof it’s a good idea that for future predictions, we use the critical attributes, mentioned above to analyze their effect on per hard drive model instead of a general prediction.\n",
    "2.\tIncluding more data source: \n",
    "a.\tUsing more than one source of information such as operating temperature, throughput, of reads and writes etc. can help build a more robust collection of data that can predict future.\n",
    "3.\tBacking up drives that are showing critical changes.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thanks!\n",
    "\n",
    "Harsh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}