Machine Learning 03 - K-NN (2).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2 align=\"center\">Machine Learning</h2> \n",
    "<h3 align=\"center\">Travis Millburn<br>Spring 2020</h3> \n",
    "\n",
    "<center>\n",
    "<img src=\"../images/logo.png\" alt=\"drawing\" style=\"width: 300px;\"/>\n",
    "</center>\n",
    "\n",
    "<h3 align=\"center\">Class 3: K Nearest Neighbors Algorithms</h3> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Outline \n",
    "1. Supervised vs Unsupervised models\n",
    "2. K-NN \n",
    "    * K-NN Classifiers\n",
    "    * Pseudocode\n",
    "    * Build K-NN in SkLearn\n",
    "    * Determine and critically compare effectiveness\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Until now, we've focused on data analysis ... \n",
    "\n",
    "# Today we build some models !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Supervised vs Unsupervised\n",
    "  \n",
    "  Within the field of machine learning, there are two main types of model types: supervised learning and unsupervised learning.\n",
    "\n",
    "\n",
    "\n",
    "<center>\n",
    "<img src=\"../images/supervised_unsupervised_tldr.png\" alt=\"drawing\" style=\"width: 600px;\"/>\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Supervised Learning\n",
    "* Built using known values from historical data  \n",
    "* Known inputs matched with known outputs  \n",
    "* Generalizing from known examples:\n",
    "    * The model will be able to create an output when presented with not-seen-before input in the future\n",
    "\n",
    "Examples:\n",
    "    * Identifying an email as spam or not\n",
    "    * Flagging a financial transaction as fraudulent or not\n",
    "    * Determining whether an elephant or monkey is present in a photo\n",
    "    * Predicting a profitable financial trade\n",
    "    \n",
    "Two Common Types:\n",
    "1. Classification  \n",
    "    A) Discrete Labels\n",
    "2. Regression  \n",
    "    A) Continuous Labels\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Unsupervised Learning\n",
    "* No known outputs, so we are learning about the structure of the data  \n",
    "* Often used for dimensionality reduction (PCA - Principle Component Analysis)  \n",
    "* Very useful for exploratory analysis, or understanding our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Today, we will focus on K-NN Models\n",
    "K Nearest Neighbors Algorithm  \n",
    "\n",
    "* K-NN is one of the simpler machine learning algorithms  \n",
    "* We will warehouse the training dataset.\n",
    "    * When we run a prediction using the model, we simply find the nearest datapoint in the training dataset  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Let's back up.  What is classification ?\n",
    "Discrete Labels:\n",
    "1. Dog | Cat\n",
    "2. Spam | Not Spam\n",
    "3. Profitable Trade | Loser Trade\n",
    "4. Apple | Orange | Pear | Banana\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Formal Classification Problem \n",
    "\n",
    "Mathematically, a classifier is a function or model $f$ that predicts the class label  for a given input example ${\\bf x}$, that is\n",
    "\n",
    "$$\\hat{y} = f({\\bf x})$$\n",
    "\n",
    "* The value $\\hat{y}$ belongs to a set $\\{c_1,c_2,...,c_k\\}$, each $c_i$ is a class label.\n",
    "\n",
    "\n",
    "The quality of the model is inherently determined by the quality and accuracy of the training set, an important consideration upon evaluating any implementation of a model. \n",
    "\n",
    "Several standardized data sets that have been tested and evaluated over many years.\n",
    "\n",
    "Scikits has a very good sample of the standardized data sets set up to be easily accessed from the standard api, as [discussed here](http://scikit-learn.org/stable/datasets/).\n",
    "\n",
    "http://scikit-learn.org/stable/datasets/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Classification Model using K-NN  \n",
    "  \n",
    "A very simple K-NN model will take only one nearest-neighbor into account.\n",
    "\n",
    "For a new input, we find the nearest input in the training dataset, and use that output\n",
    "\n",
    "<center>\n",
    "<img src=\"../images/one-neighbor-knn.png\" alt=\"drawing\" style=\"width: 600px;\"/>\n",
    " Introduction to Machine Learning; Sarah Guido, Andreas Muller\n",
    "</center>\n",
    "\n",
    "\n",
    "\n",
    "Three new inputs (stars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# It's a Party: More Neighbors\n",
    "\n",
    "We can build a more-sophisticated model by adding a few more neighbors to our prediction (not too many, though!)\n",
    "\n",
    "\n",
    "<center>\n",
    "<img src=\"../images/three_knn.png\" alt=\"drawing\" style=\"width: 600px;\"/>\n",
    " Introduction to Machine Learning; Sarah Guido, Andreas Muller\n",
    "</center>\n",
    "\n",
    "The \"K\" in K-NN refers to an algorithm that can take an arbitrary number of neighbors.\n",
    "\n",
    "Each neighbor gets one vote.  Most votes get the classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# We can display the regions of predictions\n",
    "\n",
    "<center>\n",
    "<img src=\"../images/knn_boundaries.JPG\" alt=\"drawing\" style=\"width: 1200px;\"/>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Another Example\n",
    "\n",
    "<center>\n",
    "<img src=\"../images/knn_example.jpg\" alt=\"drawing\" style=\"width: 1200px;\"/>\n",
    "</center>\n",
    "\n",
    "What happens to the prediction with one neighbor instead of 5 ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# K-NN: One Neighbor\n",
    "\n",
    "<center>\n",
    "<img src=\"../images/knn_1.jpg\" alt=\"drawing\" style=\"width: 1200px;\"/>\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# K-NN Five Neighbors\n",
    "\n",
    "<center>\n",
    "<img src=\"../images/knn_5.jpg\" alt=\"drawing\" style=\"width: 1200px;\"/>\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Algorithm\n",
    "\n",
    "The KNN algorithm is very simple to implement, as it does not need to be trained. The training phase merely stores the training data. For each test point, we calculate the distance of that data point to every existing data point and find the $K$ closest ones. What we return is the the most common amongst the top k classification nearest to the test point. Here's the pseudocode for _K_ Nearest Neighbors:\n",
    "\n",
    "```\n",
    "kNN:\n",
    "\n",
    "    Learn:\n",
    "        Store training set T to X_train: X_train <-- T\n",
    "\n",
    "\n",
    "    Predict:\n",
    "        for every point xp in X_predict:\n",
    "            for every point x in X_train:\n",
    "                calculate the distance d in D between x and xp\n",
    "            sort D in increasing order\n",
    "            take the \"k\" items in X_train with the smallest distances to x\n",
    "            return the majority class among these k items\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### KNN is a typical example of a lazy learner. It is called \"lazy\" not because of its apparent simplicity, but because it doesn't learn a discriminative function from the training data but memorizes the training dataset instead.\n",
    "\n",
    "-Python Machine Learning - Third Edition\n",
    "    Mirjalili & Raschka"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What does closest mean in this sense?  How do we calculate it?\n",
    "\n",
    "* Distance Metric\n",
    "    * Minkowski distance is generalization of Euclidean/Manhattan distance\n",
    "* sklearn allows us to use the metric parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# K-NN Models seem straightforward.  Let's build one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "ename": "ClientCannotConnectError",
     "evalue": "could not establish connection to server\r\n  CODE: 08001\r\n  LOCATION: CLIENT\r\nCONNECTION: [failed]\r\n  failures[0]:\r\n    socket('54.152.120.194', 5432)\r\n    Traceback (most recent call last):\r\n      File \"c:\\program files\\python37\\lib\\site-packages\\postgresql\\protocol\\client3.py\", line 136, in connect\r\n        self.socket = self.socket_factory(timeout = timeout)\r\n      File \"c:\\program files\\python37\\lib\\site-packages\\postgresql\\python\\socket.py\", line 64, in __call__\r\n        s.connect(self.socket_connect)\r\n    TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\r\n\r\n    The above exception was the direct cause of the following exception:\r\n\r\n    postgresql.exceptions.ConnectionRejectionError: could not connect\r\n      CODE: 08004\r\n      LOCATION: CLIENT\r\nCONNECTOR: [Host] pq://new_haven_ds_read:***@nhds.cwroivw0q1rc.us-east-1.rds.amazonaws.com:5432/nhds\r\n  category: None\r\nDRIVER: postgresql.driver.pq3.Driver",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mClientCannotConnectError\u001b[0m                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-7375f94e3313>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpreprocessing\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mNormalizer\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 11\u001b[1;33m \u001b[1;32mwith\u001b[0m \u001b[0mpostgresql\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"pq://new_haven_ds_read:new_haven_ds_secret_99@nhds.cwroivw0q1rc.us-east-1.rds.amazonaws.com/nhds\"\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mdb\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     12\u001b[0m     \u001b[0mps\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdb\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mprepare\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'select * from nhds.uci_adults'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m     \u001b[0mres\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mps\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\program files\\python37\\lib\\site-packages\\postgresql\\__init__.py\u001b[0m in \u001b[0;36mopen\u001b[1;34m(iri, prompt_title, **kw)\u001b[0m\n\u001b[0;32m     92\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     93\u001b[0m                 \u001b[0mc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mC\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 94\u001b[1;33m                 \u001b[0mc\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     95\u001b[0m                 \u001b[1;32mreturn\u001b[0m \u001b[0mc\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     96\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\program files\\python37\\lib\\site-packages\\postgresql\\driver\\pq3.py\u001b[0m in \u001b[0;36mconnect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   2425\u001b[0m                 \u001b[1;31m# It's closed.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2426\u001b[0m                 \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2427\u001b[1;33m                         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_establish\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   2428\u001b[0m                 \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2429\u001b[0m                         \u001b[1;31m# Close it up on failure.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\program files\\python37\\lib\\site-packages\\postgresql\\driver\\pq3.py\u001b[0m in \u001b[0;36m_establish\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   2551\u001b[0m                         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfailures\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfailures\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2552\u001b[0m                         \u001b[1;31m# it's over.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2553\u001b[1;33m                         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtypio\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraise_client_error\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcould_not_connect\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcreator\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcause\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mexc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   2554\u001b[0m                 \u001b[1;31m##\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2555\u001b[0m                 \u001b[1;31m# connected, now initialize connection information.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\program files\\python37\\lib\\site-packages\\postgresql\\driver\\pq3.py\u001b[0m in \u001b[0;36mraise_client_error\u001b[1;34m(self, error_message, cause, creator)\u001b[0m\n\u001b[0;32m    512\u001b[0m                         \u001b[1;32mraise\u001b[0m \u001b[0mclient_error\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mcause\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    513\u001b[0m                 \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 514\u001b[1;33m                         \u001b[1;32mraise\u001b[0m \u001b[0mclient_error\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    515\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    516\u001b[0m         \u001b[1;32mdef\u001b[0m \u001b[0mlookup_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcode\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrorlookup\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpg_exc\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mErrorLookup\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mClientCannotConnectError\u001b[0m: could not establish connection to server\r\n  CODE: 08001\r\n  LOCATION: CLIENT\r\nCONNECTION: [failed]\r\n  failures[0]:\r\n    socket('54.152.120.194', 5432)\r\n    Traceback (most recent call last):\r\n      File \"c:\\program files\\python37\\lib\\site-packages\\postgresql\\protocol\\client3.py\", line 136, in connect\r\n        self.socket = self.socket_factory(timeout = timeout)\r\n      File \"c:\\program files\\python37\\lib\\site-packages\\postgresql\\python\\socket.py\", line 64, in __call__\r\n        s.connect(self.socket_connect)\r\n    TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\r\n\r\n    The above exception was the direct cause of the following exception:\r\n\r\n    postgresql.exceptions.ConnectionRejectionError: could not connect\r\n      CODE: 08004\r\n      LOCATION: CLIENT\r\nCONNECTOR: [Host] pq://new_haven_ds_read:***@nhds.cwroivw0q1rc.us-east-1.rds.amazonaws.com:5432/nhds\r\n  category: None\r\nDRIVER: postgresql.driver.pq3.Driver"
     ]
    }
   ],
   "source": [
    "# let's import the census and income data we used in prior class\n",
    "\n",
    "# First: some imports\n",
    "import postgresql\n",
    "import pandas\n",
    "import sklearn\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.preprocessing import Normalizer\n",
    "\n",
    "with postgresql.open(\"pq://new_haven_ds_read:new_haven_ds_secret_99@nhds.cwroivw0q1rc.us-east-1.rds.amazonaws.com/nhds\") as db:\n",
    "    ps = db.prepare('select * from nhds.uci_adults')\n",
    "    res = ps()\n",
    "df = pandas.DataFrame(res, columns=ps.column_names)\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# See that income column at the end ?  That is the discrete variable we are estimating\n",
    "\n",
    "We are building a model that will estimate if an adult makes over $50k..... or not.\n",
    "\n",
    "First things first: we need to split our data into training and testing\n",
    "\n",
    "SkLearn can do this for us!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# We need X and Y\n",
    "x_df = df.drop(columns=['income'])    # We want everything but the response variable...in this case income\n",
    "y_df = df[['income']]                 # We ONLY want reponse variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Split data into training and testing\n",
    "x_train, x_test, y_train, y_test = train_test_split(x_df, y_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Build a K-NN Classifier\n",
    "clf = KNeighborsClassifier(n_neighbors=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Fit to the data.......what happened !?!? \n",
    "clf.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Let's look at that dataFrame again:\n",
    "x_train.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Let's use a little Encoding magic.....we'll come back to this later.\n",
    "for column in df.columns:\n",
    "    if df[column].dtype == type(object):\n",
    "        le = sklearn.preprocessing.LabelEncoder()\n",
    "        df[column] = le.fit_transform(df[column])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "x_df = df.drop(columns=['income'])\n",
    "y_df = df['income']\n",
    "x_train, x_test, y_train, y_test = train_test_split(x_df, y_df)\n",
    "clf = KNeighborsClassifier(n_neighbors=1)\n",
    "clf.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "clf.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print('Accuracy: {:.2f}'.format(clf.score(x_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Wow, not bad for our first model !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Do things get better or worse using more neighbors?\n",
    "\n",
    "clf_3 = KNeighborsClassifier(n_neighbors=3)\n",
    "clf_3.fit(x_train, y_train)\n",
    "clf_3.predict(x_test)\n",
    "print('Accuracy: {:.2f}'.format(clf_3.score(x_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Things have improved!  What if we do do more?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Do things get better or worse using more neighbors?\n",
    "\n",
    "clf_10 = KNeighborsClassifier(n_neighbors=10)\n",
    "clf_10.fit(x_train, y_train)\n",
    "clf_10.predict(x_test)\n",
    "print('Accuracy: {:.2f}'.format(clf_10.score(x_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Remember how many columns we have:\n",
    "len(df.columns) - 1  # one is response var"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Should we keep adding to the number of estimators?  No.\n",
    "\n",
    "Let's try anyway ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Do things get better or worse using more neighbors?\n",
    "\n",
    "clf_20 = KNeighborsClassifier(n_neighbors=20)\n",
    "clf_20.fit(x_train, y_train)\n",
    "clf_20.predict(x_test)\n",
    "print('Accuracy: {:.2f}'.format(clf_20.score(x_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Overfitting is always a concern"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### \"$K$-Nearest-Neighbor\" Algorithm - dealing with overfitting\n",
    "\n",
    "As name implies, take vote of $K$ nearest samples to address those outliers.\n",
    "\n",
    "<center>\n",
    "<img src=\"../images/kNN_1_3_5_9.png\" width=1100/>\n",
    "</center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# For the lab, let's build another one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lets go back to the Titanic Dataset from Class 1\n",
    "# https://www.kaggle.com/c/titanic/data\n",
    "\n",
    "# Build a K-NN model to predict survival\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_file = 'C:\\\\Users\\\\tmillburn\\\\OneDrive - University of New Haven\\\\Class Materials\\\\Fall 2019\\\\Week 3\\\\titanic\\\\train.csv'\n",
    "titanic_df = pandas.read_csv(train_file)\n",
    "\n",
    "test_file = 'C:\\\\Users\\\\tmillburn\\\\OneDrive - University of New Haven\\\\Class Materials\\\\Fall 2019\\\\Week 3\\\\titanic\\\\test.csv'\n",
    "test_df = pandas.read_csv(test_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_df.drop(columns=['Ticket', 'Cabin', 'Name', 'Embarked'], inplace=True)\n",
    "test_df.drop(columns=['Ticket', 'Cabin', 'Name', 'Embarked'], inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#titanic_df = titanic_df.fillna()\n",
    "titanic_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "le = sklearn.preprocessing.LabelEncoder()\n",
    "titanic_df['Sex'] = le.fit_transform(titanic_df['Sex'])\n",
    "test_df['Sex'] = le.fit_transform(test_df['Sex'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_x_df = titanic_df.drop(columns=['Survived'])\n",
    "t_y_df = titanic_df['Survived']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_x_df.fillna(0, inplace=True)\n",
    "#t_y_df.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#t_x_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_clf = KNeighborsClassifier(n_neighbors=3)\n",
    "t_clf.fit(t_x_df, t_y_df)\n",
    "#t_clf.predict(t_x_test)\n",
    "print('Accuracy: {:.2f}'.format(t_clf.score(t_x_df, t_y_df)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}