data_collection_prep_and_cleaning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kkrusere/Developing-a-Score-to-Measure-Riskiness-of-Residential-Properties-Insurance/blob/main/data_collection_prep_and_cleaning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J78MTIcOwayP",
        "outputId": "cc66733a-d2ec-4f4e-e3b7-8b26f7db3d12"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Hma08hhHTHej"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "BIHySxRsBaT3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## <center> **Developing a Score to Measure Riskiness of Residential Properties Insurance** \n",
        "\n",
        "<center><em>Developing a Score to Measure Riskiness of Residential buildings, Homes, Apartments and Condos as part of an insurance policy underwriting. Insurance underwriting is how an insurance company evaluates its risk. In this project, we identify and explore multiple data sources to collect variables that could be used to develop a score that measures the riskiness of residential buildings to aid the insurance underwriting process.</em></center>\n",
        "\n",
        "<center><img src=\"https://github.com/kkrusere/Developing-a-Score-to-Measure-Riskiness-of-Residential-Properties-Insurance/blob/main/assets/real-estate-risk.jpg?raw=1\" width=600/></center>\n",
        "\n",
        "***Project Contributors:*** Kuzi Rusere and Umair Shaikh<br>\n",
        "**MVP streamlit App URL:** https://kkrusere-developing-a-score-to-measure-prototype-mvp-app-acxav4.streamlitapp.com\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "Zyquj-u7RguB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Data collection**\n",
        "\n",
        "This notebook is for the data collection, cleaning and preparation. The first dataset that we are going to be using is from the New York City OpenDataset.\n",
        "\n",
        "<center><img src=\"https://github.com/kkrusere/Developing-a-Score-to-Measure-Riskiness-of-Residential-Properties-Insurance/blob/main/assets/nycOpenData.png?raw=1\" width=600/></center>\n",
        "\n",
        "The NYC OpenData is a data registry/repositoory of public data generated by various New York City agencies and other City organizations that is publicly available and accessible for anyone to use, participate in and improve government by conducting research and analysis gaining a better understanding of the services provided by City. The repository is an initiative to improve the accessibility, transparency, and accountability of City government.\n",
        "\n",
        "\n",
        "The datasets are available and accessible in a variety of machine-readable formats including API access. We are going to be using the NYC 311 dataset. NYC 311 gives access to non-emergency City services and info about City government programs, this tool was launched in 2003 with phone call as only contact Type, but now is also accessible through text messages, chat, a mobile application, social media and a website.\n",
        "<br>\n",
        "<br>\n",
        "<br>\n",
        "<center><img src=\"https://github.com/kkrusere/Developing-a-Score-to-Measure-Riskiness-of-Residential-Properties-Insurance/blob/main/assets/311_contact_type.png?raw=1\" width=600/><figcaption><em>Image from: https://council.nyc.gov/data/311-services/</em></figcaption></center>\n",
        "\n",
        "<br>\n",
        "<br>\n",
        "<br>\n",
        "\n",
        "We are going to be using 311 data with `Request Types` related to incidents,complaints tied to residential areas. So, right from the bet, our data in the `Location Type` will be filtered to only include entries with `Residential` as you will see from the below."
      ],
      "metadata": {
        "id": "B4EIPPy7lxfR"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "9PPSvyAXlwaB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "9ofW1wH0RXrP"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "%matplotlib inline\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "iI0jebi_iAFt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The data that we are woking with is huge, so we are going to be using pyspark for reading the dataframe/table and then use pandas to filter and clean the data.\n",
        "## **Setting up pyspark:**\n",
        "Installing PySpark on Google Colab is to use pip install"
      ],
      "metadata": {
        "id": "tm_t9QGWiAkf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install pyspark\n",
        "!pip install pyspark"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8a65SN1tiOzx",
        "outputId": "3c048091-c748-41e2-99a8-dce89ab8753c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting pyspark\n",
            "  Downloading pyspark-3.3.0.tar.gz (281.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 281.3 MB 38 kB/s \n",
            "\u001b[?25hCollecting py4j==0.10.9.5\n",
            "  Downloading py4j-0.10.9.5-py2.py3-none-any.whl (199 kB)\n",
            "\u001b[K     |████████████████████████████████| 199 kB 50.5 MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.3.0-py2.py3-none-any.whl size=281764026 sha256=8812b925a148a620fffd0f3f6b64f2b0dd373291ed010f3574f0dd0cde3fa4c2\n",
            "  Stored in directory: /root/.cache/pip/wheels/7a/8e/1b/f73a52650d2e5f337708d9f6a1750d451a7349a867f928b885\n",
            "Successfully built pyspark\n",
            "Installing collected packages: py4j, pyspark\n",
            "Successfully installed py4j-0.10.9.5 pyspark-3.3.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# # Import SparkSession\n",
        "# from pyspark.sql import SparkSession\n",
        "# # Create a Spark Session\n",
        "# spark = SparkSession.builder.master(\"local[*]\").getOrCreate()\n",
        "# # Check Spark Session Information\n",
        "# spark"
      ],
      "metadata": {
        "id": "6116GEOcimyJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "w00OW3QkimvZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use spark to read the csv dataframe "
      ],
      "metadata": {
        "id": "GUbD5aBOkd__"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#data_df = spark.read.csv(\"/content/drive/MyDrive/capstone/311_Service_Requests_from_2010_to_Present.csv\", header=True, inferSchema=True)\n",
        "data_df = pd.read_csv(\"/content/drive/MyDrive/capstone/311_Service_Requests_from_2010_to_Present.csv\")"
      ],
      "metadata": {
        "id": "xgG8IZeNiQgR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data_df.head()"
      ],
      "metadata": {
        "id": "KbaIazw3iOwK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#lets take a look atn the columns that we have in our dataset\n",
        "list(data_df.columns)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rgrho34eD1yd",
        "outputId": "29cff5c5-971a-41af-80c7-0961e4d20eab"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Unique Key',\n",
              " 'Created Date',\n",
              " 'Closed Date',\n",
              " 'Agency',\n",
              " 'Agency Name',\n",
              " 'Complaint Type',\n",
              " 'Descriptor',\n",
              " 'Location Type',\n",
              " 'Incident Zip',\n",
              " 'Incident Address',\n",
              " 'Street Name',\n",
              " 'Cross Street 1',\n",
              " 'Cross Street 2',\n",
              " 'Intersection Street 1',\n",
              " 'Intersection Street 2',\n",
              " 'Address Type',\n",
              " 'City',\n",
              " 'Landmark',\n",
              " 'Facility Type',\n",
              " 'Status',\n",
              " 'Due Date',\n",
              " 'Resolution Description',\n",
              " 'Resolution Action Updated Date',\n",
              " 'Community Board',\n",
              " 'BBL',\n",
              " 'Borough',\n",
              " 'X Coordinate (State Plane)',\n",
              " 'Y Coordinate (State Plane)',\n",
              " 'Open Data Channel Type',\n",
              " 'Park Facility Name',\n",
              " 'Park Borough',\n",
              " 'Vehicle Type',\n",
              " 'Taxi Company Borough',\n",
              " 'Taxi Pick Up Location',\n",
              " 'Bridge Highway Name',\n",
              " 'Bridge Highway Direction',\n",
              " 'Road Ramp',\n",
              " 'Bridge Highway Segment',\n",
              " 'Latitude',\n",
              " 'Longitude',\n",
              " 'Location']"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#we remove the columns/features of our dataset that are not going to be of any use for this project \n",
        "data_df = data_df[[\n",
        "    'Unique Key',\n",
        "    'Created Date',\n",
        "    'Agency',\n",
        "    'Agency Name',\n",
        "    'Complaint Type',\n",
        "    'Descriptor',\n",
        "    'Location Type',\n",
        "    'Incident Zip',\n",
        "    'Incident Address',\n",
        "    'Street Name',\n",
        "    'Address Type',\n",
        "    'City',\n",
        "    'Resolution Description',\n",
        "    'Borough',\n",
        "    'Latitude',\n",
        "    'Longitude',]]"
      ],
      "metadata": {
        "id": "JAE_7GO45Ulk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "list(data_df.columns)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vOpm4IyHGPIE",
        "outputId": "7301c9c6-4941-4c26-a0c1-d970963b2093"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Unique Key',\n",
              " 'Created Date',\n",
              " 'Agency',\n",
              " 'Agency Name',\n",
              " 'Complaint Type',\n",
              " 'Descriptor',\n",
              " 'Location Type',\n",
              " 'Incident Zip',\n",
              " 'Incident Address',\n",
              " 'Street Name',\n",
              " 'Address Type',\n",
              " 'City',\n",
              " 'Resolution Description',\n",
              " 'Borough',\n",
              " 'Latitude',\n",
              " 'Longitude']"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Data cleaning and preparation"
      ],
      "metadata": {
        "id": "Ts1wADPm5VLV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data_df.info()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TCusA_555Zjh",
        "outputId": "819de88d-8c38-44fd-f4d5-848c15a8bc2d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 6289602 entries, 0 to 6289601\n",
            "Data columns (total 16 columns):\n",
            " #   Column                  Dtype  \n",
            "---  ------                  -----  \n",
            " 0   Unique Key              int64  \n",
            " 1   Created Date            object \n",
            " 2   Agency                  object \n",
            " 3   Agency Name             object \n",
            " 4   Complaint Type          object \n",
            " 5   Descriptor              object \n",
            " 6   Location Type           object \n",
            " 7   Incident Zip            float64\n",
            " 8   Incident Address        object \n",
            " 9   Street Name             object \n",
            " 10  Address Type            object \n",
            " 11  City                    object \n",
            " 12  Resolution Description  object \n",
            " 13  Borough                 object \n",
            " 14  Latitude                float64\n",
            " 15  Longitude               float64\n",
            "dtypes: float64(3), int64(1), object(12)\n",
            "memory usage: 767.8+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#we are going to drop nan from our dataset, we have plenty enough data that we can afford to drop rows\n",
        "data_df.dropna(axis = 0, how ='any', inplace=True)\n",
        "data_df.info()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Uldd_NGM5aPt",
        "outputId": "11bf40ed-06d9-4b78-a29c-c5215f5603ad"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 6048735 entries, 0 to 6289601\n",
            "Data columns (total 16 columns):\n",
            " #   Column                  Dtype  \n",
            "---  ------                  -----  \n",
            " 0   Unique Key              int64  \n",
            " 1   Created Date            object \n",
            " 2   Agency                  object \n",
            " 3   Agency Name             object \n",
            " 4   Complaint Type          object \n",
            " 5   Descriptor              object \n",
            " 6   Location Type           object \n",
            " 7   Incident Zip            float64\n",
            " 8   Incident Address        object \n",
            " 9   Street Name             object \n",
            " 10  Address Type            object \n",
            " 11  City                    object \n",
            " 12  Resolution Description  object \n",
            " 13  Borough                 object \n",
            " 14  Latitude                float64\n",
            " 15  Longitude               float64\n",
            "dtypes: float64(3), int64(1), object(12)\n",
            "memory usage: 784.5+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#now we change datatypes for the `Unique Key`, `Created Date`, and `Incident Zip`\n",
        "data_df['Unique Key'] = data_df['Unique Key'].astype(object)\n",
        "data_df['Incident Zip'] = data_df['Incident Zip'].astype(str)\n",
        "#we are going to split the `Created Date` into dat\n",
        "data_df['Date'] = [ele.split(\" \")[0] for ele in data_df['Created Date']]\n",
        "#we drop the the `Created Date` column\n",
        "data_df.drop('Created Date', axis=1, inplace=True)\n",
        "#now we change datatypes of the `Date` to the datetime format \n",
        "data_df['Date'] = pd.to_datetime(data_df['Date'], format='%m/%d/%Y')\n"
      ],
      "metadata": {
        "id": "VPsRSRUS5aMt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data_df = data_df[[\n",
        "    'Unique Key',\n",
        "    'Date',\n",
        "    'Agency',\n",
        "    'Agency Name',\n",
        "    'Complaint Type',\n",
        "    'Descriptor',\n",
        "    'Location Type',\n",
        "    'Incident Zip',\n",
        "    'Incident Address',\n",
        "    'Street Name',\n",
        "    'Address Type',\n",
        "    'City',\n",
        "    'Resolution Description',\n",
        "    'Borough',\n",
        "    'Latitude',\n",
        "    'Longitude',]]"
      ],
      "metadata": {
        "id": "d3ezeH6iWSak"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data_df.info()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wRwf0NZ_Of7M",
        "outputId": "b200a684-1b31-4024-e233-29e8e088524c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 6048735 entries, 0 to 6289601\n",
            "Data columns (total 16 columns):\n",
            " #   Column                  Dtype         \n",
            "---  ------                  -----         \n",
            " 0   Unique Key              object        \n",
            " 1   Date                    datetime64[ns]\n",
            " 2   Agency                  object        \n",
            " 3   Agency Name             object        \n",
            " 4   Complaint Type          object        \n",
            " 5   Descriptor              object        \n",
            " 6   Location Type           object        \n",
            " 7   Incident Zip            object        \n",
            " 8   Incident Address        object        \n",
            " 9   Street Name             object        \n",
            " 10  Address Type            object        \n",
            " 11  City                    object        \n",
            " 12  Resolution Description  object        \n",
            " 13  Borough                 object        \n",
            " 14  Latitude                float64       \n",
            " 15  Longitude               float64       \n",
            "dtypes: datetime64[ns](1), float64(2), object(13)\n",
            "memory usage: 784.5+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 503
        },
        "id": "nkfnquvbIxqJ",
        "outputId": "c76331a5-b8b6-4e54-986f-3be87521d903"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  Unique Key       Date Agency  \\\n",
              "0   25595691 2013-05-23    HPD   \n",
              "1   25595692 2013-05-23    HPD   \n",
              "2   25595877 2013-05-23   DSNY   \n",
              "3   25595984 2013-05-23    HPD   \n",
              "4   25596010 2013-05-23    HPD   \n",
              "\n",
              "                                         Agency Name   Complaint Type  \\\n",
              "0  Department of Housing Preservation and Develop...  PAINT - PLASTER   \n",
              "1  Department of Housing Preservation and Develop...  PAINT - PLASTER   \n",
              "2                           Department of Sanitation         Graffiti   \n",
              "3  Department of Housing Preservation and Develop...         NONCONST   \n",
              "4  Department of Housing Preservation and Develop...  PAINT - PLASTER   \n",
              "\n",
              "  Descriptor         Location Type Incident Zip          Incident Address  \\\n",
              "0      WALLS  RESIDENTIAL BUILDING      11209.0             7207 3 AVENUE   \n",
              "1    CEILING  RESIDENTIAL BUILDING      10457.0  269 EAST BURNSIDE AVENUE   \n",
              "2   Graffiti           Residential      10472.0      1963 HAVILAND AVENUE   \n",
              "3     VERMIN  RESIDENTIAL BUILDING      11229.0             1820 AVENUE V   \n",
              "4      WALLS  RESIDENTIAL BUILDING      10467.0    3535 ROCHAMBEAU AVENUE   \n",
              "\n",
              "            Street Name Address Type      City  \\\n",
              "0              3 AVENUE      ADDRESS  BROOKLYN   \n",
              "1  EAST BURNSIDE AVENUE      ADDRESS     BRONX   \n",
              "2       HAVILAND AVENUE      ADDRESS     BRONX   \n",
              "3              AVENUE V      ADDRESS  BROOKLYN   \n",
              "4     ROCHAMBEAU AVENUE      ADDRESS     BRONX   \n",
              "\n",
              "                              Resolution Description   Borough   Latitude  \\\n",
              "0  The Department of Housing Preservation and Dev...  BROOKLYN  40.633380   \n",
              "1  The Department of Housing Preservation and Dev...     BRONX  40.851333   \n",
              "2  The City has removed the graffiti from this pr...     BRONX  40.829475   \n",
              "3  The Department of Housing Preservation and Dev...  BROOKLYN  40.597049   \n",
              "4  The Department of Housing Preservation and Dev...     BRONX  40.882408   \n",
              "\n",
              "   Longitude  \n",
              "0 -74.026993  \n",
              "1 -73.902133  \n",
              "2 -73.858298  \n",
              "3 -73.952872  \n",
              "4 -73.879058  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-068bde04-28ab-4a35-9881-442afdf98c9f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unique Key</th>\n",
              "      <th>Date</th>\n",
              "      <th>Agency</th>\n",
              "      <th>Agency Name</th>\n",
              "      <th>Complaint Type</th>\n",
              "      <th>Descriptor</th>\n",
              "      <th>Location Type</th>\n",
              "      <th>Incident Zip</th>\n",
              "      <th>Incident Address</th>\n",
              "      <th>Street Name</th>\n",
              "      <th>Address Type</th>\n",
              "      <th>City</th>\n",
              "      <th>Resolution Description</th>\n",
              "      <th>Borough</th>\n",
              "      <th>Latitude</th>\n",
              "      <th>Longitude</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>25595691</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>HPD</td>\n",
              "      <td>Department of Housing Preservation and Develop...</td>\n",
              "      <td>PAINT - PLASTER</td>\n",
              "      <td>WALLS</td>\n",
              "      <td>RESIDENTIAL BUILDING</td>\n",
              "      <td>11209.0</td>\n",
              "      <td>7207 3 AVENUE</td>\n",
              "      <td>3 AVENUE</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>The Department of Housing Preservation and Dev...</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>40.633380</td>\n",
              "      <td>-74.026993</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>25595692</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>HPD</td>\n",
              "      <td>Department of Housing Preservation and Develop...</td>\n",
              "      <td>PAINT - PLASTER</td>\n",
              "      <td>CEILING</td>\n",
              "      <td>RESIDENTIAL BUILDING</td>\n",
              "      <td>10457.0</td>\n",
              "      <td>269 EAST BURNSIDE AVENUE</td>\n",
              "      <td>EAST BURNSIDE AVENUE</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>The Department of Housing Preservation and Dev...</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>40.851333</td>\n",
              "      <td>-73.902133</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>25595877</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>DSNY</td>\n",
              "      <td>Department of Sanitation</td>\n",
              "      <td>Graffiti</td>\n",
              "      <td>Graffiti</td>\n",
              "      <td>Residential</td>\n",
              "      <td>10472.0</td>\n",
              "      <td>1963 HAVILAND AVENUE</td>\n",
              "      <td>HAVILAND AVENUE</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>The City has removed the graffiti from this pr...</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>40.829475</td>\n",
              "      <td>-73.858298</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>25595984</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>HPD</td>\n",
              "      <td>Department of Housing Preservation and Develop...</td>\n",
              "      <td>NONCONST</td>\n",
              "      <td>VERMIN</td>\n",
              "      <td>RESIDENTIAL BUILDING</td>\n",
              "      <td>11229.0</td>\n",
              "      <td>1820 AVENUE V</td>\n",
              "      <td>AVENUE V</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>The Department of Housing Preservation and Dev...</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>40.597049</td>\n",
              "      <td>-73.952872</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>25596010</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>HPD</td>\n",
              "      <td>Department of Housing Preservation and Develop...</td>\n",
              "      <td>PAINT - PLASTER</td>\n",
              "      <td>WALLS</td>\n",
              "      <td>RESIDENTIAL BUILDING</td>\n",
              "      <td>10467.0</td>\n",
              "      <td>3535 ROCHAMBEAU AVENUE</td>\n",
              "      <td>ROCHAMBEAU AVENUE</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>The Department of Housing Preservation and Dev...</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>40.882408</td>\n",
              "      <td>-73.879058</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-068bde04-28ab-4a35-9881-442afdf98c9f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-068bde04-28ab-4a35-9881-442afdf98c9f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-068bde04-28ab-4a35-9881-442afdf98c9f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "TZyKpGdcO0wv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#we install the Python SQL Toolkit and Object Relational Mapper and the python MySQL connector\n",
        "!pip install SQLAlchemy\n",
        "!pip install mysql-connector-python\n",
        "!pip install PyMySQL"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JqWHGZqG-iFR",
        "outputId": "aa91891c-80e8-4b28-ec4c-e979b7087bd4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: SQLAlchemy in /usr/local/lib/python3.7/dist-packages (1.4.41)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy) (5.0.0)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy) (1.1.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->SQLAlchemy) (3.8.1)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->SQLAlchemy) (4.1.1)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: mysql-connector-python in /usr/local/lib/python3.7/dist-packages (8.0.30)\n",
            "Requirement already satisfied: protobuf<=3.20.1,>=3.11.0 in /usr/local/lib/python3.7/dist-packages (from mysql-connector-python) (3.17.3)\n",
            "Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.7/dist-packages (from protobuf<=3.20.1,>=3.11.0->mysql-connector-python) (1.15.0)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting PyMySQL\n",
            "  Downloading PyMySQL-1.0.2-py3-none-any.whl (43 kB)\n",
            "\u001b[K     |████████████████████████████████| 43 kB 1.1 MB/s \n",
            "\u001b[?25hInstalling collected packages: PyMySQL\n",
            "Successfully installed PyMySQL-1.0.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#change directory so that we can access the config.py file \n",
        "%cd /content/drive/MyDrive/capstone"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DIV5gI0v9YHV",
        "outputId": "0f8201a6-b3f1-43e9-dba5-74aec6a2d270"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/capstone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "8DFoDYKsAx_p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Storing our cleaned data set to a MySQL AWS RDS"
      ],
      "metadata": {
        "id": "A925x_dG5al5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# import mysql.connector as connection\n",
        "# from sqlalchemy import create_engine\n",
        "# import config #this holds our credentials for the database \n",
        "\n",
        "# host= config.host\n",
        "# user= config.user\n",
        "# db_password = config.password\n",
        "# port = config.port\n",
        "\n",
        "# #create the connection to the AWS MySQL database\n",
        "# conn = connection.connect(\n",
        "#   host=host,\n",
        "#   user=user,\n",
        "#   password=db_password,\n",
        "#   port = port,\n",
        "# )\n",
        "# mycursor = conn.cursor()"
      ],
      "metadata": {
        "id": "mUvc42MF5kdi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# #we create the database to store our 311 dataset\n",
        "# mycursor.execute(\"CREATE DATABASE IF NOT EXISTS NYC311_db\")\n",
        "# database = \"NYC311_db\"\n",
        "\n",
        "# # create sqlalchemy engine and converting our pandas dataframe to an SQL table\n",
        "# engine = create_engine(f\"mysql+pymysql://{user}:{db_password}@{host}/{database}\")\n",
        "# data_df.to_sql('NYC311Open_Data', con = engine, if_exists = 'append', index=False)\n",
        "\n",
        "# #we will alse save a csv\n",
        "# data_df.to_csv(\"NYC311Open_Data.csv\", index=False)"
      ],
      "metadata": {
        "id": "qRcW7stp7XsL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "RG-1BRdx7xBc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data = pd.read_csv(\"/content/drive/MyDrive/Capstone Project/NYC311Open_Data.csv\")\n",
        "data.shape"
      ],
      "metadata": {
        "id": "dZTgwnsIZl-B",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a91f8e0f-fc64-469d-eac5-6c685f5e04ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(6048735, 16)"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data.head()"
      ],
      "metadata": {
        "id": "beG02nDdhCGk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 608
        },
        "outputId": "2af14104-dd2c-46f5-b675-4dd079be419b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Unique Key        Date Agency  \\\n",
              "0    25595691  2013-05-23    HPD   \n",
              "1    25595692  2013-05-23    HPD   \n",
              "2    25595877  2013-05-23   DSNY   \n",
              "3    25595984  2013-05-23    HPD   \n",
              "4    25596010  2013-05-23    HPD   \n",
              "\n",
              "                                         Agency Name   Complaint Type  \\\n",
              "0  Department of Housing Preservation and Develop...  PAINT - PLASTER   \n",
              "1  Department of Housing Preservation and Develop...  PAINT - PLASTER   \n",
              "2                           Department of Sanitation         Graffiti   \n",
              "3  Department of Housing Preservation and Develop...         NONCONST   \n",
              "4  Department of Housing Preservation and Develop...  PAINT - PLASTER   \n",
              "\n",
              "  Descriptor         Location Type  Incident Zip          Incident Address  \\\n",
              "0      WALLS  RESIDENTIAL BUILDING       11209.0             7207 3 AVENUE   \n",
              "1    CEILING  RESIDENTIAL BUILDING       10457.0  269 EAST BURNSIDE AVENUE   \n",
              "2   Graffiti           Residential       10472.0      1963 HAVILAND AVENUE   \n",
              "3     VERMIN  RESIDENTIAL BUILDING       11229.0             1820 AVENUE V   \n",
              "4      WALLS  RESIDENTIAL BUILDING       10467.0    3535 ROCHAMBEAU AVENUE   \n",
              "\n",
              "            Street Name Address Type      City  \\\n",
              "0              3 AVENUE      ADDRESS  BROOKLYN   \n",
              "1  EAST BURNSIDE AVENUE      ADDRESS     BRONX   \n",
              "2       HAVILAND AVENUE      ADDRESS     BRONX   \n",
              "3              AVENUE V      ADDRESS  BROOKLYN   \n",
              "4     ROCHAMBEAU AVENUE      ADDRESS     BRONX   \n",
              "\n",
              "                              Resolution Description   Borough   Latitude  \\\n",
              "0  The Department of Housing Preservation and Dev...  BROOKLYN  40.633380   \n",
              "1  The Department of Housing Preservation and Dev...     BRONX  40.851333   \n",
              "2  The City has removed the graffiti from this pr...     BRONX  40.829475   \n",
              "3  The Department of Housing Preservation and Dev...  BROOKLYN  40.597049   \n",
              "4  The Department of Housing Preservation and Dev...     BRONX  40.882408   \n",
              "\n",
              "   Longitude  \n",
              "0 -74.026993  \n",
              "1 -73.902133  \n",
              "2 -73.858298  \n",
              "3 -73.952872  \n",
              "4 -73.879058  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8a823872-b42e-4569-a14b-4f94c80f2e63\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unique Key</th>\n",
              "      <th>Date</th>\n",
              "      <th>Agency</th>\n",
              "      <th>Agency Name</th>\n",
              "      <th>Complaint Type</th>\n",
              "      <th>Descriptor</th>\n",
              "      <th>Location Type</th>\n",
              "      <th>Incident Zip</th>\n",
              "      <th>Incident Address</th>\n",
              "      <th>Street Name</th>\n",
              "      <th>Address Type</th>\n",
              "      <th>City</th>\n",
              "      <th>Resolution Description</th>\n",
              "      <th>Borough</th>\n",
              "      <th>Latitude</th>\n",
              "      <th>Longitude</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>25595691</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>HPD</td>\n",
              "      <td>Department of Housing Preservation and Develop...</td>\n",
              "      <td>PAINT - PLASTER</td>\n",
              "      <td>WALLS</td>\n",
              "      <td>RESIDENTIAL BUILDING</td>\n",
              "      <td>11209.0</td>\n",
              "      <td>7207 3 AVENUE</td>\n",
              "      <td>3 AVENUE</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>The Department of Housing Preservation and Dev...</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>40.633380</td>\n",
              "      <td>-74.026993</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>25595692</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>HPD</td>\n",
              "      <td>Department of Housing Preservation and Develop...</td>\n",
              "      <td>PAINT - PLASTER</td>\n",
              "      <td>CEILING</td>\n",
              "      <td>RESIDENTIAL BUILDING</td>\n",
              "      <td>10457.0</td>\n",
              "      <td>269 EAST BURNSIDE AVENUE</td>\n",
              "      <td>EAST BURNSIDE AVENUE</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>The Department of Housing Preservation and Dev...</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>40.851333</td>\n",
              "      <td>-73.902133</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>25595877</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>DSNY</td>\n",
              "      <td>Department of Sanitation</td>\n",
              "      <td>Graffiti</td>\n",
              "      <td>Graffiti</td>\n",
              "      <td>Residential</td>\n",
              "      <td>10472.0</td>\n",
              "      <td>1963 HAVILAND AVENUE</td>\n",
              "      <td>HAVILAND AVENUE</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>The City has removed the graffiti from this pr...</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>40.829475</td>\n",
              "      <td>-73.858298</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>25595984</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>HPD</td>\n",
              "      <td>Department of Housing Preservation and Develop...</td>\n",
              "      <td>NONCONST</td>\n",
              "      <td>VERMIN</td>\n",
              "      <td>RESIDENTIAL BUILDING</td>\n",
              "      <td>11229.0</td>\n",
              "      <td>1820 AVENUE V</td>\n",
              "      <td>AVENUE V</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>The Department of Housing Preservation and Dev...</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>40.597049</td>\n",
              "      <td>-73.952872</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>25596010</td>\n",
              "      <td>2013-05-23</td>\n",
              "      <td>HPD</td>\n",
              "      <td>Department of Housing Preservation and Develop...</td>\n",
              "      <td>PAINT - PLASTER</td>\n",
              "      <td>WALLS</td>\n",
              "      <td>RESIDENTIAL BUILDING</td>\n",
              "      <td>10467.0</td>\n",
              "      <td>3535 ROCHAMBEAU AVENUE</td>\n",
              "      <td>ROCHAMBEAU AVENUE</td>\n",
              "      <td>ADDRESS</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>The Department of Housing Preservation and Dev...</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>40.882408</td>\n",
              "      <td>-73.879058</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8a823872-b42e-4569-a14b-4f94c80f2e63')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8a823872-b42e-4569-a14b-4f94c80f2e63 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8a823872-b42e-4569-a14b-4f94c80f2e63');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data = data.sample(n = 100000)\n",
        "data.reset_index(drop=True, inplace=True)\n",
        "data.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wDZnNJ3HQtof",
        "outputId": "77f522ae-2076-4c44-af65-27a44ec29263"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(100000, 16)"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "JD_fGcKg5bsh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data.to_csv(\"/content/drive/MyDrive/Capstone Project/NYC311Open_Data100000.csv\", index=False)"
      ],
      "metadata": {
        "id": "Jh53kFz6RECm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "PF0U9pxqb21O"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## NYPD complaints Data"
      ],
      "metadata": {
        "id": "AK733Mwk5cIY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "column_Name_Description = {  'RPT_DT': 'Date event was reported to police',\n",
        "                                  'OFNS_DESC': 'Description of offense corresponding with key code',\n",
        "                                  'CRM_ATPT_CPTD_CD': 'Indicator of whether crime was successfully completed or attempted, but failed or was interrupted prematurely',\n",
        "                                  'LAW_CAT_CD': 'Level of offense: felony, misdemeanor, violation',\n",
        "                                  'BORO_NM': 'The name of the borough in which the incident occurred',\n",
        "                                  'LOC_OF_OCCUR_DESC': 'Specific location of occurrence in or around the premises; inside, opposite of, front of, rear of',\n",
        "                                  'PREM_TYP_DESC': 'Specific description of premises; grocery store, residence, street, etc.',\n",
        "                                  'Latitude': 'Midblock Latitude coordinate for Global Coordinate System, WGS 1984, decimal degrees (EPSG 4326)',\n",
        "                                  'Longitude': 'Midblock Longitude coordinate for Global Coordinate System, WGS 1984, decimal degrees (EPSG 4326)'}"
      ],
      "metadata": {
        "id": "hRZ18Vqr5bLw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "CggyI1M89lED"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "filter_list = list(column_Name_Description.keys())"
      ],
      "metadata": {
        "id": "MG_f-hBK-Deg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "g-2loAQaFBak"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data = pd.read_csv(\"/content/drive/MyDrive/Capstone Project/NYPD_Complaint_Data_Historic.csv\")\n",
        "data.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 569
        },
        "id": "TsA-ioyr9oMC",
        "outputId": "5c4dd0b2-a32b-419b-845e-5bf5f5988e13"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   CMPLNT_NUM CMPLNT_FR_DT CMPLNT_FR_TM CMPLNT_TO_DT CMPLNT_TO_TM  \\\n",
              "0   280364018   06/09/2018     21:42:00   06/09/2018     21:43:00   \n",
              "1   377132404   08/04/2018     22:15:00          NaN          NaN   \n",
              "2   336011712   11/04/2018     11:15:00          NaN          NaN   \n",
              "3   599398393   05/23/2018     23:30:00   05/24/2018     02:00:00   \n",
              "4   310389190   11/18/2018     16:00:00   11/18/2018     16:10:00   \n",
              "\n",
              "   ADDR_PCT_CD      RPT_DT  KY_CD                       OFNS_DESC  PD_CD  ...  \\\n",
              "0           10  06/10/2018    361   OFF. AGNST PUB ORD SENSBLTY &    639  ...   \n",
              "1           44  08/04/2018    344    ASSAULT 3 & RELATED OFFENSES    101  ...   \n",
              "2          103  11/04/2018    106                  FELONY ASSAULT    109  ...   \n",
              "3           47  05/24/2018    351  CRIMINAL MISCHIEF & RELATED OF    254  ...   \n",
              "4           48  11/18/2018    105                         ROBBERY    388  ...   \n",
              "\n",
              "  SUSP_SEX TRANSIT_DISTRICT   Latitude  Longitude  \\\n",
              "0        M              NaN  40.759310 -73.994706   \n",
              "1        M              NaN  40.826170 -73.916831   \n",
              "2        M              NaN  40.707858 -73.759307   \n",
              "3      NaN              NaN  40.882615 -73.851948   \n",
              "4        M              NaN  40.850357 -73.882989   \n",
              "\n",
              "                         Lat_Lon               PATROL_BORO STATION_NAME  \\\n",
              "0  (40.759310399, -73.994706072)     PATROL BORO MAN SOUTH          NaN   \n",
              "1  (40.826169612, -73.916830709)         PATROL BORO BRONX          NaN   \n",
              "2  (40.707858236, -73.759306969)  PATROL BORO QUEENS SOUTH          NaN   \n",
              "3  (40.882615325, -73.851947659)         PATROL BORO BRONX          NaN   \n",
              "4   (40.85035684, -73.882989431)         PATROL BORO BRONX          NaN   \n",
              "\n",
              "   VIC_AGE_GROUP                  VIC_RACE VIC_SEX  \n",
              "0          18-24            WHITE HISPANIC       F  \n",
              "1          25-44            WHITE HISPANIC       F  \n",
              "2          25-44                     BLACK       M  \n",
              "3          25-44  ASIAN / PACIFIC ISLANDER       F  \n",
              "4            <18            WHITE HISPANIC       M  \n",
              "\n",
              "[5 rows x 35 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5429d60f-ec20-42cd-9901-b8040c52dbb6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CMPLNT_NUM</th>\n",
              "      <th>CMPLNT_FR_DT</th>\n",
              "      <th>CMPLNT_FR_TM</th>\n",
              "      <th>CMPLNT_TO_DT</th>\n",
              "      <th>CMPLNT_TO_TM</th>\n",
              "      <th>ADDR_PCT_CD</th>\n",
              "      <th>RPT_DT</th>\n",
              "      <th>KY_CD</th>\n",
              "      <th>OFNS_DESC</th>\n",
              "      <th>PD_CD</th>\n",
              "      <th>...</th>\n",
              "      <th>SUSP_SEX</th>\n",
              "      <th>TRANSIT_DISTRICT</th>\n",
              "      <th>Latitude</th>\n",
              "      <th>Longitude</th>\n",
              "      <th>Lat_Lon</th>\n",
              "      <th>PATROL_BORO</th>\n",
              "      <th>STATION_NAME</th>\n",
              "      <th>VIC_AGE_GROUP</th>\n",
              "      <th>VIC_RACE</th>\n",
              "      <th>VIC_SEX</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>280364018</td>\n",
              "      <td>06/09/2018</td>\n",
              "      <td>21:42:00</td>\n",
              "      <td>06/09/2018</td>\n",
              "      <td>21:43:00</td>\n",
              "      <td>10</td>\n",
              "      <td>06/10/2018</td>\n",
              "      <td>361</td>\n",
              "      <td>OFF. AGNST PUB ORD SENSBLTY &amp;</td>\n",
              "      <td>639</td>\n",
              "      <td>...</td>\n",
              "      <td>M</td>\n",
              "      <td>NaN</td>\n",
              "      <td>40.759310</td>\n",
              "      <td>-73.994706</td>\n",
              "      <td>(40.759310399, -73.994706072)</td>\n",
              "      <td>PATROL BORO MAN SOUTH</td>\n",
              "      <td>NaN</td>\n",
              "      <td>18-24</td>\n",
              "      <td>WHITE HISPANIC</td>\n",
              "      <td>F</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>377132404</td>\n",
              "      <td>08/04/2018</td>\n",
              "      <td>22:15:00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>44</td>\n",
              "      <td>08/04/2018</td>\n",
              "      <td>344</td>\n",
              "      <td>ASSAULT 3 &amp; RELATED OFFENSES</td>\n",
              "      <td>101</td>\n",
              "      <td>...</td>\n",
              "      <td>M</td>\n",
              "      <td>NaN</td>\n",
              "      <td>40.826170</td>\n",
              "      <td>-73.916831</td>\n",
              "      <td>(40.826169612, -73.916830709)</td>\n",
              "      <td>PATROL BORO BRONX</td>\n",
              "      <td>NaN</td>\n",
              "      <td>25-44</td>\n",
              "      <td>WHITE HISPANIC</td>\n",
              "      <td>F</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>336011712</td>\n",
              "      <td>11/04/2018</td>\n",
              "      <td>11:15:00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>103</td>\n",
              "      <td>11/04/2018</td>\n",
              "      <td>106</td>\n",
              "      <td>FELONY ASSAULT</td>\n",
              "      <td>109</td>\n",
              "      <td>...</td>\n",
              "      <td>M</td>\n",
              "      <td>NaN</td>\n",
              "      <td>40.707858</td>\n",
              "      <td>-73.759307</td>\n",
              "      <td>(40.707858236, -73.759306969)</td>\n",
              "      <td>PATROL BORO QUEENS SOUTH</td>\n",
              "      <td>NaN</td>\n",
              "      <td>25-44</td>\n",
              "      <td>BLACK</td>\n",
              "      <td>M</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>599398393</td>\n",
              "      <td>05/23/2018</td>\n",
              "      <td>23:30:00</td>\n",
              "      <td>05/24/2018</td>\n",
              "      <td>02:00:00</td>\n",
              "      <td>47</td>\n",
              "      <td>05/24/2018</td>\n",
              "      <td>351</td>\n",
              "      <td>CRIMINAL MISCHIEF &amp; RELATED OF</td>\n",
              "      <td>254</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>40.882615</td>\n",
              "      <td>-73.851948</td>\n",
              "      <td>(40.882615325, -73.851947659)</td>\n",
              "      <td>PATROL BORO BRONX</td>\n",
              "      <td>NaN</td>\n",
              "      <td>25-44</td>\n",
              "      <td>ASIAN / PACIFIC ISLANDER</td>\n",
              "      <td>F</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>310389190</td>\n",
              "      <td>11/18/2018</td>\n",
              "      <td>16:00:00</td>\n",
              "      <td>11/18/2018</td>\n",
              "      <td>16:10:00</td>\n",
              "      <td>48</td>\n",
              "      <td>11/18/2018</td>\n",
              "      <td>105</td>\n",
              "      <td>ROBBERY</td>\n",
              "      <td>388</td>\n",
              "      <td>...</td>\n",
              "      <td>M</td>\n",
              "      <td>NaN</td>\n",
              "      <td>40.850357</td>\n",
              "      <td>-73.882989</td>\n",
              "      <td>(40.85035684, -73.882989431)</td>\n",
              "      <td>PATROL BORO BRONX</td>\n",
              "      <td>NaN</td>\n",
              "      <td>&lt;18</td>\n",
              "      <td>WHITE HISPANIC</td>\n",
              "      <td>M</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 35 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5429d60f-ec20-42cd-9901-b8040c52dbb6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5429d60f-ec20-42cd-9901-b8040c52dbb6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5429d60f-ec20-42cd-9901-b8040c52dbb6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data = data[filter_list]"
      ],
      "metadata": {
        "id": "-qBWaedrGWfH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data.info()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ctOWwHuw-kq6",
        "outputId": "ad499608-b22c-42ed-c50a-5e483819543d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 3031883 entries, 0 to 3031882\n",
            "Data columns (total 9 columns):\n",
            " #   Column             Dtype  \n",
            "---  ------             -----  \n",
            " 0   RPT_DT             object \n",
            " 1   OFNS_DESC          object \n",
            " 2   CRM_ATPT_CPTD_CD   object \n",
            " 3   LAW_CAT_CD         object \n",
            " 4   BORO_NM            object \n",
            " 5   LOC_OF_OCCUR_DESC  object \n",
            " 6   PREM_TYP_DESC      object \n",
            " 7   Latitude           float64\n",
            " 8   Longitude          float64\n",
            "dtypes: float64(2), object(7)\n",
            "memory usage: 208.2+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "QjifMvx1-Vrn",
        "outputId": "d7ec9fbd-b375-45b4-a7d1-3d801b0bb37c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       RPT_DT                       OFNS_DESC CRM_ATPT_CPTD_CD   LAW_CAT_CD  \\\n",
              "0  06/10/2018   OFF. AGNST PUB ORD SENSBLTY &        COMPLETED  MISDEMEANOR   \n",
              "1  08/04/2018    ASSAULT 3 & RELATED OFFENSES        COMPLETED  MISDEMEANOR   \n",
              "2  11/04/2018                  FELONY ASSAULT        COMPLETED       FELONY   \n",
              "3  05/24/2018  CRIMINAL MISCHIEF & RELATED OF        COMPLETED  MISDEMEANOR   \n",
              "4  11/18/2018                         ROBBERY        COMPLETED       FELONY   \n",
              "\n",
              "     BORO_NM LOC_OF_OCCUR_DESC           PREM_TYP_DESC   Latitude  Longitude  \n",
              "0  MANHATTAN            INSIDE  RESIDENCE - APT. HOUSE  40.759310 -73.994706  \n",
              "1      BRONX            INSIDE  RESIDENCE - APT. HOUSE  40.826170 -73.916831  \n",
              "2     QUEENS            INSIDE         RESIDENCE-HOUSE  40.707858 -73.759307  \n",
              "3      BRONX            INSIDE         RESIDENCE-HOUSE  40.882615 -73.851948  \n",
              "4      BRONX            INSIDE  RESIDENCE - APT. HOUSE  40.850357 -73.882989  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2edce108-9d8a-4332-aafc-18b0987dd69d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>RPT_DT</th>\n",
              "      <th>OFNS_DESC</th>\n",
              "      <th>CRM_ATPT_CPTD_CD</th>\n",
              "      <th>LAW_CAT_CD</th>\n",
              "      <th>BORO_NM</th>\n",
              "      <th>LOC_OF_OCCUR_DESC</th>\n",
              "      <th>PREM_TYP_DESC</th>\n",
              "      <th>Latitude</th>\n",
              "      <th>Longitude</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>06/10/2018</td>\n",
              "      <td>OFF. AGNST PUB ORD SENSBLTY &amp;</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>MISDEMEANOR</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>40.759310</td>\n",
              "      <td>-73.994706</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>08/04/2018</td>\n",
              "      <td>ASSAULT 3 &amp; RELATED OFFENSES</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>MISDEMEANOR</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>40.826170</td>\n",
              "      <td>-73.916831</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>11/04/2018</td>\n",
              "      <td>FELONY ASSAULT</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>QUEENS</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE-HOUSE</td>\n",
              "      <td>40.707858</td>\n",
              "      <td>-73.759307</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>05/24/2018</td>\n",
              "      <td>CRIMINAL MISCHIEF &amp; RELATED OF</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>MISDEMEANOR</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE-HOUSE</td>\n",
              "      <td>40.882615</td>\n",
              "      <td>-73.851948</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11/18/2018</td>\n",
              "      <td>ROBBERY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>40.850357</td>\n",
              "      <td>-73.882989</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2edce108-9d8a-4332-aafc-18b0987dd69d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2edce108-9d8a-4332-aafc-18b0987dd69d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2edce108-9d8a-4332-aafc-18b0987dd69d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data = pd.read_csv(\"/content/drive/MyDrive/Capstone Project/NYPD_Complaint_Data_Historic.csv\")"
      ],
      "metadata": {
        "id": "ulDBuV0QSjG-"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data = data.dropna()\n",
        "data.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hc7jKvogjb7s",
        "outputId": "720e9e88-d9f7-4ba1-98fa-e9ebd64da0ca"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2989557, 9)"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data = data.sample(n = 10000)\n",
        "data.reset_index(drop=True, inplace=True)\n",
        "data.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JicYTA5ySyJt",
        "outputId": "d8411cc1-4ca0-4dfd-f3e9-89bbf2e9f4ae"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, 9)"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "4i8zRrkRSx7q"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "fAU_Ryu_HPPU"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#converting the lattitudes and logitudes to zipcodes\n",
        "#we are going to use multiprocessing to generate the zipcodes\n",
        "from multiprocessing import  Pool\n",
        "import geopy\n",
        "\n",
        "\n",
        "geolocator = geopy.Nominatim(user_agent='my-application')\n",
        "\n",
        "\n",
        "\n",
        "data['zipcode'] = [0]*len(data)"
      ],
      "metadata": {
        "id": "7VbbdBOZLDCv"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "mivUiP1biDaU"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data.head()"
      ],
      "metadata": {
        "id": "Zrm7j_Fj6ip3",
        "outputId": "dbe2a40e-8fad-4a8b-8828-59c1ba7b5081",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       RPT_DT      OFNS_DESC CRM_ATPT_CPTD_CD   LAW_CAT_CD    BORO_NM  \\\n",
              "0  06/12/2012       BURGLARY        COMPLETED       FELONY      BRONX   \n",
              "1  06/11/2021  PETIT LARCENY        COMPLETED  MISDEMEANOR   BROOKLYN   \n",
              "2  10/24/2006  GRAND LARCENY        COMPLETED       FELONY  MANHATTAN   \n",
              "3  12/03/2008  GRAND LARCENY        COMPLETED       FELONY      BRONX   \n",
              "4  05/15/2017  GRAND LARCENY        COMPLETED       FELONY  MANHATTAN   \n",
              "\n",
              "  LOC_OF_OCCUR_DESC           PREM_TYP_DESC   Latitude  Longitude  zipcode  \n",
              "0            INSIDE  RESIDENCE - APT. HOUSE  40.851099 -73.908030        0  \n",
              "1          FRONT OF         RESIDENCE-HOUSE  40.681719 -73.866902        0  \n",
              "2            INSIDE  RESIDENCE - APT. HOUSE  40.807778 -73.959258        0  \n",
              "3          FRONT OF  RESIDENCE - APT. HOUSE  40.838450 -73.925554        0  \n",
              "4            INSIDE  RESIDENCE - APT. HOUSE  40.793548 -73.973182        0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-255733cd-5e89-49bb-af8b-33d292f7ebe9\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>RPT_DT</th>\n",
              "      <th>OFNS_DESC</th>\n",
              "      <th>CRM_ATPT_CPTD_CD</th>\n",
              "      <th>LAW_CAT_CD</th>\n",
              "      <th>BORO_NM</th>\n",
              "      <th>LOC_OF_OCCUR_DESC</th>\n",
              "      <th>PREM_TYP_DESC</th>\n",
              "      <th>Latitude</th>\n",
              "      <th>Longitude</th>\n",
              "      <th>zipcode</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>06/12/2012</td>\n",
              "      <td>BURGLARY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>40.851099</td>\n",
              "      <td>-73.908030</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>06/11/2021</td>\n",
              "      <td>PETIT LARCENY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>MISDEMEANOR</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>FRONT OF</td>\n",
              "      <td>RESIDENCE-HOUSE</td>\n",
              "      <td>40.681719</td>\n",
              "      <td>-73.866902</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>10/24/2006</td>\n",
              "      <td>GRAND LARCENY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>40.807778</td>\n",
              "      <td>-73.959258</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>12/03/2008</td>\n",
              "      <td>GRAND LARCENY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>FRONT OF</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>40.838450</td>\n",
              "      <td>-73.925554</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>05/15/2017</td>\n",
              "      <td>GRAND LARCENY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>40.793548</td>\n",
              "      <td>-73.973182</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-255733cd-5e89-49bb-af8b-33d292f7ebe9')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-255733cd-5e89-49bb-af8b-33d292f7ebe9 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-255733cd-5e89-49bb-af8b-33d292f7ebe9');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# from more_itertools import sliced\n",
        "# CHUNK_SIZE = 10000\n",
        "\n",
        "# index_slices = sliced(range(len(data)), CHUNK_SIZE)\n",
        "\n",
        "# for i, index_slice in enumerate(index_slices):\n",
        "#   chunk = data.iloc[index_slice]\n",
        "#   chunk.to_csv(f\"nypd_chuck_{i}.csv\")"
      ],
      "metadata": {
        "id": "ont6e0yxexQR"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# def get_zipcode(df, geolocator, lat_field, lon_field):\n",
        "#   location = geolocator.reverse((df[lat_field], df[lon_field]),timeout=5)\n",
        "#   return location.raw['address']['postcode']\n",
        "\n",
        "\n",
        "\n",
        "# def mapping_zipcodes(df, geolocator=geolocator):\n",
        "#   zipcodes = df.apply(get_zipcode, axis=1, geolocator=geolocator, lat_field='Latitude', lon_field='Longitude')\n",
        "\n",
        "#   df['zipcode'] = zipcodes\n",
        "\n",
        "#   return df\n",
        "\n",
        "# def parallelize_dataframe(df, func, n_cores=4):\n",
        "#     df_split = np.array_split(df, n_cores)\n",
        "#     pool = Pool(n_cores)\n",
        "#     df = pd.concat(pool.map(func, df_split))\n",
        "#     pool.close()\n",
        "#     pool.join()\n",
        "#     return df\n",
        "\n",
        "# def save_the_data(data):\n",
        "#   data = parallelize_dataframe(data, mapping_zipcodes)\n",
        "#   data.drop(['Latitude','Longitude'], axis=1, inplace=True)\n",
        "#   data.to_csv(\"/content/drive/MyDrive/Capstone Project/NYPD_Complaint_Data_Historic100000.csv\", index=False)\n",
        "#   data.head()"
      ],
      "metadata": {
        "id": "fQlzHfkeXMPy"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Et-9DJcP4gY5"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_zipcode(df, geolocator, lat_field, lon_field):\n",
        "  try:\n",
        "    location = geolocator.reverse((df[lat_field], df[lon_field]),timeout=5)\n",
        "    if 'address' in location.raw.keys():\n",
        "      if 'postcode' in location.raw['address'].keys():\n",
        "        return location.raw['address']['postcode']\n",
        "    \n",
        "  except:\n",
        "    return None\n",
        "\n",
        "data['zipcode'] = data.apply(get_zipcode, axis=1, geolocator=geolocator, lat_field='Latitude', lon_field='Longitude')\n",
        "data.drop(['Latitude','Longitude'], axis=1, inplace=True)\n",
        "data.to_csv(\"/content/drive/MyDrive/Capstone Project/NYPD_Complaint_Data_Historic10000.csv\", index=False)"
      ],
      "metadata": {
        "id": "caGzApUPSM-J"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Ng1xr1L3SZlT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data.head() "
      ],
      "metadata": {
        "id": "erDbQnwlNNAn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "dbbe3586-9e0e-4b38-ad17-96e27de985fb"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       RPT_DT      OFNS_DESC CRM_ATPT_CPTD_CD   LAW_CAT_CD    BORO_NM  \\\n",
              "0  06/12/2012       BURGLARY        COMPLETED       FELONY      BRONX   \n",
              "1  06/11/2021  PETIT LARCENY        COMPLETED  MISDEMEANOR   BROOKLYN   \n",
              "2  10/24/2006  GRAND LARCENY        COMPLETED       FELONY  MANHATTAN   \n",
              "3  12/03/2008  GRAND LARCENY        COMPLETED       FELONY      BRONX   \n",
              "4  05/15/2017  GRAND LARCENY        COMPLETED       FELONY  MANHATTAN   \n",
              "\n",
              "  LOC_OF_OCCUR_DESC           PREM_TYP_DESC zipcode  \n",
              "0            INSIDE  RESIDENCE - APT. HOUSE   10453  \n",
              "1          FRONT OF         RESIDENCE-HOUSE   11208  \n",
              "2            INSIDE  RESIDENCE - APT. HOUSE   10027  \n",
              "3          FRONT OF  RESIDENCE - APT. HOUSE   10452  \n",
              "4            INSIDE  RESIDENCE - APT. HOUSE   10025  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e4f63c10-dc2d-44d1-960f-7c529a36b518\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>RPT_DT</th>\n",
              "      <th>OFNS_DESC</th>\n",
              "      <th>CRM_ATPT_CPTD_CD</th>\n",
              "      <th>LAW_CAT_CD</th>\n",
              "      <th>BORO_NM</th>\n",
              "      <th>LOC_OF_OCCUR_DESC</th>\n",
              "      <th>PREM_TYP_DESC</th>\n",
              "      <th>zipcode</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>06/12/2012</td>\n",
              "      <td>BURGLARY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>10453</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>06/11/2021</td>\n",
              "      <td>PETIT LARCENY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>MISDEMEANOR</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>FRONT OF</td>\n",
              "      <td>RESIDENCE-HOUSE</td>\n",
              "      <td>11208</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>10/24/2006</td>\n",
              "      <td>GRAND LARCENY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>10027</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>12/03/2008</td>\n",
              "      <td>GRAND LARCENY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>BRONX</td>\n",
              "      <td>FRONT OF</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>10452</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>05/15/2017</td>\n",
              "      <td>GRAND LARCENY</td>\n",
              "      <td>COMPLETED</td>\n",
              "      <td>FELONY</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>INSIDE</td>\n",
              "      <td>RESIDENCE - APT. HOUSE</td>\n",
              "      <td>10025</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e4f63c10-dc2d-44d1-960f-7c529a36b518')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e4f63c10-dc2d-44d1-960f-7c529a36b518 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e4f63c10-dc2d-44d1-960f-7c529a36b518');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "hR3OVykENQFr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## For The FDNYC"
      ],
      "metadata": {
        "id": "xAZgFvbP8rf0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data = pd.read_csv(\"/content/drive/MyDrive/Capstone Project/Fire_Incident_Dispatch_Data.csv\")"
      ],
      "metadata": {
        "id": "DlwkD28Aqqg9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data.head()"
      ],
      "metadata": {
        "id": "OeXEI0iR9572",
        "outputId": "9d270695-66f7-41ff-fe03-3dc1f85e8755",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 399
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   STARFIRE_INCIDENT_ID       INCIDENT_DATETIME ALARM_BOX_BOROUGH  \\\n",
              "0          5.001924e+14  01/01/2005 12:07:32 AM            QUEENS   \n",
              "1          5.001149e+14  01/01/2005 12:14:40 AM         MANHATTAN   \n",
              "2          5.001065e+14  01/01/2005 12:24:58 AM          BROOKLYN   \n",
              "3          5.001165e+14  01/01/2005 12:27:19 AM         MANHATTAN   \n",
              "4          5.001165e+14  01/01/2005 12:27:19 AM         MANHATTAN   \n",
              "\n",
              "   ALARM_BOX_NUMBER                ALARM_BOX_LOCATION INCIDENT_BOROUGH  \\\n",
              "0              9237  N/SVC RD H. HARDING EXPY & 99 ST           QUEENS   \n",
              "1              1493   BWAY & W125 ST\\M.L.KING JR BLVD        MANHATTAN   \n",
              "2               653          LAFAYETTE & CLASSON AVES         BROOKLYN   \n",
              "3              1649             RIVERSIDE DR & 150 ST        MANHATTAN   \n",
              "4              1649             RIVERSIDE DR & 150 ST        MANHATTAN   \n",
              "\n",
              "   ZIPCODE  POLICEPRECINCT  CITYCOUNCILDISTRICT  COMMUNITYDISTRICT  ...  \\\n",
              "0  11368.0           110.0                 21.0              404.0  ...   \n",
              "1  10027.0            26.0                  7.0              109.0  ...   \n",
              "2  11238.0            79.0                 35.0              303.0  ...   \n",
              "3  10031.0            30.0                  7.0              109.0  ...   \n",
              "4  10031.0            30.0                  7.0              109.0  ...   \n",
              "\n",
              "   FIRST_ACTIVATION_DATETIME  FIRST_ON_SCENE_DATETIME INCIDENT_CLOSE_DATETIME  \\\n",
              "0     01/01/2005 12:09:31 AM   01/01/2005 12:13:10 AM  01/01/2005 12:33:42 AM   \n",
              "1     01/01/2005 12:15:43 AM   01/01/2005 12:19:06 AM  01/01/2005 12:35:27 AM   \n",
              "2     01/01/2005 12:25:51 AM   01/01/2005 12:28:44 AM  01/01/2005 12:47:38 AM   \n",
              "3     01/01/2005 12:28:48 AM   01/01/2005 12:31:53 AM  01/01/2005 02:25:27 AM   \n",
              "4     01/01/2005 12:28:48 AM   01/01/2005 12:31:53 AM  01/01/2005 02:25:27 AM   \n",
              "\n",
              "  VALID_DISPATCH_RSPNS_TIME_INDC VALID_INCIDENT_RSPNS_TIME_INDC  \\\n",
              "0                              N                              Y   \n",
              "1                              N                              Y   \n",
              "2                              N                              Y   \n",
              "3                              N                              Y   \n",
              "4                              N                              Y   \n",
              "\n",
              "  INCIDENT_RESPONSE_SECONDS_QY INCIDENT_TRAVEL_TM_SECONDS_QY  \\\n",
              "0                          338                           236   \n",
              "1                          266                           217   \n",
              "2                          226                           189   \n",
              "3                          274                           200   \n",
              "4                          274                           200   \n",
              "\n",
              "   ENGINES_ASSIGNED_QUANTITY LADDERS_ASSIGNED_QUANTITY  \\\n",
              "0                          3                         2   \n",
              "1                          2                         2   \n",
              "2                          3                         2   \n",
              "3                          5                         3   \n",
              "4                          5                         3   \n",
              "\n",
              "  OTHER_UNITS_ASSIGNED_QUANTITY  \n",
              "0                             2  \n",
              "1                             1  \n",
              "2                             1  \n",
              "3                             5  \n",
              "4                             5  \n",
              "\n",
              "[5 rows x 29 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2bdd2a3a-b977-4734-b9bb-e602c3ab7816\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>STARFIRE_INCIDENT_ID</th>\n",
              "      <th>INCIDENT_DATETIME</th>\n",
              "      <th>ALARM_BOX_BOROUGH</th>\n",
              "      <th>ALARM_BOX_NUMBER</th>\n",
              "      <th>ALARM_BOX_LOCATION</th>\n",
              "      <th>INCIDENT_BOROUGH</th>\n",
              "      <th>ZIPCODE</th>\n",
              "      <th>POLICEPRECINCT</th>\n",
              "      <th>CITYCOUNCILDISTRICT</th>\n",
              "      <th>COMMUNITYDISTRICT</th>\n",
              "      <th>...</th>\n",
              "      <th>FIRST_ACTIVATION_DATETIME</th>\n",
              "      <th>FIRST_ON_SCENE_DATETIME</th>\n",
              "      <th>INCIDENT_CLOSE_DATETIME</th>\n",
              "      <th>VALID_DISPATCH_RSPNS_TIME_INDC</th>\n",
              "      <th>VALID_INCIDENT_RSPNS_TIME_INDC</th>\n",
              "      <th>INCIDENT_RESPONSE_SECONDS_QY</th>\n",
              "      <th>INCIDENT_TRAVEL_TM_SECONDS_QY</th>\n",
              "      <th>ENGINES_ASSIGNED_QUANTITY</th>\n",
              "      <th>LADDERS_ASSIGNED_QUANTITY</th>\n",
              "      <th>OTHER_UNITS_ASSIGNED_QUANTITY</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5.001924e+14</td>\n",
              "      <td>01/01/2005 12:07:32 AM</td>\n",
              "      <td>QUEENS</td>\n",
              "      <td>9237</td>\n",
              "      <td>N/SVC RD H. HARDING EXPY &amp; 99 ST</td>\n",
              "      <td>QUEENS</td>\n",
              "      <td>11368.0</td>\n",
              "      <td>110.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>404.0</td>\n",
              "      <td>...</td>\n",
              "      <td>01/01/2005 12:09:31 AM</td>\n",
              "      <td>01/01/2005 12:13:10 AM</td>\n",
              "      <td>01/01/2005 12:33:42 AM</td>\n",
              "      <td>N</td>\n",
              "      <td>Y</td>\n",
              "      <td>338</td>\n",
              "      <td>236</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5.001149e+14</td>\n",
              "      <td>01/01/2005 12:14:40 AM</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>1493</td>\n",
              "      <td>BWAY &amp; W125 ST\\M.L.KING JR BLVD</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10027.0</td>\n",
              "      <td>26.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>109.0</td>\n",
              "      <td>...</td>\n",
              "      <td>01/01/2005 12:15:43 AM</td>\n",
              "      <td>01/01/2005 12:19:06 AM</td>\n",
              "      <td>01/01/2005 12:35:27 AM</td>\n",
              "      <td>N</td>\n",
              "      <td>Y</td>\n",
              "      <td>266</td>\n",
              "      <td>217</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5.001065e+14</td>\n",
              "      <td>01/01/2005 12:24:58 AM</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>653</td>\n",
              "      <td>LAFAYETTE &amp; CLASSON AVES</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>11238.0</td>\n",
              "      <td>79.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>303.0</td>\n",
              "      <td>...</td>\n",
              "      <td>01/01/2005 12:25:51 AM</td>\n",
              "      <td>01/01/2005 12:28:44 AM</td>\n",
              "      <td>01/01/2005 12:47:38 AM</td>\n",
              "      <td>N</td>\n",
              "      <td>Y</td>\n",
              "      <td>226</td>\n",
              "      <td>189</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5.001165e+14</td>\n",
              "      <td>01/01/2005 12:27:19 AM</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>1649</td>\n",
              "      <td>RIVERSIDE DR &amp; 150 ST</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10031.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>109.0</td>\n",
              "      <td>...</td>\n",
              "      <td>01/01/2005 12:28:48 AM</td>\n",
              "      <td>01/01/2005 12:31:53 AM</td>\n",
              "      <td>01/01/2005 02:25:27 AM</td>\n",
              "      <td>N</td>\n",
              "      <td>Y</td>\n",
              "      <td>274</td>\n",
              "      <td>200</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5.001165e+14</td>\n",
              "      <td>01/01/2005 12:27:19 AM</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>1649</td>\n",
              "      <td>RIVERSIDE DR &amp; 150 ST</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10031.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>109.0</td>\n",
              "      <td>...</td>\n",
              "      <td>01/01/2005 12:28:48 AM</td>\n",
              "      <td>01/01/2005 12:31:53 AM</td>\n",
              "      <td>01/01/2005 02:25:27 AM</td>\n",
              "      <td>N</td>\n",
              "      <td>Y</td>\n",
              "      <td>274</td>\n",
              "      <td>200</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 29 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2bdd2a3a-b977-4734-b9bb-e602c3ab7816')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2bdd2a3a-b977-4734-b9bb-e602c3ab7816 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2bdd2a3a-b977-4734-b9bb-e602c3ab7816');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "HnXQIzQJ954s"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data = data[[\n",
        "              'INCIDENT_DATETIME',\n",
        "              'INCIDENT_BOROUGH',\n",
        "              'ZIPCODE',\n",
        "              'HIGHEST_ALARM_LEVEL',\n",
        "              'INCIDENT_CLASSIFICATION',\n",
        "              'INCIDENT_CLASSIFICATION_GROUP',\n",
        "\n",
        "              ]]\n",
        "data.head()"
      ],
      "metadata": {
        "id": "gUB8BF9d951S",
        "outputId": "b67096f7-ae37-4290-a832-247904eb7fb7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        INCIDENT_DATETIME INCIDENT_BOROUGH  ZIPCODE HIGHEST_ALARM_LEVEL  \\\n",
              "0  01/01/2005 12:07:32 AM           QUEENS  11368.0         First Alarm   \n",
              "1  01/01/2005 12:14:40 AM        MANHATTAN  10027.0         First Alarm   \n",
              "2  01/01/2005 12:24:58 AM         BROOKLYN  11238.0         First Alarm   \n",
              "3  01/01/2005 12:27:19 AM        MANHATTAN  10031.0       Seventh Alarm   \n",
              "4  01/01/2005 12:27:19 AM        MANHATTAN  10031.0   All Hands Working   \n",
              "\n",
              "                  INCIDENT_CLASSIFICATION INCIDENT_CLASSIFICATION_GROUP  \n",
              "0      Multiple Dwelling 'A' - Other fire              Structural Fires  \n",
              "1  Multiple Dwelling 'A' - Compactor fire              Structural Fires  \n",
              "2  Multiple Dwelling 'A' - Compactor fire              Structural Fires  \n",
              "3      Multiple Dwelling 'A' - Other fire              Structural Fires  \n",
              "4      Multiple Dwelling 'A' - Other fire              Structural Fires  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a1301d00-ca66-4871-9f1c-43a8aa1f77a4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>INCIDENT_DATETIME</th>\n",
              "      <th>INCIDENT_BOROUGH</th>\n",
              "      <th>ZIPCODE</th>\n",
              "      <th>HIGHEST_ALARM_LEVEL</th>\n",
              "      <th>INCIDENT_CLASSIFICATION</th>\n",
              "      <th>INCIDENT_CLASSIFICATION_GROUP</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>01/01/2005 12:07:32 AM</td>\n",
              "      <td>QUEENS</td>\n",
              "      <td>11368.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>01/01/2005 12:14:40 AM</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10027.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Compactor fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>01/01/2005 12:24:58 AM</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>11238.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Compactor fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>01/01/2005 12:27:19 AM</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10031.0</td>\n",
              "      <td>Seventh Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>01/01/2005 12:27:19 AM</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10031.0</td>\n",
              "      <td>All Hands Working</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a1301d00-ca66-4871-9f1c-43a8aa1f77a4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a1301d00-ca66-4871-9f1c-43a8aa1f77a4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a1301d00-ca66-4871-9f1c-43a8aa1f77a4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data['INCIDENT_DATETIME'] = pd.to_datetime(data['INCIDENT_DATETIME'], infer_datetime_format=True)\n",
        "print(f\"the last date {np.max(data['INCIDENT_DATETIME'])}\")\n",
        "print(f\"the begining  date {np.min(data['INCIDENT_DATETIME'])}\")"
      ],
      "metadata": {
        "id": "05F8SqtB-EZn",
        "outputId": "14f90b9c-93b5-4f5f-a096-4ca7e7dd8e63",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "the last date 2021-08-01 23:59:54\n",
            "the begining  date 2005-01-01 00:07:32\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data.head()"
      ],
      "metadata": {
        "id": "bsDAOqR4-IGT",
        "outputId": "fb57a529-3d42-42c0-ef19-518a8862518d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    INCIDENT_DATETIME INCIDENT_BOROUGH  ZIPCODE HIGHEST_ALARM_LEVEL  \\\n",
              "0 2005-01-01 00:07:32           QUEENS  11368.0         First Alarm   \n",
              "1 2005-01-01 00:14:40        MANHATTAN  10027.0         First Alarm   \n",
              "2 2005-01-01 00:24:58         BROOKLYN  11238.0         First Alarm   \n",
              "3 2005-01-01 00:27:19        MANHATTAN  10031.0       Seventh Alarm   \n",
              "4 2005-01-01 00:27:19        MANHATTAN  10031.0   All Hands Working   \n",
              "\n",
              "                  INCIDENT_CLASSIFICATION INCIDENT_CLASSIFICATION_GROUP  \n",
              "0      Multiple Dwelling 'A' - Other fire              Structural Fires  \n",
              "1  Multiple Dwelling 'A' - Compactor fire              Structural Fires  \n",
              "2  Multiple Dwelling 'A' - Compactor fire              Structural Fires  \n",
              "3      Multiple Dwelling 'A' - Other fire              Structural Fires  \n",
              "4      Multiple Dwelling 'A' - Other fire              Structural Fires  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5d71a605-a192-44ac-8fe9-e4ab5d128650\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>INCIDENT_DATETIME</th>\n",
              "      <th>INCIDENT_BOROUGH</th>\n",
              "      <th>ZIPCODE</th>\n",
              "      <th>HIGHEST_ALARM_LEVEL</th>\n",
              "      <th>INCIDENT_CLASSIFICATION</th>\n",
              "      <th>INCIDENT_CLASSIFICATION_GROUP</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2005-01-01 00:07:32</td>\n",
              "      <td>QUEENS</td>\n",
              "      <td>11368.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2005-01-01 00:14:40</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10027.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Compactor fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2005-01-01 00:24:58</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>11238.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Compactor fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2005-01-01 00:27:19</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10031.0</td>\n",
              "      <td>Seventh Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2005-01-01 00:27:19</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10031.0</td>\n",
              "      <td>All Hands Working</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5d71a605-a192-44ac-8fe9-e4ab5d128650')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5d71a605-a192-44ac-8fe9-e4ab5d128650 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5d71a605-a192-44ac-8fe9-e4ab5d128650');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data.dropna(inplace=True)\n",
        "data.head()"
      ],
      "metadata": {
        "id": "TOHr99rL-IDM",
        "outputId": "d5def814-7cb0-4b22-e608-e31d7af8f4a0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    INCIDENT_DATETIME INCIDENT_BOROUGH  ZIPCODE HIGHEST_ALARM_LEVEL  \\\n",
              "0 2005-01-01 00:07:32           QUEENS  11368.0         First Alarm   \n",
              "1 2005-01-01 00:14:40        MANHATTAN  10027.0         First Alarm   \n",
              "2 2005-01-01 00:24:58         BROOKLYN  11238.0         First Alarm   \n",
              "3 2005-01-01 00:27:19        MANHATTAN  10031.0       Seventh Alarm   \n",
              "4 2005-01-01 00:27:19        MANHATTAN  10031.0   All Hands Working   \n",
              "\n",
              "                  INCIDENT_CLASSIFICATION INCIDENT_CLASSIFICATION_GROUP  \n",
              "0      Multiple Dwelling 'A' - Other fire              Structural Fires  \n",
              "1  Multiple Dwelling 'A' - Compactor fire              Structural Fires  \n",
              "2  Multiple Dwelling 'A' - Compactor fire              Structural Fires  \n",
              "3      Multiple Dwelling 'A' - Other fire              Structural Fires  \n",
              "4      Multiple Dwelling 'A' - Other fire              Structural Fires  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a71732a9-ccc6-4943-9e7b-e08275c2c96c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>INCIDENT_DATETIME</th>\n",
              "      <th>INCIDENT_BOROUGH</th>\n",
              "      <th>ZIPCODE</th>\n",
              "      <th>HIGHEST_ALARM_LEVEL</th>\n",
              "      <th>INCIDENT_CLASSIFICATION</th>\n",
              "      <th>INCIDENT_CLASSIFICATION_GROUP</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2005-01-01 00:07:32</td>\n",
              "      <td>QUEENS</td>\n",
              "      <td>11368.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2005-01-01 00:14:40</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10027.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Compactor fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2005-01-01 00:24:58</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>11238.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Compactor fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2005-01-01 00:27:19</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10031.0</td>\n",
              "      <td>Seventh Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2005-01-01 00:27:19</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10031.0</td>\n",
              "      <td>All Hands Working</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a71732a9-ccc6-4943-9e7b-e08275c2c96c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a71732a9-ccc6-4943-9e7b-e08275c2c96c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a71732a9-ccc6-4943-9e7b-e08275c2c96c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "JjXp5ym6-IAl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data.info()"
      ],
      "metadata": {
        "id": "skDQ5CjC-H9u",
        "outputId": "202b8e8c-a168-4852-8a11-f681b4dca145",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 386662 entries, 0 to 388531\n",
            "Data columns (total 6 columns):\n",
            " #   Column                         Non-Null Count   Dtype         \n",
            "---  ------                         --------------   -----         \n",
            " 0   INCIDENT_DATETIME              386662 non-null  datetime64[ns]\n",
            " 1   INCIDENT_BOROUGH               386662 non-null  object        \n",
            " 2   ZIPCODE                        386662 non-null  float64       \n",
            " 3   HIGHEST_ALARM_LEVEL            386662 non-null  object        \n",
            " 4   INCIDENT_CLASSIFICATION        386662 non-null  object        \n",
            " 5   INCIDENT_CLASSIFICATION_GROUP  386662 non-null  object        \n",
            "dtypes: datetime64[ns](1), float64(1), object(4)\n",
            "memory usage: 20.6+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data = pd.read_csv(\"/content/drive/MyDrive/Capstone Project/Fire_Incident_Dispatch_Data.csv\")\n",
        "data = data.sample(n = 100000)\n",
        "data.reset_index(drop=True, inplace=True)\n",
        "data.shape"
      ],
      "metadata": {
        "id": "u6wcYkeU-QTD",
        "outputId": "0bef4039-0240-4eaa-cfbb-283ce9a54d8d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(100000, 6)"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data.head()"
      ],
      "metadata": {
        "id": "QwGmOWI3FHMS",
        "outputId": "12fce7c6-536c-40c5-e021-163a6de5b3b5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     INCIDENT_DATETIME INCIDENT_BOROUGH  ZIPCODE HIGHEST_ALARM_LEVEL  \\\n",
              "0  2014-03-15 06:39:53        MANHATTAN  10010.0         First Alarm   \n",
              "1  2021-03-18 08:53:00         BROOKLYN  11232.0         First Alarm   \n",
              "2  2020-07-15 19:26:00        MANHATTAN  10023.0         First Alarm   \n",
              "3  2012-02-10 12:22:46        MANHATTAN  10023.0         First Alarm   \n",
              "4  2014-06-18 18:05:02         BROOKLYN  11212.0         First Alarm   \n",
              "\n",
              "                          INCIDENT_CLASSIFICATION  \\\n",
              "0              Multiple Dwelling 'A' - Other fire   \n",
              "1  Multiple Dwelling 'A' - Food on the stove fire   \n",
              "2  Multiple Dwelling 'A' - Food on the stove fire   \n",
              "3              Multiple Dwelling 'A' - Other fire   \n",
              "4                      Multiple Dwelling 'B' Fire   \n",
              "\n",
              "  INCIDENT_CLASSIFICATION_GROUP  \n",
              "0              Structural Fires  \n",
              "1              Structural Fires  \n",
              "2              Structural Fires  \n",
              "3              Structural Fires  \n",
              "4              Structural Fires  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7d4860b7-d0fe-46b6-8621-ee26c52fe9ba\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>INCIDENT_DATETIME</th>\n",
              "      <th>INCIDENT_BOROUGH</th>\n",
              "      <th>ZIPCODE</th>\n",
              "      <th>HIGHEST_ALARM_LEVEL</th>\n",
              "      <th>INCIDENT_CLASSIFICATION</th>\n",
              "      <th>INCIDENT_CLASSIFICATION_GROUP</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2014-03-15 06:39:53</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10010.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2021-03-18 08:53:00</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>11232.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Food on the stove fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2020-07-15 19:26:00</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10023.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Food on the stove fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2012-02-10 12:22:46</td>\n",
              "      <td>MANHATTAN</td>\n",
              "      <td>10023.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'A' - Other fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2014-06-18 18:05:02</td>\n",
              "      <td>BROOKLYN</td>\n",
              "      <td>11212.0</td>\n",
              "      <td>First Alarm</td>\n",
              "      <td>Multiple Dwelling 'B' Fire</td>\n",
              "      <td>Structural Fires</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7d4860b7-d0fe-46b6-8621-ee26c52fe9ba')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-7d4860b7-d0fe-46b6-8621-ee26c52fe9ba button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-7d4860b7-d0fe-46b6-8621-ee26c52fe9ba');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data.to_csv(\"/content/drive/MyDrive/Capstone Project/Fire_Incident_Dispatch_Data100000.csv\", index=False)"
      ],
      "metadata": {
        "id": "uY8Hr2Nj-QQO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "13_8YTuy-8Yh"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3.9.5 ('capstone')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.5"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "6c67f8adbebdea49f93c63d2a2dcd5cb4fed3a722249876be11f90d0fd0a91fc"
      }
    },
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}