Obtaining_and_Scrubbing_the_Data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Obtaining_and_Scrubbing_the_Data.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm",
      "mount_file_id": "1eZHXz7JQz3jfmS8vbEBGmO9w5a7CjLQ1",
      "authorship_tag": "ABX9TyNAsyZ817hlJmaWSNd2Rzf6",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AvonleaFisher/Analyzing-NYC-311-Service-Requests/blob/main/Obtaining_and_Scrubbing_the_Data.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDSXHbtye-LG"
      },
      "source": [
        "# Introduction\n",
        "\n",
        "This project aims to explore trends in recent data on NYC 311 calls, and to train a neural network that can classifify the agency that responded to a given call. Government agencies and local policy stakeholders can more effectively serve NYC residents with a fuller understanding of the content, timing, frequency, and location of the issues that 311 callers raise. This project seeks to contribute to that understanding through the exploratory analysis and visualization of public 311 data. The modeling section, which focuses on classifying the responding agency, attempts to serve as a proof-of-concept for a tool that could directly and efficiently connect residents to appropriate government services. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vH6ovDoqfFdv"
      },
      "source": [
        "# Data Sources\n",
        "\n",
        "The data used in this project was obtained from two sources: \n",
        "\n",
        "- [NYC Open Data's 311 Service Requests from 2010 to Present](https://data.cityofnewyork.us/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9)\n",
        "  - This dataset contains information about the time, location, complaint type, and status of more than 24 million 311 service requests made in New York City within the past decade. This project uses a subset of the data from 2020 that was accessed with the [Socrata Open Data (SODA) API](https://dev.socrata.com/consumers/getting-started.html). \n",
        "- [NYC Department of City Planning’s Community District Profiles](https://communityprofiles.planning.nyc.gov/)\n",
        "  - After navigating to any profile on the Community District Profiles website,  the Indicators Data can be obtained under \"Download the Data.\" This dataset contains development and population information for each Community District in New York City. Community board names, which correspond to community districts, can also be found in the 311 dataset. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "97g6-Nn9ferV"
      },
      "source": [
        "# Loading Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aIf7ni3WfBbM"
      },
      "source": [
        "pip install sodapy\n",
        "from sodapy import Socrata\n",
        "import pandas as pd\n",
        "from google.colab import drive\n",
        "import re"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9P4Xt5fYflXN"
      },
      "source": [
        "# Obtaining the 311 Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SUqfuf94fpEr"
      },
      "source": [
        "The [API Documentation](https://dev.socrata.com/foundry/data.cityofnewyork.us/erm2-nwe9) for this dataset contains further information about how to obtain filtered versions of the data. Below, we simply request the 1500000 most recent 311 calls. The line ``client.timeout = 1000`` prevents the Socrata connection from timing out after its default setting of 10 seconds. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6YD-bMvUfAY2"
      },
      "source": [
        "client = Socrata(\"data.cityofnewyork.us\", None)\n",
        "\n",
        "client.timeout = 1000\n",
        "results = client.get(\"erm2-nwe9\", limit=1500000)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "20JGEyd9f40p"
      },
      "source": [
        "Below, the results are stored into a pandas dataframe:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m6x8AG89fzSl"
      },
      "source": [
        "df = pd.DataFrame.from_records(results)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TIW9rM9Wf2lK"
      },
      "source": [
        "# Scrubbing the Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rEDtgSQeu0gD"
      },
      "source": [
        "The NYC 311 Service Requests and Community District Indicators have both been obtained and exported. In this section, we'll merge the datasets on the community district/board columns, deal with missing values, reformat values for consistency, and exclude superfluous columns. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SY5Riq7eEg6C"
      },
      "source": [
        "### Dealing with Missing and Superfluous Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eY45fsVnvMRI"
      },
      "source": [
        "Let's check the column data types and identify columns with missing values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dQmZetA1rTu3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bfc04897-c8b1-4816-ef81-ad5414f7b30d"
      },
      "source": [
        "df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 1500000 entries, 0 to 1499999\n",
            "Data columns (total 46 columns):\n",
            " #   Column                          Non-Null Count    Dtype  \n",
            "---  ------                          --------------    -----  \n",
            " 0   unique_key                      1500000 non-null  int64  \n",
            " 1   created_date                    1500000 non-null  object \n",
            " 2   agency                          1500000 non-null  object \n",
            " 3   agency_name                     1500000 non-null  object \n",
            " 4   complaint_type                  1500000 non-null  object \n",
            " 5   descriptor                      1422438 non-null  object \n",
            " 6   location_type                   1237278 non-null  object \n",
            " 7   incident_zip                    1467586 non-null  float64\n",
            " 8   incident_address                1437587 non-null  object \n",
            " 9   street_name                     1437527 non-null  object \n",
            " 10  cross_street_1                  1217249 non-null  object \n",
            " 11  cross_street_2                  1216469 non-null  object \n",
            " 12  intersection_street_1           1128813 non-null  object \n",
            " 13  intersection_street_2           1128837 non-null  object \n",
            " 14  city                            1389041 non-null  object \n",
            " 15  landmark                        991719 non-null   object \n",
            " 16  status                          1500000 non-null  object \n",
            " 17  community_board                 1500000 non-null  object \n",
            " 18  bbl                             1298580 non-null  float64\n",
            " 19  borough                         1500000 non-null  object \n",
            " 20  x_coordinate_state_plane        1460622 non-null  float64\n",
            " 21  y_coordinate_state_plane        1460622 non-null  float64\n",
            " 22  open_data_channel_type          1500000 non-null  object \n",
            " 23  park_facility_name              1499380 non-null  object \n",
            " 24  park_borough                    1500000 non-null  object \n",
            " 25  latitude                        1460510 non-null  float64\n",
            " 26  longitude                       1460510 non-null  float64\n",
            " 27  location                        1460510 non-null  object \n",
            " 28  :@computed_region_efsh_h5xi     1455794 non-null  float64\n",
            " 29  :@computed_region_f5dn_yrer     1460131 non-null  float64\n",
            " 30  :@computed_region_yeji_bk3q     1460130 non-null  float64\n",
            " 31  :@computed_region_92fq_4b7q     1460130 non-null  float64\n",
            " 32  :@computed_region_sbqj_enih     1460131 non-null  float64\n",
            " 33  resolution_description          1418604 non-null  object \n",
            " 34  resolution_action_updated_date  1419722 non-null  object \n",
            " 35  closed_date                     1382171 non-null  object \n",
            " 36  address_type                    446566 non-null   object \n",
            " 37  facility_type                   38080 non-null    object \n",
            " 38  bridge_highway_name             181 non-null      object \n",
            " 39  bridge_highway_direction        1699 non-null     object \n",
            " 40  road_ramp                       1687 non-null     object \n",
            " 41  bridge_highway_segment          1687 non-null     object \n",
            " 42  taxi_pick_up_location           5144 non-null     object \n",
            " 43  taxi_company_borough            637 non-null      object \n",
            " 44  vehicle_type                    271 non-null      object \n",
            " 45  due_date                        233 non-null      object \n",
            "dtypes: float64(11), int64(1), object(34)\n",
            "memory usage: 537.9+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "URQiF0QM0QY6"
      },
      "source": [
        "The dataset consists of float, integer, and object data types. Most of the columns that contain missing values appear to be associated with particular complaint types. Next, we overwrite the dataframe to include only the rows and columns of interest."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lBZdbiQMEO9C"
      },
      "source": [
        "#eliminate rows without lat/long and location type data\n",
        "df = df[df.latitude.notnull()]\n",
        "df = df[df.longitude.notnull()]\n",
        "df = df[df.descriptor.notnull()]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YADBHHe7Ga7M"
      },
      "source": [
        "#exclude unnecessary columns\n",
        "df = df[['created_date', 'complaint_type', 'agency', 'community_board', \n",
        "         'latitude', 'longitude', 'descriptor', 'borough']]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-V7_gIVsUc0m"
      },
      "source": [
        "\n",
        "### Descriptions of Remaining Columns\n",
        "\n",
        "The descriptions for every column are available on the dataset's NYCOpenData profile. The descriptions for the columns included in this analysis have been pasted below.\n",
        "*   **Created Date**: Date service request(SR) was created\n",
        "*   **Complaint Type**: This is the first level of a hierarchy identifying the topic of the incident or condition. Complaint Type may have a corresponding Descriptor (below) or may stand alone.\n",
        "*   **Agency**: Acronym of responding City Government Agency\n",
        "*   **Community Board**: Provided by geovalidation.\n",
        "*   **Latitude**: Geo based Lat of the incident location\n",
        "*   **Longitude**: eo based Long of the incident location\n",
        "*   **Descriptor**: This is associated to the Complaint Type, and provides further detail on the incident or condition. Descriptor values are dependent on the Complaint Type, and are not always required in SR.\n",
        "*   **Borough**: Provided by the submitter and confirmed by geovalidation.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j5U47ypncjtN"
      },
      "source": [
        "### Reformatting Values for Consistency"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1sCXBiV7rrlK"
      },
      "source": [
        "If we look at unique values in the 'complaint type' column, we'll see that some values are in title case while others are in upper case.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vAvU33uQtDYO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "af3e9a7b-f081-432f-e52f-024dee0bb4b6"
      },
      "source": [
        "df.complaint_type.unique()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['Noise - Residential', 'Illegal Parking',\n",
              "       'Noise - Street/Sidewalk', 'NonCompliance with Phased Reopening',\n",
              "       'Blocked Driveway', 'Noise - Vehicle', 'Street Sign - Damaged',\n",
              "       'Overgrown Tree/Branches', 'Animal-Abuse', 'Noise - Commercial',\n",
              "       'Mass Gathering Complaint', 'Traffic Signal Condition',\n",
              "       'Non-Emergency Police Matter', 'Rodent', 'Street Sign - Missing',\n",
              "       'Noise - Park', 'Maintenance or Facility', 'Illegal Tree Damage',\n",
              "       'Food Establishment', 'Street Condition', 'Street Light Condition',\n",
              "       'Noise', 'HEAT/HOT WATER', 'UNSANITARY CONDITION',\n",
              "       'Construction Lead Dust', 'GENERAL', 'Air Quality', 'ELECTRIC',\n",
              "       'DOOR/WINDOW', 'PLUMBING', 'APPLIANCE',\n",
              "       'Unsanitary Animal Facility', 'Drinking', 'Sidewalk Condition',\n",
              "       'PAINT/PLASTER', 'Traffic', 'Water System', 'Curb Condition',\n",
              "       'Indoor Sewage', 'WATER LEAK', 'Sewer', 'Outdoor Dining',\n",
              "       'FLOORING/STAIRS', 'Drug Activity', 'Violation of Park Rules',\n",
              "       'Noise - Helicopter', 'Abandoned Vehicle', 'New Tree Request',\n",
              "       'For Hire Vehicle Complaint', 'Hazardous Materials', 'Lead',\n",
              "       'Indoor Air Quality', 'Lost Property', 'SAFETY', 'Taxi Complaint',\n",
              "       'Cooling Tower', 'Dead/Dying Tree', 'Consumer Complaint',\n",
              "       'Smoking', 'Root/Sewer/Sidewalk Condition', 'Vending',\n",
              "       'School Maintenance', 'ELEVATOR', 'For Hire Vehicle Report',\n",
              "       'Damaged Tree', 'Elevator', 'Graffiti',\n",
              "       'General Construction/Plumbing', 'Emergency Response Team (ERT)',\n",
              "       'Wood Pile Remaining', 'Boilers', 'Building/Use',\n",
              "       'Broken Parking Meter', 'Asbestos',\n",
              "       'COVID-19 Non-essential Construction', 'Street Sign - Dangling',\n",
              "       'Water Conservation', 'Water Quality',\n",
              "       'Unsanitary Animal Pvt Property', 'Animal in a Park',\n",
              "       'Food Poisoning', 'Plumbing',\n",
              "       'Special Projects Inspection Team (SPIT)', 'Day Care',\n",
              "       'Missed Collection (All Materials)', 'Dirty Conditions',\n",
              "       'Mobile Food Vendor', 'Industrial Waste', 'Unleashed Dog',\n",
              "       'Derelict Bicycle', 'Electrical', 'Illegal Animal Sold',\n",
              "       'Bus Stop Shelter Complaint', 'Scaffold Safety', 'Uprooted Stump',\n",
              "       'Non-Residential Heat', 'OUTSIDE BUILDING', 'Sanitation Condition',\n",
              "       'Other Enforcement', 'Illegal Animal Kept as Pet',\n",
              "       'Unsanitary Pigeon Condition', 'Taxi Report', 'Drinking Water',\n",
              "       'Noise - House of Worship', 'Investigations and Discipline (IAD)',\n",
              "       'Cranes and Derricks', 'DEP Street Condition',\n",
              "       'Green Taxi Complaint', 'Mold', 'LinkNYC', 'Bike Rack Condition',\n",
              "       'Tattooing', 'BEST/Site Safety', 'Bus Stop Shelter Placement',\n",
              "       'Mosquitoes', 'Vacant Lot', 'Peeling Paint',\n",
              "       'Public Payphone Complaint', 'Beach/Pool/Sauna Complaint',\n",
              "       'Special Natural Area District (SNAD)', 'Snow',\n",
              "       'Recycling Enforcement', 'DSNY Spillage', 'Derelict Vehicles',\n",
              "       'Disorderly Youth', 'Plant', 'Pet Shop', 'Internal Code',\n",
              "       'Green Taxi Report', 'Taxi Compliment', 'Public Toilet',\n",
              "       'X-Ray Machine/Equipment', 'Bottled Water',\n",
              "       'Posting Advertisement', 'Window Guard', 'Harboring Bees/Wasps',\n",
              "       'Radioactive Material', 'Overflowing Litter Baskets',\n",
              "       'Litter Basket / Request', 'Sweeping/Missed', 'Standing Water',\n",
              "       'Borough Office', 'Found Property', 'Collection Truck Noise',\n",
              "       'DEP Sidewalk Condition', 'Calorie Labeling', 'Highway Condition',\n",
              "       'Highway Sign - Damaged', 'Dispatched Taxi Complaint',\n",
              "       'Building Condition', 'Employee Behavior',\n",
              "       'Dept of Investigations', 'Lifeguard', 'Miscellaneous Categories',\n",
              "       'FHV Licensee Complaint', 'Sweeping/Inadequate',\n",
              "       'Bridge Condition'], dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r11h7v9PsuNx"
      },
      "source": [
        "Below, all of the values in the column are converted to title case."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rLB_9HDEHnHF"
      },
      "source": [
        "df.complaint_type = df.complaint_type.str.title()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uRaIpitq5u0J"
      },
      "source": [
        "The values in the 'agengy' column also have inconsistent formatting: all but one have abbreviated names."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7jXZOUF_95SS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a65b6c00-559b-44b0-cf4c-6ffa8f65500f"
      },
      "source": [
        "df.agency.unique()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['NYPD', 'MAYORâ\\x80\\x99S OFFICE OF SPECIAL ENFORCEMENT', 'DOT',\n",
              "       'DPR', 'DOHMH', 'DEP', 'HPD', 'EDC', 'TLC', 'DCA', 'DOE', 'DOB',\n",
              "       'DSNY', 'DOITT'], dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jbX1O3zs98vG"
      },
      "source": [
        "We'll convert the ''MAYOR'S OFFICE OF SPECIAL ENFORCEMENT'value to its abbreviated name, OSE:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VFBHDvCs5NfH"
      },
      "source": [
        "def update_OSE(agency):\n",
        "\n",
        "  \"\"\"Takes in an agency name and converts to abbreviated form, if the agency is\n",
        "     the Office of Special Enforcement.\"\"\"\n",
        "\n",
        "  if agency == 'MAYORâ\\x80\\x99S OFFICE OF SPECIAL ENFORCEMENT':\n",
        "      return \"OSE\"\n",
        "  else:\n",
        "      return agency\n",
        "    \n",
        "df.agency = df.agency.map(lambda agency: update_OSE(agency))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K82RP0ee-9zi"
      },
      "source": [
        "Among noise-related complaints, only one does not specify the noise source:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cISjPxvk-SVl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7df3a928-6a48-4de8-dc11-caead827ac6f"
      },
      "source": [
        "noise_columns = []\n",
        "\n",
        "for i in df.complaint_type:\n",
        "   if i.startswith('Noise') and i not in noise_columns:\n",
        "     noise_columns.append(i)\n",
        "noise_columns"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Noise - Residential',\n",
              " 'Noise - Street/Sidewalk',\n",
              " 'Noise - Vehicle',\n",
              " 'Noise - Commercial',\n",
              " 'Noise - Park',\n",
              " 'Noise',\n",
              " 'Noise - Helicopter',\n",
              " 'Noise - House Of Worship']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ASZQsk4_MJb"
      },
      "source": [
        "We'll convert the generic 'Noise' value to 'Noise - Unspecified' so that its formatting matches that of similar complaint types."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LvREvg4JIbht"
      },
      "source": [
        "#Label unspecified noise complaints as 'Noise - Unspecified'\n",
        "\n",
        "def update_noise(complaint):\n",
        "\n",
        "  \"\"\"Takes in an complaint name and updates it, if it is\n",
        "     an unspecified noise complaint.\"\"\"\n",
        "\n",
        "  if complaint == 'Noise':\n",
        "    \n",
        "    return \"Noise - Unspecified\"\n",
        "\n",
        "  else:\n",
        "\n",
        "    return complaint\n",
        "    \n",
        "df.complaint_type = df.complaint_type.map(\n",
        "    lambda complaint: update_noise(complaint))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m3mN6YiI_rNZ"
      },
      "source": [
        "The 'created_date' column currently contains string values:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O991scyBFamg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2b12360c-923b-4018-efba-b09fdab7582c"
      },
      "source": [
        "type(df.created_date[1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "str"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Puu10BYNF3lU"
      },
      "source": [
        "The ```pandas.to_datetime()``` method converts each string value to a datetime object below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dn6AghINJulZ"
      },
      "source": [
        "#convert date column to datetime \n",
        "df['created_date'] = pd.to_datetime(df['created_date'], \n",
        "                                    format='%Y-%m-%dT%H:%M:%S.%f')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OrdYL0PlGmpL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f13166b5-6e31-4c28-e268-2d30b8bbbd3f"
      },
      "source": [
        "#check new data type for created_date column\n",
        "type(df['created_date'][1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pandas._libs.tslibs.timestamps.Timestamp"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AM_hMyBzHaoU"
      },
      "source": [
        "Using the datetime format, we can now create new columns with the day, month, day of week, and hour of each call. This will allow us to explore the independent relationship of each variable to other characteristics of the calls."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L5MP2yt7OfnR"
      },
      "source": [
        "df['day'] = [str(i.date()) for i in df.created_date]\n",
        "df['month'] = [int(i.month) for i in df.created_date]\n",
        "df['day_of_week'] = [int(i.weekday()) for i in df.created_date]\n",
        "df['hour'] = [int(i.hour) for i in df.created_date]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZ0ntkHoIKkt"
      },
      "source": [
        "The calls are currently ordered in the dataframe from most-least recent. For data visualization purposes, we'll reindex the calls so that the linear, left-to-right presentation of time on an x-axis runs from least-most recent."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZnIpVS5AKHhp"
      },
      "source": [
        "#reorder df from least-most recent calls\n",
        "df = df.reindex(index=df.index[::-1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e926SCqvJ9pI"
      },
      "source": [
        "To numerically encode the descriptor column in the modeling phase, we'll first need to exclude all values that aren't letters. Below, an empty space is substituted for all numbers and non-letter symbols."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hb4xUDA8ockq"
      },
      "source": [
        "num_pattern = r'[-()0-9]'\n",
        "df.descriptor = df.descriptor.map(lambda x: re.sub(num_pattern, '', x).lower())\n",
        "df.descriptor = df.descriptor.map(lambda x: re.sub('/', ' ', x))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZXnnB68RxEs_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4424c18f-764a-430f-c572-10c57749a198"
      },
      "source": [
        "#preview sample of unique descriptors with non-letters removed\n",
        "df.descriptor.unique()[0:20]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['construction', 'no access', 'blocked hydrant',\n",
              "       'posted parking sign violation', ' derelict vehicles',\n",
              "       'manhole cover missing emergency sa',\n",
              "       'noise: construction before after hours nm', 'banging pounding',\n",
              "       'loud talking', 'social distancing', 'leak use comments wa',\n",
              "       'noise: private carting noise nq', 'engine idling',\n",
              "       'e dirty sidewalk', 'rodent sighting',\n",
              "       'possible water main break use comments wa', 'loud music party',\n",
              "       'rat sighting', 'dumpster  construction waste', 'dirty water we'],\n",
              "      dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NMZD3kM5D9MT"
      },
      "source": [
        "### Merging the Community District Indicators Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sw9jjyM_D9MT"
      },
      "source": [
        "Now let's read in and preview the community district indicators data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_iuHbvsJD9MT"
      },
      "source": [
        "path = '//content/drive/MyDrive/Colab Notebooks/indicators.csv'\n",
        "ind_df = pd.read_csv(path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VrvOBDnyD9MT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 563
        },
        "outputId": "e180269e-d2fc-4558-c1e7-be8c8e258c04"
      },
      "source": [
        "ind_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>the_geom</th>\n",
              "      <th>cartodb_id</th>\n",
              "      <th>the_geom_webmercator</th>\n",
              "      <th>acres</th>\n",
              "      <th>acs_tooltip</th>\n",
              "      <th>acs_tooltip_2</th>\n",
              "      <th>acs_tooltip_3</th>\n",
              "      <th>area_sqmi</th>\n",
              "      <th>borocd</th>\n",
              "      <th>cb_email</th>\n",
              "      <th>cb_website</th>\n",
              "      <th>cd_full_title</th>\n",
              "      <th>cd_short_title</th>\n",
              "      <th>cd_son_fy2018</th>\n",
              "      <th>cd_tot_bldgs</th>\n",
              "      <th>cd_tot_resunits</th>\n",
              "      <th>count_hosp_clinic</th>\n",
              "      <th>count_libraries</th>\n",
              "      <th>count_parks</th>\n",
              "      <th>count_public_schools</th>\n",
              "      <th>crime_count</th>\n",
              "      <th>crime_count_boro</th>\n",
              "      <th>crime_count_nyc</th>\n",
              "      <th>crime_per_1000</th>\n",
              "      <th>crime_per_1000_boro</th>\n",
              "      <th>crime_per_1000_nyc</th>\n",
              "      <th>female_10_14</th>\n",
              "      <th>female_15_19</th>\n",
              "      <th>female_20_24</th>\n",
              "      <th>female_25_29</th>\n",
              "      <th>female_30_34</th>\n",
              "      <th>female_35_39</th>\n",
              "      <th>female_40_44</th>\n",
              "      <th>female_45_49</th>\n",
              "      <th>female_5_9</th>\n",
              "      <th>female_50_54</th>\n",
              "      <th>female_55_59</th>\n",
              "      <th>female_60_64</th>\n",
              "      <th>female_65_69</th>\n",
              "      <th>female_70_74</th>\n",
              "      <th>...</th>\n",
              "      <th>pct_asian_nh</th>\n",
              "      <th>pct_bach_deg</th>\n",
              "      <th>pct_bach_deg_boro</th>\n",
              "      <th>pct_bach_deg_nyc</th>\n",
              "      <th>pct_black_nh</th>\n",
              "      <th>pct_clean_strts</th>\n",
              "      <th>pct_clean_strts_boro</th>\n",
              "      <th>pct_clean_strts_nyc</th>\n",
              "      <th>pct_foreign_born</th>\n",
              "      <th>pct_hh_rent_burd</th>\n",
              "      <th>pct_hh_rent_burd_boro</th>\n",
              "      <th>pct_hh_rent_burd_nyc</th>\n",
              "      <th>pct_hispanic</th>\n",
              "      <th>pct_other_nh</th>\n",
              "      <th>pct_served_parks</th>\n",
              "      <th>pct_white_nh</th>\n",
              "      <th>pop_2000</th>\n",
              "      <th>pop_2010</th>\n",
              "      <th>pop_acs</th>\n",
              "      <th>pop_change_00_10</th>\n",
              "      <th>poverty_rate</th>\n",
              "      <th>poverty_rate_boro</th>\n",
              "      <th>poverty_rate_nyc</th>\n",
              "      <th>puma</th>\n",
              "      <th>shared_puma</th>\n",
              "      <th>shared_puma_cd</th>\n",
              "      <th>son_issue_1</th>\n",
              "      <th>son_issue_2</th>\n",
              "      <th>son_issue_3</th>\n",
              "      <th>total_lot_area</th>\n",
              "      <th>under18_rate</th>\n",
              "      <th>under18_rate_boro</th>\n",
              "      <th>under18_rate_nyc</th>\n",
              "      <th>unemployment_boro</th>\n",
              "      <th>unemployment</th>\n",
              "      <th>unemployment_nyc</th>\n",
              "      <th>v_pluto</th>\n",
              "      <th>v_acs</th>\n",
              "      <th>v_facdb</th>\n",
              "      <th>v_crime</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>976.3</td>\n",
              "      <td>American Community Survey 2014-2018 5-Year Est...</td>\n",
              "      <td>American Community Survey (ACS) 2013-2017 5-ye...</td>\n",
              "      <td>2010 Census population counts for floodplain a...</td>\n",
              "      <td>1.5</td>\n",
              "      <td>101</td>\n",
              "      <td>man01@cb.nyc.gov</td>\n",
              "      <td>www1.nyc.gov/site/manhattancb1/index.page</td>\n",
              "      <td>Manhattan Community District 1</td>\n",
              "      <td>Manhattan CD 1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1725</td>\n",
              "      <td>45629</td>\n",
              "      <td>10</td>\n",
              "      <td>2</td>\n",
              "      <td>13</td>\n",
              "      <td>19</td>\n",
              "      <td>1055</td>\n",
              "      <td>26271</td>\n",
              "      <td>92480</td>\n",
              "      <td>6.8</td>\n",
              "      <td>16.1</td>\n",
              "      <td>11</td>\n",
              "      <td>1.1</td>\n",
              "      <td>2.4</td>\n",
              "      <td>3.9</td>\n",
              "      <td>7.2</td>\n",
              "      <td>6.5</td>\n",
              "      <td>4.7</td>\n",
              "      <td>3.6</td>\n",
              "      <td>2.9</td>\n",
              "      <td>1.8</td>\n",
              "      <td>2.3</td>\n",
              "      <td>2.4</td>\n",
              "      <td>2.6</td>\n",
              "      <td>2.4</td>\n",
              "      <td>1.8</td>\n",
              "      <td>...</td>\n",
              "      <td>15.5</td>\n",
              "      <td>82.2</td>\n",
              "      <td>60.8</td>\n",
              "      <td>37.4</td>\n",
              "      <td>2.2</td>\n",
              "      <td>95.4</td>\n",
              "      <td>94.1</td>\n",
              "      <td>95.1</td>\n",
              "      <td>23.3</td>\n",
              "      <td>30.1</td>\n",
              "      <td>36.4</td>\n",
              "      <td>44.2</td>\n",
              "      <td>6.5</td>\n",
              "      <td>3.6</td>\n",
              "      <td>100</td>\n",
              "      <td>72.2</td>\n",
              "      <td>34420</td>\n",
              "      <td>60978</td>\n",
              "      <td>154636</td>\n",
              "      <td>0.77</td>\n",
              "      <td>8.8</td>\n",
              "      <td>14.4</td>\n",
              "      <td>19.8</td>\n",
              "      <td>3810</td>\n",
              "      <td>True</td>\n",
              "      <td>Manhattan CD 2</td>\n",
              "      <td>Infrastructure resiliency</td>\n",
              "      <td>Traffic</td>\n",
              "      <td>Other</td>\n",
              "      <td>2881166.373</td>\n",
              "      <td>12.7</td>\n",
              "      <td>14.4</td>\n",
              "      <td>20.9</td>\n",
              "      <td>3.8</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.4</td>\n",
              "      <td>20v4</td>\n",
              "      <td>Y2014-2018</td>\n",
              "      <td>6/24/20</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>865.9</td>\n",
              "      <td>American Community Survey 2014-2018 5-Year Est...</td>\n",
              "      <td>American Community Survey (ACS) 2013-2017 5-ye...</td>\n",
              "      <td>2010 Census population counts for floodplain a...</td>\n",
              "      <td>1.4</td>\n",
              "      <td>102</td>\n",
              "      <td>bgormley@cb.nyc.gov</td>\n",
              "      <td>www.nyc.gov/html/mancb2</td>\n",
              "      <td>Manhattan Community District 2</td>\n",
              "      <td>Manhattan CD 2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5256</td>\n",
              "      <td>60610</td>\n",
              "      <td>19</td>\n",
              "      <td>3</td>\n",
              "      <td>6</td>\n",
              "      <td>8</td>\n",
              "      <td>2313</td>\n",
              "      <td>26271</td>\n",
              "      <td>92480</td>\n",
              "      <td>15.0</td>\n",
              "      <td>16.1</td>\n",
              "      <td>11</td>\n",
              "      <td>1.1</td>\n",
              "      <td>2.4</td>\n",
              "      <td>3.9</td>\n",
              "      <td>7.2</td>\n",
              "      <td>6.5</td>\n",
              "      <td>4.7</td>\n",
              "      <td>3.6</td>\n",
              "      <td>2.9</td>\n",
              "      <td>1.8</td>\n",
              "      <td>2.3</td>\n",
              "      <td>2.4</td>\n",
              "      <td>2.6</td>\n",
              "      <td>2.4</td>\n",
              "      <td>1.8</td>\n",
              "      <td>...</td>\n",
              "      <td>15.5</td>\n",
              "      <td>82.2</td>\n",
              "      <td>60.8</td>\n",
              "      <td>37.4</td>\n",
              "      <td>2.2</td>\n",
              "      <td>96.3</td>\n",
              "      <td>94.1</td>\n",
              "      <td>95.1</td>\n",
              "      <td>23.3</td>\n",
              "      <td>30.1</td>\n",
              "      <td>36.4</td>\n",
              "      <td>44.2</td>\n",
              "      <td>6.5</td>\n",
              "      <td>3.6</td>\n",
              "      <td>100</td>\n",
              "      <td>72.2</td>\n",
              "      <td>93119</td>\n",
              "      <td>90016</td>\n",
              "      <td>154636</td>\n",
              "      <td>-0.03</td>\n",
              "      <td>8.8</td>\n",
              "      <td>14.4</td>\n",
              "      <td>19.8</td>\n",
              "      <td>3810</td>\n",
              "      <td>True</td>\n",
              "      <td>Manhattan CD 1</td>\n",
              "      <td>Parks</td>\n",
              "      <td>Schools</td>\n",
              "      <td>Senior services</td>\n",
              "      <td>2231472.120</td>\n",
              "      <td>12.7</td>\n",
              "      <td>14.4</td>\n",
              "      <td>20.9</td>\n",
              "      <td>3.8</td>\n",
              "      <td>2.8</td>\n",
              "      <td>4.4</td>\n",
              "      <td>20v4</td>\n",
              "      <td>Y2014-2018</td>\n",
              "      <td>6/24/20</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1076.9</td>\n",
              "      <td>American Community Survey 2014-2018 5-Year Est...</td>\n",
              "      <td>American Community Survey (ACS) 2013-2017 5-ye...</td>\n",
              "      <td>2010 Census population counts for floodplain a...</td>\n",
              "      <td>1.7</td>\n",
              "      <td>103</td>\n",
              "      <td>mn03@cb.nyc.gov</td>\n",
              "      <td>www.nyc.gov/html/mancb3</td>\n",
              "      <td>Manhattan Community District 3</td>\n",
              "      <td>Manhattan CD 3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4644</td>\n",
              "      <td>82557</td>\n",
              "      <td>41</td>\n",
              "      <td>5</td>\n",
              "      <td>13</td>\n",
              "      <td>42</td>\n",
              "      <td>2360</td>\n",
              "      <td>26271</td>\n",
              "      <td>92480</td>\n",
              "      <td>15.2</td>\n",
              "      <td>16.1</td>\n",
              "      <td>11</td>\n",
              "      <td>1.5</td>\n",
              "      <td>2.9</td>\n",
              "      <td>4.2</td>\n",
              "      <td>6.4</td>\n",
              "      <td>4.1</td>\n",
              "      <td>3.2</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.2</td>\n",
              "      <td>1.2</td>\n",
              "      <td>2.8</td>\n",
              "      <td>3.3</td>\n",
              "      <td>3.4</td>\n",
              "      <td>2.9</td>\n",
              "      <td>2.1</td>\n",
              "      <td>...</td>\n",
              "      <td>30.6</td>\n",
              "      <td>43.3</td>\n",
              "      <td>60.8</td>\n",
              "      <td>37.4</td>\n",
              "      <td>8.4</td>\n",
              "      <td>91.0</td>\n",
              "      <td>94.1</td>\n",
              "      <td>95.1</td>\n",
              "      <td>34.5</td>\n",
              "      <td>38.4</td>\n",
              "      <td>36.4</td>\n",
              "      <td>44.2</td>\n",
              "      <td>25.1</td>\n",
              "      <td>2.6</td>\n",
              "      <td>100</td>\n",
              "      <td>33.3</td>\n",
              "      <td>164407</td>\n",
              "      <td>163277</td>\n",
              "      <td>154995</td>\n",
              "      <td>-0.01</td>\n",
              "      <td>19.3</td>\n",
              "      <td>14.4</td>\n",
              "      <td>19.8</td>\n",
              "      <td>3809</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Affordable housing</td>\n",
              "      <td>Senior services</td>\n",
              "      <td>Homelessness</td>\n",
              "      <td>2985434.526</td>\n",
              "      <td>11.4</td>\n",
              "      <td>14.4</td>\n",
              "      <td>20.9</td>\n",
              "      <td>3.8</td>\n",
              "      <td>3.6</td>\n",
              "      <td>4.4</td>\n",
              "      <td>20v4</td>\n",
              "      <td>Y2014-2018</td>\n",
              "      <td>6/24/20</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1131.6</td>\n",
              "      <td>American Community Survey 2014-2018 5-Year Est...</td>\n",
              "      <td>American Community Survey (ACS) 2013-2017 5-ye...</td>\n",
              "      <td>2010 Census population counts for floodplain a...</td>\n",
              "      <td>1.8</td>\n",
              "      <td>104</td>\n",
              "      <td>jbodine@cb.nyc.gov</td>\n",
              "      <td>www.nyc.gov/mcb4</td>\n",
              "      <td>Manhattan Community District 4</td>\n",
              "      <td>Manhattan CD 4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3829</td>\n",
              "      <td>85706</td>\n",
              "      <td>38</td>\n",
              "      <td>2</td>\n",
              "      <td>7</td>\n",
              "      <td>29</td>\n",
              "      <td>2792</td>\n",
              "      <td>26271</td>\n",
              "      <td>92480</td>\n",
              "      <td>18.1</td>\n",
              "      <td>16.1</td>\n",
              "      <td>11</td>\n",
              "      <td>0.9</td>\n",
              "      <td>1.7</td>\n",
              "      <td>4.0</td>\n",
              "      <td>7.2</td>\n",
              "      <td>6.4</td>\n",
              "      <td>3.8</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.9</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.3</td>\n",
              "      <td>2.7</td>\n",
              "      <td>2.7</td>\n",
              "      <td>2.1</td>\n",
              "      <td>2.1</td>\n",
              "      <td>...</td>\n",
              "      <td>17.2</td>\n",
              "      <td>74.1</td>\n",
              "      <td>60.8</td>\n",
              "      <td>37.4</td>\n",
              "      <td>5.2</td>\n",
              "      <td>95.6</td>\n",
              "      <td>94.1</td>\n",
              "      <td>95.1</td>\n",
              "      <td>30.8</td>\n",
              "      <td>33.9</td>\n",
              "      <td>36.4</td>\n",
              "      <td>44.2</td>\n",
              "      <td>15.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>94</td>\n",
              "      <td>59.1</td>\n",
              "      <td>87479</td>\n",
              "      <td>103245</td>\n",
              "      <td>154496</td>\n",
              "      <td>0.18</td>\n",
              "      <td>11.3</td>\n",
              "      <td>14.4</td>\n",
              "      <td>19.8</td>\n",
              "      <td>3807</td>\n",
              "      <td>True</td>\n",
              "      <td>Manhattan CD 5</td>\n",
              "      <td>Affordable housing</td>\n",
              "      <td>Land use trends (zoning, development, neighbor...</td>\n",
              "      <td>Traffic</td>\n",
              "      <td>3110433.364</td>\n",
              "      <td>8.2</td>\n",
              "      <td>14.4</td>\n",
              "      <td>20.9</td>\n",
              "      <td>3.8</td>\n",
              "      <td>3.5</td>\n",
              "      <td>4.4</td>\n",
              "      <td>20v4</td>\n",
              "      <td>Y2014-2018</td>\n",
              "      <td>6/24/20</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1005.4</td>\n",
              "      <td>American Community Survey 2014-2018 5-Year Est...</td>\n",
              "      <td>American Community Survey (ACS) 2013-2017 5-ye...</td>\n",
              "      <td>2010 Census population counts for floodplain a...</td>\n",
              "      <td>1.6</td>\n",
              "      <td>105</td>\n",
              "      <td>office@cb5.org</td>\n",
              "      <td>www.cb5.org</td>\n",
              "      <td>Manhattan Community District 5</td>\n",
              "      <td>Manhattan CD 5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3127</td>\n",
              "      <td>49436</td>\n",
              "      <td>23</td>\n",
              "      <td>7</td>\n",
              "      <td>3</td>\n",
              "      <td>13</td>\n",
              "      <td>4538</td>\n",
              "      <td>26271</td>\n",
              "      <td>92480</td>\n",
              "      <td>29.4</td>\n",
              "      <td>16.1</td>\n",
              "      <td>11</td>\n",
              "      <td>0.9</td>\n",
              "      <td>1.7</td>\n",
              "      <td>4.0</td>\n",
              "      <td>7.2</td>\n",
              "      <td>6.4</td>\n",
              "      <td>3.8</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.9</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.3</td>\n",
              "      <td>2.7</td>\n",
              "      <td>2.7</td>\n",
              "      <td>2.1</td>\n",
              "      <td>2.1</td>\n",
              "      <td>...</td>\n",
              "      <td>17.2</td>\n",
              "      <td>74.1</td>\n",
              "      <td>60.8</td>\n",
              "      <td>37.4</td>\n",
              "      <td>5.2</td>\n",
              "      <td>94.4</td>\n",
              "      <td>94.1</td>\n",
              "      <td>95.1</td>\n",
              "      <td>30.8</td>\n",
              "      <td>33.9</td>\n",
              "      <td>36.4</td>\n",
              "      <td>44.2</td>\n",
              "      <td>15.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>95</td>\n",
              "      <td>59.1</td>\n",
              "      <td>44028</td>\n",
              "      <td>51673</td>\n",
              "      <td>154496</td>\n",
              "      <td>0.17</td>\n",
              "      <td>11.3</td>\n",
              "      <td>14.4</td>\n",
              "      <td>19.8</td>\n",
              "      <td>3807</td>\n",
              "      <td>True</td>\n",
              "      <td>Manhattan CD 4</td>\n",
              "      <td>Traffic</td>\n",
              "      <td>Trash removal &amp; cleanliness</td>\n",
              "      <td>Homelessness</td>\n",
              "      <td>2741102.661</td>\n",
              "      <td>8.2</td>\n",
              "      <td>14.4</td>\n",
              "      <td>20.9</td>\n",
              "      <td>3.8</td>\n",
              "      <td>3.5</td>\n",
              "      <td>4.4</td>\n",
              "      <td>20v4</td>\n",
              "      <td>Y2014-2018</td>\n",
              "      <td>6/24/20</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 188 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   the_geom  cartodb_id  the_geom_webmercator  ...       v_acs  v_facdb v_crime\n",
              "0       NaN           1                   NaN  ...  Y2014-2018  6/24/20    2019\n",
              "1       NaN           2                   NaN  ...  Y2014-2018  6/24/20    2019\n",
              "2       NaN           3                   NaN  ...  Y2014-2018  6/24/20    2019\n",
              "3       NaN           4                   NaN  ...  Y2014-2018  6/24/20    2019\n",
              "4       NaN           5                   NaN  ...  Y2014-2018  6/24/20    2019\n",
              "\n",
              "[5 rows x 188 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "glOQgml_D9MT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2a2f674b-83e7-4017-a684-2b6b33a94742"
      },
      "source": [
        "ind_df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 59 entries, 0 to 58\n",
            "Columns: 188 entries, the_geom to v_crime\n",
            "dtypes: bool(1), float64(121), int64(51), object(15)\n",
            "memory usage: 86.4+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9oDC8G7pD9MT"
      },
      "source": [
        "This dataset contains many columns, which have been narrowed down to a selection of development, demographic, economic and social variables below. Most of the column names are self-explanatory, but a a full list of column descriptions can be found in the 'Indicators Data Dictionary' file on any [community district profile](https://communityprofiles.planning.nyc.gov/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oFwkLPNWD9MT"
      },
      "source": [
        "ind_df = ind_df[[\n",
        "'cd_short_title', \n",
        "'area_sqmi',\n",
        "'count_hosp_clinic',\n",
        "'count_libraries',\n",
        "'count_parks',\n",
        "'count_public_schools',\n",
        "'crime_per_1000',\n",
        "'female_15_19',\n",
        "'female_20_24',\n",
        "'female_25_29',\n",
        "'female_30_34',\n",
        "'female_35_39',\n",
        "'female_40_44',\n",
        "'female_45_49',\n",
        "'female_50_54',\n",
        "'female_55_59',\n",
        "'female_60_64',\n",
        "'male_10_14',\n",
        "'male_15_19',\n",
        "'male_20_24',\n",
        "'male_25_29',\n",
        "'male_30_34',\n",
        "'male_35_39',\n",
        "'male_40_44',\n",
        "'male_45_49',\n",
        "'male_50_54',\n",
        "'male_55_59',\n",
        "'male_60_64',\n",
        "'under18_rate',\n",
        "'over65_rate',\n",
        "'lots_total',\n",
        "'lots_vacant',\n",
        "'pct_white_nh',\n",
        "'pct_black_nh',\n",
        "'pct_asian_nh',\n",
        "'pct_other_nh',\n",
        "'pct_hispanic',\n",
        "'mean_commute',\n",
        "'pop_change_00_10',\n",
        "'pct_clean_strts',\n",
        "'unemployment',\n",
        "'poverty_rate',\n",
        "'lep_rate_boro',\n",
        "'pct_hh_rent_burd'\n",
        "]]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o4sHtp4vD9MU"
      },
      "source": [
        "Let's look at the summary statistics for each of the numeric variables in the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gr8RyCiED9MU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 182
        },
        "outputId": "0eb4a322-af1a-4ad2-d972-82eac08d82cd"
      },
      "source": [
        "ind_df.describe().loc[['mean','std', 'min', 'max']]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area_sqmi</th>\n",
              "      <th>count_hosp_clinic</th>\n",
              "      <th>count_libraries</th>\n",
              "      <th>count_parks</th>\n",
              "      <th>count_public_schools</th>\n",
              "      <th>crime_per_1000</th>\n",
              "      <th>female_15_19</th>\n",
              "      <th>female_20_24</th>\n",
              "      <th>female_25_29</th>\n",
              "      <th>female_30_34</th>\n",
              "      <th>female_35_39</th>\n",
              "      <th>female_40_44</th>\n",
              "      <th>female_45_49</th>\n",
              "      <th>female_50_54</th>\n",
              "      <th>female_55_59</th>\n",
              "      <th>female_60_64</th>\n",
              "      <th>male_10_14</th>\n",
              "      <th>male_15_19</th>\n",
              "      <th>male_20_24</th>\n",
              "      <th>male_25_29</th>\n",
              "      <th>male_30_34</th>\n",
              "      <th>male_35_39</th>\n",
              "      <th>male_40_44</th>\n",
              "      <th>male_45_49</th>\n",
              "      <th>male_50_54</th>\n",
              "      <th>male_55_59</th>\n",
              "      <th>male_60_64</th>\n",
              "      <th>under18_rate</th>\n",
              "      <th>over65_rate</th>\n",
              "      <th>lots_total</th>\n",
              "      <th>lots_vacant</th>\n",
              "      <th>pct_white_nh</th>\n",
              "      <th>pct_black_nh</th>\n",
              "      <th>pct_asian_nh</th>\n",
              "      <th>pct_other_nh</th>\n",
              "      <th>pct_hispanic</th>\n",
              "      <th>mean_commute</th>\n",
              "      <th>pop_change_00_10</th>\n",
              "      <th>pct_clean_strts</th>\n",
              "      <th>unemployment</th>\n",
              "      <th>poverty_rate</th>\n",
              "      <th>lep_rate_boro</th>\n",
              "      <th>pct_hh_rent_burd</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>4.613559</td>\n",
              "      <td>22.898305</td>\n",
              "      <td>3.677966</td>\n",
              "      <td>6.915254</td>\n",
              "      <td>25.694915</td>\n",
              "      <td>10.205085</td>\n",
              "      <td>2.720339</td>\n",
              "      <td>3.540678</td>\n",
              "      <td>4.883051</td>\n",
              "      <td>4.398305</td>\n",
              "      <td>3.757627</td>\n",
              "      <td>3.313559</td>\n",
              "      <td>3.308475</td>\n",
              "      <td>3.318644</td>\n",
              "      <td>3.172881</td>\n",
              "      <td>2.988136</td>\n",
              "      <td>2.794915</td>\n",
              "      <td>2.720339</td>\n",
              "      <td>3.342373</td>\n",
              "      <td>4.601695</td>\n",
              "      <td>4.237288</td>\n",
              "      <td>3.538983</td>\n",
              "      <td>3.159322</td>\n",
              "      <td>3.083051</td>\n",
              "      <td>3.045763</td>\n",
              "      <td>2.861017</td>\n",
              "      <td>2.513559</td>\n",
              "      <td>20.872881</td>\n",
              "      <td>13.832203</td>\n",
              "      <td>14531.593220</td>\n",
              "      <td>458.000000</td>\n",
              "      <td>32.264407</td>\n",
              "      <td>21.822034</td>\n",
              "      <td>13.088136</td>\n",
              "      <td>2.927119</td>\n",
              "      <td>29.898305</td>\n",
              "      <td>41.238983</td>\n",
              "      <td>0.038475</td>\n",
              "      <td>94.064407</td>\n",
              "      <td>4.513559</td>\n",
              "      <td>19.940678</td>\n",
              "      <td>23.049153</td>\n",
              "      <td>44.723729</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>4.295164</td>\n",
              "      <td>13.101323</td>\n",
              "      <td>1.665478</td>\n",
              "      <td>4.035567</td>\n",
              "      <td>8.987068</td>\n",
              "      <td>4.627594</td>\n",
              "      <td>0.683243</td>\n",
              "      <td>0.844887</td>\n",
              "      <td>1.364848</td>\n",
              "      <td>1.113629</td>\n",
              "      <td>0.547787</td>\n",
              "      <td>0.368809</td>\n",
              "      <td>0.402273</td>\n",
              "      <td>0.563081</td>\n",
              "      <td>0.541973</td>\n",
              "      <td>0.529178</td>\n",
              "      <td>0.797285</td>\n",
              "      <td>0.788447</td>\n",
              "      <td>0.844904</td>\n",
              "      <td>1.219453</td>\n",
              "      <td>1.238450</td>\n",
              "      <td>0.827338</td>\n",
              "      <td>0.655249</td>\n",
              "      <td>0.448043</td>\n",
              "      <td>0.453463</td>\n",
              "      <td>0.467944</td>\n",
              "      <td>0.480466</td>\n",
              "      <td>5.383334</td>\n",
              "      <td>3.643816</td>\n",
              "      <td>11987.065263</td>\n",
              "      <td>581.954495</td>\n",
              "      <td>24.355979</td>\n",
              "      <td>22.333160</td>\n",
              "      <td>12.294884</td>\n",
              "      <td>2.074129</td>\n",
              "      <td>20.544053</td>\n",
              "      <td>6.309572</td>\n",
              "      <td>0.111801</td>\n",
              "      <td>3.076146</td>\n",
              "      <td>1.465518</td>\n",
              "      <td>6.824982</td>\n",
              "      <td>5.305637</td>\n",
              "      <td>7.550322</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.400000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>2.600000</td>\n",
              "      <td>1.100000</td>\n",
              "      <td>1.800000</td>\n",
              "      <td>2.900000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>2.900000</td>\n",
              "      <td>2.600000</td>\n",
              "      <td>2.100000</td>\n",
              "      <td>1.800000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>0.800000</td>\n",
              "      <td>1.100000</td>\n",
              "      <td>1.200000</td>\n",
              "      <td>2.700000</td>\n",
              "      <td>2.500000</td>\n",
              "      <td>2.500000</td>\n",
              "      <td>1.800000</td>\n",
              "      <td>2.100000</td>\n",
              "      <td>2.100000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1.400000</td>\n",
              "      <td>8.200000</td>\n",
              "      <td>8.200000</td>\n",
              "      <td>1461.000000</td>\n",
              "      <td>25.000000</td>\n",
              "      <td>1.100000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.700000</td>\n",
              "      <td>1.200000</td>\n",
              "      <td>6.500000</td>\n",
              "      <td>25.400000</td>\n",
              "      <td>-0.100000</td>\n",
              "      <td>85.100000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>7.200000</td>\n",
              "      <td>10.900000</td>\n",
              "      <td>28.800000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>21.500000</td>\n",
              "      <td>54.000000</td>\n",
              "      <td>9.000000</td>\n",
              "      <td>20.000000</td>\n",
              "      <td>47.000000</td>\n",
              "      <td>29.400000</td>\n",
              "      <td>4.200000</td>\n",
              "      <td>6.900000</td>\n",
              "      <td>8.400000</td>\n",
              "      <td>6.900000</td>\n",
              "      <td>5.600000</td>\n",
              "      <td>4.300000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>4.400000</td>\n",
              "      <td>4.200000</td>\n",
              "      <td>3.900000</td>\n",
              "      <td>4.800000</td>\n",
              "      <td>4.200000</td>\n",
              "      <td>5.600000</td>\n",
              "      <td>8.100000</td>\n",
              "      <td>7.300000</td>\n",
              "      <td>5.600000</td>\n",
              "      <td>5.300000</td>\n",
              "      <td>4.200000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>3.800000</td>\n",
              "      <td>3.300000</td>\n",
              "      <td>36.800000</td>\n",
              "      <td>23.900000</td>\n",
              "      <td>50434.000000</td>\n",
              "      <td>3364.000000</td>\n",
              "      <td>82.900000</td>\n",
              "      <td>86.600000</td>\n",
              "      <td>52.700000</td>\n",
              "      <td>14.900000</td>\n",
              "      <td>71.500000</td>\n",
              "      <td>51.100000</td>\n",
              "      <td>0.770000</td>\n",
              "      <td>99.100000</td>\n",
              "      <td>7.800000</td>\n",
              "      <td>35.600000</td>\n",
              "      <td>28.800000</td>\n",
              "      <td>60.300000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      area_sqmi  count_hosp_clinic  ...  lep_rate_boro  pct_hh_rent_burd\n",
              "mean   4.613559          22.898305  ...      23.049153         44.723729\n",
              "std    4.295164          13.101323  ...       5.305637          7.550322\n",
              "min    1.400000           1.000000  ...      10.900000         28.800000\n",
              "max   21.500000          54.000000  ...      28.800000         60.300000\n",
              "\n",
              "[4 rows x 43 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZVEEZ-mUD9MU"
      },
      "source": [
        "We can see that the average size of a community district is 4.6 miles, and that the average percentage of men and women in their 20s and 30s are higher than percentages for other age groups. On average, white residents make up about a third of the community districts' population, followed by hispanic, black, and asian residents. The average poverty rate is 19%, the average unemployment rate is 4.5%, and the average crime rate (for 7 major felonies) per 1000 residents is about 10. \n",
        "\n",
        "We'll format the community district column in this dataset such that it can be merged with the community board column in the 311 dataset. In NYC, community districts and their corresponding boards share the same number and borough. According to the webpage for [Queens Community Board 3](http://www.cb3qn.nyc.gov/1652), \"each community district has a Community Board to oversee the delivery of city services and facilitate local governance.\" Currently, the values for community boards in df and community districts in ind_df are formatted differently:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cwq7zFRpD9MU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9cb1eb4e-49d8-42de-b09e-2d9788238678"
      },
      "source": [
        "df.community_board[0:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1499999        08 BRONX\n",
              "1499998    11 MANHATTAN\n",
              "1499997       06 QUEENS\n",
              "1499996       05 QUEENS\n",
              "1499995     05 BROOKLYN\n",
              "1499994     05 BROOKLYN\n",
              "1499993     17 BROOKLYN\n",
              "1499992    08 MANHATTAN\n",
              "1499991    04 MANHATTAN\n",
              "1499990    10 MANHATTAN\n",
              "Name: community_board, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "30bgoKzoD9MU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "237d05ca-e26a-4d59-a156-a2ebf116d02d"
      },
      "source": [
        "ind_df.cd_short_title[0:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0     Manhattan CD 1\n",
              "1     Manhattan CD 2\n",
              "2     Manhattan CD 3\n",
              "3     Manhattan CD 4\n",
              "4     Manhattan CD 5\n",
              "5     Manhattan CD 6\n",
              "6     Manhattan CD 7\n",
              "7     Manhattan CD 8\n",
              "8     Manhattan CD 9\n",
              "9    Manhattan CD 10\n",
              "Name: cd_short_title, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u4sBCfWqD9MU"
      },
      "source": [
        "Below, each community district short title is converted to uppercase, 'CD' is removed, and the number is moved to the front of the string, as in the 311 dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "23ervTsXD9MU"
      },
      "source": [
        "#format community district string values so that they match community board vals \n",
        "\n",
        "pd.options.mode.chained_assignment = None \n",
        "\n",
        "for row in ind_df.index:\n",
        "    upper = ind_df.at[row, \"cd_short_title\"].upper().replace(' CD','')\n",
        "    num_first = \"{}\".format(upper[-2:] + ' ' + upper[:-2]).strip()\n",
        "    ind_df.cd_short_title.loc[row] = num_first"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_UDDO5_cD9MU"
      },
      "source": [
        "The below function removes all 0's from the beginning of single-digit community board numbers in the 311 dataset. The subsequent list comprehension uses the function to update each value in the community board column."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "btQIgGmiD9MU"
      },
      "source": [
        "def format_cb(cb_string):\n",
        "\n",
        "  \"\"\"Takes in a string value for a community board in the 311 dataframe. If the \n",
        "     string starts with '0', updates the string to contain every letter after 0.\"\"\"  \n",
        "  \n",
        "  if cb_string.startswith('0'):\n",
        "        return cb_string[1:]\n",
        "  else:\n",
        "        return cb_string"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3k9T3XZID9MU"
      },
      "source": [
        "df.community_board = [format_cb(i) for i in df.community_board]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WcrmKY5BD9MU"
      },
      "source": [
        "The community board column contains more unique values than the community district column, as shown below. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QGPAAdtyD9MU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c83d56db-24af-492d-9bbe-3721dd77401e"
      },
      "source": [
        "len(df.community_board.unique())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "76"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ztC7nSA5D9MU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a1b5f077-ef10-4d70-ce68-915f04117444"
      },
      "source": [
        "len(ind_df.cd_short_title.unique())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "59"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oqhd82KvD9MU"
      },
      "source": [
        "We'll remove every row without a matching value by first labelling the non-matches and then overwriting the 311 dataset to exclude them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MRiOeyy6D9MU"
      },
      "source": [
        "def label_non_match(cb):\n",
        "\n",
        "  \"\"\"Takes in a community board value in the 311 dataset. If the value does not \n",
        "     match one of the unique values in the indicators dataset, returns the \n",
        "     string 'Non-match.'\"\"\"\n",
        "    \n",
        "  if cb in ind_df.cd_short_title.unique():\n",
        "        return cb \n",
        "  \n",
        "  else:\n",
        "        return 'Non-match'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GOJeGGcxD9MU"
      },
      "source": [
        "df.community_board = df.community_board.map(lambda cb: label_non_match(cb))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e7fDtK3ND9MU"
      },
      "source": [
        "df = df[df.community_board != 'Non-match']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2BWJxHsD9MU"
      },
      "source": [
        "Now that all of the community district/board values have a match in both columns, we can create a column with a matching title in the indicators dataframe and perform an inner join with the 311 dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hs9nBCk5D9MU"
      },
      "source": [
        "#create new column with matching name to join data, and drop duplicate column\n",
        "ind_df['community_board'] = ind_df.cd_short_title\n",
        "df = pd.merge(df, ind_df, how='inner')\n",
        "\n",
        "df.drop(\"cd_short_title\", axis=1, inplace=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w1aqOpzVgN-K"
      },
      "source": [
        "# Exporting the Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pVwlxP1EDGeW"
      },
      "source": [
        "The code below can be used to export the dataframes to Google Drive. Simply change the file paths to paths of your choice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ohqG_j6YQ5s4"
      },
      "source": [
        "drive.mount(\"/content/gdrive\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lpK4u6_jQAGG"
      },
      "source": [
        "df.to_csv('/content/gdrive/My Drive/Colab Notebooks/community_board_311.csv', header=True)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}