02-intro-to-hvplot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "72daec7f-addf-4640-a7f9-6f3f4d90db75",
   "metadata": {},
   "source": [
    "<img src=\"https://hvplot.holoviz.org/_static/logo_horizontal.svg\" width=\"25%\" align=\"right\"/>\n",
    "\n",
    "# Introduction to interactive data visualization with `hvplot`\n",
    "\n",
    "In this notebook, we'll continue to explore the dataset, but with visuals! We will learn to use `hvplot` with pandas to create some quick interactive visualizations.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "724f8bcc-770b-483e-be86-e08321aa4eeb",
   "metadata": {},
   "source": [
    "## What is hvPlot?\n",
    "\n",
    "hvPlot is library that provides a high-level API for data exploration and visualization. \n",
    " \n",
    "It is a powerful and interactive version of the pandas' `.plot()` API.\n",
    "**By replacing .plot() with .hvplot() you get an interactive figure.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1755b76-9259-4581-aaa2-7778f6431d40",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13b49b64-058a-4080-b29f-17b42097e673",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Note: Extension setup needs to be in a separate cell to avoid a JupyterHub race conditions. If you see an error/warning, please re-run the cell.\n",
    "\n",
    "import hvplot\n",
    "import hvplot.pandas # noqa\n",
    "\n",
    "hvplot.extension('bokeh')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e17be552-28b9-4353-96ff-e9f5640aacc4",
   "metadata": {},
   "source": [
    "## Read a subset into pandas\n",
    "\n",
    "Let's read in 1 year of data as into a pandas DataFrame. \n",
    "\n",
    "We'll read the Parquet dataset (details in a future notebook!) and only the first few columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75bc10ad-86bd-4a7a-9b9a-2dcfadd7a611",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "columns = [\n",
    "    'MONTH', 'DAY_OF_MONTH', 'DAY_OF_WEEK', 'FL_DATE', 'OP_CARRIER', \n",
    "    'TAIL_NUM', 'OP_CARRIER_FL_NUM', 'ORIGIN', 'DEST', 'CRS_DEP_TIME', \n",
    "    'DEP_TIME', 'DEP_DELAY', 'ARR_TIME', 'ARR_DELAY', 'CANCELLED', \n",
    "    'CANCELLATION_CODE', 'DIVERTED', 'AIR_TIME', 'FLIGHTS', 'DISTANCE',\n",
    "    'CARRIER_DELAY', 'WEATHER_DELAY', 'NAS_DELAY', 'SECURITY_DELAY', \n",
    "    'LATE_AIRCRAFT_DELAY', 'DIV_ARR_DELAY'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df48beb6-7799-42b3-8ab4-b78996391536",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights = pd.read_parquet(\n",
    "        f\"gcs://quansight-datasets/airline-ontime-performance/sorted/parquet_by_year\", \n",
    "        filters=[('YEAR', '=', 2022)],\n",
    "        columns=columns,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0320d703-d55e-4645-8ebc-2956ed4cac45",
   "metadata": {},
   "source": [
    "### Create a smaller DataFrame\n",
    "\n",
    "Let's reduce the dataset to only have information about 4 airlines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1167cdb9-8c05-4bde-b1bb-b4e16927255f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"Before: {len(flights)} rows\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "369b42e1-57eb-4f6c-8818-8aabb1f6d2d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset = flights[flights.OP_CARRIER.isin(['AA', 'UA', 'WN', 'DL'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad4b4cf4-484b-4b53-9102-1172d4d8abe9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"After: {len(flights_subset)} rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c383fcaa-232d-46fc-9731-983911e20bd1",
   "metadata": {},
   "source": [
    "##  hvPlot as a `pandas.plot` replacement\n",
    "\n",
    "hvPlot gives you an interactive plot very quickly and out-of-the-box.\n",
    "\n",
    "Let's see this in action by plotting the average departure delay each day:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e43934e6-b741-4f00-aaf2-aca8482799bb",
   "metadata": {},
   "source": [
    "**pandas.plot:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc00e3e1-fec8-4de0-a8bf-c5e580ca68ef",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.groupby('FL_DATE')[\"DEP_DELAY\"].mean().plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "913d0ba0-12d7-46dd-922f-b02a751d3568",
   "metadata": {},
   "source": [
    "**hvPlot:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "711c1745-6ef9-4c29-af43-e263db79e14c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.groupby('FL_DATE')[\"DEP_DELAY\"].mean().hvplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e8daa1-08c9-41b1-8ccb-6ab54fb79452",
   "metadata": {},
   "source": [
    "### Bokeh plot tools\n",
    "\n",
    "The above plot is rendered using Bokeh, hover/click on the pan, box zoom, wheel zoom, save, reset, and help tools on the right to interact with your plot!\n",
    "\n",
    "Learn more about the tools in the [Bokeh documentation](https://docs.bokeh.org/en/latest/docs/first_steps/first_steps_1.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca4784d0-8dbf-4436-9c86-4250c99852df",
   "metadata": {},
   "source": [
    "### 💻 Your turn: Include arrival delays to the same plot\n",
    "\n",
    "Note that your DataFrame will now need both departure and arrival delay columns, and when done, make sure to click on the legend labels to show/hide each plot!\n",
    "\n",
    "Optionally, you can plot the maximum or cumulative delays as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0ddd32c-ffd4-47c9-ab52-bef10a974496",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots for the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "406b53a3-07c5-4321-b251-336591e67310",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights.groupby('MONTH')[[\"DEP_DELAY\", \"ARR_DELAY\"]].mean().hvplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afcfe82d-4f12-43bc-983a-d89b66c88561",
   "metadata": {},
   "source": [
    "### Histograms\n",
    "\n",
    "hvPlot makes all the `pandas.plot` options more powerful, let's look at histograms for instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adb18ac8-66c3-49ca-a887-2b2aa0c1eedb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.hvplot.hist('DEP_DELAY', by='OP_CARRIER', bins=20, bin_range=(-20, 100), width=300, subplots=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9da8c29-84d1-44e7-aafe-2b86fbd7cff7",
   "metadata": {},
   "source": [
    "You can hover over the bars in the plots to view more details."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae45126f-b7a6-4540-a864-b0ee521c91ae",
   "metadata": {
    "tags": []
   },
   "source": [
    "## More interactivity with quick widgets\n",
    "\n",
    "Zoom, pan, and hover are just the tip of the iceberg for interactivity, widgets open up a whole new world of interaction. Some examples of widgets are dropdown selectors, range/date/color selectors, radio buttons, text fields, etc.\n",
    "\n",
    "hvPlot automatically includes the best widgets for your visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7405fd4a-df16-4263-a834-73c4c00d939a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights.hvplot.hist('DEP_DELAY', groupby='OP_CARRIER', bins=20, bin_range=(-20, 100), width=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aff1920-db3d-47f5-84b6-dcae574894b6",
   "metadata": {},
   "source": [
    "Here we see a dropdown for selecting the carrier, try it out!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43302426-754e-4386-b2c4-d6d5d3a05af5",
   "metadata": {},
   "source": [
    "### 💻 Your turn: Create violin plots for the different types of \"DELAYS\" for each \n",
    "\n",
    "Hint: You can look for columns associated with Delays (i.e. \"DEL\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5dd53e9-acf3-4716-b4bb-5a1c158b5032",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots for the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b7cfe9d-07ea-4121-a581-1b78b9507a91",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "columns = [col for col in flights.columns if \"DEL\" in col]\n",
    "flights.hvplot.violin(y=columns, group_label='Type of Delay', value_label='Delay in Minutes', invert=True, groupby=\"OP_CARRIER\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e497dc-4918-46e8-9e49-228d8a2227ea",
   "metadata": {},
   "source": [
    "## Compose and overlay plots \n",
    "\n",
    "With hvPlot, you can compose and overlay your plots easily with the `+` or `*` operations, respectively.\n",
    "\n",
    "Let's plot the minimum, maximum, and mean departure delays per week for each carrier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6835c344-5f7f-4cbc-988c-7353b79b1b73",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5804fb6-6578-47d1-9894-e88abf0e8477",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "delays = flights.groupby(['DAY_OF_WEEK', 'OP_CARRIER'])['DEP_DELAY'].agg([\"min\", \"mean\", \"max\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04144fb0-1ff4-457a-85da-3dd576a8fe41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "delays.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d77b6fd1-350f-47eb-9f9f-e209ce54217f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "min_max_plot = delays.hvplot.area(x='DAY_OF_WEEK', y='min', y2='max', alpha=0.2, groupby=\"OP_CARRIER\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a241f52c-1b1a-45af-a0c1-b6acdee5a31f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mean_plot = delays['mean'].hvplot.line(x='DAY_OF_WEEK', groupby=\"OP_CARRIER\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62828b6f-d59c-486d-a37c-1765043c4a2a",
   "metadata": {},
   "source": [
    "The + operation creates a layout, displaying the plots side-by-side:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9181d6a-56c0-466b-8f73-65bcf2e25f26",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "min_max_plot + mean_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa3fa6bf-a999-4914-b7be-a6fbfb47bf53",
   "metadata": {},
   "source": [
    "The * operation overlays one plot on top of the other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2b491df-de54-419c-af48-ad50e8b21d42",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "min_max_plot * mean_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd6e8b03-6ea7-4a06-9050-bfb16f54c83f",
   "metadata": {},
   "source": [
    "### 💻 Your turn: Plot the mean and max departure delay by time (hour) of day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28ae6633-834e-46db-ad23-ea0dc239e277",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots for the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1485521f-c4c9-4391-8a3a-9415500cc721",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights['DEP_HOUR'] = flights.CRS_DEP_TIME.astype(int) // 100\n",
    "\n",
    "flights.groupby('DEP_HOUR')['DEP_DELAY'].mean().hvplot.bar() + flights.groupby('DEP_HOUR')['DEP_DELAY'].max().hvplot.bar()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa954704-bbcc-48e0-87fe-5dc484d6faea",
   "metadata": {},
   "source": [
    "## Explorer\n",
    "\n",
    "For creating all of our previous plots, we needed some preliminary knowledge of the dataset.\n",
    "\n",
    "What if you want to explore a dataset visually from scratch? hvPlot's data explorer can help you explore and create interactive visualizations using a graphical UI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed02cf8d-070f-451b-87ec-90c5040930e4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# takes a few minutes\n",
    "\n",
    "explorer = hvplot.explorer(flights_subset)\n",
    "explorer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b0de41d-d6d2-4a8d-a15e-a2de62f4b71a",
   "metadata": {},
   "source": [
    "You can use the above GUI to create a plot you want!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc9c083c-846b-49f5-a0ef-7777bbc51ffd",
   "metadata": {},
   "source": [
    "### Save your plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76f0d4b-3bd3-4ca0-b6c1-fd0c86f48fb7",
   "metadata": {},
   "source": [
    "You can then save the selected visualization using `save()`, or generate the code to create the specific viz using `plot_code`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2b5014f-5c4f-497e-809f-3456071b6148",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "explorer.plot_code()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9df050f5-8d7d-459d-968f-ecc6a07db65e",
   "metadata": {},
   "source": [
    "### 💻 Your turn: Use the explorer to plot the flights cancellations per day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2d6e567-15a5-4dc9-b08a-84326f48484f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots for the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffc70dff-d9cc-417c-b2ad-f363d4d613a2",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.groupby('FL_DATE')['CANCELLED'].count().hvplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcc002c6-be3d-4334-a488-6be260f144d3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-04-08T05:23:34.526542Z",
     "iopub.status.busy": "2023-04-08T05:23:34.525475Z",
     "iopub.status.idle": "2023-04-08T05:23:34.530800Z",
     "shell.execute_reply": "2023-04-08T05:23:34.529489Z",
     "shell.execute_reply.started": "2023-04-08T05:23:34.526507Z"
    },
    "tags": []
   },
   "source": [
    "## Geographic plots\n",
    "\n",
    "To plot data on geographic maps, we need the latitude and longitude values. `ip2location` has created a list of lat/lon values for US airports here: https://github.com/ip2location/ip2location-iata-icao\n",
    "\n",
    "We'll use this information to plot the departure delays on a world map!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ece50ca6-3c75-4c85-ab23-7cd86a1b40f8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore') # Ignore some HoloviewsDeprecationWarning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ba547f9-ea4b-4165-b207-ecd420f41b0e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airports = pd.read_csv('https://github.com/raw/ip2location/ip2location-iata-icao/master/iata-icao.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "443d8e40-f4dc-477b-bf25-f073cff6b538",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airports = airports.set_index('iata')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a774d66-0418-434d-87a4-4a15e66c5a6c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airports.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33128334-e926-4885-9885-2e283528631c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airport_delays = flights.groupby('ORIGIN')['DEP_DELAY'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8149bdbc-7d5f-47e1-9890-f58e675f9868",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airport_delays = pd.merge(airport_delays, airports, left_on='ORIGIN', right_on='iata')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fae8feb-cf6e-4b9d-ad19-8b65f7211f3c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airport_delays.hvplot.points('longitude', 'latitude', geo=True, c='DEP_DELAY', alpha=1, xlim=(-180, -30), ylim=(0, 72), tiles='ESRI')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccee1b87-3f06-4615-ae24-13e103628ce5",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Next →\n",
    "\n",
    "Let's dive into [Dask](./03-intro-to-dask.ipynb)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0979d916-a23e-4d8a-ad6e-c80be04a01c9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy-scipy-data-of-unusual-size",
   "language": "python",
   "name": "conda-env-scipy-scipy-data-of-unusual-size-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}