notebooks/dtreeviz_sklearn_visualisations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `dtreeviz` scikit-learn Examples\n",
    "\n",
    "([View this notebook in Colab](https://colab.research.google.com/github/parrt/dtreeviz/blob/master/notebooks/dtreeviz_sklearn_visualisations.ipynb))\n",
    "\n",
    "The [dtreeviz](https://github.com/parrt/dtreeviz) library is designed to help machine learning practitioners visualize and interpret decision trees and decision-tree-based models, such as gradient boosting machines.  \n",
    "\n",
    "The purpose of this notebook is to illustrate the main capabilities and functions of the dtreeviz API. To do that, we will use scikit-learn and the toy but well-known Titanic data set for illustrative purposes.  Currently, dtreeviz supports the following decision tree libraries:\n",
    "\n",
    "* [scikit-learn](https://scikit-learn.org/stable)\n",
    "* [XGBoost](https://xgboost.readthedocs.io/en/latest)\n",
    "* [Spark MLlib](https://spark.apache.org/mllib/)\n",
    "* [LightGBM](https://lightgbm.readthedocs.io/en/latest/)\n",
    "* [Tensorflow](https://www.tensorflow.org/decision_forests)\n",
    "\n",
    "To interopt with these different libraries, dtreeviz uses an adaptor object, obtained from function `dtreeviz.model()`, to extract model information necessary for visualization. Given such an adaptor object, all of the dtreeviz functionality is available to you using the same programmer interface. The basic dtreeviz usage recipe is:\n",
    "\n",
    "1. Import dtreeviz and your decision tree library\n",
    "2. Acquire and load data into memory\n",
    "3. Train a classifier or regressor model using your decision tree library\n",
    "4. Obtain a dtreeviz adaptor model using<br>`viz_model = dtreeviz.model(your_trained_model,...)`\n",
    "5. Call dtreeviz functions, such as<br>`viz_model.view()` or `viz_model.explain_prediction_path(sample_x)`\n",
    "\n",
    "The four categories of dtreeviz functionality are:\n",
    "\n",
    "1. Tree visualizations\n",
    "2. Prediction path explanations\n",
    "3. Leaf information\n",
    "4. Feature space exploration\n",
    "\n",
    "We have grouped code examples by [classifiers](#Classifiers) and [regressors](#Regressors), with a follow up section on [partitioning feature space](#Feature-Space-Partitioning).\n",
    "\n",
    "*These examples require dtreeviz 2.0 or above because the code uses the new API introduced in 2.0.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_format = 'retina' # Make visualizations look good\n",
    "#%config InlineBackend.figure_format = 'svg' \n",
    "%matplotlib inline\n",
    "\n",
    "if 'google.colab' in sys.modules:\n",
    "  !pip install -q dtreeviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor\n",
    "\n",
    "import dtreeviz\n",
    "\n",
    "random_state = 1234 # get reproducible trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Sample Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_url = \"https://github.com/raw/parrt/dtreeviz/master/data/titanic/titanic.csv\"\n",
    "dataset = pd.read_csv(dataset_url)\n",
    "# Fill missing values for Age\n",
    "dataset.fillna({\"Age\":dataset.Age.mean()}, inplace=True)\n",
    "# Encode categorical variables\n",
    "dataset[\"Sex_label\"] = dataset.Sex.astype(\"category\").cat.codes\n",
    "dataset[\"Cabin_label\"] = dataset.Cabin.astype(\"category\").cat.codes\n",
    "dataset[\"Embarked_label\"] = dataset.Embarked.astype(\"category\").cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifiers\n",
    "\n",
    "To demonstrate classifier decision trees, we trying to model using six features to predict the boolean survived target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = [\"Pclass\", \"Age\", \"Fare\", \"Sex_label\", \"Cabin_label\", \"Embarked_label\"]\n",
    "target = \"Survived\"\n",
    "\n",
    "tree_classifier = DecisionTreeClassifier(max_depth=3, random_state=random_state)\n",
    "tree_classifier.fit(dataset[features].values, dataset[target].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Initialize dtreeviz model (adaptor)\n",
    "\n",
    "To adapt dtreeviz to a specific model, use the `model()` function to get an adaptor.  You'll need to provide the model, X/y data, feature names, target name, and target class names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model = dtreeviz.model(tree_classifier,\n",
    "                           X_train=dataset[features], y_train=dataset[target],\n",
    "                           feature_names=features,\n",
    "                           target_name=target, class_names=[\"survive\", \"perish\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use this model to demonstrate dtreeviz functionality in the following sections; the code will look the same for any decision tree library once we have this model adaptor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tree structure visualizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To show the decision tree structure using the default visualization, call `view()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "viz_model.view(scale=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To change the visualization, you can pass parameters, such as changing the orientation to left-to-right:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.view(orientation=\"LR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize larger trees, you can reduce the amount of detail by turning off the fancy view:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.view(fancy=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to reduce the visualization size is to specify the tree depths of interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.view(depth_range_to_display=(1, 2)) # root is level 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction path explanations\n",
    "\n",
    "For interpretation purposes, we often want to understand how a tree behaves for a specific instance. Let's pick a specific instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = dataset[features].iloc[10]\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and then display the path through the tree structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.view(x=x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.view(x=x, show_just_path=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "You can also get a string representation explaining the comparisons made as an instance is run down the tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(viz_model.explain_prediction_path(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you'd like the feature importance for a specific instance, as calculated by the underlying decision tree library, use `instance_feature_importance()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.instance_feature_importance(x, figsize=(3.5,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leaf info\n",
    "\n",
    "There are a number of functions to get information about the leaves of the tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.leaf_sizes(figsize=(3.5,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.ctree_leaf_distributions(figsize=(3.5,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.node_stats(node_id=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.leaf_purity(figsize=(3.5,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regressors\n",
    "\n",
    "To demonstrate regressor tree visualization, we start by creating a regressors model that predicts age instead of survival:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_reg = [\"Pclass\", \"Fare\", \"Sex_label\", \"Cabin_label\", \"Embarked_label\", \"Survived\"]\n",
    "target_reg = \"Age\"\n",
    "\n",
    "tree_regressor = DecisionTreeRegressor(max_depth=3, random_state=random_state, criterion=\"absolute_error\")\n",
    "tree_regressor.fit(dataset[features_reg].values, dataset[target_reg].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize dtreeviz model (adaptor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel = dtreeviz.model(model=tree_regressor, \n",
    "                            X_train=dataset[features_reg], \n",
    "                            y_train=dataset[target_reg], \n",
    "                            feature_names=features_reg, \n",
    "                            target_name=target_reg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tree structure visualisations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.view()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.view(orientation=\"LR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.view(fancy=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.view(depth_range_to_display=(0, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction path explanations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = dataset[features_reg].iloc[10]\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.view(x = x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.view(show_just_path=True, x = x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(viz_rmodel.explain_prediction_path(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.instance_feature_importance(x, figsize=(3.5,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leaf info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.leaf_sizes(figsize=(3.5,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.rtree_leaf_distributions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.node_stats(node_id=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.leaf_purity(figsize=(3.5,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Feature Space Partitioning\n",
    "\n",
    "Decision trees partition feature space in such a way as to maximize target value purity for the instances associated with a node. It's often useful to visualize the feature space partitioning, although it's not feasible to visualize more than a couple of dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "\n",
    "To visualize how it decision tree partitions a single feature, let's train a shallow decision tree classifier using the toy Iris data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "class_names = iris.target_names\n",
    "\n",
    "X = X[:,2].reshape(-1,1) # petal length (cm)\n",
    "y = iris.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtc_univar = DecisionTreeClassifier(max_depth=2, min_samples_leaf=1)\n",
    "dtc_univar.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model = dtreeviz.model(dtc_univar,\n",
    "                           X_train=X, y_train=y,\n",
    "                           feature_names=['petal length (cm)'],\n",
    "                           target_name='iris',\n",
    "                           class_names=class_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following diagram indicates that the decision tree splits the petal length feature into three mostly-pure regions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model.ctree_feature_space(nbins=40, gtype='barstacked', show={'splits','title'},\n",
    "                             figsize=(5,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A deeper tree gives this finer grand partitioning of the single feature space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtc_univar = DecisionTreeClassifier(max_depth=3, min_samples_leaf=1)\n",
    "dtc_univar.fit(X, y)\n",
    "\n",
    "viz_model = dtreeviz.model(dtc_univar,\n",
    "                           X_train=X, y_train=y,\n",
    "                           feature_names=['petal length (cm)'],\n",
    "                           target_name='iris',\n",
    "                           class_names=class_names)\n",
    "\n",
    "viz_model.ctree_feature_space(nbins=40, gtype='barstacked', show={'splits','title'}, figsize=(5,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at how a decision tree partitions two-dimensional feature space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = iris.data\n",
    "X = X[:,[0,3]] # 'sepal length (cm)', 'petal width (cm)'\n",
    "y = iris.target\n",
    "dtc_bivar = DecisionTreeClassifier(max_depth=3, min_samples_leaf=1)\n",
    "dtc_bivar.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model = dtreeviz.model(dtc_bivar,\n",
    "                           X_train=X, y_train=y,\n",
    "                           feature_names=['sepal length (cm)', 'petal width (cm)'], \n",
    "                           target_name='iris',\n",
    "                           class_names=class_names)\n",
    "\n",
    "viz_model.ctree_feature_space(nbins=40, gtype='barstacked', show={'splits','title'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Regression\n",
    "\n",
    "To demonstrate regression, let's load a toy Cars data set and visualize the partitioning of univariate and bivariate feature spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_url = \"https://github.com/raw/parrt/dtreeviz/master/data/cars.csv\"\n",
    "df_cars = pd.read_csv(dataset_url)\n",
    "X = df_cars[['WGT']]\n",
    "y = df_cars['MPG']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtr_univar = DecisionTreeRegressor(max_depth=3, criterion=\"absolute_error\")\n",
    "dtr_univar.fit(X.values, y.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel = dtreeviz.model(dtr_univar, X, y,\n",
    "                           feature_names=['WGT'],\n",
    "                           target_name='MPG')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following visualization illustrates how the decision tree breaks up the `WGT` (car weight) in order to get relatively pure `MPG` (miles per gallon) target values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.rtree_feature_space()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to visualize two-dimensional feature space, we can draw in three dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_cars[[\"WGT\", \"ENG\"]]\n",
    "y = df_cars['MPG']\n",
    "dtr_bivar_3d = DecisionTreeRegressor(max_depth=3, criterion=\"absolute_error\")\n",
    "dtr_bivar_3d.fit(X.values, y.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel = dtreeviz.model(dtr_bivar_3d, X, y,\n",
    "                           feature_names=[\"WGT\", \"ENG\"],\n",
    "                           target_name='MPG')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.rtree_feature_space3D(fontsize=10,\n",
    "                        elev=30, azim=20, \n",
    "                        show={'splits','title'},\n",
    "                        colors={'tesselation_alpha':.5})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Equivalently, we can show a heat map as if we were looking at the three-dimensional plot from the top down:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_rmodel.rtree_feature_space()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}