python/1_data_preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictive Process Monitoring - Data Preparation\n",
    "\n",
    "This notebook is part of the [starter package](https://github.com/fmannhardt/starter-predictive-process-monitoring) for predictive process monitoring. It contains examples for prefix extraction and prefix encoding from event logs for the purpose of developing and applying predictive process monitoring techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "The following Python libraries are used, please refer to the installation instructions to prepare your environment:\n",
    "\n",
    "* [PM4Py](https://pm4py.fit.fraunhofer.de/)\n",
    "* [Pandas](https://pandas.pydata.org/)\n",
    "* [Numpy](https://numpy.org/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pm4py\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Event Log & Data Loading\n",
    "\n",
    "We continue with the data loaded in the [previous notebook](./0_data_loading.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "import os\n",
    "\n",
    "# download from 4tu.nl\n",
    "urlretrieve('https://data.4tu.nl/file/33632f3c-5c48-40cf-8d8f-2db57f5a6ce7/643dccf2-985a-459e-835c-a82bce1c0339', 'sepsis1.xes.gz')\n",
    "sepsis_log = pm4py.read_xes('sepsis1.xes.gz')\n",
    "os.unlink('sepsis1.xes.gz') # clean up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prefix Extraction\n",
    "\n",
    "Many different prediction tasks are possible based on an event log. Often, the assumption is made that only a prefix of a trace is known and that a prediction on some future state of the process instance represented by that trace should be made.\n",
    "\n",
    "The first step is to generate suitable prefixes of the traces contained in the event log to be used as the training samples. As a *simple example*, we may be interested in predicting whether the patient in the process returns ot the emergency room indicated by the event *Return ER* as the last event. Since the event *Return ER* is part of the event log, we need to remove that event and remember in which trace it occurred. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_returns = [len(list(filter(lambda e: e[\"concept:name\"] == \"Return ER\" ,trace))) > 0 for trace in sepsis_log]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check if this worked\n",
    "print(sepsis_log[3][-1])\n",
    "print(sepsis_returns[3])\n",
    "\n",
    "print(sepsis_log[0][-1])\n",
    "print(sepsis_returns[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the same time, we may be interested in how well we can predict whether a patient returns for different sizes of the prefix, e.g., we can generate a new event log keeping only prefixes of each trace with at most size 10 (*10-prefix*).\n",
    "\n",
    "**Note that this is just a simple example with 10 chosen as arbitrary prefix length and in the general case you generate not only prefixes of a specific size but of variables or all sizes. Also, some traces are less than 10 events long in which case we would use the full trace for the prediction, which would not be very useful in practice.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove Return ER event\n",
    "sepsis_log = pm4py.filter_event_attribute_values(sepsis_log, \"concept:name\", \"Return ER\", level = \"event\", retain=False)\n",
    "\n",
    "from pm4py.objects.log.obj import EventLog, Trace\n",
    "# generate prefixes, note that we need to add the casts to EventLog and Trace to make sure that the result is a PM4Py EventLog object\n",
    "sepsis_prefixes = EventLog([Trace(trace[0:10], attributes = trace.attributes) for trace in sepsis_log])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check the trace length\n",
    "print([len(trace) for trace in sepsis_log][0:15])\n",
    "print([len(trace) for trace in sepsis_prefixes][0:15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prefix Encoding\n",
    "\n",
    "For training a prediction model, the traces or sequences of events need to be often transformed to a vector representation. We show how to compute three basic encodings+ using the built-in PM4Py [feature selection and processing](https://pm4py.fit.fraunhofer.de/documentation#decision-trees) functionality.\n",
    "\n",
    "Of course, more complex encodings such as representing each trace as a sequence of features are possible, e.g., for sequential models such as LSTMs. This is left as exercise. \n",
    "\n",
    "### Feature Selection \\& Engineering\n",
    "\n",
    "Before we do prefix encoding, we need to select which features we will use for the prediction. In this example we will only use the \"activity\" of the events as feature. Depending on your prediction problem, you might want to include additional trace/event attributes.\n",
    "\n",
    "Additionally, you can also derive new trace-level features (e.g., day of week, time since case start) or log-based features (e.g., workload of resources, number of active cases at a certain time). This is left as exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding as Set of Events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pm4py.algo.transformation.log_to_features import algorithm as log_to_features\n",
    "\n",
    "# log_to_feature provides a flexible interface to compute features on an event and trace level\n",
    "# see the documentation for more information: https://pm4py.fit.fraunhofer.de/documentation#item-7-0-2 \n",
    "data, feature_names = log_to_features.apply(sepsis_prefixes, parameters={\"str_ev_attr\": [\"concept:name\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard encoding of the `concept:name` attribute (i.e., the event label) is a one-hot encoded vector. Let us have a look at the encoding. The index of the number corresponds to the index in the feature label vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pm4py.objects.log.util.log import project_traces\n",
    "def project_nth(log, index):\n",
    "    print(str(project_traces(log)[index]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_nth(sepsis_prefixes, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The overall data shape is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.asarray(data).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, PM4Py gives us a *one-hot encoding* of the so called *set abstraction* of the event log. This means there are 16 distinct activities in the event log and the feature vector simply encodes whether that activity is present or not in the data. \n",
    "\n",
    "Let us have a look at the distribution of these feature vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# look at the unique vectors and their occurrence frequency/count\n",
    "dist_features = np.unique(data, return_counts= True, axis = 0)\n",
    "print(dist_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the most common feature vector?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# argmax give use the index of the most frequent vector\n",
    "dist_features[0][np.argmax(dist_features[1])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Makes sense, almost all activities actually are bound to occur in this process. There are only few choices.\n",
    "So, this encoding is likely not the most useful one but a very simple one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding as Bi-Grams / Succession Relation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_2gram, feature_names = log_to_features.apply(sepsis_prefixes, \n",
    "                                                  parameters={\"str_ev_attr\": [], \n",
    "                                                        \"str_tr_attr\": [], \n",
    "                                                        \"num_ev_attr\": [], \n",
    "                                                        \"num_tr_attr\": [], \n",
    "                                                        \"str_evsucc_attr\": [\"concept:name\"]})\n",
    "feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each feature represents the succession relation (or bigram) between any two activities of the event log. We transform the features into a tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_2gram = np.asarray(data_2gram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us, again, have a look at the encoding of the first trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_nth(sepsis_log, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data_2gram[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding as Bag of Words / Multiset of Events"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option would be to use the encoding known as [bag-of-words model](https://en.wikipedia.org/wiki/Bag-of-words_model) in Natural Language Processing, which is constructing a multiset of the one-hot encoded events. So, the frequency with which each activity occurs is reflected. This encoding is not provided in PM4Py but can be easily computed with Pandas and Numpy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first need to transform the PM4Py event log to a Pandas data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_df = pm4py.convert_to_dataframe(sepsis_prefixes)\n",
    "sepsis_df.head(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We build a bag of words representation by grouping our data and then counting the number of events refering to the individual activities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# concept:name refers to the activity\n",
    "# case:concept:name refers to the case identifier\n",
    "sepsis_case_act = sepsis_df.loc[:,[\"case:concept:name\", \"concept:name\"]]\n",
    "sepsis_case_act"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the occurrence of activities in a trace (no sorting to keep order of traces stable!)\n",
    "sepsis_act_count = sepsis_case_act.groupby([\"case:concept:name\", \"concept:name\"], sort=False).size()\n",
    "sepsis_act_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have the count of each activity for each trace and still need to convert this to a tensor format such that we have one feature vector (columns) per case (row)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_bag = np.asarray(sepsis_act_count.unstack(fill_value=0))\n",
    "sepsis_bag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_bag.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us, again, have a look at the encoding of the first trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_nth(sepsis_log, 0)\n",
    "print(sepsis_bag[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_nth(sepsis_log, 1)\n",
    "print(sepsis_bag[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This already gives us much more information to work with."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "28aff1567d8aae5536826c1be921f2ff2e204808293d43dc67bdcb73bd29110e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}