python/lecture4-prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applied Process Mining Module\n",
    "\n",
    "This notebook is part of an Applied Process Mining module. The collection of notebooks is a *living document* and subject to change. \n",
    "\n",
    "# Lecture 4 - 'Predictive Process Mining' (Python / PM4Py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "<img src=\"https://pm4py.fit.fraunhofer.de/static/assets/images/pm4py-site-logo-padded.png\" alt=\"PM4Py\" style=\"width: 200px;\"/>\n",
    "\n",
    "In this notebook, we are using the [PM4Py library](https://pm4py.fit.fraunhofer.de/) in combination with several standard Python data science libraries:\n",
    "\n",
    "* [pandas](https://pandas.pydata.org/)\n",
    "* [PyTorch](https://pytorch.org/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Perform the commented out commands to install the dependencies\n",
    "# %pip install pandas\n",
    "# %pip install matplotlib\n",
    "# %pip install pm4py\n",
    "# %pip install torch\n",
    "# %pip install tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pm4py\n",
    "import os\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from tqdm.autonotebook import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictive Process Mining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Event Log \n",
    "\n",
    "We are using the Sepsis event log as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "import os\n",
    "\n",
    "# download from 4tu.nl\n",
    "urlretrieve('https://data.4tu.nl/ndownloader/files/24061976', 'sepsis.xes.gz')\n",
    "sepsis_log = pm4py.read_xes('sepsis.xes.gz')\n",
    "os.unlink('sepsis.xes.gz') # clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(sepsis_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Prefix Extraction\n",
    "\n",
    "Many different prediction tasks are possible based on an event log. Often, the assumption is made that only a prefix of a trace is known and that a prediction on some future state of the process instance represented by that trace should be made.\n",
    "\n",
    "The first step is to generate suitable prefixes of the traces contained in the event log to be used as the training samples. As a *simple example*, we may be interested in predicting whether the patient in the process returns ot the emergency room indicated by the event *Return ER* as the last event. Since the event *Return ER* is part of the event log, we need to remove that event and remember in which trace it occurred. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_returns = [len(list(filter(lambda e: e[\"concept:name\"] == \"Return ER\" ,trace))) > 0 for trace in sepsis_log]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check if this worked\n",
    "print(sepsis_log[3][-1])\n",
    "print(sepsis_returns[3])\n",
    "\n",
    "print(sepsis_log[0][-1])\n",
    "print(sepsis_returns[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the same time, we may be interested in how well we can predict whether a patient returns for different sizes of the prefix, e.g., we can generate a new event log keeping only prefixes of each trace with at most size 10 (*10-prefix*).\n",
    "\n",
    "**Note that this is just a simple example with 10 chosen as arbitrary prefix length and in the general case you generate not only prefixes of a specific size but of variables or all sizes. Also, some traces are less than 10 events long in which case we would use the full trace for the prediction, which would not be very useful in practice.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove Return ER event\n",
    "sepsis_log = pm4py.filter_event_attribute_values(sepsis_log, \"concept:name\", \"Return ER\", level = \"event\", retain=False)\n",
    "\n",
    "from pm4py.objects.log.obj import EventLog, Trace\n",
    "# generate prefixes, note that we need to add the casts to EventLog and Trace to make sure that the result is a PM4Py EventLog object\n",
    "sepsis_prefixes = EventLog([Trace(trace[0:10], attributes = trace.attributes) for trace in sepsis_log])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check the trace length\n",
    "print([len(trace) for trace in sepsis_log][0:15])\n",
    "print([len(trace) for trace in sepsis_prefixes][0:15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prefix Encoding\n",
    "\n",
    "For training a prediction model, the traces or sequences of events need to be often transformed to a vector representation. We show how to compute three basic encodings+ using the built-in PM4Py [feature selection and processing](https://pm4py.fit.fraunhofer.de/documentation#decision-trees) functionality.\n",
    "\n",
    "Of course, more complex encodings such as representing each trace as a sequence of features are possible, e.g., for sequential models such as LSTMs. This is left as exercise. \n",
    "\n",
    "### Feature Selection \\& Engineering\n",
    "\n",
    "Before we do prefix encoding, we need to select which features we will use for the prediction. In this example we will only use the \"activity\" of the events as feature. Depending on your prediction problem, you might want to include additional trace/event attributes.\n",
    "\n",
    "Additionally, you can also derive new trace-level features (e.g., day of week, time since case start) or log-based features (e.g., workload of resources, number of active cases at a certain time). This is left as exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding as Set of Events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pm4py.algo.transformation.log_to_features import algorithm as log_to_features\n",
    "\n",
    "# log_to_feature provides a flexible interface to compute features on an event and trace level\n",
    "# see the documentation for more information: https://pm4py.fit.fraunhofer.de/documentation#item-7-0-2 \n",
    "data, feature_names = log_to_features.apply(sepsis_prefixes, parameters={\"str_ev_attr\": [\"concept:name\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard encoding of the `concept:name` attribute (i.e., the event label) is a one-hot encoded vector. Let us have a look at the encoding. The index of the number corresponds to the index in the feature label vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pm4py.objects.log.util.log import project_traces\n",
    "def project_nth(log, index):\n",
    "    print(str(project_traces(log)[index]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_nth(sepsis_prefixes, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The overall data shape is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.asarray(data).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, PM4Py gives us a *one-hot encoding* of the so called *set abstraction* of the event log. This means there are 16 distinct activities in the event log and the feature vector simply encodes whether that activity is present or not in the data. \n",
    "\n",
    "Let us have a look at the distribution of these feature vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# look at the unique vectors and their occurrence frequency/count\n",
    "dist_features = np.unique(data, return_counts= True, axis = 0)\n",
    "print(dist_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the most common feature vector?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# argmax give use the index of the most frequent vector\n",
    "dist_features[0][np.argmax(dist_features[1])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Makes sense, almost all activities actually are bound to occur in this process. There are only few choices.\n",
    "So, this encoding is likely not the most useful one but a very simple one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding as Bi-Grams / Succession Relation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_2gram, feature_names = log_to_features.apply(sepsis_prefixes, \n",
    "                                                  parameters={\"str_ev_attr\": [], \n",
    "                                                        \"str_tr_attr\": [], \n",
    "                                                        \"num_ev_attr\": [], \n",
    "                                                        \"num_tr_attr\": [], \n",
    "                                                        \"str_evsucc_attr\": [\"concept:name\"]})\n",
    "feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each feature represents the succession relation (or bigram) between any two activities of the event log. We transform the features into a tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_2gram = np.asarray(data_2gram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us, again, have a look at the encoding of the first trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_nth(sepsis_log, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data_2gram[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding as Bag of Words / Multiset of Events"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option would be to use the encoding known as [bag-of-words model](https://en.wikipedia.org/wiki/Bag-of-words_model) in Natural Language Processing, which is constructing a multiset of the one-hot encoded events. So, the frequency with which each activity occurs is reflected. This encoding is not provided in PM4Py but can be easily computed with Pandas and Numpy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first need to transform the PM4Py event log to a Pandas data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_df = pm4py.convert_to_dataframe(sepsis_prefixes)\n",
    "sepsis_df.head(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We build a bag of words representation by grouping our data and then counting the number of events refering to the individual activities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# concept:name refers to the activity\n",
    "# case:concept:name refers to the case identifier\n",
    "sepsis_case_act = sepsis_df.loc[:,[\"case:concept:name\", \"concept:name\"]]\n",
    "sepsis_case_act"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the occurrence of activities in a trace (no sorting to keep order of traces stable!)\n",
    "sepsis_act_count = sepsis_case_act.groupby([\"case:concept:name\", \"concept:name\"], sort=False).size()\n",
    "sepsis_act_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have the count of each activity for each trace and still need to convert this to a tensor format such that we have one feature vector (columns) per case (row)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_bag = np.asarray(sepsis_act_count.unstack(fill_value=0))\n",
    "sepsis_bag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_bag.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us, again, have a look at the encoding of the first trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_nth(sepsis_log, 0)\n",
    "print(sepsis_bag[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_nth(sepsis_log, 1)\n",
    "print(sepsis_bag[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This already gives us much more information to work with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "Let us try to build a basic prediction model based on this information. In this example, we aim to predict the binary outcome whether the event `Return ER` occurred or not. \n",
    "\n",
    "**Disclaimer: here *basic* means that the model and encoding is not expected to be of any quality. Also note that the prediction task, while useful, may not be feasible based on the prefix encoding that we chose. Treat the following code as an example and starting point only!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Target Variable\n",
    "\n",
    "Let us look at the distribution of the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(sepsis_returns, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For future processing we need 0 and 1 instead of True and False\n",
    "sepsis_returns = np.asarray(sepsis_returns).astype(int)\n",
    "sepsis_returns.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Scaling & Loading\n",
    "\n",
    "This often helps prediction models to perform better.\n",
    "\n",
    "**Important:** make sure to not compute the scaling with the test set included since there is a risk of data leakage otherwise. In other words, the test set should be separated before any pre-processing, which may use a property of the dataset, is applied. Of course, the test set is scaled as well but with the scaler *trained* only on the training set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import FunctionTransformer, MinMaxScaler\n",
    "\n",
    "scaler_x = MinMaxScaler()\n",
    "data_scaled = scaler_x.fit_transform(sepsis_bag)\n",
    "\n",
    "scaler_y = FunctionTransformer() # for binary values scaling does not make sense at all but we keep it for symetry and apply the \"NoOp\" scaler\n",
    "target_scaled = scaler_y.fit_transform(sepsis_returns.reshape(-1, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Definition\n",
    "\n",
    "Let's define a simple network and try to overfit. We make use of PyTorch to build a simple Neural Network. \n",
    "\n",
    "**Disclaimer: Again, this is just a simple example and not in anyway meant as a recommendation for a model.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralNetworkBinaryOutcome(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(NeuralNetworkBinaryOutcome, self).__init__()\n",
    "        self.linear_relu_stack = nn.Sequential(            \n",
    "            torch.nn.Linear(x.shape[1], 64),\n",
    "            nn.BatchNorm1d(num_features=64),\n",
    "            nn.LeakyReLU(),            \n",
    "            torch.nn.Linear(64, 128),\n",
    "            nn.BatchNorm1d(num_features=128),          \n",
    "            torch.nn.Linear(128, 1),\n",
    "            nn.Sigmoid()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        logits = self.linear_relu_stack(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a standard training loop in PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(dataloader, model, \n",
    "          loss_fn, measure_fn, \n",
    "          optimizer, device, epochs): \n",
    "    \n",
    "    losses = []\n",
    "    size = len(dataloader.dataset)\n",
    "    \n",
    "    loop = tqdm(range(epochs))\n",
    "    \n",
    "    for epoch in loop:\n",
    "    \n",
    "        for batch, (X, y) in enumerate(dataloader):\n",
    "            X, y = X.to(device), y.to(device)\n",
    "\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            # Compute prediction error\n",
    "            pred = model(X)\n",
    "            \n",
    "            loss = loss_fn(pred, y)\n",
    "            measure = measure_fn(pred, y)\n",
    "\n",
    "            # Backpropagation\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            \n",
    "            losses.append([loss.item(), measure.item()])\n",
    "            \n",
    "        loop.set_description('Epoch {}/{}'.format(epoch + 1, epochs))\n",
    "        loop.set_postfix(loss=loss.item(), measure=measure.item())\n",
    "    \n",
    "    return losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And can use the following function to get all evaluation results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_all(dataloader, model, device):    \n",
    "    size = len(dataloader.dataset)\n",
    "    num_batches = len(dataloader)\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    result = []\n",
    "    original = []\n",
    "\n",
    "    with torch.no_grad(): \n",
    "        for X, y in tqdm(dataloader):  \n",
    "            X, y = X.to(device), y.to(device) \n",
    "            pred = model(X)          \n",
    "                        \n",
    "            result.extend(pred.flatten().numpy())\n",
    "            original.extend(y.flatten().numpy())\n",
    "                           \n",
    "    return np.asarray(result), np.asarray(original)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Prepare the data for the PyTorch data loading mechanism."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "# We need float32 data\n",
    "x = torch.from_numpy(data_scaled.astype('float32'))\n",
    "y = torch.from_numpy(target_scaled.astype('float32'))\n",
    "\n",
    "# Always check the shapes\n",
    "print(x.shape)\n",
    "print(y.shape)\n",
    "\n",
    "ds = TensorDataset(x, y)\n",
    "train_dataloader = DataLoader(ds, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us check a random single sample from our data loader (always a good idea!) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs, classes = next(iter(train_dataloader))\n",
    "print(inputs[0])\n",
    "print(classes[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train the model using cross entropy as loss function accuracy as easier to interpret measure to report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## if you want ot use a GPU you need to tweak the requirements.txt to include the GPU-enabled PyTorch\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print('Using {} device'.format(device))\n",
    "\n",
    "# fix a seed to get reproducible results\n",
    "torch.manual_seed(42)\n",
    "\n",
    "model = NeuralNetworkBinaryOutcome().to(device)\n",
    "print(model)\n",
    "\n",
    "def get_accuracy(y_prob, y_true):    \n",
    "    y_true = y_true.flatten()\n",
    "    y_prob = y_prob.flatten()\n",
    "    assert y_true.ndim == 1 and y_true.size() == y_prob.size()\n",
    "    y_prob = y_prob > 0.5\n",
    "    return (y_true == y_prob).sum() / y_true.size(0)\n",
    "measure_fn = get_accuracy\n",
    "\n",
    "results = train(train_dataloader, model, \n",
    "                nn.BCELoss(), # crossentropy for binary target \n",
    "                get_accuracy, \n",
    "                torch.optim.Adam(model.parameters()), \n",
    "                device, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "results_data = pd.DataFrame(results)\n",
    "results_data.columns = ['loss', 'measure']\n",
    "ax = results_data.plot(subplots=True);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Accuracy: \" + str(results[len(results)-1][1]))\n",
    "\n",
    "true_returns = np.unique(sepsis_returns, return_counts=True)[1][0]\n",
    "true_not_returns = np.unique(sepsis_returns, return_counts=True)[1][1]\n",
    "\n",
    "print(\"Accuracy (never returns)\" + str(true_returns / len(sepsis_returns)))\n",
    "print(\"Accuracy (always returns)\" + str(true_not_returns / len(sepsis_returns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Brief Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, that is a bit better compared to simply always saying that the patient does not return. But the accuracy on the training set (**not even considering a test set!**) is still varying a lot and the variation of the log and accuracy over the epochs trained does not look good either. So, let us still have a look at the individual predictions and their score depending on the ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataloader = DataLoader(ds, batch_size=256, shuffle=False)\n",
    "result, original = evaluate_all(test_dataloader, model, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd_pos = pd.DataFrame({'Returns': result[original == 1]})\n",
    "pd_neg = pd.DataFrame({'Does not return': result[original == 0]})\n",
    "pd.concat([pd_pos, pd_neg],axis=1).boxplot().set_ylabel('Score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There seems to be some separation but likely the prediction model will give us many false positives when used to identify returning patients in practice.\n",
    "\n",
    "**Of course, you should now compute the usual measures for classification tasks and the threshold for making a decision: recall, precision, confusion matrices, area under the curve and many other ways to deeply evaluate a prediction model. Always consider what would be the use case of your prediction.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is this so bad? Let us have a look at the data distribution we put in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# count the unique vectors\n",
    "dist_bags = np.unique(sepsis_bag, return_counts=True, axis=0)\n",
    "\n",
    "# sort them with numpy\n",
    "unique_vectors = dist_bags[0][np.argsort(-dist_bags[1])]\n",
    "count_vectors = dist_bags[1][np.argsort(-dist_bags[1])]\n",
    "\n",
    "pd.DataFrame({'Occurrence of unique sample vectors': count_vectors}).boxplot().set_ylabel('Frequency')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many of the traces result in the exact same sample. Let us check what is the \"return status\" for the most common sample that represents more than 175 traces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# most frequently used vector\n",
    "unique_vectors[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the sample indicies for this vector\n",
    "sample_indicies = np.where((sepsis_bag == unique_vectors[0]).all(axis=1)) \n",
    "sample_durations = target_scaled[sample_indicies]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(sample_durations, return_counts=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is clear that, without additional information, there is nothing the prediction model can learn to represent this division for the exact same feature values. We can look at further examples, but it seems we simply cannot reliably predict whether a patient will return from the bag-of-words / multiset of events model in the Sepsis event log.\n",
    "\n",
    "This was just an example on how to use a predictive model with an event log to predict a binary process characteristic based on events contained in the event log."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "986d19a5f30747571c3041dc5ec04d8eebcc8ad808bb611da7ac1a06db10f3a6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}