RF_CWRU_EvaluationFramework.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "RF CWRU-EvaluationFramework.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/fboldt/SignalProcessing/blob/master/RF_CWRU_EvaluationFramework.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "llMoY5AQGXQY",
        "colab_type": "text"
      },
      "source": [
        "The code is separated into four sections.\n",
        "\n",
        "* In section \"CWRU dataset\" the CWRU Matlab files are downloaded, the acquisitions are extracted from the Matlab files into Numpy arrays. Then, the data is segmented and the samples are selected. The samples are selected by their labels using regular expressions. \n",
        "\n",
        "* Section \"Experimenter\" defines the splitters mentioned in this work, i.e. GroupShuffleKFold and BySeverityKFold, and the setup of each experiment. The samples are grouped by their original labels also using regular expressions. For instance, to group samples by load, the regular expression '_\\d' may be used. A list of evaluation methods is defined in this section. \n",
        "\n",
        "* Section \"Classification Models\" defines the estimators and their feature extraction methods. To instantiate a classification method with feature extraction, a Pipeline must be made. A list of classification methods is defined in this section. \n",
        "\n",
        "* Finally, section \"Performing Experiments\" executes the experiments as they were defined in the previous sections, showing and saving  their results. It iterates one list of classification methods and one list of evaluation methods in $r$ rounds. In this work, four classification methods were tested by three evaluation methods in four rounds, resulting in $4\\times 3\\times 4 = 48$  experiments. New classification or evaluation methods can be tested by adding them in their respective list.\n",
        "\n",
        "The code can be executed direct in the Colab environment when few samples and simple classifier methods are tested. For the experiments presented in this work, it must be run in a local GPU, with enough memory and processing capacity.\n",
        "The results are presented for each round of each experiment of each classification method. The average and standard deviation of the rounds is also presented, as well, the average of the differences among the classification methods.\n",
        "\n",
        "New feature extraction methods must receive a 3-D Numpy array and returns a 2-D Numpy array. This is necessary because the same raw dataset is used for methods that need feature extraction of the signal acquisitions, like K-NN, SVM and Random Forest, and convolutional neural networks that deal with 3-D arrays. The experiments presented here used just one channel of each acquisition, but newer experiments may use more channels.\n",
        "\n",
        "New classifications methods that need feature extraction may be easily added. It is only required a Pipeline with the feature extraction method and the classifier, like those presented in the code. A new neural network base architecture must be wrapped in a scikit-learn estimator, and its definition must be in the method *fit*. It cannot be defined in the method *\\_\\_init\\_\\_* due to the implementation of the Keras library. If the network architecture is defined in the method *\\_\\_init\\_\\_*, it will remember the samples across the folds and the rounds, giving outstanding results, that will be never replicated in a real scenario. It is worth to highlight that some parameters of the network, like kernel size and number of filters, should be selected by a GridSearchCV method to provide fairer results when compared with other methods."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sm5t8TYBqkDu",
        "colab_type": "text"
      },
      "source": [
        "#CWRU database"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bSSOMru17Z6c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "debug = False"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pMQoq6dvStey",
        "colab_type": "text"
      },
      "source": [
        "## CWRU files.\n",
        "\n",
        "Associate each Matlab file name to a bearing condition in a Python dictionary.\n",
        "The dictionary keys identify the conditions.\n",
        "\n",
        "There are only four normal conditions, with loads of 0, 1, 2 and 3 hp.\n",
        "All conditions end with an underscore character followed by an algarism representing the load applied during the acquisitions.\n",
        "The remaining conditions follow the pattern:\n",
        "\n",
        "\n",
        "* First two characters represent the bearing location, i.e. drive end (DE) and fan end (FE).\n",
        "* The following two characters represent the failure location in the bearing, i.e. ball (BA), Inner Race (IR) and Outer Race (OR).\n",
        "* The next three algarisms indicate the severity of the failure, where 007 stands for 0.007 inches and 0021 for 0.021 inches.\n",
        "* For Outer Race failures, the character @ is followed by a number that indicates different load zones. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K6mp2QrP1lmR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cwru_12khz():\n",
        "  '''\n",
        "  Retuns a dictionary with the names of all Matlab files read in 12kHz located in\n",
        "  http://csegroups.case.edu/sites/default/files/bearingdatacenter/files/Datafiles/.\n",
        "  The dictionary keys represent the bearing condition.\n",
        "  '''\n",
        "  matlab_files_name = {}\n",
        "  # Normal\n",
        "  matlab_files_name[\"Normal_0\"] = \"97.mat\"\n",
        "  matlab_files_name[\"Normal_1\"] = \"98.mat\"\n",
        "  matlab_files_name[\"Normal_2\"] = \"99.mat\"\n",
        "  matlab_files_name[\"Normal_3\"] = \"100.mat\"\n",
        "  # DE Inner Race 0.007 inches\n",
        "  matlab_files_name[\"DEIR.007_0\"] = \"105.mat\"\n",
        "  matlab_files_name[\"DEIR.007_1\"] = \"106.mat\"\n",
        "  matlab_files_name[\"DEIR.007_2\"] = \"107.mat\"\n",
        "  matlab_files_name[\"DEIR.007_3\"] = \"108.mat\"\n",
        "  # DE Ball 0.007 inches\n",
        "  matlab_files_name[\"DEB.007_0\"] = \"118.mat\"\n",
        "  matlab_files_name[\"DEB.007_1\"] = \"119.mat\"\n",
        "  matlab_files_name[\"DEB.007_2\"] = \"120.mat\"\n",
        "  matlab_files_name[\"DEB.007_3\"] = \"121.mat\"\n",
        "  # DE Outer race 0.007 inches centered @6:00\n",
        "  matlab_files_name[\"DEOR.007@6_0\"] = \"130.mat\"\n",
        "  matlab_files_name[\"DEOR.007@6_1\"] = \"131.mat\"\n",
        "  matlab_files_name[\"DEOR.007@6_2\"] = \"132.mat\"\n",
        "  matlab_files_name[\"DEOR.007@6_3\"] = \"133.mat\"\n",
        "  # DE Outer race 0.007 inches centered @3:00\n",
        "  matlab_files_name[\"DEOR.007@3_0\"] = \"144.mat\"\n",
        "  matlab_files_name[\"DEOR.007@3_1\"] = \"145.mat\"\n",
        "  matlab_files_name[\"DEOR.007@3_2\"] = \"146.mat\"\n",
        "  matlab_files_name[\"DEOR.007@3_3\"] = \"147.mat\"\n",
        "  # DE Outer race 0.007 inches centered @12:00\n",
        "  matlab_files_name[\"DEOR.007@12_0\"] = \"156.mat\"\n",
        "  matlab_files_name[\"DEOR.007@12_1\"] = \"158.mat\"\n",
        "  matlab_files_name[\"DEOR.007@12_2\"] = \"159.mat\"\n",
        "  matlab_files_name[\"DEOR.007@12_3\"] = \"160.mat\"\n",
        "  # DE Inner Race 0.014 inches\n",
        "  matlab_files_name[\"DEIR.014_0\"] = \"169.mat\"\n",
        "  matlab_files_name[\"DEIR.014_1\"] = \"170.mat\"\n",
        "  matlab_files_name[\"DEIR.014_2\"] = \"171.mat\"\n",
        "  matlab_files_name[\"DEIR.014_3\"] = \"172.mat\"\n",
        "  # DE Ball 0.014 inches\n",
        "  matlab_files_name[\"DEB.014_0\"] = \"185.mat\"\n",
        "  matlab_files_name[\"DEB.014_1\"] = \"186.mat\"\n",
        "  matlab_files_name[\"DEB.014_2\"] = \"187.mat\"\n",
        "  matlab_files_name[\"DEB.014_3\"] = \"188.mat\"\n",
        "  # DE Outer race 0.014 inches centered @6:00\n",
        "  matlab_files_name[\"DEOR.014@6_0\"] = \"197.mat\"\n",
        "  matlab_files_name[\"DEOR.014@6_1\"] = \"198.mat\"\n",
        "  matlab_files_name[\"DEOR.014@6_2\"] = \"199.mat\"\n",
        "  matlab_files_name[\"DEOR.014@6_3\"] = \"200.mat\"\n",
        "  # DE Ball 0.021 inches\n",
        "  matlab_files_name[\"DEB.021_0\"] = \"222.mat\"\n",
        "  matlab_files_name[\"DEB.021_1\"] = \"223.mat\"\n",
        "  matlab_files_name[\"DEB.021_2\"] = \"224.mat\"\n",
        "  matlab_files_name[\"DEB.021_3\"] = \"225.mat\"\n",
        "  # FE Inner Race 0.021 inches\n",
        "  matlab_files_name[\"FEIR.021_0\"] = \"270.mat\"\n",
        "  matlab_files_name[\"FEIR.021_1\"] = \"271.mat\"\n",
        "  matlab_files_name[\"FEIR.021_2\"] = \"272.mat\"\n",
        "  matlab_files_name[\"FEIR.021_3\"] = \"273.mat\"\n",
        "  # FE Inner Race 0.014 inches\n",
        "  matlab_files_name[\"FEIR.014_0\"] = \"274.mat\"\n",
        "  matlab_files_name[\"FEIR.014_1\"] = \"275.mat\"\n",
        "  matlab_files_name[\"FEIR.014_2\"] = \"276.mat\"\n",
        "  matlab_files_name[\"FEIR.014_3\"] = \"277.mat\"\n",
        "  # FE Ball 0.007 inches\n",
        "  matlab_files_name[\"FEB.007_0\"] = \"282.mat\"\n",
        "  matlab_files_name[\"FEB.007_1\"] = \"283.mat\"\n",
        "  matlab_files_name[\"FEB.007_2\"] = \"284.mat\"\n",
        "  matlab_files_name[\"FEB.007_3\"] = \"285.mat\"\n",
        "  # DE Inner Race 0.021 inches\n",
        "  matlab_files_name[\"DEIR.021_0\"] = \"209.mat\"\n",
        "  matlab_files_name[\"DEIR.021_1\"] = \"210.mat\"\n",
        "  matlab_files_name[\"DEIR.021_2\"] = \"211.mat\"\n",
        "  matlab_files_name[\"DEIR.021_3\"] = \"212.mat\"\n",
        "  # DE Outer race 0.021 inches centered @6:00\n",
        "  matlab_files_name[\"DEOR.021@6_0\"] = \"234.mat\"\n",
        "  matlab_files_name[\"DEOR.021@6_1\"] = \"235.mat\"\n",
        "  matlab_files_name[\"DEOR.021@6_2\"] = \"236.mat\"\n",
        "  matlab_files_name[\"DEOR.021@6_3\"] = \"237.mat\"\n",
        "  # DE Outer race 0.021 inches centered @3:00\n",
        "  matlab_files_name[\"DEOR.021@3_0\"] = \"246.mat\"\n",
        "  matlab_files_name[\"DEOR.021@3_1\"] = \"247.mat\"\n",
        "  matlab_files_name[\"DEOR.021@3_2\"] = \"248.mat\"\n",
        "  matlab_files_name[\"DEOR.021@3_3\"] = \"249.mat\"\n",
        "  # DE Outer race 0.021 inches centered @12:00\n",
        "  matlab_files_name[\"DEOR.021@12_0\"] = \"258.mat\"\n",
        "  matlab_files_name[\"DEOR.021@12_1\"] = \"259.mat\"\n",
        "  matlab_files_name[\"DEOR.021@12_2\"] = \"260.mat\"\n",
        "  matlab_files_name[\"DEOR.021@12_3\"] = \"261.mat\"\n",
        "  # FE Inner Race 0.007 inches\n",
        "  matlab_files_name[\"FEIR.007_0\"] = \"278.mat\"\n",
        "  matlab_files_name[\"FEIR.007_1\"] = \"279.mat\"\n",
        "  matlab_files_name[\"FEIR.007_2\"] = \"280.mat\"\n",
        "  matlab_files_name[\"FEIR.007_3\"] = \"281.mat\"\n",
        "  # FE Ball 0.014 inches\n",
        "  matlab_files_name[\"FEB.014_0\"] = \"286.mat\"\n",
        "  matlab_files_name[\"FEB.014_1\"] = \"287.mat\"\n",
        "  matlab_files_name[\"FEB.014_2\"] = \"288.mat\"\n",
        "  matlab_files_name[\"FEB.014_3\"] = \"289.mat\"\n",
        "  # FE Ball 0.021 inches\n",
        "  matlab_files_name[\"FEB.021_0\"] = \"290.mat\"\n",
        "  matlab_files_name[\"FEB.021_1\"] = \"291.mat\"\n",
        "  matlab_files_name[\"FEB.021_2\"] = \"292.mat\"\n",
        "  matlab_files_name[\"FEB.021_3\"] = \"293.mat\"\n",
        "  # FE Outer race 0.007 inches centered @6:00\n",
        "  matlab_files_name[\"FEOR.007@6_0\"] = \"294.mat\"\n",
        "  matlab_files_name[\"FEOR.007@6_1\"] = \"295.mat\"\n",
        "  matlab_files_name[\"FEOR.007@6_2\"] = \"296.mat\"\n",
        "  matlab_files_name[\"FEOR.007@6_3\"] = \"297.mat\"\n",
        "  # FE Outer race 0.007 inches centered @3:00\n",
        "  matlab_files_name[\"FEOR.007@3_0\"] = \"298.mat\"\n",
        "  matlab_files_name[\"FEOR.007@3_1\"] = \"299.mat\"\n",
        "  matlab_files_name[\"FEOR.007@3_2\"] = \"300.mat\"\n",
        "  matlab_files_name[\"FEOR.007@3_3\"] = \"301.mat\"\n",
        "  # FE Outer race 0.007 inches centered @12:00\n",
        "  matlab_files_name[\"FEOR.007@12_0\"] = \"302.mat\"\n",
        "  matlab_files_name[\"FEOR.007@12_1\"] = \"305.mat\"\n",
        "  matlab_files_name[\"FEOR.007@12_2\"] = \"306.mat\"\n",
        "  matlab_files_name[\"FEOR.007@12_3\"] = \"307.mat\"\n",
        "  # FE Outer race 0.014 inches centered @3:00\n",
        "  matlab_files_name[\"FEOR.014@3_0\"] = \"310.mat\"\n",
        "  matlab_files_name[\"FEOR.014@3_1\"] = \"309.mat\"\n",
        "  matlab_files_name[\"FEOR.014@3_2\"] = \"311.mat\"\n",
        "  matlab_files_name[\"FEOR.014@3_3\"] = \"312.mat\"\n",
        "  # FE Outer race 0.014 inches centered @6:00\n",
        "  matlab_files_name[\"FEOR.014@6_0\"] = \"313.mat\"\n",
        "  # FE Outer race 0.021 inches centered @6:00\n",
        "  matlab_files_name[\"FEOR.021@6_0\"] = \"315.mat\"\n",
        "  # FE Outer race 0.021 inches centered @3:00\n",
        "  matlab_files_name[\"FEOR.021@3_1\"] = \"316.mat\"\n",
        "  matlab_files_name[\"FEOR.021@3_2\"] = \"317.mat\"\n",
        "  matlab_files_name[\"FEOR.021@3_3\"] = \"318.mat\"\n",
        "  # DE Inner Race 0.028 inches\n",
        "  matlab_files_name[\"DEIR.028_0\"] = \"3001.mat\"\n",
        "  matlab_files_name[\"DEIR.028_1\"] = \"3002.mat\"\n",
        "  matlab_files_name[\"DEIR.028_2\"] = \"3003.mat\"\n",
        "  matlab_files_name[\"DEIR.028_3\"] = \"3004.mat\"\n",
        "  # DE Ball 0.028 inches\n",
        "  matlab_files_name[\"DEB.028_0\"] = \"3005.mat\"\n",
        "  matlab_files_name[\"DEB.028_1\"] = \"3006.mat\"\n",
        "  matlab_files_name[\"DEB.028_2\"] = \"3007.mat\"\n",
        "  matlab_files_name[\"DEB.028_3\"] = \"3008.mat\"\n",
        "  return matlab_files_name\n",
        "\n",
        "def files_debug():\n",
        "  \"\"\"\n",
        "  Associate each Matlab file name to a bearing condition in a Python dictionary. \n",
        "  The dictionary keys identify the conditions.\n",
        "  \n",
        "  NOTE: Used only for debug.\n",
        "  \"\"\"\n",
        "  matlab_files_name = {}\n",
        "  # Normal\n",
        "  matlab_files_name[\"Normal_0\"] = \"97.mat\"\n",
        "  matlab_files_name[\"Normal_1\"] = \"98.mat\"\n",
        "  matlab_files_name[\"Normal_2\"] = \"99.mat\"\n",
        "  matlab_files_name[\"Normal_3\"] = \"100.mat\"\n",
        "  # FE Inner Race 0.007 inches\n",
        "  matlab_files_name[\"FEIR.007_2\"] = \"280.mat\"\n",
        "  # DE Outer race 0.014 inches centered @6:00\n",
        "  matlab_files_name[\"DEOR.014@6_1\"] = \"198.mat\"\n",
        "  # FE Outer race 0.021 inches centered @6:00\n",
        "  matlab_files_name[\"FEOR.021@6_0\"] = \"315.mat\"\n",
        "  # DE Ball 0.028 inches\n",
        "  matlab_files_name[\"DEB.028_3\"] = \"3008.mat\"\n",
        "  return matlab_files_name"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K9y9byVeSz_u",
        "colab_type": "text"
      },
      "source": [
        "##Download Matlab files\n",
        "Downloads the Matlab files in the dictionary matlab_files_name."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wPSGH1401-W2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import urllib.request\n",
        "import os.path\n",
        "\n",
        "def download_cwrufiles(matlab_files_name):\n",
        "  '''\n",
        "  Downloads the Matlab files in the dictionary matlab_files_name.\n",
        "  '''\n",
        "  url=\"http://csegroups.case.edu/sites/default/files/bearingdatacenter/files/Datafiles/\"\n",
        "  n = len(matlab_files_name)\n",
        "  for i,key in enumerate(matlab_files_name):\n",
        "    file_name = matlab_files_name[key]\n",
        "    if not os.path.exists(file_name):\n",
        "      urllib.request.urlretrieve(url+file_name, file_name)\n",
        "    print(\"{}/{}\\t{}\\t{}\".format(i+1, n, key, file_name))\n"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FRijKbOjS-JZ",
        "colab_type": "text"
      },
      "source": [
        "##Extract data from Matlab files\n",
        "Extracts the acquisitions of each Matlab file in the dictionary matlab_files_name."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BbpFkSI12CUe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import scipy.io\n",
        "import numpy as np\n",
        "\n",
        "def get_tensors_from_matlab(matlab_files_name):\n",
        "  '''\n",
        "  Extracts the acquisitions of each Matlab file in the dictionary matlab_files_name.\n",
        "  '''\n",
        "  acquisitions = {}\n",
        "  for key in matlab_files_name:\n",
        "    file_name = matlab_files_name[key]\n",
        "    matlab_file = scipy.io.loadmat(file_name)\n",
        "    for position in ['DE','FE', 'BA']:\n",
        "      keys = [key for key in matlab_file if key.endswith(position+\"_time\")]\n",
        "      if len(keys)>0:\n",
        "        array_key = keys[0]\n",
        "        acquisitions[key+position.lower()] = matlab_file[array_key].reshape(1,-1)[0]\n",
        "  return acquisitions\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x-2lcukz5Nyk",
        "colab_type": "text"
      },
      "source": [
        "##Downloading pickle file\n",
        "Following, some auxiliary functions to download a pickle file in a google drive account.\n",
        "The pickle file already has the acquisitions propertly extracted.\n",
        "Therefore, these functions might speed up the whole process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZJkpaFxn1xtR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import requests\n",
        "import os.path\n",
        "\n",
        "def download_file_from_google_drive(id, destination):\n",
        "    URL = \"https://docs.google.com/uc?export=download\"\n",
        "    session = requests.Session()\n",
        "    response = session.get(URL, params = { 'id' : id }, stream = True)\n",
        "    token = get_confirm_token(response)\n",
        "    if token:\n",
        "        params = { 'id' : id, 'confirm' : token }\n",
        "        response = session.get(URL, params = params, stream = True)\n",
        "    save_response_content(response, destination)    \n",
        "\n",
        "def get_confirm_token(response):\n",
        "    for key, value in response.cookies.items():\n",
        "        if key.startswith('download_warning'):\n",
        "            return value\n",
        "    return None\n",
        "\n",
        "def save_response_content(response, destination):\n",
        "    CHUNK_SIZE = 32768\n",
        "    with open(destination, \"wb\") as f:\n",
        "        for chunk in response.iter_content(CHUNK_SIZE):\n",
        "            if chunk: # filter out keep-alive new chunks\n",
        "                f.write(chunk)\n",
        "\n",
        "# https://drive.google.com/file/d/1qJezMiROz9NAYafPUDPh9BFkxYF4nOi2/view?usp=sharing\n",
        "file_id = \"1qJezMiROz9NAYafPUDPh9BFkxYF4nOi2\"\n",
        "if not debug:\n",
        "  pickle_file = 'cwru.pickle'\n",
        "else:\n",
        "  pickle_file = 'debug.pickle'\n",
        "\n",
        "if not os.path.isfile(pickle_file) and not debug:\n",
        "  try:\n",
        "    download_file_from_google_drive(file_id, destination)\n",
        "  except:\n",
        "    print(\"Download failed!\")"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oEzRboCpgtlx",
        "colab_type": "text"
      },
      "source": [
        "##Save/Load data\n",
        "If the cwru pickle file is already download, it will not be downloaded again, and the dictionary with the acquisitions will be loaded.\n",
        "Otherwise, the desired files are downloaded and the acquisitions are extrated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z1m5Q3OUbvqa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pickle\n",
        "import os\n",
        "\n",
        "if not debug:\n",
        "  matlab_files_name = cwru_12khz()\n",
        "else:\n",
        "  matlab_files_name = files_debug()\n",
        "\n",
        "if os.path.isfile(pickle_file) and  not debug:\n",
        "  with open(pickle_file, 'rb') as handle:\n",
        "    acquisitions = pickle.load(handle)\n",
        "else:\n",
        "  download_cwrufiles(matlab_files_name)\n",
        "  acquisitions = get_tensors_from_matlab(matlab_files_name)\n",
        "  with open(pickle_file, 'wb') as handle:\n",
        "    pickle.dump(acquisitions, handle, protocol=pickle.HIGHEST_PROTOCOL)\n"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AT7hgDnzTcNP",
        "colab_type": "text"
      },
      "source": [
        "##Segment data\n",
        "Segments the acquisitions.\n",
        "  sample_size is the size of each segment.\n",
        "  max_samples is used for debug purpouses and \n",
        "  reduces the number of samples from each acquisition.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7BKfioJFzAKA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "d7326aeb-d45f-434a-8a0f-fdffc08aa75f"
      },
      "source": [
        "import numpy as np\n",
        "def cwru_segmentation(acquisitions, sample_size=512, max_samples=None):\n",
        "  '''\n",
        "  Segments the acquisitions.\n",
        "  sample_size is the size of each segment.\n",
        "  max_samples is used for debug purpouses and \n",
        "  reduces the number of samples from each acquisition.\n",
        "  '''\n",
        "  origin = []\n",
        "  data = np.empty((0,sample_size,1))\n",
        "  n = len(acquisitions)\n",
        "  for i,key in enumerate(acquisitions):\n",
        "    acquisition_size = len(acquisitions[key])\n",
        "    n_samples = acquisition_size//sample_size\n",
        "    if max_samples is not None and max_samples > 0 and n_samples > max_samples:\n",
        "      n_samples = max_samples\n",
        "    print('{}/{} --- {}: {}'.format(i+1, n, key, n_samples))\n",
        "    origin.extend([key for _ in range(n_samples)])\n",
        "    data = np.concatenate((data,\n",
        "           acquisitions[key][:(n_samples*sample_size)].reshape(\n",
        "               (n_samples,sample_size,1))))\n",
        "  return data,origin\n",
        "\n",
        "if not debug:\n",
        "  signal_data,signal_origin = cwru_segmentation(acquisitions, 512)\n",
        "else:\n",
        "  signal_data,signal_origin = cwru_segmentation(acquisitions, 512, 15)\n",
        "signal_data.shape"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1/307 --- Normal_0de: 476\n",
            "2/307 --- Normal_0fe: 476\n",
            "3/307 --- Normal_1de: 945\n",
            "4/307 --- Normal_1fe: 945\n",
            "5/307 --- Normal_2de: 945\n",
            "6/307 --- Normal_2fe: 945\n",
            "7/307 --- Normal_3de: 948\n",
            "8/307 --- Normal_3fe: 948\n",
            "9/307 --- DEIR.007_0de: 236\n",
            "10/307 --- DEIR.007_0fe: 236\n",
            "11/307 --- DEIR.007_0ba: 236\n",
            "12/307 --- DEIR.007_1de: 238\n",
            "13/307 --- DEIR.007_1fe: 238\n",
            "14/307 --- DEIR.007_1ba: 238\n",
            "15/307 --- DEIR.007_2de: 238\n",
            "16/307 --- DEIR.007_2fe: 238\n",
            "17/307 --- DEIR.007_2ba: 238\n",
            "18/307 --- DEIR.007_3de: 240\n",
            "19/307 --- DEIR.007_3fe: 240\n",
            "20/307 --- DEIR.007_3ba: 240\n",
            "21/307 --- DEB.007_0de: 239\n",
            "22/307 --- DEB.007_0fe: 239\n",
            "23/307 --- DEB.007_0ba: 239\n",
            "24/307 --- DEB.007_1de: 237\n",
            "25/307 --- DEB.007_1fe: 237\n",
            "26/307 --- DEB.007_1ba: 237\n",
            "27/307 --- DEB.007_2de: 237\n",
            "28/307 --- DEB.007_2fe: 237\n",
            "29/307 --- DEB.007_2ba: 237\n",
            "30/307 --- DEB.007_3de: 237\n",
            "31/307 --- DEB.007_3fe: 237\n",
            "32/307 --- DEB.007_3ba: 237\n",
            "33/307 --- DEOR.007@6_0de: 238\n",
            "34/307 --- DEOR.007@6_0fe: 238\n",
            "35/307 --- DEOR.007@6_0ba: 238\n",
            "36/307 --- DEOR.007@6_1de: 239\n",
            "37/307 --- DEOR.007@6_1fe: 239\n",
            "38/307 --- DEOR.007@6_1ba: 239\n",
            "39/307 --- DEOR.007@6_2de: 237\n",
            "40/307 --- DEOR.007@6_2fe: 237\n",
            "41/307 --- DEOR.007@6_2ba: 237\n",
            "42/307 --- DEOR.007@6_3de: 239\n",
            "43/307 --- DEOR.007@6_3fe: 239\n",
            "44/307 --- DEOR.007@6_3ba: 239\n",
            "45/307 --- DEOR.007@3_0de: 238\n",
            "46/307 --- DEOR.007@3_0fe: 238\n",
            "47/307 --- DEOR.007@3_0ba: 238\n",
            "48/307 --- DEOR.007@3_1de: 237\n",
            "49/307 --- DEOR.007@3_1fe: 237\n",
            "50/307 --- DEOR.007@3_1ba: 237\n",
            "51/307 --- DEOR.007@3_2de: 237\n",
            "52/307 --- DEOR.007@3_2fe: 237\n",
            "53/307 --- DEOR.007@3_2ba: 237\n",
            "54/307 --- DEOR.007@3_3de: 238\n",
            "55/307 --- DEOR.007@3_3fe: 238\n",
            "56/307 --- DEOR.007@3_3ba: 238\n",
            "57/307 --- DEOR.007@12_0de: 238\n",
            "58/307 --- DEOR.007@12_0fe: 238\n",
            "59/307 --- DEOR.007@12_0ba: 238\n",
            "60/307 --- DEOR.007@12_1de: 238\n",
            "61/307 --- DEOR.007@12_1fe: 238\n",
            "62/307 --- DEOR.007@12_1ba: 238\n",
            "63/307 --- DEOR.007@12_2de: 238\n",
            "64/307 --- DEOR.007@12_2fe: 238\n",
            "65/307 --- DEOR.007@12_2ba: 238\n",
            "66/307 --- DEOR.007@12_3de: 238\n",
            "67/307 --- DEOR.007@12_3fe: 238\n",
            "68/307 --- DEOR.007@12_3ba: 238\n",
            "69/307 --- DEIR.014_0de: 237\n",
            "70/307 --- DEIR.014_0fe: 237\n",
            "71/307 --- DEIR.014_0ba: 237\n",
            "72/307 --- DEIR.014_1de: 237\n",
            "73/307 --- DEIR.014_1fe: 237\n",
            "74/307 --- DEIR.014_1ba: 237\n",
            "75/307 --- DEIR.014_2de: 237\n",
            "76/307 --- DEIR.014_2fe: 237\n",
            "77/307 --- DEIR.014_2ba: 237\n",
            "78/307 --- DEIR.014_3de: 237\n",
            "79/307 --- DEIR.014_3fe: 237\n",
            "80/307 --- DEIR.014_3ba: 237\n",
            "81/307 --- DEB.014_0de: 237\n",
            "82/307 --- DEB.014_0fe: 237\n",
            "83/307 --- DEB.014_0ba: 237\n",
            "84/307 --- DEB.014_1de: 238\n",
            "85/307 --- DEB.014_1fe: 238\n",
            "86/307 --- DEB.014_1ba: 238\n",
            "87/307 --- DEB.014_2de: 238\n",
            "88/307 --- DEB.014_2fe: 238\n",
            "89/307 --- DEB.014_2ba: 238\n",
            "90/307 --- DEB.014_3de: 238\n",
            "91/307 --- DEB.014_3fe: 238\n",
            "92/307 --- DEB.014_3ba: 238\n",
            "93/307 --- DEOR.014@6_0de: 237\n",
            "94/307 --- DEOR.014@6_0fe: 237\n",
            "95/307 --- DEOR.014@6_0ba: 237\n",
            "96/307 --- DEOR.014@6_1de: 238\n",
            "97/307 --- DEOR.014@6_1fe: 238\n",
            "98/307 --- DEOR.014@6_1ba: 238\n",
            "99/307 --- DEOR.014@6_2de: 237\n",
            "100/307 --- DEOR.014@6_2fe: 237\n",
            "101/307 --- DEOR.014@6_2ba: 237\n",
            "102/307 --- DEOR.014@6_3de: 238\n",
            "103/307 --- DEOR.014@6_3fe: 238\n",
            "104/307 --- DEOR.014@6_3ba: 238\n",
            "105/307 --- DEB.021_0de: 238\n",
            "106/307 --- DEB.021_0fe: 238\n",
            "107/307 --- DEB.021_0ba: 238\n",
            "108/307 --- DEB.021_1de: 237\n",
            "109/307 --- DEB.021_1fe: 237\n",
            "110/307 --- DEB.021_1ba: 237\n",
            "111/307 --- DEB.021_2de: 238\n",
            "112/307 --- DEB.021_2fe: 238\n",
            "113/307 --- DEB.021_2ba: 238\n",
            "114/307 --- DEB.021_3de: 238\n",
            "115/307 --- DEB.021_3fe: 238\n",
            "116/307 --- DEB.021_3ba: 238\n",
            "117/307 --- FEIR.021_0de: 236\n",
            "118/307 --- FEIR.021_0fe: 236\n",
            "119/307 --- FEIR.021_0ba: 236\n",
            "120/307 --- FEIR.021_1de: 236\n",
            "121/307 --- FEIR.021_1fe: 236\n",
            "122/307 --- FEIR.021_1ba: 236\n",
            "123/307 --- FEIR.021_2de: 236\n",
            "124/307 --- FEIR.021_2fe: 236\n",
            "125/307 --- FEIR.021_2ba: 236\n",
            "126/307 --- FEIR.021_3de: 236\n",
            "127/307 --- FEIR.021_3fe: 236\n",
            "128/307 --- FEIR.021_3ba: 236\n",
            "129/307 --- FEIR.014_0de: 237\n",
            "130/307 --- FEIR.014_0fe: 237\n",
            "131/307 --- FEIR.014_0ba: 237\n",
            "132/307 --- FEIR.014_1de: 237\n",
            "133/307 --- FEIR.014_1fe: 237\n",
            "134/307 --- FEIR.014_1ba: 237\n",
            "135/307 --- FEIR.014_2de: 237\n",
            "136/307 --- FEIR.014_2fe: 237\n",
            "137/307 --- FEIR.014_2ba: 237\n",
            "138/307 --- FEIR.014_3de: 236\n",
            "139/307 --- FEIR.014_3fe: 236\n",
            "140/307 --- FEIR.014_3ba: 236\n",
            "141/307 --- FEB.007_0de: 236\n",
            "142/307 --- FEB.007_0fe: 236\n",
            "143/307 --- FEB.007_0ba: 236\n",
            "144/307 --- FEB.007_1de: 235\n",
            "145/307 --- FEB.007_1fe: 235\n",
            "146/307 --- FEB.007_1ba: 235\n",
            "147/307 --- FEB.007_2de: 237\n",
            "148/307 --- FEB.007_2fe: 237\n",
            "149/307 --- FEB.007_2ba: 237\n",
            "150/307 --- FEB.007_3de: 236\n",
            "151/307 --- FEB.007_3fe: 236\n",
            "152/307 --- FEB.007_3ba: 236\n",
            "153/307 --- DEIR.021_0de: 238\n",
            "154/307 --- DEIR.021_0fe: 238\n",
            "155/307 --- DEIR.021_0ba: 238\n",
            "156/307 --- DEIR.021_1de: 237\n",
            "157/307 --- DEIR.021_1fe: 237\n",
            "158/307 --- DEIR.021_1ba: 237\n",
            "159/307 --- DEIR.021_2de: 237\n",
            "160/307 --- DEIR.021_2fe: 237\n",
            "161/307 --- DEIR.021_2ba: 237\n",
            "162/307 --- DEIR.021_3de: 238\n",
            "163/307 --- DEIR.021_3fe: 238\n",
            "164/307 --- DEIR.021_3ba: 238\n",
            "165/307 --- DEOR.021@6_0de: 239\n",
            "166/307 --- DEOR.021@6_0fe: 239\n",
            "167/307 --- DEOR.021@6_0ba: 239\n",
            "168/307 --- DEOR.021@6_1de: 238\n",
            "169/307 --- DEOR.021@6_1fe: 238\n",
            "170/307 --- DEOR.021@6_1ba: 238\n",
            "171/307 --- DEOR.021@6_2de: 238\n",
            "172/307 --- DEOR.021@6_2fe: 238\n",
            "173/307 --- DEOR.021@6_2ba: 238\n",
            "174/307 --- DEOR.021@6_3de: 238\n",
            "175/307 --- DEOR.021@6_3fe: 238\n",
            "176/307 --- DEOR.021@6_3ba: 238\n",
            "177/307 --- DEOR.021@3_0de: 237\n",
            "178/307 --- DEOR.021@3_0fe: 237\n",
            "179/307 --- DEOR.021@3_0ba: 237\n",
            "180/307 --- DEOR.021@3_1de: 238\n",
            "181/307 --- DEOR.021@3_1fe: 238\n",
            "182/307 --- DEOR.021@3_1ba: 238\n",
            "183/307 --- DEOR.021@3_2de: 238\n",
            "184/307 --- DEOR.021@3_2fe: 238\n",
            "185/307 --- DEOR.021@3_2ba: 238\n",
            "186/307 --- DEOR.021@3_3de: 238\n",
            "187/307 --- DEOR.021@3_3fe: 238\n",
            "188/307 --- DEOR.021@3_3ba: 238\n",
            "189/307 --- DEOR.021@12_0de: 237\n",
            "190/307 --- DEOR.021@12_0fe: 237\n",
            "191/307 --- DEOR.021@12_0ba: 237\n",
            "192/307 --- DEOR.021@12_1de: 239\n",
            "193/307 --- DEOR.021@12_1fe: 239\n",
            "194/307 --- DEOR.021@12_1ba: 239\n",
            "195/307 --- DEOR.021@12_2de: 239\n",
            "196/307 --- DEOR.021@12_2fe: 239\n",
            "197/307 --- DEOR.021@12_2ba: 239\n",
            "198/307 --- DEOR.021@12_3de: 237\n",
            "199/307 --- DEOR.021@12_3fe: 237\n",
            "200/307 --- DEOR.021@12_3ba: 237\n",
            "201/307 --- FEIR.007_0de: 237\n",
            "202/307 --- FEIR.007_0fe: 237\n",
            "203/307 --- FEIR.007_0ba: 237\n",
            "204/307 --- FEIR.007_1de: 237\n",
            "205/307 --- FEIR.007_1fe: 237\n",
            "206/307 --- FEIR.007_1ba: 237\n",
            "207/307 --- FEIR.007_2de: 237\n",
            "208/307 --- FEIR.007_2fe: 237\n",
            "209/307 --- FEIR.007_2ba: 237\n",
            "210/307 --- FEIR.007_3de: 237\n",
            "211/307 --- FEIR.007_3fe: 237\n",
            "212/307 --- FEIR.007_3ba: 237\n",
            "213/307 --- FEB.014_0de: 238\n",
            "214/307 --- FEB.014_0fe: 238\n",
            "215/307 --- FEB.014_0ba: 238\n",
            "216/307 --- FEB.014_1de: 237\n",
            "217/307 --- FEB.014_1fe: 237\n",
            "218/307 --- FEB.014_1ba: 237\n",
            "219/307 --- FEB.014_2de: 238\n",
            "220/307 --- FEB.014_2fe: 238\n",
            "221/307 --- FEB.014_2ba: 238\n",
            "222/307 --- FEB.014_3de: 236\n",
            "223/307 --- FEB.014_3fe: 236\n",
            "224/307 --- FEB.014_3ba: 236\n",
            "225/307 --- FEB.021_0de: 237\n",
            "226/307 --- FEB.021_0fe: 237\n",
            "227/307 --- FEB.021_0ba: 237\n",
            "228/307 --- FEB.021_1de: 237\n",
            "229/307 --- FEB.021_1fe: 237\n",
            "230/307 --- FEB.021_1ba: 237\n",
            "231/307 --- FEB.021_2de: 237\n",
            "232/307 --- FEB.021_2fe: 237\n",
            "233/307 --- FEB.021_2ba: 237\n",
            "234/307 --- FEB.021_3de: 235\n",
            "235/307 --- FEB.021_3fe: 235\n",
            "236/307 --- FEB.021_3ba: 235\n",
            "237/307 --- FEOR.007@6_0de: 236\n",
            "238/307 --- FEOR.007@6_0fe: 236\n",
            "239/307 --- FEOR.007@6_0ba: 236\n",
            "240/307 --- FEOR.007@6_1de: 237\n",
            "241/307 --- FEOR.007@6_1fe: 237\n",
            "242/307 --- FEOR.007@6_1ba: 237\n",
            "243/307 --- FEOR.007@6_2de: 236\n",
            "244/307 --- FEOR.007@6_2fe: 236\n",
            "245/307 --- FEOR.007@6_2ba: 236\n",
            "246/307 --- FEOR.007@6_3de: 238\n",
            "247/307 --- FEOR.007@6_3fe: 238\n",
            "248/307 --- FEOR.007@6_3ba: 238\n",
            "249/307 --- FEOR.007@3_0de: 236\n",
            "250/307 --- FEOR.007@3_0fe: 236\n",
            "251/307 --- FEOR.007@3_0ba: 236\n",
            "252/307 --- FEOR.007@3_1de: 236\n",
            "253/307 --- FEOR.007@3_1fe: 236\n",
            "254/307 --- FEOR.007@3_1ba: 236\n",
            "255/307 --- FEOR.007@3_2de: 238\n",
            "256/307 --- FEOR.007@3_2fe: 238\n",
            "257/307 --- FEOR.007@3_2ba: 238\n",
            "258/307 --- FEOR.007@3_3de: 237\n",
            "259/307 --- FEOR.007@3_3fe: 237\n",
            "260/307 --- FEOR.007@3_3ba: 237\n",
            "261/307 --- FEOR.007@12_0de: 236\n",
            "262/307 --- FEOR.007@12_0fe: 236\n",
            "263/307 --- FEOR.007@12_0ba: 236\n",
            "264/307 --- FEOR.007@12_1de: 236\n",
            "265/307 --- FEOR.007@12_1fe: 236\n",
            "266/307 --- FEOR.007@12_1ba: 236\n",
            "267/307 --- FEOR.007@12_2de: 237\n",
            "268/307 --- FEOR.007@12_2fe: 237\n",
            "269/307 --- FEOR.007@12_2ba: 237\n",
            "270/307 --- FEOR.007@12_3de: 236\n",
            "271/307 --- FEOR.007@12_3fe: 236\n",
            "272/307 --- FEOR.007@12_3ba: 236\n",
            "273/307 --- FEOR.014@3_0de: 237\n",
            "274/307 --- FEOR.014@3_0fe: 237\n",
            "275/307 --- FEOR.014@3_0ba: 237\n",
            "276/307 --- FEOR.014@3_1de: 236\n",
            "277/307 --- FEOR.014@3_1fe: 236\n",
            "278/307 --- FEOR.014@3_1ba: 236\n",
            "279/307 --- FEOR.014@3_2de: 236\n",
            "280/307 --- FEOR.014@3_2fe: 236\n",
            "281/307 --- FEOR.014@3_2ba: 236\n",
            "282/307 --- FEOR.014@3_3de: 236\n",
            "283/307 --- FEOR.014@3_3fe: 236\n",
            "284/307 --- FEOR.014@3_3ba: 236\n",
            "285/307 --- FEOR.014@6_0de: 236\n",
            "286/307 --- FEOR.014@6_0fe: 236\n",
            "287/307 --- FEOR.014@6_0ba: 236\n",
            "288/307 --- FEOR.021@6_0de: 235\n",
            "289/307 --- FEOR.021@6_0fe: 235\n",
            "290/307 --- FEOR.021@6_0ba: 235\n",
            "291/307 --- FEOR.021@3_1de: 235\n",
            "292/307 --- FEOR.021@3_1fe: 235\n",
            "293/307 --- FEOR.021@3_1ba: 235\n",
            "294/307 --- FEOR.021@3_2de: 235\n",
            "295/307 --- FEOR.021@3_2fe: 235\n",
            "296/307 --- FEOR.021@3_2ba: 235\n",
            "297/307 --- FEOR.021@3_3de: 237\n",
            "298/307 --- FEOR.021@3_3fe: 237\n",
            "299/307 --- FEOR.021@3_3ba: 237\n",
            "300/307 --- DEIR.028_0de: 235\n",
            "301/307 --- DEIR.028_1de: 237\n",
            "302/307 --- DEIR.028_2de: 237\n",
            "303/307 --- DEIR.028_3de: 237\n",
            "304/307 --- DEB.028_0de: 235\n",
            "305/307 --- DEB.028_1de: 237\n",
            "306/307 --- DEB.028_2de: 236\n",
            "307/307 --- DEB.028_3de: 236\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(77527, 512, 1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jCN8XZ5dOF3",
        "colab_type": "text"
      },
      "source": [
        "## Clean dataset functions\n",
        "The functions below help to select samples from acquisitions and form groups according to these acquisitions, using regular expressions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eOOP9H2c3AaZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "import numpy as np\n",
        "\n",
        "def select_samples(regex, X, y):\n",
        "  '''\n",
        "  Selects samples wich has some regex pattern in its name.\n",
        "  '''\n",
        "  mask = [re.search(regex,label) is not None for label in y]\n",
        "  return X[mask],y[mask]\n",
        "\n",
        "def join_labels(regex, y):\n",
        "  '''\n",
        "  Excludes some regex patterns from the labels, \n",
        "  making some samples to have the same label.\n",
        "  '''\n",
        "  return np.array([re.sub(regex, '', label) for label in y])\n",
        "\n",
        "def get_groups(regex, y):\n",
        "  '''\n",
        "  Generates a list of groups of samples with \n",
        "  the same regex patten in its label.\n",
        "  '''\n",
        "  groups = list(range(len(y)))\n",
        "  for i,label in enumerate(y):\n",
        "    match = re.search(regex,label)\n",
        "    groups[i] = match.group(0) if match else None\n",
        "  return groups"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yFA-8l02RplD",
        "colab_type": "text"
      },
      "source": [
        "##Selecting samples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r89dYOJm8gzW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "cf803ad6-d628-407b-f001-9f1ea90213f0"
      },
      "source": [
        "#DE from 'de', FE from 'fe', Normal from 'de' and 'fe'\n",
        "samples = '^(DE).*(de)|^(FE).*(fe)|(Normal).*'\n",
        "X,y = select_samples(samples, signal_data, np.array(signal_origin))\n",
        "print(len(set(y)),set(y))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "113 {'DEB.028_2de', 'DEB.014_1de', 'FEIR.021_2fe', 'FEIR.007_2fe', 'FEOR.007@6_3fe', 'DEIR.014_3de', 'DEB.021_2de', 'FEIR.014_2fe', 'Normal_1de', 'FEOR.007@3_1fe', 'FEB.007_3fe', 'DEOR.014@6_3de', 'Normal_2fe', 'DEOR.021@12_1de', 'DEOR.007@12_3de', 'DEB.007_2de', 'DEB.014_2de', 'DEOR.007@6_2de', 'Normal_1fe', 'DEB.007_3de', 'DEOR.007@3_3de', 'DEIR.021_3de', 'FEOR.007@6_1fe', 'DEOR.007@6_1de', 'Normal_3de', 'DEOR.007@12_0de', 'Normal_0fe', 'FEOR.007@6_2fe', 'FEB.007_1fe', 'DEIR.014_1de', 'DEB.014_3de', 'DEB.021_3de', 'FEB.021_0fe', 'DEIR.028_1de', 'FEOR.007@3_2fe', 'DEIR.007_1de', 'DEOR.021@3_2de', 'FEIR.007_1fe', 'FEOR.014@3_1fe', 'DEOR.007@12_1de', 'DEIR.007_0de', 'FEIR.007_3fe', 'DEOR.021@6_3de', 'FEIR.014_3fe', 'DEOR.021@12_3de', 'DEIR.021_2de', 'FEIR.021_3fe', 'DEOR.021@6_1de', 'DEIR.028_2de', 'DEB.007_1de', 'DEB.028_1de', 'FEOR.007@12_0fe', 'FEOR.007@12_1fe', 'FEB.014_2fe', 'FEOR.007@6_0fe', 'FEOR.007@12_3fe', 'DEOR.021@12_0de', 'FEOR.014@3_2fe', 'FEOR.014@3_0fe', 'DEB.021_1de', 'DEOR.021@3_0de', 'DEB.028_0de', 'DEIR.021_1de', 'DEIR.028_3de', 'Normal_2de', 'DEOR.021@3_3de', 'FEOR.007@3_0fe', 'DEOR.014@6_2de', 'DEOR.021@6_0de', 'FEOR.014@3_3fe', 'DEOR.007@3_1de', 'DEB.007_0de', 'DEIR.007_2de', 'FEOR.021@3_2fe', 'DEOR.014@6_0de', 'DEIR.014_0de', 'Normal_0de', 'DEB.028_3de', 'FEOR.007@12_2fe', 'FEB.007_2fe', 'Normal_3fe', 'DEIR.028_0de', 'DEOR.021@6_2de', 'FEB.021_2fe', 'FEOR.014@6_0fe', 'DEIR.007_3de', 'FEB.007_0fe', 'DEOR.021@12_2de', 'FEB.014_1fe', 'DEOR.007@6_3de', 'FEOR.021@3_3fe', 'DEOR.007@3_2de', 'DEOR.014@6_1de', 'DEOR.021@3_1de', 'FEIR.014_0fe', 'DEB.021_0de', 'FEB.021_3fe', 'DEOR.007@3_0de', 'FEOR.021@3_1fe', 'FEB.014_3fe', 'FEIR.007_0fe', 'FEIR.021_0fe', 'DEIR.014_2de', 'FEOR.007@3_3fe', 'DEOR.007@6_0de', 'FEOR.021@6_0fe', 'DEB.014_0de', 'FEIR.014_1fe', 'FEB.014_0fe', 'FEB.021_1fe', 'DEOR.007@12_2de', 'DEIR.021_0de', 'FEIR.021_1fe'}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5pRQQK0Mhm1_",
        "colab_type": "text"
      },
      "source": [
        "#Experimenter"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GE4TTG1-hmH7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "4998c6d3-b105-49e9-ac97-17178f0b4adb"
      },
      "source": [
        "from sklearn.model_selection import cross_validate, KFold, PredefinedSplit\n",
        "\n",
        "def experimenter(model, X, y, groups=None, scoring=None, cv=KFold(4, True), verbose=0):\n",
        "  '''\n",
        "  Performs a experiment with some estimator (model) and validation.\n",
        "  It works like a cross_validate function from sklearn, however, \n",
        "  when a estimator has an internal validation with groups, \n",
        "  it maintains the groups from the external validation.\n",
        "  '''\n",
        "  if hasattr(model,'cv') or (hasattr(model,'steps') and any(['gs' in step[0] for step in model.steps])):\n",
        "    scores = {}\n",
        "    lstval = list(validation.split(X,y,groups))\n",
        "    for tr,te in lstval:\n",
        "      if groups is not None:\n",
        "        innercv = list(GroupShuffleKFold(validation.n_splits-1).split(X[tr],y[tr],np.array(groups)[tr]))\n",
        "      else:\n",
        "        innercv = list(KFold(validation.n_splits-1, True).split(X[tr],y[tr]))\n",
        "      if hasattr(model,'cv'):\n",
        "        model.cv = innercv\n",
        "      else:\n",
        "        for step in model.steps:\n",
        "          if 'gs' in step[0]:\n",
        "            step[1].cv = innercv\n",
        "      test_fold = np.zeros((len(y),), dtype=int)\n",
        "      test_fold[tr] = -1\n",
        "      score = cross_validate(model, X, y, groups, scoring, \n",
        "                             PredefinedSplit(test_fold), verbose=verbose)\n",
        "      for k in score.keys():\n",
        "        if k not in scores:\n",
        "          scores[k] = []\n",
        "        scores[k].extend(score[k])\n",
        "    return scores\n",
        "  return cross_validate(model, X, y, groups, scoring, cv, verbose=verbose)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/home/francisco/Jupyter/.venv/lib/python3.8/site-packages/sklearn/utils/validation.py:68: FutureWarning: Pass shuffle=True as keyword args. From version 0.25 passing these as positional arguments will result in an error\n",
            "  warnings.warn(\"Pass {} as keyword args. From version 0.25 \"\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dWYfuxcxFjt8",
        "colab_type": "text"
      },
      "source": [
        "## Custom Splitter"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xRdfG-uzhPm4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import KFold\n",
        "from sklearn.utils import shuffle\n",
        "from sklearn.utils.validation import check_array\n",
        "import numpy as np\n",
        "\n",
        "class GroupShuffleKFold(KFold):\n",
        "  '''\n",
        "  Neither GroupShuffleSplit nor GroupKFold are good splitters for this case.\n",
        "  A custom splitter must be made.\n",
        "  '''\n",
        "  def __init__(self, n_splits=4, shuffle=False, random_state=None):\n",
        "    super().__init__(n_splits, shuffle=shuffle, random_state=random_state)\n",
        "  def get_n_splits(self, X, y, groups=None):\n",
        "    return self.n_splits\n",
        "  def _iter_test_indices(self, X=None, y=None, groups=None):\n",
        "    if groups is None:\n",
        "      raise ValueError(\"The 'groups' parameter should not be None.\")\n",
        "    groups = check_array(groups, ensure_2d=False, dtype=None)\n",
        "    unique_groups, groups = np.unique(groups, return_inverse=True)\n",
        "    n_groups = len(unique_groups)\n",
        "    if self.n_splits > n_groups:\n",
        "      raise ValueError(\"Cannot have number of splits n_splits=%d greater\"\n",
        "                        \" than the number of groups: %d.\"\n",
        "                        % (self.n_splits, n_groups))\n",
        "    # Distribute groups\n",
        "    indices = np.arange(n_groups)\n",
        "    if self.shuffle:\n",
        "      for i in range(n_groups//self.n_splits):\n",
        "        if self.random_state is None:\n",
        "          indices[self.n_splits*i:self.n_splits*(i+1)] = shuffle(\n",
        "              indices[self.n_splits*i:self.n_splits*(i+1)])\n",
        "        else:\n",
        "          indices[self.n_splits*i:self.n_splits*(i+1)] = shuffle(\n",
        "              indices[self.n_splits*i:self.n_splits*(i+1)],\n",
        "              random_state=self.random_state+i)\n",
        "    #print(unique_groups[indices]) #Debug purpose\n",
        "    # Total weight of each fold\n",
        "    n_samples_per_fold = np.zeros(self.n_splits)\n",
        "    # Mapping from group index to fold index\n",
        "    group_to_fold = np.zeros(len(unique_groups))\n",
        "    # Distribute samples \n",
        "    for group_index in indices:\n",
        "      group_to_fold[indices[group_index]] = group_index%(self.n_splits)\n",
        "    indices = group_to_fold[groups]\n",
        "    for f in range(self.n_splits):\n",
        "      yield np.where(indices == f)[0]"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Frmm3FVivQvg",
        "colab_type": "text"
      },
      "source": [
        "## BySeverity Splitter"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZFDWy6zqvGTF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import KFold\n",
        "from sklearn.utils import shuffle\n",
        "from sklearn.utils.validation import check_array\n",
        "import numpy as np\n",
        "\n",
        "class BySeverityKFold(KFold):\n",
        "  '''\n",
        "  Splits the CWRU dataset in severities.\n",
        "  '''\n",
        "  # Compatibility constructor\n",
        "  def __init__(self, n_splits=4, shuffle=False, random_state=None):\n",
        "    super().__init__(n_splits=4, shuffle=False, random_state=None)\n",
        "    self.nround = random_state\n",
        "  def _iter_test_indices(self, X=None, y=None, groups=None):\n",
        "    if groups is None:\n",
        "      raise ValueError(\"The 'groups' parameter should not be None.\")\n",
        "    groups = check_array(groups, ensure_2d=False, dtype=None)\n",
        "    unique_groups, groups = np.unique(groups, return_inverse=True)\n",
        "    n_groups = len(unique_groups)\n",
        "    if self.n_splits > n_groups:\n",
        "      raise ValueError(\"Cannot have number of splits n_splits=%d greater\"\n",
        "                        \" than the number of groups: %d.\"\n",
        "                        % (self.n_splits, n_groups))\n",
        "    # Distribute groups\n",
        "    indices = np.arange(n_groups)\n",
        "    nround = self.nround - random_state\n",
        "    for i in range(nround//4):\n",
        "      indices[i],indices[i+1] = indices[i+1],indices[i]\n",
        "    for i in range(self.n_splits):      \n",
        "      indices[i+self.n_splits] = (i+nround)%self.n_splits+self.n_splits\n",
        "    #print(unique_groups[indices]) #Debug purpose\n",
        "    # Total weight of each fold\n",
        "    n_samples_per_fold = np.zeros(self.n_splits)\n",
        "    # Mapping from group index to fold index\n",
        "    group_to_fold = np.zeros(len(unique_groups))\n",
        "    # Distribute samples \n",
        "    for group_index in indices:\n",
        "      group_to_fold[indices[group_index]] = group_index%(self.n_splits)\n",
        "    print(group_to_fold)\n",
        "    indices = group_to_fold[groups]\n",
        "    for f in range(self.n_splits):\n",
        "      yield np.where(indices == f)[0]"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e48W6KkIhesw",
        "colab_type": "text"
      },
      "source": [
        "##Experiment setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K9dImVH3hh_Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from collections import namedtuple\n",
        "\n",
        "ExperimentSetup = namedtuple('ExperimentSetup', \n",
        "                             'groups, splitter_name, shuffle, rounds')\n",
        "\n",
        "validations = {\n",
        "    # Samples with the same load cannot be in the trainning fold and\n",
        "    # the test folds simultaneously. \n",
        "    \"By Load\": ExperimentSetup(groups = get_groups('_\\d',y), \n",
        "                               splitter_name = 'GroupShuffleKFold',\n",
        "                               shuffle = False,\n",
        "                               rounds=1,\n",
        "                               ),\n",
        "    # Samples with the same severity cannot be in the trainning folds and\n",
        "    # the test folds simultaneously.\n",
        "    \"By Severity\": ExperimentSetup(groups = get_groups('(\\.\\d{3})|(Normal_\\d)',y),\n",
        "                                   splitter_name = 'BySeverityKFold',\n",
        "                                   shuffle = False,\n",
        "                                   rounds=8),\n",
        "    # Validation usually seen in publications with CWRU bearing dataset.\n",
        "    \"Usual K-Fold\": ExperimentSetup(groups = None, \n",
        "                                    splitter_name = 'KFold',\n",
        "                                    shuffle = True,\n",
        "                                    rounds=8), \n",
        "}"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E9yn44VAoRFo",
        "colab_type": "text"
      },
      "source": [
        "##Common Variables"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ogpdDde4oTsS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Only four conditions are considered: Normal, Ball, Inner Race and Outer Race.\n",
        "selected_y = join_labels('_\\d|@\\d{1,3}|(de)|(fe)|\\.\\d{3}|(DE)|(FE)',y)\n",
        "verbose = 0 #if not debug else 3\n",
        "random_state = 42\n",
        "scoring = ['accuracy', 'f1_macro']#, 'precision_macro', 'recall_macro']"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mq30RtWYToeu",
        "colab_type": "text"
      },
      "source": [
        "#Classification Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IstS2gTeY7pg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NLeArq-uThHW",
        "colab_type": "text"
      },
      "source": [
        "##Feature Extraction Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WuSNj6YIEhu0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.base import TransformerMixin"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mm95T4CsDxaN",
        "colab_type": "text"
      },
      "source": [
        "###Statistical functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vWCPUON8D1A8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import scipy.stats as stats\n",
        "\n",
        "def rms(x):\n",
        "  '''\n",
        "  root mean square\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.sqrt(np.mean(np.square(x)))\n",
        "\n",
        "def sra(x):\n",
        "  '''\n",
        "  square root amplitude\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.mean(np.sqrt(np.absolute(x)))**2\n",
        "\n",
        "def ppv(x):\n",
        "  '''\n",
        "  peak to peak value\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.max(x)-np.min(x)\n",
        "\n",
        "def cf(x):\n",
        "  '''\n",
        "  crest factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.max(np.absolute(x))/rms(x)\n",
        "\n",
        "def ifa(x):\n",
        "  '''\n",
        "  impact factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.max(np.absolute(x))/np.mean(np.absolute(x))\n",
        "\n",
        "def mf(x):\n",
        "  '''\n",
        "  margin factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.max(np.absolute(x))/sra(x)\n",
        "\n",
        "def sf(x):\n",
        "  '''\n",
        "  shape factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return rms(x)/np.mean(np.absolute(x))\n",
        "\n",
        "def kf(x):\n",
        "  '''\n",
        "  kurtosis factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return stats.kurtosis(x)/(np.mean(x**2)**2)\n",
        "\n"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "njMb9HtUEBrI",
        "colab_type": "text"
      },
      "source": [
        "### Statistical Features from Time Domain"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oSN2_c28D_Zr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class StatisticalTime(TransformerMixin):\n",
        "  '''\n",
        "  Extracts statistical features from the time domain.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    return np.array([\n",
        "                     [\n",
        "                      rms(x), # root mean square\n",
        "                      sra(x), # square root amplitude\n",
        "                      stats.kurtosis(x), # kurtosis\n",
        "                      stats.skew(x), # skewness\n",
        "                      ppv(x), # peak to peak value\n",
        "                      cf(x), # crest factor\n",
        "                      ifa(x), # impact factor\n",
        "                      mf(x), # margin factor\n",
        "                      sf(x), # shape factor\n",
        "                      kf(x), # kurtosis factor\n",
        "                      ] for x in X[:,:,0]\n",
        "                     ])\n"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dXDWD3JZEnep",
        "colab_type": "text"
      },
      "source": [
        "### Statistical Features from Frequency Domain"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sj3XTpVTEvAp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class StatisticalFrequency(TransformerMixin):\n",
        "  '''\n",
        "  Extracts statistical features from the frequency domain.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    sig = []\n",
        "    for x in X[:,:,0]:\n",
        "      fx = np.absolute(np.fft.fft(x)) # transform x from time to frequency domain\n",
        "      fc = np.mean(fx) # frequency center\n",
        "      sig.append([\n",
        "                  fc, # frequency center\n",
        "                  rms(fx), # RMS from the frequency domain\n",
        "                  rms(fx-fc), # Root Variance Frequency\n",
        "                  ])\n",
        "    return np.array(sig)\n"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0YBmzTb6ARb",
        "colab_type": "text"
      },
      "source": [
        "###Statistical Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kep4ubkR6DR0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Statistical(TransformerMixin):\n",
        "  '''\n",
        "  Extracts statistical features from both time and frequency domain.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    st = StatisticalTime()\n",
        "    stfeats = st.transform(X)\n",
        "    sf = StatisticalFrequency()\n",
        "    sffeats = sf.transform(X)\n",
        "    return np.concatenate((stfeats,sffeats),axis=1)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZuiVsHNzFORr",
        "colab_type": "text"
      },
      "source": [
        "###Wavelet Package Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oPd92xtJhaH3",
        "colab_type": "code",
        "cellView": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pywt\n",
        "\n",
        "class WaveletPackage(TransformerMixin):\n",
        "  '''\n",
        "  Extracts Wavelet Package features.\n",
        "  The features are calculated by the energy of the recomposed signal\n",
        "  of the leaf nodes coeficients.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    def Energy(coeffs, k):\n",
        "      return np.sqrt(np.sum(np.array(coeffs[-k]) ** 2)) / len(coeffs[-k])\n",
        "    def getEnergy(wp):\n",
        "      coefs = np.asarray([n.data for n in wp.get_leaf_nodes(True)])\n",
        "      return np.asarray([Energy(coefs,i) for i in range(2**wp.maxlevel)])\n",
        "    return np.array([getEnergy(pywt.WaveletPacket(data=x, wavelet='db4', \n",
        "                                                  mode='symmetric', maxlevel=4)\n",
        "                                                  ) for x in X[:,:,0]])"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_sonHkjFYbB",
        "colab_type": "text"
      },
      "source": [
        "###Heterogeneus Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IZsZhuVfFZsQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Heterogeneous(TransformerMixin):\n",
        "  '''\n",
        "  Mixes Statistical and Wavelet Package features.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    st = StatisticalTime()\n",
        "    stfeats = st.transform(X)\n",
        "    sf = StatisticalFrequency()\n",
        "    sffeats = sf.transform(X)\n",
        "    wp = WaveletPackage()\n",
        "    wpfeats = wp.transform(X)\n",
        "    return np.concatenate((stfeats,sffeats,wpfeats),axis=1)\n"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1bnyL67EUcxV",
        "colab_type": "text"
      },
      "source": [
        "##K-NN with Heterogeneous Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F--sjKZRUh5G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "a529b03f-6b14-45da-e1ad-e43ba45715cb"
      },
      "source": [
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "knn = Pipeline([\n",
        "                ('FeatureExtraction', Heterogeneous()),\n",
        "                ('scaler', StandardScaler()),\n",
        "                ('knn', KNeighborsClassifier()),\n",
        "                ])\n",
        "\n",
        "parameters = {'knn__n_neighbors': list(range(1,16,2))}\n",
        "if not debug:\n",
        "  knn = GridSearchCV(knn, parameters, verbose=verbose)\n",
        "else:\n",
        "  knn = GridSearchCV(knn, {'knn__n_neighbors': list(range(1,4,2))}, verbose=verbose)\n",
        "knn"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=Pipeline(steps=[('FeatureExtraction',\n",
              "                                        <__main__.Heterogeneous object at 0x7fce0bb38a00>),\n",
              "                                       ('scaler', StandardScaler()),\n",
              "                                       ('knn', KNeighborsClassifier())]),\n",
              "             param_grid={'knn__n_neighbors': [1, 3, 5, 7, 9, 11, 13, 15]})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d7DfMTS_ujeE",
        "colab_type": "text"
      },
      "source": [
        "##SVM with Heterogeneous Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6v_wXxiDupvF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "3425f85e-66bc-4f7b-b24c-9b762d0ec6ea"
      },
      "source": [
        "from sklearn.svm import SVC\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "svm = Pipeline([\n",
        "                ('FeatureExtraction', Heterogeneous()),\n",
        "                ('scaler', StandardScaler()),\n",
        "                ('svc', SVC()),\n",
        "                ])\n",
        "\n",
        "parameters = {\n",
        "    'svc__C': [10**x for x in range(-3,2)],\n",
        "    'svc__gamma': [10**x for x in range(-3,1)],\n",
        "    }\n",
        "if not debug:\n",
        "  svm = GridSearchCV(svm, parameters, verbose=verbose)\n",
        "svm"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=Pipeline(steps=[('FeatureExtraction',\n",
              "                                        <__main__.Heterogeneous object at 0x7fce0c12c850>),\n",
              "                                       ('scaler', StandardScaler()),\n",
              "                                       ('svc', SVC())]),\n",
              "             param_grid={'svc__C': [0.001, 0.01, 0.1, 1, 10],\n",
              "                         'svc__gamma': [0.001, 0.01, 0.1, 1]})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yU36xsi4JGZv",
        "colab_type": "text"
      },
      "source": [
        "##Random Forest with Heterogeneous Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GXABo6HpJJY_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "adbcbdd1-a17c-4382-d30f-42d8ce4b71c1"
      },
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "rf = Pipeline([\n",
        "               ('FeatureExtraction', Heterogeneous()),\n",
        "               ('scaler', StandardScaler()),\n",
        "               ('rf', RandomForestClassifier()),\n",
        "               ])\n",
        "\n",
        "parameters = {\n",
        "    \"rf__n_estimators\": [10, 20, 50, 100, 200, 500],\n",
        "    \"rf__max_features\": [1, 5, 10, 15, 20], #list(range(1,21)),\n",
        "    }\n",
        "if not debug:\n",
        "  rf = GridSearchCV(rf, parameters, verbose=verbose)\n",
        "rf"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=Pipeline(steps=[('FeatureExtraction',\n",
              "                                        <__main__.Heterogeneous object at 0x7fce0c12c310>),\n",
              "                                       ('scaler', StandardScaler()),\n",
              "                                       ('rf', RandomForestClassifier())]),\n",
              "             param_grid={'rf__max_features': [1, 5, 10, 15, 20],\n",
              "                         'rf__n_estimators': [10, 20, 50, 100, 200, 500]})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "REy6ykvWSbJc",
        "colab_type": "text"
      },
      "source": [
        "##Convolutional Neural Network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8YpHSjvNcEx5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "864dcdcc-6858-488e-ac3a-e42fb81492e8"
      },
      "source": [
        "try:\n",
        "  %tensorflow_version 2.x\n",
        "except:\n",
        "  print(\"Out of Colab\")"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Out of Colab\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FF1hCxJ9b5h0",
        "colab_type": "text"
      },
      "source": [
        "###F1-score macro averaged implemented for Keras"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LCJErrQIcIZ0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras import backend as K\n",
        "\n",
        "def f1_score_macro(y_true,y_pred):\n",
        "    def recall(y_true, y_pred):\n",
        "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
        "        recall = true_positives / (possible_positives + K.epsilon())\n",
        "        return recall\n",
        "    def precision(y_true, y_pred):\n",
        "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
        "        precision = true_positives / (predicted_positives + K.epsilon())\n",
        "        return precision\n",
        "    precision = precision(y_true, y_pred)\n",
        "    recall = recall(y_true, y_pred)\n",
        "    return 2*((precision*recall)/(precision+recall+K.epsilon()))"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PeLXSU7-cLfk",
        "colab_type": "text"
      },
      "source": [
        "###ANN wrapped in a scikit-learn estimator."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ny7otiW6Siz_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "from sklearn.base import BaseEstimator, ClassifierMixin\n",
        "import numpy as np\n",
        "from tensorflow.keras.callbacks import EarlyStopping,ReduceLROnPlateau\n",
        "\n",
        "class ANN(BaseEstimator, ClassifierMixin):\n",
        "  def __init__(self, \n",
        "               dense_layer_sizes=[64], \n",
        "               kernel_size=32, \n",
        "               filters=32, \n",
        "               n_conv_layers=2,\n",
        "               pool_size=8,\n",
        "               dropout=0.25,\n",
        "               epochs=50,\n",
        "               validation_split=0.05,\n",
        "               optimizer='sgd'#'nadam'#'rmsprop'#\n",
        "               ):\n",
        "    self.dense_layer_sizes = dense_layer_sizes\n",
        "    self.kernel_size = kernel_size\n",
        "    self.filters = filters\n",
        "    self.n_conv_layers = n_conv_layers\n",
        "    self.pool_size = pool_size\n",
        "    self.dropout = dropout\n",
        "    self.epochs = epochs\n",
        "    self.validation_split = validation_split\n",
        "    self.optimizer = optimizer\n",
        "  \n",
        "  def fit(self, X, y=None):\n",
        "    dense_layer_sizes = self.dense_layer_sizes\n",
        "    kernel_size = self.kernel_size\n",
        "    filters = self.filters\n",
        "    n_conv_layers = self.n_conv_layers\n",
        "    pool_size = self.pool_size\n",
        "    dropout = self.dropout\n",
        "    epochs = self.epochs\n",
        "    optimizer = self.optimizer\n",
        "    validation_split = self.validation_split\n",
        "\n",
        "    self.labels, ids = np.unique(y, return_inverse=True)\n",
        "    y_cat = to_categorical(ids)\n",
        "    num_classes = y_cat.shape[1]\n",
        "    \n",
        "    self.model = Sequential()\n",
        "    self.model.add(layers.InputLayer(input_shape=(X.shape[1],X.shape[-1])))\n",
        "    for _ in range(n_conv_layers):\n",
        "      self.model.add(layers.Conv1D(filters, kernel_size))#, padding='valid'))\n",
        "      self.model.add(layers.Activation('relu'))\n",
        "      if pool_size>1:\n",
        "        self.model.add(layers.MaxPooling1D(pool_size=pool_size))\n",
        "    #self.model.add(layers.Dropout(0.25))\n",
        "    self.model.add(layers.Flatten())\n",
        "    for layer_size in dense_layer_sizes:\n",
        "        self.model.add(layers.Dense(layer_size))\n",
        "        self.model.add(layers.Activation('relu'))\n",
        "    if dropout>0 and dropout<1:\n",
        "      self.model.add(layers.Dropout(dropout))\n",
        "    self.model.add(layers.Dense(num_classes))\n",
        "    self.model.add(layers.Activation('softmax'))\n",
        "    self.model.compile(loss='categorical_crossentropy',\n",
        "                       optimizer=optimizer,\n",
        "                       metrics=[f1_score_macro])\n",
        "    if validation_split>0 and validation_split<1:\n",
        "      prop = int(1/validation_split)\n",
        "      mask = np.array([i%prop==0 for i in range(len(y))])\n",
        "      self.history = self.model.fit(X[~mask], y_cat[~mask], epochs=epochs, \n",
        "                                    validation_data=(X[mask],y_cat[mask]),\n",
        "                                    callbacks=[EarlyStopping(patience=3), ReduceLROnPlateau()],\n",
        "                                    verbose=False\n",
        "                                    )  \n",
        "    else:\n",
        "      self.history = self.model.fit(X, y_cat, epochs=epochs, verbose=False)  \n",
        "  \n",
        "  def predict_proba(self, X, y=None):\n",
        "    return self.model.predict(X)\n",
        "\n",
        "  def predict(self, X, y=None):\n",
        "    predictions = self.model.predict(X)\n",
        "    return self.labels[np.argmax(predictions,axis=1)]"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UL5Zv2-xcgEG",
        "colab_type": "text"
      },
      "source": [
        "###ANN instantiation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MmQe7jeRcb6F",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "34ed789f-7b58-48ac-9eb5-6b6ebbf9a09a"
      },
      "source": [
        "parameters = {\n",
        "    'filters': [16, 32],\n",
        "    'kernel_size': [16, 32],\n",
        "    'n_conv_layers': [1, 2],\n",
        "    #'pool_size': [2, 4, 6, 8],\n",
        "    }\n",
        "ann = ANN()\n",
        "if not debug:\n",
        "  ann = GridSearchCV(ann, parameters, verbose=verbose)\n",
        "ann"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=ANN(),\n",
              "             param_grid={'filters': [16, 32], 'kernel_size': [16, 32],\n",
              "                         'n_conv_layers': [1, 2]})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4SQenKxoSkME",
        "colab_type": "text"
      },
      "source": [
        "##List of Estimators"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vb6AGKFJJqvL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "clfs = [\n",
        "        # ('KNN - KNeighborsClassifier, Heterogeneous Features', knn),\n",
        "        # ('SVM - SVC with Heterogeneous Features', svm),\n",
        "        # ('ANN - Artificial Neural Network with Convolutional Layers', ann),\n",
        "        ('RF - RandomForestClassifier with Heterogeneous Features', rf),\n",
        "        ]\n",
        "if not debug:\n",
        "  dirres = 'cwru_rf'\n",
        "  # dirres = 'cwru_res'\n",
        "else:\n",
        "  dirres = 'debugres'"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dEl_vSYaq-s2"
      },
      "source": [
        "#Performing Experiments"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CH4LVC3Zj3jC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "54d0b9bb-3544-451f-cba3-1e3a578b2571"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "scores = {}\n",
        "trtime = {}\n",
        "tetime = {}\n",
        "# Estimators\n",
        "for clf_name, estimator in clfs:\n",
        "  if clf_name not in scores:\n",
        "    scores[clf_name] = {}\n",
        "    trtime[clf_name] = {}\n",
        "    tetime[clf_name] = {}\n",
        "  print(\"*\"*(len(clf_name)+8),'\\n***',clf_name,'***\\n'+\"*\"*(len(clf_name)+8))\n",
        "  # Validation forms\n",
        "  for val_name in validations.keys():\n",
        "    print(\"#\"*(len(val_name)+8),'\\n###',val_name,'###\\n'+\"#\"*(len(val_name)+8))\n",
        "    # Number of repetitions\n",
        "    for r in range(validations[val_name].rounds):\n",
        "      round_str = \"Round {}\".format(r+1)\n",
        "      print(\"@\"*(len(round_str)+8),'\\n@@@',round_str,'@@@\\n'+\"@\"*(len(round_str)+8))\n",
        "      groups = validations[val_name].groups\n",
        "      if val_name not in scores[clf_name]:\n",
        "        scores[clf_name][val_name] = {}\n",
        "      validation = eval(validations[val_name].splitter_name\n",
        "                        +'(4,shuffle='+str(validations[val_name].shuffle)\n",
        "                        +',random_state='+str(random_state+r)+')')\n",
        "      score = experimenter(estimator, X, selected_y, groups, \n",
        "                           scoring, validation, verbose)\n",
        "      for metric,s in score.items():\n",
        "        print(metric, ' \\t', s)\n",
        "        if metric not in scores[clf_name][val_name]:\n",
        "          scores[clf_name][val_name][metric] = []\n",
        "        scores[clf_name][val_name][metric].append(s)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "*************************************************************** \n",
            "*** RF - RandomForestClassifier with Heterogeneous Features ***\n",
            "***************************************************************\n",
            "############### \n",
            "### By Load ###\n",
            "###############\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 1 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [4072.0048599243164, 3971.1273851394653, 4007.050894498825, 3923.62695813179]\n",
            "score_time  \t [9.202571630477905, 9.959919691085815, 9.863134145736694, 9.595574140548706]\n",
            "test_accuracy  \t [0.9480130647795318, 0.9812515520238391, 0.9662363455809335, 0.9365000620116581]\n",
            "test_f1_macro  \t [0.9541007434900952, 0.981709376022543, 0.9663312387552976, 0.9356125065034845]\n",
            "################### \n",
            "### By Severity ###\n",
            "###################\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 1 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "[0. 1. 2. 3. 0. 1. 2. 3.]\n",
            "fit_time  \t [3301.591542005539, 3844.1913075447083, 3531.6875348091125, 4570.869230747223]\n",
            "score_time  \t [12.796550512313843, 9.3943030834198, 11.179861783981323, 4.535309314727783]\n",
            "test_accuracy  \t [0.5101532567049808, 0.539150460593654, 0.5631395716847769, 0.49841521394611726]\n",
            "test_f1_macro  \t [0.5280373901589933, 0.5056137271948168, 0.5454059975595691, 0.24940523394131642]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 2 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "[0. 1. 2. 3. 3. 0. 1. 2.]\n",
            "fit_time  \t [3100.424046754837, 3838.168055295944, 3521.3323690891266, 4736.201282501221]\n",
            "score_time  \t [13.206766366958618, 9.292587041854858, 12.063929080963135, 3.476682424545288]\n",
            "test_accuracy  \t [0.6051151344700298, 0.5381269191402251, 0.5546652609383237, 0.33497536945812806]\n",
            "test_f1_macro  \t [0.5525010407714559, 0.5046672033816783, 0.53943790799564, 0.25]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 3 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "[0. 1. 2. 3. 2. 3. 0. 1.]\n",
            "fit_time  \t [3122.9404289722443, 3821.518924474716, 3800.3353536129, 4884.549899816513]\n",
            "score_time  \t [13.460021734237671, 9.806526184082031, 10.750390529632568, 4.655071973800659]\n",
            "test_accuracy  \t [0.596765688170153, 0.5278701099463053, 0.5162158997775436, 0.5002645502645503]\n",
            "test_f1_macro  \t [0.5530607585702343, 0.4955428274727212, 0.5467125252922331, 0.25052798310454066]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 4 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "[0. 1. 2. 3. 1. 2. 3. 0.]\n",
            "fit_time  \t [3124.4579796791077, 4080.3560025691986, 3529.207218170166, 4621.081122875214]\n",
            "score_time  \t [13.211638689041138, 8.480123043060303, 11.16506052017212, 4.515885353088379]\n",
            "test_accuracy  \t [0.5346099789177794, 0.48197150334399536, 0.5544888701339804, 0.5]\n",
            "test_f1_macro  \t [0.5174391652041983, 0.5102736015712057, 0.5296003549190389, 0.25]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 5 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "[1. 0. 2. 3. 0. 1. 2. 3.]\n",
            "fit_time  \t [3838.603193283081, 2956.779947280884, 3339.363736629486, 3658.5913541316986]\n",
            "score_time  \t [7.868077516555786, 13.02027153968811, 10.6065034866333, 3.2213878631591797]\n",
            "test_accuracy  \t [0.47484733934283224, 0.6205835823519071, 0.553117417449098, 0.5007923930269413]\n",
            "test_f1_macro  \t [0.5051672636289304, 0.561606318832955, 0.5410200655302071, 0.25]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 6 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "[1. 0. 2. 3. 3. 0. 1. 2.]\n",
            "fit_time  \t [2700.387570619583, 2234.948934316635, 2493.2651419639587, 3344.0244669914246]\n",
            "score_time  \t [6.557081699371338, 9.528404474258423, 7.923113822937012, 2.4130032062530518]\n",
            "test_accuracy  \t [0.5547594677584442, 0.612673580594129, 0.5499209277807063, 0.33497536945812806]\n",
            "test_f1_macro  \t [0.5171233375499282, 0.557289673850635, 0.529242166332487, 0.25]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 7 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "[1. 0. 2. 3. 2. 3. 0. 1.]\n",
            "fit_time  \t [2675.2328929901123, 2205.2475955486298, 2588.479674100876, 3264.188013315201]\n",
            "score_time  \t [6.503838300704956, 9.486050844192505, 7.428351163864136, 3.1671860218048096]\n",
            "test_accuracy  \t [0.5496417604912999, 0.600843288826423, 0.5130546774382391, 0.5007936507936508]\n",
            "test_f1_macro  \t [0.5110375700866188, 0.5495968715665601, 0.543148044285606, 0.25158061116965225]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 8 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "[1. 0. 2. 3. 1. 2. 3. 0.]\n",
            "fit_time  \t [2667.922997713089, 2251.012385368347, 2483.071825027466, 3267.6658494472504]\n",
            "score_time  \t [6.51307225227356, 8.647204160690308, 8.156021356582642, 3.1517128944396973]\n",
            "test_accuracy  \t [0.5230120173868575, 0.5183908045977011, 0.5542778774132292, 0.5023809523809524]\n",
            "test_f1_macro  \t [0.49040240376973293, 0.5409535640281503, 0.5401308960145214, 0.2547120418848168]\n",
            "#################### \n",
            "### Usual K-Fold ###\n",
            "####################\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 1 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [2747.5283024311066, 2732.700756549835, 2704.8834381103516, 2718.086138010025]\n",
            "score_time  \t [6.65735936164856, 6.700840950012207, 6.540146827697754, 6.679693937301636]\n",
            "test_accuracy  \t [0.9887070168760309, 0.9902284263959391, 0.9855329949238578, 0.9889593908629442]\n",
            "test_f1_macro  \t [0.9885769878934064, 0.9903998814835775, 0.9857619457772442, 0.9890759919576393]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 2 @@@\n",
            "@@@@@@@@@@@@@@@\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-31-d38d8daa6147>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     24\u001b[0m                         \u001b[0;34m+\u001b[0m\u001b[0;34m'(4,shuffle='\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalidations\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mval_name\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshuffle\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     25\u001b[0m                         +',random_state='+str(random_state+r)+')')\n\u001b[0;32m---> 26\u001b[0;31m       score = experimenter(estimator, X, selected_y, groups, \n\u001b[0m\u001b[1;32m     27\u001b[0m                            scoring, validation, verbose)\n\u001b[1;32m     28\u001b[0m       \u001b[0;32mfor\u001b[0m \u001b[0mmetric\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0ms\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mscore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-10-8f9893f1cb3d>\u001b[0m in \u001b[0;36mexperimenter\u001b[0;34m(model, X, y, groups, scoring, cv, verbose)\u001b[0m\n\u001b[1;32m     24\u001b[0m       \u001b[0mtest_fold\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     25\u001b[0m       \u001b[0mtest_fold\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m       score = cross_validate(model, X, y, groups, scoring, \n\u001b[0m\u001b[1;32m     27\u001b[0m                              PredefinedSplit(test_fold), verbose=verbose)\n\u001b[1;32m     28\u001b[0m       \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mscore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36minner_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     71\u001b[0m                           FutureWarning)\n\u001b[1;32m     72\u001b[0m         \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 73\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     74\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0minner_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     75\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/model_selection/_validation.py\u001b[0m in \u001b[0;36mcross_validate\u001b[0;34m(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, return_train_score, return_estimator, error_score)\u001b[0m\n\u001b[1;32m    240\u001b[0m     parallel = Parallel(n_jobs=n_jobs, verbose=verbose,\n\u001b[1;32m    241\u001b[0m                         pre_dispatch=pre_dispatch)\n\u001b[0;32m--> 242\u001b[0;31m     scores = parallel(\n\u001b[0m\u001b[1;32m    243\u001b[0m         delayed(_fit_and_score)(\n\u001b[1;32m    244\u001b[0m             \u001b[0mclone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mestimator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscorers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, iterable)\u001b[0m\n\u001b[1;32m   1027\u001b[0m             \u001b[0;31m# remaining jobs.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1028\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_iterating\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1029\u001b[0;31m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispatch_one_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1030\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_iterating\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_original_iterator\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1031\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36mdispatch_one_batch\u001b[0;34m(self, iterator)\u001b[0m\n\u001b[1;32m    845\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    846\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 847\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dispatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtasks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    848\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    849\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m_dispatch\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    763\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    764\u001b[0m             \u001b[0mjob_idx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 765\u001b[0;31m             \u001b[0mjob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    766\u001b[0m             \u001b[0;31m# A job can complete so quickly than its callback is\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    767\u001b[0m             \u001b[0;31m# called before we get here, causing self._jobs to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36mapply_async\u001b[0;34m(self, func, callback)\u001b[0m\n\u001b[1;32m    204\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    205\u001b[0m         \u001b[0;34m\"\"\"Schedule a func to be run\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 206\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mImmediateResult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    207\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    208\u001b[0m             \u001b[0mcallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    568\u001b[0m         \u001b[0;31m# Don't delay the application, to avoid keeping the input\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    569\u001b[0m         \u001b[0;31m# arguments in memory\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 570\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    571\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    572\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    250\u001b[0m         \u001b[0;31m# change the default number of processes to -1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m             return [func(*args, **kwargs)\n\u001b[0m\u001b[1;32m    253\u001b[0m                     for func, args, kwargs in self.items]\n\u001b[1;32m    254\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    250\u001b[0m         \u001b[0;31m# change the default number of processes to -1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m             return [func(*args, **kwargs)\n\u001b[0m\u001b[1;32m    253\u001b[0m                     for func, args, kwargs in self.items]\n\u001b[1;32m    254\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/model_selection/_validation.py\u001b[0m in \u001b[0;36m_fit_and_score\u001b[0;34m(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, return_n_test_samples, return_times, return_estimator, error_score)\u001b[0m\n\u001b[1;32m    529\u001b[0m             \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    530\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 531\u001b[0;31m             \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    532\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    533\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36minner_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     71\u001b[0m                           FutureWarning)\n\u001b[1;32m     72\u001b[0m         \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 73\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     74\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0minner_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     75\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/model_selection/_search.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, groups, **fit_params)\u001b[0m\n\u001b[1;32m    734\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresults\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    735\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 736\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_search\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mevaluate_candidates\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    737\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    738\u001b[0m         \u001b[0;31m# For multi-metric evaluation, store the best_index_, best_params_ and\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/model_selection/_search.py\u001b[0m in \u001b[0;36m_run_search\u001b[0;34m(self, evaluate_candidates)\u001b[0m\n\u001b[1;32m   1186\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_run_search\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mevaluate_candidates\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1187\u001b[0m         \u001b[0;34m\"\"\"Search all candidates in param_grid\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1188\u001b[0;31m         \u001b[0mevaluate_candidates\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mParameterGrid\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparam_grid\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1189\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1190\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/model_selection/_search.py\u001b[0m in \u001b[0;36mevaluate_candidates\u001b[0;34m(candidate_params)\u001b[0m\n\u001b[1;32m    706\u001b[0m                               n_splits, n_candidates, n_candidates * n_splits))\n\u001b[1;32m    707\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 708\u001b[0;31m                 out = parallel(delayed(_fit_and_score)(clone(base_estimator),\n\u001b[0m\u001b[1;32m    709\u001b[0m                                                        \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    710\u001b[0m                                                        \u001b[0mtrain\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtest\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, iterable)\u001b[0m\n\u001b[1;32m   1030\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_iterating\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_original_iterator\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1031\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1032\u001b[0;31m             \u001b[0;32mwhile\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispatch_one_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1033\u001b[0m                 \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1034\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36mdispatch_one_batch\u001b[0;34m(self, iterator)\u001b[0m\n\u001b[1;32m    845\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    846\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 847\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dispatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtasks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    848\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    849\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m_dispatch\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    763\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    764\u001b[0m             \u001b[0mjob_idx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 765\u001b[0;31m             \u001b[0mjob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    766\u001b[0m             \u001b[0;31m# A job can complete so quickly than its callback is\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    767\u001b[0m             \u001b[0;31m# called before we get here, causing self._jobs to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36mapply_async\u001b[0;34m(self, func, callback)\u001b[0m\n\u001b[1;32m    204\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    205\u001b[0m         \u001b[0;34m\"\"\"Schedule a func to be run\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 206\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mImmediateResult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    207\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    208\u001b[0m             \u001b[0mcallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    568\u001b[0m         \u001b[0;31m# Don't delay the application, to avoid keeping the input\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    569\u001b[0m         \u001b[0;31m# arguments in memory\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 570\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    571\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    572\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    250\u001b[0m         \u001b[0;31m# change the default number of processes to -1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m             return [func(*args, **kwargs)\n\u001b[0m\u001b[1;32m    253\u001b[0m                     for func, args, kwargs in self.items]\n\u001b[1;32m    254\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    250\u001b[0m         \u001b[0;31m# change the default number of processes to -1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m             return [func(*args, **kwargs)\n\u001b[0m\u001b[1;32m    253\u001b[0m                     for func, args, kwargs in self.items]\n\u001b[1;32m    254\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/model_selection/_validation.py\u001b[0m in \u001b[0;36m_fit_and_score\u001b[0;34m(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, return_n_test_samples, return_times, return_estimator, error_score)\u001b[0m\n\u001b[1;32m    529\u001b[0m             \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    530\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 531\u001b[0;31m             \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    532\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    533\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/pipeline.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, **fit_params)\u001b[0m\n\u001b[1;32m    333\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_final_estimator\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;34m'passthrough'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    334\u001b[0m                 \u001b[0mfit_params_last_step\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfit_params_steps\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msteps\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 335\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_final_estimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mXt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params_last_step\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    336\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    337\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/ensemble/_forest.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[1;32m    384\u001b[0m             \u001b[0;31m# parallel_backend contexts set at a higher level,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    385\u001b[0m             \u001b[0;31m# since correctness does not rely on using threads.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 386\u001b[0;31m             trees = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,\n\u001b[0m\u001b[1;32m    387\u001b[0m                              \u001b[0;34m**\u001b[0m\u001b[0m_joblib_parallel_args\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'threads'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    388\u001b[0m                 delayed(_parallel_build_trees)(\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, iterable)\u001b[0m\n\u001b[1;32m   1030\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_iterating\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_original_iterator\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1031\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1032\u001b[0;31m             \u001b[0;32mwhile\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispatch_one_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1033\u001b[0m                 \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1034\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36mdispatch_one_batch\u001b[0;34m(self, iterator)\u001b[0m\n\u001b[1;32m    845\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    846\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 847\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dispatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtasks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    848\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    849\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m_dispatch\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    763\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    764\u001b[0m             \u001b[0mjob_idx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 765\u001b[0;31m             \u001b[0mjob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    766\u001b[0m             \u001b[0;31m# A job can complete so quickly than its callback is\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    767\u001b[0m             \u001b[0;31m# called before we get here, causing self._jobs to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36mapply_async\u001b[0;34m(self, func, callback)\u001b[0m\n\u001b[1;32m    204\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    205\u001b[0m         \u001b[0;34m\"\"\"Schedule a func to be run\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 206\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mImmediateResult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    207\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    208\u001b[0m             \u001b[0mcallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    568\u001b[0m         \u001b[0;31m# Don't delay the application, to avoid keeping the input\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    569\u001b[0m         \u001b[0;31m# arguments in memory\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 570\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    571\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    572\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    250\u001b[0m         \u001b[0;31m# change the default number of processes to -1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m             return [func(*args, **kwargs)\n\u001b[0m\u001b[1;32m    253\u001b[0m                     for func, args, kwargs in self.items]\n\u001b[1;32m    254\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    250\u001b[0m         \u001b[0;31m# change the default number of processes to -1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m             return [func(*args, **kwargs)\n\u001b[0m\u001b[1;32m    253\u001b[0m                     for func, args, kwargs in self.items]\n\u001b[1;32m    254\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/ensemble/_forest.py\u001b[0m in \u001b[0;36m_parallel_build_trees\u001b[0;34m(tree, forest, X, y, sample_weight, tree_idx, n_trees, verbose, class_weight, n_samples_bootstrap)\u001b[0m\n\u001b[1;32m    166\u001b[0m                                                         indices=indices)\n\u001b[1;32m    167\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 168\u001b[0;31m         \u001b[0mtree\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msample_weight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcurr_sample_weight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcheck_input\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    169\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    170\u001b[0m         \u001b[0mtree\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msample_weight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msample_weight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcheck_input\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/tree/_classes.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight, check_input, X_idx_sorted)\u001b[0m\n\u001b[1;32m    888\u001b[0m         \"\"\"\n\u001b[1;32m    889\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 890\u001b[0;31m         super().fit(\n\u001b[0m\u001b[1;32m    891\u001b[0m             \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    892\u001b[0m             \u001b[0msample_weight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msample_weight\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/Jupyter/.venv/lib/python3.8/site-packages/sklearn/tree/_classes.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight, check_input, X_idx_sorted)\u001b[0m\n\u001b[1;32m    373\u001b[0m                                            min_impurity_split)\n\u001b[1;32m    374\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 375\u001b[0;31m         \u001b[0mbuilder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuild\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtree_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msample_weight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX_idx_sorted\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    376\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    377\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_outputs_\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mis_classifier\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QJ-qe0MIhM-z",
        "colab_type": "text"
      },
      "source": [
        "##Save results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qrp8uvOonKpd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 901
        },
        "outputId": "a31cf2b7-5929-476d-9b7c-3a2b7f303d36"
      },
      "source": [
        "from pathlib import Path\n",
        "\n",
        "clf = {}\n",
        "val = {}\n",
        "src = {}\n",
        "for c, clf_name in enumerate(scores.keys()):\n",
        "  if c not in clf:\n",
        "    clf[c] = clf_name\n",
        "  for v, val_name in enumerate(scores[clf_name].keys()):\n",
        "    if v not in val:\n",
        "      val[v] = val_name\n",
        "    for s, scr_name in enumerate(scores[clf_name][val_name].keys()):\n",
        "      scores[clf_name][val_name][scr_name] = np.array(scores[clf_name][val_name][scr_name])\n",
        "      if s not in src:\n",
        "        src[s] = scr_name\n",
        "      Path(dirres).mkdir(parents=True, exist_ok=True)\n",
        "      np.savetxt('{}/{}-{}-{}.txt'.format(dirres,clf_name,val_name,scr_name), \n",
        "                 scores[clf_name][val_name][scr_name], delimiter=',')\n",
        "      print('{}/{} - {} - {}\\n'.format(dirres,clf_name.split('-')[0],val_name,scr_name),\n",
        "            scores[clf_name][val_name][scr_name])\n"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cwru_rf/RF  - By Load - fit_time\n",
            " [[4072.00485992 3971.12738514 4007.0508945  3923.62695813]]\n",
            "cwru_rf/RF  - By Load - score_time\n",
            " [[9.20257163 9.95991969 9.86313415 9.59557414]]\n",
            "cwru_rf/RF  - By Load - test_accuracy\n",
            " [[0.94801306 0.98125155 0.96623635 0.93650006]]\n",
            "cwru_rf/RF  - By Load - test_f1_macro\n",
            " [[0.95410074 0.98170938 0.96633124 0.93561251]]\n",
            "cwru_rf/RF  - By Severity - fit_time\n",
            " [[3301.59154201 3844.19130754 3531.68753481 4570.86923075]\n",
            " [3100.42404675 3838.1680553  3521.33236909 4736.2012825 ]\n",
            " [3122.94042897 3821.51892447 3800.33535361 4884.54989982]\n",
            " [3124.45797968 4080.35600257 3529.20721817 4621.08112288]\n",
            " [3838.60319328 2956.77994728 3339.36373663 3658.59135413]\n",
            " [2700.38757062 2234.94893432 2493.26514196 3344.02446699]\n",
            " [2675.23289299 2205.24759555 2588.4796741  3264.18801332]\n",
            " [2667.92299771 2251.01238537 2483.07182503 3267.66584945]]\n",
            "cwru_rf/RF  - By Severity - score_time\n",
            " [[12.79655051  9.39430308 11.17986178  4.53530931]\n",
            " [13.20676637  9.29258704 12.06392908  3.47668242]\n",
            " [13.46002173  9.80652618 10.75039053  4.65507197]\n",
            " [13.21163869  8.48012304 11.16506052  4.51588535]\n",
            " [ 7.86807752 13.02027154 10.60650349  3.22138786]\n",
            " [ 6.5570817   9.52840447  7.92311382  2.41300321]\n",
            " [ 6.5038383   9.48605084  7.42835116  3.16718602]\n",
            " [ 6.51307225  8.64720416  8.15602136  3.15171289]]\n",
            "cwru_rf/RF  - By Severity - test_accuracy\n",
            " [[0.51015326 0.53915046 0.56313957 0.49841521]\n",
            " [0.60511513 0.53812692 0.55466526 0.33497537]\n",
            " [0.59676569 0.52787011 0.5162159  0.50026455]\n",
            " [0.53460998 0.4819715  0.55448887 0.5       ]\n",
            " [0.47484734 0.62058358 0.55311742 0.50079239]\n",
            " [0.55475947 0.61267358 0.54992093 0.33497537]\n",
            " [0.54964176 0.60084329 0.51305468 0.50079365]\n",
            " [0.52301202 0.5183908  0.55427788 0.50238095]]\n",
            "cwru_rf/RF  - By Severity - test_f1_macro\n",
            " [[0.52803739 0.50561373 0.545406   0.24940523]\n",
            " [0.55250104 0.5046672  0.53943791 0.25      ]\n",
            " [0.55306076 0.49554283 0.54671253 0.25052798]\n",
            " [0.51743917 0.5102736  0.52960035 0.25      ]\n",
            " [0.50516726 0.56160632 0.54102007 0.25      ]\n",
            " [0.51712334 0.55728967 0.52924217 0.25      ]\n",
            " [0.51103757 0.54959687 0.54314804 0.25158061]\n",
            " [0.4904024  0.54095356 0.5401309  0.25471204]]\n",
            "cwru_rf/RF  - Usual K-Fold - fit_time\n",
            " [[2747.52830243 2732.70075655 2704.88343811 2718.08613801]]\n",
            "cwru_rf/RF  - Usual K-Fold - score_time\n",
            " [[6.65735936 6.70084095 6.54014683 6.67969394]]\n",
            "cwru_rf/RF  - Usual K-Fold - test_accuracy\n",
            " [[0.98870702 0.99022843 0.98553299 0.98895939]]\n",
            "cwru_rf/RF  - Usual K-Fold - test_f1_macro\n",
            " [[0.98857699 0.99039988 0.98576195 0.98907599]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkb8XMN-Ht58",
        "colab_type": "text"
      },
      "source": [
        "##Average & Standard Deviation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hJxdjboqtuNb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "b3866451-e2aa-489f-dd26-1f88f1271042"
      },
      "source": [
        "c,v,s = len(clf),len(val),len(src)\n",
        "for i in range(s):\n",
        "  print(src[i])\n",
        "  for k in range(v):\n",
        "    print('\\t'+val[k]+' ', end='')\n",
        "  print()\n",
        "  for j in range(c):\n",
        "    print(clf[j].split('-')[0], end='\\t')\n",
        "    for k in range(v):\n",
        "      print(\"{0:.3f} ({1:.3f})\".format(\n",
        "          scores[clf[j]][val[k]][src[i]].mean(),\n",
        "          scores[clf[j]][val[k]][src[i]].std()), end='\\t')\n",
        "    print()\n",
        "  print()"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fit_time\n",
            "\tBy Load \tBy Severity \tUsual K-Fold \n",
            "RF \t3993.453 (54.151)\t3356.178 (723.628)\t2725.800 (15.943)\t\n",
            "\n",
            "score_time\n",
            "\tBy Load \tBy Severity \tUsual K-Fold \n",
            "RF \t9.655 (0.293)\t8.318 (3.334)\t6.645 (0.062)\t\n",
            "\n",
            "test_accuracy\n",
            "\tBy Load \tBy Severity \tUsual K-Fold \n",
            "RF \t0.958 (0.017)\t0.526 (0.062)\t0.988 (0.002)\t\n",
            "\n",
            "test_f1_macro\n",
            "\tBy Load \tBy Severity \tUsual K-Fold \n",
            "RF \t0.959 (0.017)\t0.460 (0.122)\t0.988 (0.002)\t\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qZG2ZnLQAeJL",
        "colab_type": "text"
      },
      "source": [
        "## Experiment results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "manFfTfh_9Ta",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "```\n",
        "*************************************************************** \n",
        "*** RF - RandomForestClassifier with Heterogeneous Features ***\n",
        "***************************************************************\n",
        "############### \n",
        "### By Load ###\n",
        "###############\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 1 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "fit_time  \t [4072.0048599243164, 3971.1273851394653, 4007.050894498825, 3923.62695813179]\n",
        "score_time  \t [9.202571630477905, 9.959919691085815, 9.863134145736694, 9.595574140548706]\n",
        "test_accuracy  \t [0.9480130647795318, 0.9812515520238391, 0.9662363455809335, 0.9365000620116581]\n",
        "test_f1_macro  \t [0.9541007434900952, 0.981709376022543, 0.9663312387552976, 0.9356125065034845]\n",
        "################### \n",
        "### By Severity ###\n",
        "###################\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 1 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "[0. 1. 2. 3. 0. 1. 2. 3.]\n",
        "fit_time  \t [3301.591542005539, 3844.1913075447083, 3531.6875348091125, 4570.869230747223]\n",
        "score_time  \t [12.796550512313843, 9.3943030834198, 11.179861783981323, 4.535309314727783]\n",
        "test_accuracy  \t [0.5101532567049808, 0.539150460593654, 0.5631395716847769, 0.49841521394611726]\n",
        "test_f1_macro  \t [0.5280373901589933, 0.5056137271948168, 0.5454059975595691, 0.24940523394131642]\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 2 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "[0. 1. 2. 3. 3. 0. 1. 2.]\n",
        "fit_time  \t [3100.424046754837, 3838.168055295944, 3521.3323690891266, 4736.201282501221]\n",
        "score_time  \t [13.206766366958618, 9.292587041854858, 12.063929080963135, 3.476682424545288]\n",
        "test_accuracy  \t [0.6051151344700298, 0.5381269191402251, 0.5546652609383237, 0.33497536945812806]\n",
        "test_f1_macro  \t [0.5525010407714559, 0.5046672033816783, 0.53943790799564, 0.25]\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 3 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "[0. 1. 2. 3. 2. 3. 0. 1.]\n",
        "fit_time  \t [3122.9404289722443, 3821.518924474716, 3800.3353536129, 4884.549899816513]\n",
        "score_time  \t [13.460021734237671, 9.806526184082031, 10.750390529632568, 4.655071973800659]\n",
        "test_accuracy  \t [0.596765688170153, 0.5278701099463053, 0.5162158997775436, 0.5002645502645503]\n",
        "test_f1_macro  \t [0.5530607585702343, 0.4955428274727212, 0.5467125252922331, 0.25052798310454066]\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 4 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "[0. 1. 2. 3. 1. 2. 3. 0.]\n",
        "fit_time  \t [3124.4579796791077, 4080.3560025691986, 3529.207218170166, 4621.081122875214]\n",
        "score_time  \t [13.211638689041138, 8.480123043060303, 11.16506052017212, 4.515885353088379]\n",
        "test_accuracy  \t [0.5346099789177794, 0.48197150334399536, 0.5544888701339804, 0.5]\n",
        "test_f1_macro  \t [0.5174391652041983, 0.5102736015712057, 0.5296003549190389, 0.25]\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 5 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "[1. 0. 2. 3. 0. 1. 2. 3.]\n",
        "fit_time  \t [3838.603193283081, 2956.779947280884, 3339.363736629486, 3658.5913541316986]\n",
        "score_time  \t [7.868077516555786, 13.02027153968811, 10.6065034866333, 3.2213878631591797]\n",
        "test_accuracy  \t [0.47484733934283224, 0.6205835823519071, 0.553117417449098, 0.5007923930269413]\n",
        "test_f1_macro  \t [0.5051672636289304, 0.561606318832955, 0.5410200655302071, 0.25]\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 6 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "[1. 0. 2. 3. 3. 0. 1. 2.]\n",
        "fit_time  \t [2700.387570619583, 2234.948934316635, 2493.2651419639587, 3344.0244669914246]\n",
        "score_time  \t [6.557081699371338, 9.528404474258423, 7.923113822937012, 2.4130032062530518]\n",
        "test_accuracy  \t [0.5547594677584442, 0.612673580594129, 0.5499209277807063, 0.33497536945812806]\n",
        "test_f1_macro  \t [0.5171233375499282, 0.557289673850635, 0.529242166332487, 0.25]\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 7 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "[1. 0. 2. 3. 2. 3. 0. 1.]\n",
        "fit_time  \t [2675.2328929901123, 2205.2475955486298, 2588.479674100876, 3264.188013315201]\n",
        "score_time  \t [6.503838300704956, 9.486050844192505, 7.428351163864136, 3.1671860218048096]\n",
        "test_accuracy  \t [0.5496417604912999, 0.600843288826423, 0.5130546774382391, 0.5007936507936508]\n",
        "test_f1_macro  \t [0.5110375700866188, 0.5495968715665601, 0.543148044285606, 0.25158061116965225]\n",
        "@@@@@@@@@@@@@@@ \n",
        "@@@ Round 8 @@@\n",
        "@@@@@@@@@@@@@@@\n",
        "[1. 0. 2. 3. 1. 2. 3. 0.]\n",
        "fit_time  \t [2667.922997713089, 2251.012385368347, 2483.071825027466, 3267.6658494472504]\n",
        "score_time  \t [6.51307225227356, 8.647204160690308, 8.156021356582642, 3.1517128944396973]\n",
        "test_accuracy  \t [0.5230120173868575, 0.5183908045977011, 0.5542778774132292, 0.5023809523809524]\n",
        "test_f1_macro  \t [0.49040240376973293, 0.5409535640281503, 0.5401308960145214, 0.2547120418848168]\n",
        "```\n",
        "\n"
      ]
    }
  ]
}