KFold_RF_CWRU_EvaluationFramework.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "KFold RF CWRU-EvaluationFramework.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/fboldt/SignalProcessing/blob/master/KFold_RF_CWRU_EvaluationFramework.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "llMoY5AQGXQY",
        "colab_type": "text"
      },
      "source": [
        "The code is separated into four sections.\n",
        "\n",
        "* In section \"CWRU dataset\" the CWRU Matlab files are downloaded, the acquisitions are extracted from the Matlab files into Numpy arrays. Then, the data is segmented and the samples are selected. The samples are selected by their labels using regular expressions. \n",
        "\n",
        "* Section \"Experimenter\" defines the splitters mentioned in this work, i.e. GroupShuffleKFold and BySeverityKFold, and the setup of each experiment. The samples are grouped by their original labels also using regular expressions. For instance, to group samples by load, the regular expression '_\\d' may be used. A list of evaluation methods is defined in this section. \n",
        "\n",
        "* Section \"Classification Models\" defines the estimators and their feature extraction methods. To instantiate a classification method with feature extraction, a Pipeline must be made. A list of classification methods is defined in this section. \n",
        "\n",
        "* Finally, section \"Performing Experiments\" executes the experiments as they were defined in the previous sections, showing and saving  their results. It iterates one list of classification methods and one list of evaluation methods in $r$ rounds. In this work, four classification methods were tested by three evaluation methods in four rounds, resulting in $4\\times 3\\times 4 = 48$  experiments. New classification or evaluation methods can be tested by adding them in their respective list.\n",
        "\n",
        "The code can be executed direct in the Colab environment when few samples and simple classifier methods are tested. For the experiments presented in this work, it must be run in a local GPU, with enough memory and processing capacity.\n",
        "The results are presented for each round of each experiment of each classification method. The average and standard deviation of the rounds is also presented, as well, the average of the differences among the classification methods.\n",
        "\n",
        "New feature extraction methods must receive a 3-D Numpy array and returns a 2-D Numpy array. This is necessary because the same raw dataset is used for methods that need feature extraction of the signal acquisitions, like K-NN, SVM and Random Forest, and convolutional neural networks that deal with 3-D arrays. The experiments presented here used just one channel of each acquisition, but newer experiments may use more channels.\n",
        "\n",
        "New classifications methods that need feature extraction may be easily added. It is only required a Pipeline with the feature extraction method and the classifier, like those presented in the code. A new neural network base architecture must be wrapped in a scikit-learn estimator, and its definition must be in the method *fit*. It cannot be defined in the method *\\_\\_init\\_\\_* due to the implementation of the Keras library. If the network architecture is defined in the method *\\_\\_init\\_\\_*, it will remember the samples across the folds and the rounds, giving outstanding results, that will be never replicated in a real scenario. It is worth to highlight that some parameters of the network, like kernel size and number of filters, should be selected by a GridSearchCV method to provide fairer results when compared with other methods."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sm5t8TYBqkDu",
        "colab_type": "text"
      },
      "source": [
        "#CWRU database"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bSSOMru17Z6c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "debug = False"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pMQoq6dvStey",
        "colab_type": "text"
      },
      "source": [
        "## CWRU files.\n",
        "\n",
        "Associate each Matlab file name to a bearing condition in a Python dictionary.\n",
        "The dictionary keys identify the conditions.\n",
        "\n",
        "There are only four normal conditions, with loads of 0, 1, 2 and 3 hp.\n",
        "All conditions end with an underscore character followed by an algarism representing the load applied during the acquisitions.\n",
        "The remaining conditions follow the pattern:\n",
        "\n",
        "\n",
        "* First two characters represent the bearing location, i.e. drive end (DE) and fan end (FE).\n",
        "* The following two characters represent the failure location in the bearing, i.e. ball (BA), Inner Race (IR) and Outer Race (OR).\n",
        "* The next three algarisms indicate the severity of the failure, where 007 stands for 0.007 inches and 0021 for 0.021 inches.\n",
        "* For Outer Race failures, the character @ is followed by a number that indicates different load zones. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K6mp2QrP1lmR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cwru_12khz():\n",
        "  '''\n",
        "  Retuns a dictionary with the names of all Matlab files read in 12kHz located in\n",
        "  http://csegroups.case.edu/sites/default/files/bearingdatacenter/files/Datafiles/.\n",
        "  The dictionary keys represent the bearing condition.\n",
        "  '''\n",
        "  matlab_files_name = {}\n",
        "  # Normal\n",
        "  matlab_files_name[\"Normal_0\"] = \"97.mat\"\n",
        "  matlab_files_name[\"Normal_1\"] = \"98.mat\"\n",
        "  matlab_files_name[\"Normal_2\"] = \"99.mat\"\n",
        "  matlab_files_name[\"Normal_3\"] = \"100.mat\"\n",
        "  # DE Inner Race 0.007 inches\n",
        "  matlab_files_name[\"DEIR.007_0\"] = \"105.mat\"\n",
        "  matlab_files_name[\"DEIR.007_1\"] = \"106.mat\"\n",
        "  matlab_files_name[\"DEIR.007_2\"] = \"107.mat\"\n",
        "  matlab_files_name[\"DEIR.007_3\"] = \"108.mat\"\n",
        "  # DE Ball 0.007 inches\n",
        "  matlab_files_name[\"DEB.007_0\"] = \"118.mat\"\n",
        "  matlab_files_name[\"DEB.007_1\"] = \"119.mat\"\n",
        "  matlab_files_name[\"DEB.007_2\"] = \"120.mat\"\n",
        "  matlab_files_name[\"DEB.007_3\"] = \"121.mat\"\n",
        "  # DE Outer race 0.007 inches centered @6:00\n",
        "  matlab_files_name[\"DEOR.007@6_0\"] = \"130.mat\"\n",
        "  matlab_files_name[\"DEOR.007@6_1\"] = \"131.mat\"\n",
        "  matlab_files_name[\"DEOR.007@6_2\"] = \"132.mat\"\n",
        "  matlab_files_name[\"DEOR.007@6_3\"] = \"133.mat\"\n",
        "  # DE Outer race 0.007 inches centered @3:00\n",
        "  matlab_files_name[\"DEOR.007@3_0\"] = \"144.mat\"\n",
        "  matlab_files_name[\"DEOR.007@3_1\"] = \"145.mat\"\n",
        "  matlab_files_name[\"DEOR.007@3_2\"] = \"146.mat\"\n",
        "  matlab_files_name[\"DEOR.007@3_3\"] = \"147.mat\"\n",
        "  # DE Outer race 0.007 inches centered @12:00\n",
        "  matlab_files_name[\"DEOR.007@12_0\"] = \"156.mat\"\n",
        "  matlab_files_name[\"DEOR.007@12_1\"] = \"158.mat\"\n",
        "  matlab_files_name[\"DEOR.007@12_2\"] = \"159.mat\"\n",
        "  matlab_files_name[\"DEOR.007@12_3\"] = \"160.mat\"\n",
        "  # DE Inner Race 0.014 inches\n",
        "  matlab_files_name[\"DEIR.014_0\"] = \"169.mat\"\n",
        "  matlab_files_name[\"DEIR.014_1\"] = \"170.mat\"\n",
        "  matlab_files_name[\"DEIR.014_2\"] = \"171.mat\"\n",
        "  matlab_files_name[\"DEIR.014_3\"] = \"172.mat\"\n",
        "  # DE Ball 0.014 inches\n",
        "  matlab_files_name[\"DEB.014_0\"] = \"185.mat\"\n",
        "  matlab_files_name[\"DEB.014_1\"] = \"186.mat\"\n",
        "  matlab_files_name[\"DEB.014_2\"] = \"187.mat\"\n",
        "  matlab_files_name[\"DEB.014_3\"] = \"188.mat\"\n",
        "  # DE Outer race 0.014 inches centered @6:00\n",
        "  matlab_files_name[\"DEOR.014@6_0\"] = \"197.mat\"\n",
        "  matlab_files_name[\"DEOR.014@6_1\"] = \"198.mat\"\n",
        "  matlab_files_name[\"DEOR.014@6_2\"] = \"199.mat\"\n",
        "  matlab_files_name[\"DEOR.014@6_3\"] = \"200.mat\"\n",
        "  # DE Ball 0.021 inches\n",
        "  matlab_files_name[\"DEB.021_0\"] = \"222.mat\"\n",
        "  matlab_files_name[\"DEB.021_1\"] = \"223.mat\"\n",
        "  matlab_files_name[\"DEB.021_2\"] = \"224.mat\"\n",
        "  matlab_files_name[\"DEB.021_3\"] = \"225.mat\"\n",
        "  # FE Inner Race 0.021 inches\n",
        "  matlab_files_name[\"FEIR.021_0\"] = \"270.mat\"\n",
        "  matlab_files_name[\"FEIR.021_1\"] = \"271.mat\"\n",
        "  matlab_files_name[\"FEIR.021_2\"] = \"272.mat\"\n",
        "  matlab_files_name[\"FEIR.021_3\"] = \"273.mat\"\n",
        "  # FE Inner Race 0.014 inches\n",
        "  matlab_files_name[\"FEIR.014_0\"] = \"274.mat\"\n",
        "  matlab_files_name[\"FEIR.014_1\"] = \"275.mat\"\n",
        "  matlab_files_name[\"FEIR.014_2\"] = \"276.mat\"\n",
        "  matlab_files_name[\"FEIR.014_3\"] = \"277.mat\"\n",
        "  # FE Ball 0.007 inches\n",
        "  matlab_files_name[\"FEB.007_0\"] = \"282.mat\"\n",
        "  matlab_files_name[\"FEB.007_1\"] = \"283.mat\"\n",
        "  matlab_files_name[\"FEB.007_2\"] = \"284.mat\"\n",
        "  matlab_files_name[\"FEB.007_3\"] = \"285.mat\"\n",
        "  # DE Inner Race 0.021 inches\n",
        "  matlab_files_name[\"DEIR.021_0\"] = \"209.mat\"\n",
        "  matlab_files_name[\"DEIR.021_1\"] = \"210.mat\"\n",
        "  matlab_files_name[\"DEIR.021_2\"] = \"211.mat\"\n",
        "  matlab_files_name[\"DEIR.021_3\"] = \"212.mat\"\n",
        "  # DE Outer race 0.021 inches centered @6:00\n",
        "  matlab_files_name[\"DEOR.021@6_0\"] = \"234.mat\"\n",
        "  matlab_files_name[\"DEOR.021@6_1\"] = \"235.mat\"\n",
        "  matlab_files_name[\"DEOR.021@6_2\"] = \"236.mat\"\n",
        "  matlab_files_name[\"DEOR.021@6_3\"] = \"237.mat\"\n",
        "  # DE Outer race 0.021 inches centered @3:00\n",
        "  matlab_files_name[\"DEOR.021@3_0\"] = \"246.mat\"\n",
        "  matlab_files_name[\"DEOR.021@3_1\"] = \"247.mat\"\n",
        "  matlab_files_name[\"DEOR.021@3_2\"] = \"248.mat\"\n",
        "  matlab_files_name[\"DEOR.021@3_3\"] = \"249.mat\"\n",
        "  # DE Outer race 0.021 inches centered @12:00\n",
        "  matlab_files_name[\"DEOR.021@12_0\"] = \"258.mat\"\n",
        "  matlab_files_name[\"DEOR.021@12_1\"] = \"259.mat\"\n",
        "  matlab_files_name[\"DEOR.021@12_2\"] = \"260.mat\"\n",
        "  matlab_files_name[\"DEOR.021@12_3\"] = \"261.mat\"\n",
        "  # FE Inner Race 0.007 inches\n",
        "  matlab_files_name[\"FEIR.007_0\"] = \"278.mat\"\n",
        "  matlab_files_name[\"FEIR.007_1\"] = \"279.mat\"\n",
        "  matlab_files_name[\"FEIR.007_2\"] = \"280.mat\"\n",
        "  matlab_files_name[\"FEIR.007_3\"] = \"281.mat\"\n",
        "  # FE Ball 0.014 inches\n",
        "  matlab_files_name[\"FEB.014_0\"] = \"286.mat\"\n",
        "  matlab_files_name[\"FEB.014_1\"] = \"287.mat\"\n",
        "  matlab_files_name[\"FEB.014_2\"] = \"288.mat\"\n",
        "  matlab_files_name[\"FEB.014_3\"] = \"289.mat\"\n",
        "  # FE Ball 0.021 inches\n",
        "  matlab_files_name[\"FEB.021_0\"] = \"290.mat\"\n",
        "  matlab_files_name[\"FEB.021_1\"] = \"291.mat\"\n",
        "  matlab_files_name[\"FEB.021_2\"] = \"292.mat\"\n",
        "  matlab_files_name[\"FEB.021_3\"] = \"293.mat\"\n",
        "  # FE Outer race 0.007 inches centered @6:00\n",
        "  matlab_files_name[\"FEOR.007@6_0\"] = \"294.mat\"\n",
        "  matlab_files_name[\"FEOR.007@6_1\"] = \"295.mat\"\n",
        "  matlab_files_name[\"FEOR.007@6_2\"] = \"296.mat\"\n",
        "  matlab_files_name[\"FEOR.007@6_3\"] = \"297.mat\"\n",
        "  # FE Outer race 0.007 inches centered @3:00\n",
        "  matlab_files_name[\"FEOR.007@3_0\"] = \"298.mat\"\n",
        "  matlab_files_name[\"FEOR.007@3_1\"] = \"299.mat\"\n",
        "  matlab_files_name[\"FEOR.007@3_2\"] = \"300.mat\"\n",
        "  matlab_files_name[\"FEOR.007@3_3\"] = \"301.mat\"\n",
        "  # FE Outer race 0.007 inches centered @12:00\n",
        "  matlab_files_name[\"FEOR.007@12_0\"] = \"302.mat\"\n",
        "  matlab_files_name[\"FEOR.007@12_1\"] = \"305.mat\"\n",
        "  matlab_files_name[\"FEOR.007@12_2\"] = \"306.mat\"\n",
        "  matlab_files_name[\"FEOR.007@12_3\"] = \"307.mat\"\n",
        "  # FE Outer race 0.014 inches centered @3:00\n",
        "  matlab_files_name[\"FEOR.014@3_0\"] = \"310.mat\"\n",
        "  matlab_files_name[\"FEOR.014@3_1\"] = \"309.mat\"\n",
        "  matlab_files_name[\"FEOR.014@3_2\"] = \"311.mat\"\n",
        "  matlab_files_name[\"FEOR.014@3_3\"] = \"312.mat\"\n",
        "  # FE Outer race 0.014 inches centered @6:00\n",
        "  matlab_files_name[\"FEOR.014@6_0\"] = \"313.mat\"\n",
        "  # FE Outer race 0.021 inches centered @6:00\n",
        "  matlab_files_name[\"FEOR.021@6_0\"] = \"315.mat\"\n",
        "  # FE Outer race 0.021 inches centered @3:00\n",
        "  matlab_files_name[\"FEOR.021@3_1\"] = \"316.mat\"\n",
        "  matlab_files_name[\"FEOR.021@3_2\"] = \"317.mat\"\n",
        "  matlab_files_name[\"FEOR.021@3_3\"] = \"318.mat\"\n",
        "  # DE Inner Race 0.028 inches\n",
        "  matlab_files_name[\"DEIR.028_0\"] = \"3001.mat\"\n",
        "  matlab_files_name[\"DEIR.028_1\"] = \"3002.mat\"\n",
        "  matlab_files_name[\"DEIR.028_2\"] = \"3003.mat\"\n",
        "  matlab_files_name[\"DEIR.028_3\"] = \"3004.mat\"\n",
        "  # DE Ball 0.028 inches\n",
        "  matlab_files_name[\"DEB.028_0\"] = \"3005.mat\"\n",
        "  matlab_files_name[\"DEB.028_1\"] = \"3006.mat\"\n",
        "  matlab_files_name[\"DEB.028_2\"] = \"3007.mat\"\n",
        "  matlab_files_name[\"DEB.028_3\"] = \"3008.mat\"\n",
        "  return matlab_files_name\n",
        "\n",
        "def files_debug():\n",
        "  \"\"\"\n",
        "  Associate each Matlab file name to a bearing condition in a Python dictionary. \n",
        "  The dictionary keys identify the conditions.\n",
        "  \n",
        "  NOTE: Used only for debug.\n",
        "  \"\"\"\n",
        "  matlab_files_name = {}\n",
        "  # Normal\n",
        "  matlab_files_name[\"Normal_0\"] = \"97.mat\"\n",
        "  matlab_files_name[\"Normal_1\"] = \"98.mat\"\n",
        "  matlab_files_name[\"Normal_2\"] = \"99.mat\"\n",
        "  matlab_files_name[\"Normal_3\"] = \"100.mat\"\n",
        "  # FE Inner Race 0.007 inches\n",
        "  matlab_files_name[\"FEIR.007_2\"] = \"280.mat\"\n",
        "  # DE Outer race 0.014 inches centered @6:00\n",
        "  matlab_files_name[\"DEOR.014@6_1\"] = \"198.mat\"\n",
        "  # FE Outer race 0.021 inches centered @6:00\n",
        "  matlab_files_name[\"FEOR.021@6_0\"] = \"315.mat\"\n",
        "  # DE Ball 0.028 inches\n",
        "  matlab_files_name[\"DEB.028_3\"] = \"3008.mat\"\n",
        "  return matlab_files_name"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K9y9byVeSz_u",
        "colab_type": "text"
      },
      "source": [
        "##Download Matlab files\n",
        "Downloads the Matlab files in the dictionary matlab_files_name."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wPSGH1401-W2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import urllib.request\n",
        "import os.path\n",
        "\n",
        "def download_cwrufiles(matlab_files_name):\n",
        "  '''\n",
        "  Downloads the Matlab files in the dictionary matlab_files_name.\n",
        "  '''\n",
        "  url=\"http://csegroups.case.edu/sites/default/files/bearingdatacenter/files/Datafiles/\"\n",
        "  n = len(matlab_files_name)\n",
        "  for i,key in enumerate(matlab_files_name):\n",
        "    file_name = matlab_files_name[key]\n",
        "    if not os.path.exists(file_name):\n",
        "      urllib.request.urlretrieve(url+file_name, file_name)\n",
        "    print(\"{}/{}\\t{}\\t{}\".format(i+1, n, key, file_name))\n"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FRijKbOjS-JZ",
        "colab_type": "text"
      },
      "source": [
        "##Extract data from Matlab files\n",
        "Extracts the acquisitions of each Matlab file in the dictionary matlab_files_name."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BbpFkSI12CUe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import scipy.io\n",
        "import numpy as np\n",
        "\n",
        "def get_tensors_from_matlab(matlab_files_name):\n",
        "  '''\n",
        "  Extracts the acquisitions of each Matlab file in the dictionary matlab_files_name.\n",
        "  '''\n",
        "  acquisitions = {}\n",
        "  for key in matlab_files_name:\n",
        "    file_name = matlab_files_name[key]\n",
        "    matlab_file = scipy.io.loadmat(file_name)\n",
        "    for position in ['DE','FE', 'BA']:\n",
        "      keys = [key for key in matlab_file if key.endswith(position+\"_time\")]\n",
        "      if len(keys)>0:\n",
        "        array_key = keys[0]\n",
        "        acquisitions[key+position.lower()] = matlab_file[array_key].reshape(1,-1)[0]\n",
        "  return acquisitions\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x-2lcukz5Nyk",
        "colab_type": "text"
      },
      "source": [
        "##Downloading pickle file\n",
        "Following, some auxiliary functions to download a pickle file in a google drive account.\n",
        "The pickle file already has the acquisitions propertly extracted.\n",
        "Therefore, these functions might speed up the whole process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZJkpaFxn1xtR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import requests\n",
        "import os.path\n",
        "\n",
        "def download_file_from_google_drive(id, destination):\n",
        "    URL = \"https://docs.google.com/uc?export=download\"\n",
        "    session = requests.Session()\n",
        "    response = session.get(URL, params = { 'id' : id }, stream = True)\n",
        "    token = get_confirm_token(response)\n",
        "    if token:\n",
        "        params = { 'id' : id, 'confirm' : token }\n",
        "        response = session.get(URL, params = params, stream = True)\n",
        "    save_response_content(response, destination)    \n",
        "\n",
        "def get_confirm_token(response):\n",
        "    for key, value in response.cookies.items():\n",
        "        if key.startswith('download_warning'):\n",
        "            return value\n",
        "    return None\n",
        "\n",
        "def save_response_content(response, destination):\n",
        "    CHUNK_SIZE = 32768\n",
        "    with open(destination, \"wb\") as f:\n",
        "        for chunk in response.iter_content(CHUNK_SIZE):\n",
        "            if chunk: # filter out keep-alive new chunks\n",
        "                f.write(chunk)\n",
        "\n",
        "# https://drive.google.com/file/d/1qJezMiROz9NAYafPUDPh9BFkxYF4nOi2/view?usp=sharing\n",
        "file_id = \"1qJezMiROz9NAYafPUDPh9BFkxYF4nOi2\"\n",
        "if not debug:\n",
        "  pickle_file = 'cwru.pickle'\n",
        "else:\n",
        "  pickle_file = 'debug.pickle'\n",
        "\n",
        "if not os.path.isfile(pickle_file) and not debug:\n",
        "  try:\n",
        "    download_file_from_google_drive(file_id, destination)\n",
        "  except:\n",
        "    print(\"Download failed!\")"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oEzRboCpgtlx",
        "colab_type": "text"
      },
      "source": [
        "##Save/Load data\n",
        "If the cwru pickle file is already download, it will not be downloaded again, and the dictionary with the acquisitions will be loaded.\n",
        "Otherwise, the desired files are downloaded and the acquisitions are extrated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z1m5Q3OUbvqa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pickle\n",
        "import os\n",
        "\n",
        "if not debug:\n",
        "  matlab_files_name = cwru_12khz()\n",
        "else:\n",
        "  matlab_files_name = files_debug()\n",
        "\n",
        "if os.path.isfile(pickle_file) and  not debug:\n",
        "  with open(pickle_file, 'rb') as handle:\n",
        "    acquisitions = pickle.load(handle)\n",
        "else:\n",
        "  download_cwrufiles(matlab_files_name)\n",
        "  acquisitions = get_tensors_from_matlab(matlab_files_name)\n",
        "  with open(pickle_file, 'wb') as handle:\n",
        "    pickle.dump(acquisitions, handle, protocol=pickle.HIGHEST_PROTOCOL)\n"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AT7hgDnzTcNP",
        "colab_type": "text"
      },
      "source": [
        "##Segment data\n",
        "Segments the acquisitions.\n",
        "  sample_size is the size of each segment.\n",
        "  max_samples is used for debug purpouses and \n",
        "  reduces the number of samples from each acquisition.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7BKfioJFzAKA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a072ec32-a722-46a0-f908-e80d943e4699"
      },
      "source": [
        "import numpy as np\n",
        "def cwru_segmentation(acquisitions, sample_size=512, max_samples=None):\n",
        "  '''\n",
        "  Segments the acquisitions.\n",
        "  sample_size is the size of each segment.\n",
        "  max_samples is used for debug purpouses and \n",
        "  reduces the number of samples from each acquisition.\n",
        "  '''\n",
        "  origin = []\n",
        "  data = np.empty((0,sample_size,1))\n",
        "  n = len(acquisitions)\n",
        "  for i,key in enumerate(acquisitions):\n",
        "    acquisition_size = len(acquisitions[key])\n",
        "    n_samples = acquisition_size//sample_size\n",
        "    if max_samples is not None and max_samples > 0 and n_samples > max_samples:\n",
        "      n_samples = max_samples\n",
        "    print('{}/{} --- {}: {}'.format(i+1, n, key, n_samples))\n",
        "    origin.extend([key for _ in range(n_samples)])\n",
        "    data = np.concatenate((data,\n",
        "           acquisitions[key][:(n_samples*sample_size)].reshape(\n",
        "               (n_samples,sample_size,1))))\n",
        "  return data,origin\n",
        "\n",
        "if not debug:\n",
        "  signal_data,signal_origin = cwru_segmentation(acquisitions, 512)\n",
        "else:\n",
        "  signal_data,signal_origin = cwru_segmentation(acquisitions, 512, 15)\n",
        "signal_data.shape"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1/307 --- Normal_0de: 476\n",
            "2/307 --- Normal_0fe: 476\n",
            "3/307 --- Normal_1de: 945\n",
            "4/307 --- Normal_1fe: 945\n",
            "5/307 --- Normal_2de: 945\n",
            "6/307 --- Normal_2fe: 945\n",
            "7/307 --- Normal_3de: 948\n",
            "8/307 --- Normal_3fe: 948\n",
            "9/307 --- DEIR.007_0de: 236\n",
            "10/307 --- DEIR.007_0fe: 236\n",
            "11/307 --- DEIR.007_0ba: 236\n",
            "12/307 --- DEIR.007_1de: 238\n",
            "13/307 --- DEIR.007_1fe: 238\n",
            "14/307 --- DEIR.007_1ba: 238\n",
            "15/307 --- DEIR.007_2de: 238\n",
            "16/307 --- DEIR.007_2fe: 238\n",
            "17/307 --- DEIR.007_2ba: 238\n",
            "18/307 --- DEIR.007_3de: 240\n",
            "19/307 --- DEIR.007_3fe: 240\n",
            "20/307 --- DEIR.007_3ba: 240\n",
            "21/307 --- DEB.007_0de: 239\n",
            "22/307 --- DEB.007_0fe: 239\n",
            "23/307 --- DEB.007_0ba: 239\n",
            "24/307 --- DEB.007_1de: 237\n",
            "25/307 --- DEB.007_1fe: 237\n",
            "26/307 --- DEB.007_1ba: 237\n",
            "27/307 --- DEB.007_2de: 237\n",
            "28/307 --- DEB.007_2fe: 237\n",
            "29/307 --- DEB.007_2ba: 237\n",
            "30/307 --- DEB.007_3de: 237\n",
            "31/307 --- DEB.007_3fe: 237\n",
            "32/307 --- DEB.007_3ba: 237\n",
            "33/307 --- DEOR.007@6_0de: 238\n",
            "34/307 --- DEOR.007@6_0fe: 238\n",
            "35/307 --- DEOR.007@6_0ba: 238\n",
            "36/307 --- DEOR.007@6_1de: 239\n",
            "37/307 --- DEOR.007@6_1fe: 239\n",
            "38/307 --- DEOR.007@6_1ba: 239\n",
            "39/307 --- DEOR.007@6_2de: 237\n",
            "40/307 --- DEOR.007@6_2fe: 237\n",
            "41/307 --- DEOR.007@6_2ba: 237\n",
            "42/307 --- DEOR.007@6_3de: 239\n",
            "43/307 --- DEOR.007@6_3fe: 239\n",
            "44/307 --- DEOR.007@6_3ba: 239\n",
            "45/307 --- DEOR.007@3_0de: 238\n",
            "46/307 --- DEOR.007@3_0fe: 238\n",
            "47/307 --- DEOR.007@3_0ba: 238\n",
            "48/307 --- DEOR.007@3_1de: 237\n",
            "49/307 --- DEOR.007@3_1fe: 237\n",
            "50/307 --- DEOR.007@3_1ba: 237\n",
            "51/307 --- DEOR.007@3_2de: 237\n",
            "52/307 --- DEOR.007@3_2fe: 237\n",
            "53/307 --- DEOR.007@3_2ba: 237\n",
            "54/307 --- DEOR.007@3_3de: 238\n",
            "55/307 --- DEOR.007@3_3fe: 238\n",
            "56/307 --- DEOR.007@3_3ba: 238\n",
            "57/307 --- DEOR.007@12_0de: 238\n",
            "58/307 --- DEOR.007@12_0fe: 238\n",
            "59/307 --- DEOR.007@12_0ba: 238\n",
            "60/307 --- DEOR.007@12_1de: 238\n",
            "61/307 --- DEOR.007@12_1fe: 238\n",
            "62/307 --- DEOR.007@12_1ba: 238\n",
            "63/307 --- DEOR.007@12_2de: 238\n",
            "64/307 --- DEOR.007@12_2fe: 238\n",
            "65/307 --- DEOR.007@12_2ba: 238\n",
            "66/307 --- DEOR.007@12_3de: 238\n",
            "67/307 --- DEOR.007@12_3fe: 238\n",
            "68/307 --- DEOR.007@12_3ba: 238\n",
            "69/307 --- DEIR.014_0de: 237\n",
            "70/307 --- DEIR.014_0fe: 237\n",
            "71/307 --- DEIR.014_0ba: 237\n",
            "72/307 --- DEIR.014_1de: 237\n",
            "73/307 --- DEIR.014_1fe: 237\n",
            "74/307 --- DEIR.014_1ba: 237\n",
            "75/307 --- DEIR.014_2de: 237\n",
            "76/307 --- DEIR.014_2fe: 237\n",
            "77/307 --- DEIR.014_2ba: 237\n",
            "78/307 --- DEIR.014_3de: 237\n",
            "79/307 --- DEIR.014_3fe: 237\n",
            "80/307 --- DEIR.014_3ba: 237\n",
            "81/307 --- DEB.014_0de: 237\n",
            "82/307 --- DEB.014_0fe: 237\n",
            "83/307 --- DEB.014_0ba: 237\n",
            "84/307 --- DEB.014_1de: 238\n",
            "85/307 --- DEB.014_1fe: 238\n",
            "86/307 --- DEB.014_1ba: 238\n",
            "87/307 --- DEB.014_2de: 238\n",
            "88/307 --- DEB.014_2fe: 238\n",
            "89/307 --- DEB.014_2ba: 238\n",
            "90/307 --- DEB.014_3de: 238\n",
            "91/307 --- DEB.014_3fe: 238\n",
            "92/307 --- DEB.014_3ba: 238\n",
            "93/307 --- DEOR.014@6_0de: 237\n",
            "94/307 --- DEOR.014@6_0fe: 237\n",
            "95/307 --- DEOR.014@6_0ba: 237\n",
            "96/307 --- DEOR.014@6_1de: 238\n",
            "97/307 --- DEOR.014@6_1fe: 238\n",
            "98/307 --- DEOR.014@6_1ba: 238\n",
            "99/307 --- DEOR.014@6_2de: 237\n",
            "100/307 --- DEOR.014@6_2fe: 237\n",
            "101/307 --- DEOR.014@6_2ba: 237\n",
            "102/307 --- DEOR.014@6_3de: 238\n",
            "103/307 --- DEOR.014@6_3fe: 238\n",
            "104/307 --- DEOR.014@6_3ba: 238\n",
            "105/307 --- DEB.021_0de: 238\n",
            "106/307 --- DEB.021_0fe: 238\n",
            "107/307 --- DEB.021_0ba: 238\n",
            "108/307 --- DEB.021_1de: 237\n",
            "109/307 --- DEB.021_1fe: 237\n",
            "110/307 --- DEB.021_1ba: 237\n",
            "111/307 --- DEB.021_2de: 238\n",
            "112/307 --- DEB.021_2fe: 238\n",
            "113/307 --- DEB.021_2ba: 238\n",
            "114/307 --- DEB.021_3de: 238\n",
            "115/307 --- DEB.021_3fe: 238\n",
            "116/307 --- DEB.021_3ba: 238\n",
            "117/307 --- FEIR.021_0de: 236\n",
            "118/307 --- FEIR.021_0fe: 236\n",
            "119/307 --- FEIR.021_0ba: 236\n",
            "120/307 --- FEIR.021_1de: 236\n",
            "121/307 --- FEIR.021_1fe: 236\n",
            "122/307 --- FEIR.021_1ba: 236\n",
            "123/307 --- FEIR.021_2de: 236\n",
            "124/307 --- FEIR.021_2fe: 236\n",
            "125/307 --- FEIR.021_2ba: 236\n",
            "126/307 --- FEIR.021_3de: 236\n",
            "127/307 --- FEIR.021_3fe: 236\n",
            "128/307 --- FEIR.021_3ba: 236\n",
            "129/307 --- FEIR.014_0de: 237\n",
            "130/307 --- FEIR.014_0fe: 237\n",
            "131/307 --- FEIR.014_0ba: 237\n",
            "132/307 --- FEIR.014_1de: 237\n",
            "133/307 --- FEIR.014_1fe: 237\n",
            "134/307 --- FEIR.014_1ba: 237\n",
            "135/307 --- FEIR.014_2de: 237\n",
            "136/307 --- FEIR.014_2fe: 237\n",
            "137/307 --- FEIR.014_2ba: 237\n",
            "138/307 --- FEIR.014_3de: 236\n",
            "139/307 --- FEIR.014_3fe: 236\n",
            "140/307 --- FEIR.014_3ba: 236\n",
            "141/307 --- FEB.007_0de: 236\n",
            "142/307 --- FEB.007_0fe: 236\n",
            "143/307 --- FEB.007_0ba: 236\n",
            "144/307 --- FEB.007_1de: 235\n",
            "145/307 --- FEB.007_1fe: 235\n",
            "146/307 --- FEB.007_1ba: 235\n",
            "147/307 --- FEB.007_2de: 237\n",
            "148/307 --- FEB.007_2fe: 237\n",
            "149/307 --- FEB.007_2ba: 237\n",
            "150/307 --- FEB.007_3de: 236\n",
            "151/307 --- FEB.007_3fe: 236\n",
            "152/307 --- FEB.007_3ba: 236\n",
            "153/307 --- DEIR.021_0de: 238\n",
            "154/307 --- DEIR.021_0fe: 238\n",
            "155/307 --- DEIR.021_0ba: 238\n",
            "156/307 --- DEIR.021_1de: 237\n",
            "157/307 --- DEIR.021_1fe: 237\n",
            "158/307 --- DEIR.021_1ba: 237\n",
            "159/307 --- DEIR.021_2de: 237\n",
            "160/307 --- DEIR.021_2fe: 237\n",
            "161/307 --- DEIR.021_2ba: 237\n",
            "162/307 --- DEIR.021_3de: 238\n",
            "163/307 --- DEIR.021_3fe: 238\n",
            "164/307 --- DEIR.021_3ba: 238\n",
            "165/307 --- DEOR.021@6_0de: 239\n",
            "166/307 --- DEOR.021@6_0fe: 239\n",
            "167/307 --- DEOR.021@6_0ba: 239\n",
            "168/307 --- DEOR.021@6_1de: 238\n",
            "169/307 --- DEOR.021@6_1fe: 238\n",
            "170/307 --- DEOR.021@6_1ba: 238\n",
            "171/307 --- DEOR.021@6_2de: 238\n",
            "172/307 --- DEOR.021@6_2fe: 238\n",
            "173/307 --- DEOR.021@6_2ba: 238\n",
            "174/307 --- DEOR.021@6_3de: 238\n",
            "175/307 --- DEOR.021@6_3fe: 238\n",
            "176/307 --- DEOR.021@6_3ba: 238\n",
            "177/307 --- DEOR.021@3_0de: 237\n",
            "178/307 --- DEOR.021@3_0fe: 237\n",
            "179/307 --- DEOR.021@3_0ba: 237\n",
            "180/307 --- DEOR.021@3_1de: 238\n",
            "181/307 --- DEOR.021@3_1fe: 238\n",
            "182/307 --- DEOR.021@3_1ba: 238\n",
            "183/307 --- DEOR.021@3_2de: 238\n",
            "184/307 --- DEOR.021@3_2fe: 238\n",
            "185/307 --- DEOR.021@3_2ba: 238\n",
            "186/307 --- DEOR.021@3_3de: 238\n",
            "187/307 --- DEOR.021@3_3fe: 238\n",
            "188/307 --- DEOR.021@3_3ba: 238\n",
            "189/307 --- DEOR.021@12_0de: 237\n",
            "190/307 --- DEOR.021@12_0fe: 237\n",
            "191/307 --- DEOR.021@12_0ba: 237\n",
            "192/307 --- DEOR.021@12_1de: 239\n",
            "193/307 --- DEOR.021@12_1fe: 239\n",
            "194/307 --- DEOR.021@12_1ba: 239\n",
            "195/307 --- DEOR.021@12_2de: 239\n",
            "196/307 --- DEOR.021@12_2fe: 239\n",
            "197/307 --- DEOR.021@12_2ba: 239\n",
            "198/307 --- DEOR.021@12_3de: 237\n",
            "199/307 --- DEOR.021@12_3fe: 237\n",
            "200/307 --- DEOR.021@12_3ba: 237\n",
            "201/307 --- FEIR.007_0de: 237\n",
            "202/307 --- FEIR.007_0fe: 237\n",
            "203/307 --- FEIR.007_0ba: 237\n",
            "204/307 --- FEIR.007_1de: 237\n",
            "205/307 --- FEIR.007_1fe: 237\n",
            "206/307 --- FEIR.007_1ba: 237\n",
            "207/307 --- FEIR.007_2de: 237\n",
            "208/307 --- FEIR.007_2fe: 237\n",
            "209/307 --- FEIR.007_2ba: 237\n",
            "210/307 --- FEIR.007_3de: 237\n",
            "211/307 --- FEIR.007_3fe: 237\n",
            "212/307 --- FEIR.007_3ba: 237\n",
            "213/307 --- FEB.014_0de: 238\n",
            "214/307 --- FEB.014_0fe: 238\n",
            "215/307 --- FEB.014_0ba: 238\n",
            "216/307 --- FEB.014_1de: 237\n",
            "217/307 --- FEB.014_1fe: 237\n",
            "218/307 --- FEB.014_1ba: 237\n",
            "219/307 --- FEB.014_2de: 238\n",
            "220/307 --- FEB.014_2fe: 238\n",
            "221/307 --- FEB.014_2ba: 238\n",
            "222/307 --- FEB.014_3de: 236\n",
            "223/307 --- FEB.014_3fe: 236\n",
            "224/307 --- FEB.014_3ba: 236\n",
            "225/307 --- FEB.021_0de: 237\n",
            "226/307 --- FEB.021_0fe: 237\n",
            "227/307 --- FEB.021_0ba: 237\n",
            "228/307 --- FEB.021_1de: 237\n",
            "229/307 --- FEB.021_1fe: 237\n",
            "230/307 --- FEB.021_1ba: 237\n",
            "231/307 --- FEB.021_2de: 237\n",
            "232/307 --- FEB.021_2fe: 237\n",
            "233/307 --- FEB.021_2ba: 237\n",
            "234/307 --- FEB.021_3de: 235\n",
            "235/307 --- FEB.021_3fe: 235\n",
            "236/307 --- FEB.021_3ba: 235\n",
            "237/307 --- FEOR.007@6_0de: 236\n",
            "238/307 --- FEOR.007@6_0fe: 236\n",
            "239/307 --- FEOR.007@6_0ba: 236\n",
            "240/307 --- FEOR.007@6_1de: 237\n",
            "241/307 --- FEOR.007@6_1fe: 237\n",
            "242/307 --- FEOR.007@6_1ba: 237\n",
            "243/307 --- FEOR.007@6_2de: 236\n",
            "244/307 --- FEOR.007@6_2fe: 236\n",
            "245/307 --- FEOR.007@6_2ba: 236\n",
            "246/307 --- FEOR.007@6_3de: 238\n",
            "247/307 --- FEOR.007@6_3fe: 238\n",
            "248/307 --- FEOR.007@6_3ba: 238\n",
            "249/307 --- FEOR.007@3_0de: 236\n",
            "250/307 --- FEOR.007@3_0fe: 236\n",
            "251/307 --- FEOR.007@3_0ba: 236\n",
            "252/307 --- FEOR.007@3_1de: 236\n",
            "253/307 --- FEOR.007@3_1fe: 236\n",
            "254/307 --- FEOR.007@3_1ba: 236\n",
            "255/307 --- FEOR.007@3_2de: 238\n",
            "256/307 --- FEOR.007@3_2fe: 238\n",
            "257/307 --- FEOR.007@3_2ba: 238\n",
            "258/307 --- FEOR.007@3_3de: 237\n",
            "259/307 --- FEOR.007@3_3fe: 237\n",
            "260/307 --- FEOR.007@3_3ba: 237\n",
            "261/307 --- FEOR.007@12_0de: 236\n",
            "262/307 --- FEOR.007@12_0fe: 236\n",
            "263/307 --- FEOR.007@12_0ba: 236\n",
            "264/307 --- FEOR.007@12_1de: 236\n",
            "265/307 --- FEOR.007@12_1fe: 236\n",
            "266/307 --- FEOR.007@12_1ba: 236\n",
            "267/307 --- FEOR.007@12_2de: 237\n",
            "268/307 --- FEOR.007@12_2fe: 237\n",
            "269/307 --- FEOR.007@12_2ba: 237\n",
            "270/307 --- FEOR.007@12_3de: 236\n",
            "271/307 --- FEOR.007@12_3fe: 236\n",
            "272/307 --- FEOR.007@12_3ba: 236\n",
            "273/307 --- FEOR.014@3_0de: 237\n",
            "274/307 --- FEOR.014@3_0fe: 237\n",
            "275/307 --- FEOR.014@3_0ba: 237\n",
            "276/307 --- FEOR.014@3_1de: 236\n",
            "277/307 --- FEOR.014@3_1fe: 236\n",
            "278/307 --- FEOR.014@3_1ba: 236\n",
            "279/307 --- FEOR.014@3_2de: 236\n",
            "280/307 --- FEOR.014@3_2fe: 236\n",
            "281/307 --- FEOR.014@3_2ba: 236\n",
            "282/307 --- FEOR.014@3_3de: 236\n",
            "283/307 --- FEOR.014@3_3fe: 236\n",
            "284/307 --- FEOR.014@3_3ba: 236\n",
            "285/307 --- FEOR.014@6_0de: 236\n",
            "286/307 --- FEOR.014@6_0fe: 236\n",
            "287/307 --- FEOR.014@6_0ba: 236\n",
            "288/307 --- FEOR.021@6_0de: 235\n",
            "289/307 --- FEOR.021@6_0fe: 235\n",
            "290/307 --- FEOR.021@6_0ba: 235\n",
            "291/307 --- FEOR.021@3_1de: 235\n",
            "292/307 --- FEOR.021@3_1fe: 235\n",
            "293/307 --- FEOR.021@3_1ba: 235\n",
            "294/307 --- FEOR.021@3_2de: 235\n",
            "295/307 --- FEOR.021@3_2fe: 235\n",
            "296/307 --- FEOR.021@3_2ba: 235\n",
            "297/307 --- FEOR.021@3_3de: 237\n",
            "298/307 --- FEOR.021@3_3fe: 237\n",
            "299/307 --- FEOR.021@3_3ba: 237\n",
            "300/307 --- DEIR.028_0de: 235\n",
            "301/307 --- DEIR.028_1de: 237\n",
            "302/307 --- DEIR.028_2de: 237\n",
            "303/307 --- DEIR.028_3de: 237\n",
            "304/307 --- DEB.028_0de: 235\n",
            "305/307 --- DEB.028_1de: 237\n",
            "306/307 --- DEB.028_2de: 236\n",
            "307/307 --- DEB.028_3de: 236\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(77527, 512, 1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jCN8XZ5dOF3",
        "colab_type": "text"
      },
      "source": [
        "## Clean dataset functions\n",
        "The functions below help to select samples from acquisitions and form groups according to these acquisitions, using regular expressions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eOOP9H2c3AaZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "import numpy as np\n",
        "\n",
        "def select_samples(regex, X, y):\n",
        "  '''\n",
        "  Selects samples wich has some regex pattern in its name.\n",
        "  '''\n",
        "  mask = [re.search(regex,label) is not None for label in y]\n",
        "  return X[mask],y[mask]\n",
        "\n",
        "def join_labels(regex, y):\n",
        "  '''\n",
        "  Excludes some regex patterns from the labels, \n",
        "  making some samples to have the same label.\n",
        "  '''\n",
        "  return np.array([re.sub(regex, '', label) for label in y])\n",
        "\n",
        "def get_groups(regex, y):\n",
        "  '''\n",
        "  Generates a list of groups of samples with \n",
        "  the same regex patten in its label.\n",
        "  '''\n",
        "  groups = list(range(len(y)))\n",
        "  for i,label in enumerate(y):\n",
        "    match = re.search(regex,label)\n",
        "    groups[i] = match.group(0) if match else None\n",
        "  return groups"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yFA-8l02RplD",
        "colab_type": "text"
      },
      "source": [
        "##Selecting samples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r89dYOJm8gzW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "3834f56c-049c-46cf-9be4-5aa9f9e34908"
      },
      "source": [
        "#DE from 'de', FE from 'fe', Normal from 'de' and 'fe'\n",
        "samples = '^(DE).*(de)|^(FE).*(fe)|(Normal).*'\n",
        "X,y = select_samples(samples, signal_data, np.array(signal_origin))\n",
        "print(len(set(y)),set(y))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "113 {'FEIR.021_2fe', 'DEOR.021@12_3de', 'FEOR.007@3_2fe', 'DEB.021_3de', 'DEB.007_2de', 'DEIR.007_2de', 'DEOR.021@6_3de', 'FEOR.014@3_2fe', 'DEB.028_2de', 'FEIR.021_1fe', 'FEB.007_2fe', 'DEIR.007_1de', 'DEOR.014@6_2de', 'DEOR.021@12_1de', 'FEB.007_1fe', 'DEIR.021_2de', 'FEOR.021@3_3fe', 'FEIR.014_0fe', 'FEOR.007@12_3fe', 'FEOR.007@12_1fe', 'DEOR.014@6_0de', 'FEOR.014@6_0fe', 'DEOR.007@6_3de', 'FEOR.007@3_0fe', 'DEOR.007@6_0de', 'DEOR.007@12_1de', 'FEB.014_3fe', 'FEB.014_2fe', 'FEOR.021@3_2fe', 'DEIR.028_3de', 'DEOR.007@6_2de', 'DEIR.021_1de', 'FEOR.007@3_3fe', 'DEB.028_1de', 'DEB.028_3de', 'DEIR.021_0de', 'FEOR.007@3_1fe', 'DEB.007_0de', 'DEB.014_1de', 'FEB.007_3fe', 'Normal_0de', 'FEB.021_2fe', 'FEOR.014@3_1fe', 'FEOR.014@3_0fe', 'DEOR.014@6_1de', 'DEIR.021_3de', 'DEOR.007@3_2de', 'FEB.021_0fe', 'DEOR.007@12_0de', 'DEB.021_0de', 'DEOR.021@3_2de', 'DEIR.007_3de', 'FEOR.007@12_2fe', 'FEIR.014_1fe', 'DEB.021_2de', 'FEOR.007@6_3fe', 'FEB.014_1fe', 'FEB.007_0fe', 'DEOR.014@6_3de', 'FEIR.014_3fe', 'FEIR.007_0fe', 'Normal_2de', 'FEOR.007@6_0fe', 'DEIR.014_0de', 'FEIR.007_3fe', 'DEB.014_2de', 'FEOR.021@3_1fe', 'DEB.028_0de', 'DEOR.021@12_0de', 'FEIR.021_0fe', 'DEOR.021@6_2de', 'DEOR.007@12_3de', 'DEOR.021@3_1de', 'FEOR.007@6_1fe', 'Normal_1fe', 'DEOR.021@3_3de', 'FEOR.007@12_0fe', 'DEIR.014_1de', 'DEOR.021@6_0de', 'FEIR.014_2fe', 'FEB.014_0fe', 'DEIR.007_0de', 'DEIR.028_0de', 'DEIR.028_2de', 'Normal_3fe', 'DEIR.014_3de', 'FEB.021_3fe', 'FEIR.007_1fe', 'FEOR.021@6_0fe', 'FEIR.021_3fe', 'DEIR.028_1de', 'DEB.014_3de', 'DEOR.007@3_3de', 'DEIR.014_2de', 'FEOR.007@6_2fe', 'DEB.007_1de', 'DEB.014_0de', 'FEOR.014@3_3fe', 'DEOR.021@3_0de', 'DEOR.021@12_2de', 'Normal_1de', 'Normal_0fe', 'FEB.021_1fe', 'DEOR.007@3_1de', 'DEOR.007@6_1de', 'Normal_3de', 'Normal_2fe', 'DEOR.007@3_0de', 'DEOR.007@12_2de', 'DEB.021_1de', 'DEOR.021@6_1de', 'DEB.007_3de', 'FEIR.007_2fe'}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5pRQQK0Mhm1_",
        "colab_type": "text"
      },
      "source": [
        "#Experimenter"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GE4TTG1-hmH7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "324eda7a-47d9-4366-f6fd-62a1808ae70f"
      },
      "source": [
        "from sklearn.model_selection import cross_validate, KFold, PredefinedSplit\n",
        "\n",
        "def experimenter(model, X, y, groups=None, scoring=None, cv=KFold(4, True), verbose=0):\n",
        "  '''\n",
        "  Performs a experiment with some estimator (model) and validation.\n",
        "  It works like a cross_validate function from sklearn, however, \n",
        "  when a estimator has an internal validation with groups, \n",
        "  it maintains the groups from the external validation.\n",
        "  '''\n",
        "  if hasattr(model,'cv') or (hasattr(model,'steps') and any(['gs' in step[0] for step in model.steps])):\n",
        "    scores = {}\n",
        "    lstval = list(validation.split(X,y,groups))\n",
        "    for tr,te in lstval:\n",
        "      if groups is not None:\n",
        "        innercv = list(GroupShuffleKFold(validation.n_splits-1).split(X[tr],y[tr],np.array(groups)[tr]))\n",
        "      else:\n",
        "        innercv = list(KFold(validation.n_splits-1, True).split(X[tr],y[tr]))\n",
        "      if hasattr(model,'cv'):\n",
        "        model.cv = innercv\n",
        "      else:\n",
        "        for step in model.steps:\n",
        "          if 'gs' in step[0]:\n",
        "            step[1].cv = innercv\n",
        "      test_fold = np.zeros((len(y),), dtype=int)\n",
        "      test_fold[tr] = -1\n",
        "      score = cross_validate(model, X, y, groups, scoring, \n",
        "                             PredefinedSplit(test_fold), verbose=verbose)\n",
        "      for k in score.keys():\n",
        "        if k not in scores:\n",
        "          scores[k] = []\n",
        "        scores[k].extend(score[k])\n",
        "    return scores\n",
        "  return cross_validate(model, X, y, groups, scoring, cv, verbose=verbose)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/home/francisco/Jupyter/.venv/lib/python3.8/site-packages/sklearn/utils/validation.py:68: FutureWarning: Pass shuffle=True as keyword args. From version 0.25 passing these as positional arguments will result in an error\n",
            "  warnings.warn(\"Pass {} as keyword args. From version 0.25 \"\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dWYfuxcxFjt8",
        "colab_type": "text"
      },
      "source": [
        "## Custom Splitter"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xRdfG-uzhPm4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import KFold\n",
        "from sklearn.utils import shuffle\n",
        "from sklearn.utils.validation import check_array\n",
        "import numpy as np\n",
        "\n",
        "class GroupShuffleKFold(KFold):\n",
        "  '''\n",
        "  Neither GroupShuffleSplit nor GroupKFold are good splitters for this case.\n",
        "  A custom splitter must be made.\n",
        "  '''\n",
        "  def __init__(self, n_splits=4, shuffle=False, random_state=None):\n",
        "    super().__init__(n_splits, shuffle=shuffle, random_state=random_state)\n",
        "  def get_n_splits(self, X, y, groups=None):\n",
        "    return self.n_splits\n",
        "  def _iter_test_indices(self, X=None, y=None, groups=None):\n",
        "    if groups is None:\n",
        "      raise ValueError(\"The 'groups' parameter should not be None.\")\n",
        "    groups = check_array(groups, ensure_2d=False, dtype=None)\n",
        "    unique_groups, groups = np.unique(groups, return_inverse=True)\n",
        "    n_groups = len(unique_groups)\n",
        "    if self.n_splits > n_groups:\n",
        "      raise ValueError(\"Cannot have number of splits n_splits=%d greater\"\n",
        "                        \" than the number of groups: %d.\"\n",
        "                        % (self.n_splits, n_groups))\n",
        "    # Distribute groups\n",
        "    indices = np.arange(n_groups)\n",
        "    if self.shuffle:\n",
        "      for i in range(n_groups//self.n_splits):\n",
        "        if self.random_state is None:\n",
        "          indices[self.n_splits*i:self.n_splits*(i+1)] = shuffle(\n",
        "              indices[self.n_splits*i:self.n_splits*(i+1)])\n",
        "        else:\n",
        "          indices[self.n_splits*i:self.n_splits*(i+1)] = shuffle(\n",
        "              indices[self.n_splits*i:self.n_splits*(i+1)],\n",
        "              random_state=self.random_state+i)\n",
        "    #print(unique_groups[indices]) #Debug purpose\n",
        "    # Total weight of each fold\n",
        "    n_samples_per_fold = np.zeros(self.n_splits)\n",
        "    # Mapping from group index to fold index\n",
        "    group_to_fold = np.zeros(len(unique_groups))\n",
        "    # Distribute samples \n",
        "    for group_index in indices:\n",
        "      group_to_fold[indices[group_index]] = group_index%(self.n_splits)\n",
        "    indices = group_to_fold[groups]\n",
        "    for f in range(self.n_splits):\n",
        "      yield np.where(indices == f)[0]"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Frmm3FVivQvg",
        "colab_type": "text"
      },
      "source": [
        "## BySeverity Splitter"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZFDWy6zqvGTF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import KFold\n",
        "from sklearn.utils import shuffle\n",
        "from sklearn.utils.validation import check_array\n",
        "import numpy as np\n",
        "\n",
        "class BySeverityKFold(KFold):\n",
        "  '''\n",
        "  Splits the CWRU dataset in severities.\n",
        "  '''\n",
        "  # Compatibility constructor\n",
        "  def __init__(self, n_splits=4, shuffle=False, random_state=None):\n",
        "    super().__init__(n_splits=4, shuffle=False, random_state=None)\n",
        "    self.nround = random_state\n",
        "  def _iter_test_indices(self, X=None, y=None, groups=None):\n",
        "    if groups is None:\n",
        "      raise ValueError(\"The 'groups' parameter should not be None.\")\n",
        "    groups = check_array(groups, ensure_2d=False, dtype=None)\n",
        "    unique_groups, groups = np.unique(groups, return_inverse=True)\n",
        "    n_groups = len(unique_groups)\n",
        "    if self.n_splits > n_groups:\n",
        "      raise ValueError(\"Cannot have number of splits n_splits=%d greater\"\n",
        "                        \" than the number of groups: %d.\"\n",
        "                        % (self.n_splits, n_groups))\n",
        "    # Distribute groups\n",
        "    indices = np.arange(n_groups)\n",
        "    nround = self.nround - random_state\n",
        "    for i in range(nround//4):\n",
        "      indices[i],indices[i+1] = indices[i+1],indices[i]\n",
        "    for i in range(self.n_splits):      \n",
        "      indices[i+self.n_splits] = (i+nround)%self.n_splits+self.n_splits\n",
        "    #print(unique_groups[indices]) #Debug purpose\n",
        "    # Total weight of each fold\n",
        "    n_samples_per_fold = np.zeros(self.n_splits)\n",
        "    # Mapping from group index to fold index\n",
        "    group_to_fold = np.zeros(len(unique_groups))\n",
        "    # Distribute samples \n",
        "    for group_index in indices:\n",
        "      group_to_fold[indices[group_index]] = group_index%(self.n_splits)\n",
        "    print(group_to_fold)\n",
        "    indices = group_to_fold[groups]\n",
        "    for f in range(self.n_splits):\n",
        "      yield np.where(indices == f)[0]"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e48W6KkIhesw",
        "colab_type": "text"
      },
      "source": [
        "##Experiment setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K9dImVH3hh_Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from collections import namedtuple\n",
        "\n",
        "ExperimentSetup = namedtuple('ExperimentSetup', \n",
        "                             'groups, splitter_name, shuffle, rounds')\n",
        "\n",
        "validations = {\n",
        "    # # Samples with the same load cannot be in the trainning fold and\n",
        "    # # the test folds simultaneously. \n",
        "    # \"By Load\": ExperimentSetup(groups = get_groups('_\\d',y), \n",
        "    #                            splitter_name = 'GroupShuffleKFold',\n",
        "    #                            shuffle = False,\n",
        "    #                            rounds=1,\n",
        "    #                            ),\n",
        "    # # Samples with the same severity cannot be in the trainning folds and\n",
        "    # # the test folds simultaneously.\n",
        "    # \"By Severity\": ExperimentSetup(groups = get_groups('(\\.\\d{3})|(Normal_\\d)',y),\n",
        "    #                                splitter_name = 'BySeverityKFold',\n",
        "    #                                shuffle = False,\n",
        "    #                                rounds=8),\n",
        "    # Validation usually seen in publications with CWRU bearing dataset.\n",
        "    \"Usual K-Fold\": ExperimentSetup(groups = None, \n",
        "                                    splitter_name = 'KFold',\n",
        "                                    shuffle = True,\n",
        "                                    rounds=8), \n",
        "}"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E9yn44VAoRFo",
        "colab_type": "text"
      },
      "source": [
        "##Common Variables"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ogpdDde4oTsS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Only four conditions are considered: Normal, Ball, Inner Race and Outer Race.\n",
        "selected_y = join_labels('_\\d|@\\d{1,3}|(de)|(fe)|\\.\\d{3}|(DE)|(FE)',y)\n",
        "verbose = 0 #if not debug else 3\n",
        "random_state = 42\n",
        "scoring = ['accuracy', 'f1_macro']#, 'precision_macro', 'recall_macro']"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mq30RtWYToeu",
        "colab_type": "text"
      },
      "source": [
        "#Classification Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IstS2gTeY7pg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NLeArq-uThHW",
        "colab_type": "text"
      },
      "source": [
        "##Feature Extraction Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WuSNj6YIEhu0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.base import TransformerMixin"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mm95T4CsDxaN",
        "colab_type": "text"
      },
      "source": [
        "###Statistical functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vWCPUON8D1A8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import scipy.stats as stats\n",
        "\n",
        "def rms(x):\n",
        "  '''\n",
        "  root mean square\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.sqrt(np.mean(np.square(x)))\n",
        "\n",
        "def sra(x):\n",
        "  '''\n",
        "  square root amplitude\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.mean(np.sqrt(np.absolute(x)))**2\n",
        "\n",
        "def ppv(x):\n",
        "  '''\n",
        "  peak to peak value\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.max(x)-np.min(x)\n",
        "\n",
        "def cf(x):\n",
        "  '''\n",
        "  crest factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.max(np.absolute(x))/rms(x)\n",
        "\n",
        "def ifa(x):\n",
        "  '''\n",
        "  impact factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.max(np.absolute(x))/np.mean(np.absolute(x))\n",
        "\n",
        "def mf(x):\n",
        "  '''\n",
        "  margin factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return np.max(np.absolute(x))/sra(x)\n",
        "\n",
        "def sf(x):\n",
        "  '''\n",
        "  shape factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return rms(x)/np.mean(np.absolute(x))\n",
        "\n",
        "def kf(x):\n",
        "  '''\n",
        "  kurtosis factor\n",
        "  '''\n",
        "  x = np.array(x)\n",
        "  return stats.kurtosis(x)/(np.mean(x**2)**2)\n",
        "\n"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "njMb9HtUEBrI",
        "colab_type": "text"
      },
      "source": [
        "### Statistical Features from Time Domain"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oSN2_c28D_Zr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class StatisticalTime(TransformerMixin):\n",
        "  '''\n",
        "  Extracts statistical features from the time domain.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    return np.array([\n",
        "                     [\n",
        "                      rms(x), # root mean square\n",
        "                      sra(x), # square root amplitude\n",
        "                      stats.kurtosis(x), # kurtosis\n",
        "                      stats.skew(x), # skewness\n",
        "                      ppv(x), # peak to peak value\n",
        "                      cf(x), # crest factor\n",
        "                      ifa(x), # impact factor\n",
        "                      mf(x), # margin factor\n",
        "                      sf(x), # shape factor\n",
        "                      kf(x), # kurtosis factor\n",
        "                      ] for x in X[:,:,0]\n",
        "                     ])\n"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dXDWD3JZEnep",
        "colab_type": "text"
      },
      "source": [
        "### Statistical Features from Frequency Domain"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sj3XTpVTEvAp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class StatisticalFrequency(TransformerMixin):\n",
        "  '''\n",
        "  Extracts statistical features from the frequency domain.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    sig = []\n",
        "    for x in X[:,:,0]:\n",
        "      fx = np.absolute(np.fft.fft(x)) # transform x from time to frequency domain\n",
        "      fc = np.mean(fx) # frequency center\n",
        "      sig.append([\n",
        "                  fc, # frequency center\n",
        "                  rms(fx), # RMS from the frequency domain\n",
        "                  rms(fx-fc), # Root Variance Frequency\n",
        "                  ])\n",
        "    return np.array(sig)\n"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0YBmzTb6ARb",
        "colab_type": "text"
      },
      "source": [
        "###Statistical Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kep4ubkR6DR0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Statistical(TransformerMixin):\n",
        "  '''\n",
        "  Extracts statistical features from both time and frequency domain.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    st = StatisticalTime()\n",
        "    stfeats = st.transform(X)\n",
        "    sf = StatisticalFrequency()\n",
        "    sffeats = sf.transform(X)\n",
        "    return np.concatenate((stfeats,sffeats),axis=1)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZuiVsHNzFORr",
        "colab_type": "text"
      },
      "source": [
        "###Wavelet Package Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oPd92xtJhaH3",
        "colab_type": "code",
        "cellView": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pywt\n",
        "\n",
        "class WaveletPackage(TransformerMixin):\n",
        "  '''\n",
        "  Extracts Wavelet Package features.\n",
        "  The features are calculated by the energy of the recomposed signal\n",
        "  of the leaf nodes coeficients.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    def Energy(coeffs, k):\n",
        "      return np.sqrt(np.sum(np.array(coeffs[-k]) ** 2)) / len(coeffs[-k])\n",
        "    def getEnergy(wp):\n",
        "      coefs = np.asarray([n.data for n in wp.get_leaf_nodes(True)])\n",
        "      return np.asarray([Energy(coefs,i) for i in range(2**wp.maxlevel)])\n",
        "    return np.array([getEnergy(pywt.WaveletPacket(data=x, wavelet='db4', \n",
        "                                                  mode='symmetric', maxlevel=4)\n",
        "                                                  ) for x in X[:,:,0]])"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_sonHkjFYbB",
        "colab_type": "text"
      },
      "source": [
        "###Heterogeneus Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IZsZhuVfFZsQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Heterogeneous(TransformerMixin):\n",
        "  '''\n",
        "  Mixes Statistical and Wavelet Package features.\n",
        "  '''\n",
        "  def fit(self, X, y=None):\n",
        "    return self\n",
        "  def transform(self, X, y=None):\n",
        "    st = StatisticalTime()\n",
        "    stfeats = st.transform(X)\n",
        "    sf = StatisticalFrequency()\n",
        "    sffeats = sf.transform(X)\n",
        "    wp = WaveletPackage()\n",
        "    wpfeats = wp.transform(X)\n",
        "    return np.concatenate((stfeats,sffeats,wpfeats),axis=1)\n"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1bnyL67EUcxV",
        "colab_type": "text"
      },
      "source": [
        "##K-NN with Heterogeneous Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F--sjKZRUh5G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "fcb36a08-9664-4d60-a877-2f8a68a32bc6"
      },
      "source": [
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "knn = Pipeline([\n",
        "                ('FeatureExtraction', Heterogeneous()),\n",
        "                ('scaler', StandardScaler()),\n",
        "                ('knn', KNeighborsClassifier()),\n",
        "                ])\n",
        "\n",
        "parameters = {'knn__n_neighbors': list(range(1,16,2))}\n",
        "if not debug:\n",
        "  knn = GridSearchCV(knn, parameters, verbose=verbose)\n",
        "else:\n",
        "  knn = GridSearchCV(knn, {'knn__n_neighbors': list(range(1,4,2))}, verbose=verbose)\n",
        "knn"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=Pipeline(steps=[('FeatureExtraction',\n",
              "                                        <__main__.Heterogeneous object at 0x7ffaee193d30>),\n",
              "                                       ('scaler', StandardScaler()),\n",
              "                                       ('knn', KNeighborsClassifier())]),\n",
              "             param_grid={'knn__n_neighbors': [1, 3, 5, 7, 9, 11, 13, 15]})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d7DfMTS_ujeE",
        "colab_type": "text"
      },
      "source": [
        "##SVM with Heterogeneous Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6v_wXxiDupvF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "d198d0b0-4b20-4037-940d-c793b9d1c9b5"
      },
      "source": [
        "from sklearn.svm import SVC\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "svm = Pipeline([\n",
        "                ('FeatureExtraction', Heterogeneous()),\n",
        "                ('scaler', StandardScaler()),\n",
        "                ('svc', SVC()),\n",
        "                ])\n",
        "\n",
        "parameters = {\n",
        "    'svc__C': [10**x for x in range(-3,2)],\n",
        "    'svc__gamma': [10**x for x in range(-3,1)],\n",
        "    }\n",
        "if not debug:\n",
        "  svm = GridSearchCV(svm, parameters, verbose=verbose)\n",
        "svm"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=Pipeline(steps=[('FeatureExtraction',\n",
              "                                        <__main__.Heterogeneous object at 0x7ffaee403550>),\n",
              "                                       ('scaler', StandardScaler()),\n",
              "                                       ('svc', SVC())]),\n",
              "             param_grid={'svc__C': [0.001, 0.01, 0.1, 1, 10],\n",
              "                         'svc__gamma': [0.001, 0.01, 0.1, 1]})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yU36xsi4JGZv",
        "colab_type": "text"
      },
      "source": [
        "##Random Forest with Heterogeneous Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GXABo6HpJJY_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "41e9b1ad-c008-4c83-b8b6-a8d9385e9f4a"
      },
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "rf = Pipeline([\n",
        "               ('FeatureExtraction', Heterogeneous()),\n",
        "               ('scaler', StandardScaler()),\n",
        "               ('rf', RandomForestClassifier()),\n",
        "               ])\n",
        "\n",
        "parameters = {\n",
        "    \"rf__n_estimators\": [10, 20, 50, 100, 200, 500],\n",
        "    \"rf__max_features\": [1, 5, 10, 15, 20], #list(range(1,21)),\n",
        "    }\n",
        "if not debug:\n",
        "  rf = GridSearchCV(rf, parameters, verbose=verbose)\n",
        "rf"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=Pipeline(steps=[('FeatureExtraction',\n",
              "                                        <__main__.Heterogeneous object at 0x7ffaff7a9910>),\n",
              "                                       ('scaler', StandardScaler()),\n",
              "                                       ('rf', RandomForestClassifier())]),\n",
              "             param_grid={'rf__max_features': [1, 5, 10, 15, 20],\n",
              "                         'rf__n_estimators': [10, 20, 50, 100, 200, 500]})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "REy6ykvWSbJc",
        "colab_type": "text"
      },
      "source": [
        "##Convolutional Neural Network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8YpHSjvNcEx5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8a111264-ff86-4a96-ca5b-3e82add0d389"
      },
      "source": [
        "try:\n",
        "  %tensorflow_version 2.x\n",
        "except:\n",
        "  print(\"Out of Colab\")"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Out of Colab\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FF1hCxJ9b5h0",
        "colab_type": "text"
      },
      "source": [
        "###F1-score macro averaged implemented for Keras"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LCJErrQIcIZ0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras import backend as K\n",
        "\n",
        "def f1_score_macro(y_true,y_pred):\n",
        "    def recall(y_true, y_pred):\n",
        "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
        "        recall = true_positives / (possible_positives + K.epsilon())\n",
        "        return recall\n",
        "    def precision(y_true, y_pred):\n",
        "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
        "        precision = true_positives / (predicted_positives + K.epsilon())\n",
        "        return precision\n",
        "    precision = precision(y_true, y_pred)\n",
        "    recall = recall(y_true, y_pred)\n",
        "    return 2*((precision*recall)/(precision+recall+K.epsilon()))"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PeLXSU7-cLfk",
        "colab_type": "text"
      },
      "source": [
        "###ANN wrapped in a scikit-learn estimator."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ny7otiW6Siz_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "from sklearn.base import BaseEstimator, ClassifierMixin\n",
        "import numpy as np\n",
        "from tensorflow.keras.callbacks import EarlyStopping,ReduceLROnPlateau\n",
        "\n",
        "class ANN(BaseEstimator, ClassifierMixin):\n",
        "  def __init__(self, \n",
        "               dense_layer_sizes=[64], \n",
        "               kernel_size=32, \n",
        "               filters=32, \n",
        "               n_conv_layers=2,\n",
        "               pool_size=8,\n",
        "               dropout=0.25,\n",
        "               epochs=50,\n",
        "               validation_split=0.05,\n",
        "               optimizer='sgd'#'nadam'#'rmsprop'#\n",
        "               ):\n",
        "    self.dense_layer_sizes = dense_layer_sizes\n",
        "    self.kernel_size = kernel_size\n",
        "    self.filters = filters\n",
        "    self.n_conv_layers = n_conv_layers\n",
        "    self.pool_size = pool_size\n",
        "    self.dropout = dropout\n",
        "    self.epochs = epochs\n",
        "    self.validation_split = validation_split\n",
        "    self.optimizer = optimizer\n",
        "  \n",
        "  def fit(self, X, y=None):\n",
        "    dense_layer_sizes = self.dense_layer_sizes\n",
        "    kernel_size = self.kernel_size\n",
        "    filters = self.filters\n",
        "    n_conv_layers = self.n_conv_layers\n",
        "    pool_size = self.pool_size\n",
        "    dropout = self.dropout\n",
        "    epochs = self.epochs\n",
        "    optimizer = self.optimizer\n",
        "    validation_split = self.validation_split\n",
        "\n",
        "    self.labels, ids = np.unique(y, return_inverse=True)\n",
        "    y_cat = to_categorical(ids)\n",
        "    num_classes = y_cat.shape[1]\n",
        "    \n",
        "    self.model = Sequential()\n",
        "    self.model.add(layers.InputLayer(input_shape=(X.shape[1],X.shape[-1])))\n",
        "    for _ in range(n_conv_layers):\n",
        "      self.model.add(layers.Conv1D(filters, kernel_size))#, padding='valid'))\n",
        "      self.model.add(layers.Activation('relu'))\n",
        "      if pool_size>1:\n",
        "        self.model.add(layers.MaxPooling1D(pool_size=pool_size))\n",
        "    #self.model.add(layers.Dropout(0.25))\n",
        "    self.model.add(layers.Flatten())\n",
        "    for layer_size in dense_layer_sizes:\n",
        "        self.model.add(layers.Dense(layer_size))\n",
        "        self.model.add(layers.Activation('relu'))\n",
        "    if dropout>0 and dropout<1:\n",
        "      self.model.add(layers.Dropout(dropout))\n",
        "    self.model.add(layers.Dense(num_classes))\n",
        "    self.model.add(layers.Activation('softmax'))\n",
        "    self.model.compile(loss='categorical_crossentropy',\n",
        "                       optimizer=optimizer,\n",
        "                       metrics=[f1_score_macro])\n",
        "    if validation_split>0 and validation_split<1:\n",
        "      prop = int(1/validation_split)\n",
        "      mask = np.array([i%prop==0 for i in range(len(y))])\n",
        "      self.history = self.model.fit(X[~mask], y_cat[~mask], epochs=epochs, \n",
        "                                    validation_data=(X[mask],y_cat[mask]),\n",
        "                                    callbacks=[EarlyStopping(patience=3), ReduceLROnPlateau()],\n",
        "                                    verbose=False\n",
        "                                    )  \n",
        "    else:\n",
        "      self.history = self.model.fit(X, y_cat, epochs=epochs, verbose=False)  \n",
        "  \n",
        "  def predict_proba(self, X, y=None):\n",
        "    return self.model.predict(X)\n",
        "\n",
        "  def predict(self, X, y=None):\n",
        "    predictions = self.model.predict(X)\n",
        "    return self.labels[np.argmax(predictions,axis=1)]"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UL5Zv2-xcgEG",
        "colab_type": "text"
      },
      "source": [
        "###ANN instantiation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MmQe7jeRcb6F",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "84682a4e-d768-439d-ddf4-63853c6f3b15"
      },
      "source": [
        "parameters = {\n",
        "    'filters': [16, 32],\n",
        "    'kernel_size': [16, 32],\n",
        "    'n_conv_layers': [1, 2],\n",
        "    #'pool_size': [2, 4, 6, 8],\n",
        "    }\n",
        "ann = ANN()\n",
        "if not debug:\n",
        "  ann = GridSearchCV(ann, parameters, verbose=verbose)\n",
        "ann"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=ANN(),\n",
              "             param_grid={'filters': [16, 32], 'kernel_size': [16, 32],\n",
              "                         'n_conv_layers': [1, 2]})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4SQenKxoSkME",
        "colab_type": "text"
      },
      "source": [
        "##List of Estimators"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vb6AGKFJJqvL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "clfs = [\n",
        "        # ('KNN - KNeighborsClassifier, Heterogeneous Features', knn),\n",
        "        # ('SVM - SVC with Heterogeneous Features', svm),\n",
        "        # ('ANN - Artificial Neural Network with Convolutional Layers', ann),\n",
        "        ('RF - RandomForestClassifier with Heterogeneous Features', rf),\n",
        "        ]\n",
        "if not debug:\n",
        "  dirres = 'cwru_rf'\n",
        "  # dirres = 'cwru_res'\n",
        "else:\n",
        "  dirres = 'debugres'"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dEl_vSYaq-s2"
      },
      "source": [
        "#Performing Experiments"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CH4LVC3Zj3jC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "6c9dbb69-d06b-4067-c7ff-3671ed2e8497"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "scores = {}\n",
        "trtime = {}\n",
        "tetime = {}\n",
        "# Estimators\n",
        "for clf_name, estimator in clfs:\n",
        "  if clf_name not in scores:\n",
        "    scores[clf_name] = {}\n",
        "    trtime[clf_name] = {}\n",
        "    tetime[clf_name] = {}\n",
        "  print(\"*\"*(len(clf_name)+8),'\\n***',clf_name,'***\\n'+\"*\"*(len(clf_name)+8))\n",
        "  # Validation forms\n",
        "  for val_name in validations.keys():\n",
        "    print(\"#\"*(len(val_name)+8),'\\n###',val_name,'###\\n'+\"#\"*(len(val_name)+8))\n",
        "    # Number of repetitions\n",
        "    for r in range(validations[val_name].rounds):\n",
        "      round_str = \"Round {}\".format(r+1)\n",
        "      print(\"@\"*(len(round_str)+8),'\\n@@@',round_str,'@@@\\n'+\"@\"*(len(round_str)+8))\n",
        "      groups = validations[val_name].groups\n",
        "      if val_name not in scores[clf_name]:\n",
        "        scores[clf_name][val_name] = {}\n",
        "      validation = eval(validations[val_name].splitter_name\n",
        "                        +'(4,shuffle='+str(validations[val_name].shuffle)\n",
        "                        +',random_state='+str(random_state+r)+')')\n",
        "      score = experimenter(estimator, X, selected_y, groups, \n",
        "                           scoring, validation, verbose)\n",
        "      for metric,s in score.items():\n",
        "        print(metric, ' \\t', s)\n",
        "        if metric not in scores[clf_name][val_name]:\n",
        "          scores[clf_name][val_name][metric] = []\n",
        "        scores[clf_name][val_name][metric].append(s)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "*************************************************************** \n",
            "*** RF - RandomForestClassifier with Heterogeneous Features ***\n",
            "***************************************************************\n",
            "#################### \n",
            "### Usual K-Fold ###\n",
            "####################\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 1 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [3927.1396749019623, 3926.9082491397858, 3922.4425542354584, 3890.117686986923]\n",
            "score_time  \t [9.353840112686157, 9.795709609985352, 9.666334867477417, 9.215996742248535]\n",
            "test_accuracy  \t [0.9887070168760309, 0.98997461928934, 0.9866751269035533, 0.9881979695431472]\n",
            "test_f1_macro  \t [0.9886136086699508, 0.9901947681580123, 0.986953524688857, 0.9883360840427271]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 2 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [3897.857188463211, 3916.1644649505615, 3834.143933534622, 3819.244938135147]\n",
            "score_time  \t [9.50651240348816, 9.435830116271973, 9.096308469772339, 9.4651358127594]\n",
            "test_accuracy  \t [0.9883263545235377, 0.9871827411167513, 0.9871827411167513, 0.9901015228426396]\n",
            "test_f1_macro  \t [0.9883608348253965, 0.9874413624249705, 0.9873425362681483, 0.9899966059721018]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 3 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [3814.7653987407684, 3840.2795662879944, 3819.407868862152, 3839.219763278961]\n",
            "score_time  \t [9.151491403579712, 9.516995429992676, 9.212405443191528, 9.516690492630005]\n",
            "test_accuracy  \t [0.9883263545235377, 0.9875634517766497, 0.9884517766497461, 0.9885786802030457]\n",
            "test_f1_macro  \t [0.9883440181460252, 0.9877101771624337, 0.9888442981541188, 0.9886192131365508]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 4 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [3808.983441591263, 3969.9939017295837, 4037.284947872162, 3986.3975813388824]\n",
            "score_time  \t [9.403703689575195, 9.974312543869019, 9.711014032363892, 9.435908555984497]\n",
            "test_accuracy  \t [0.9876919172693821, 0.9902284263959391, 0.9869289340101522, 0.9887055837563452]\n",
            "test_f1_macro  \t [0.9876078716592412, 0.990513192151286, 0.987057167184373, 0.9888626833159644]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 5 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [3947.0644967556, 3906.2969613075256, 3875.4539771080017, 3888.7648227214813]\n",
            "score_time  \t [9.5697500705719, 9.350503921508789, 9.444551229476929, 9.442347764968872]\n",
            "test_accuracy  \t [0.9892145666793554, 0.9860406091370558, 0.9902284263959391, 0.9881979695431472]\n",
            "test_f1_macro  \t [0.9892790984399402, 0.9862119307489339, 0.9902912120810636, 0.9883950548411021]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 6 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [3820.27006816864, 3681.8083748817444, 3659.199379682541, 2798.5435979366302]\n",
            "score_time  \t [9.140054702758789, 9.149131536483765, 7.1341307163238525, 6.6482696533203125]\n",
            "test_accuracy  \t [0.987565029818551, 0.98997461928934, 0.9869289340101522, 0.9874365482233503]\n",
            "test_f1_macro  \t [0.9876976020722907, 0.9898417825129663, 0.9869844556366362, 0.9878141195353268]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 7 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [2804.366559743881, 2842.63476061821, 2831.7397742271423, 2773.038544178009]\n",
            "score_time  \t [6.896226406097412, 6.899433374404907, 6.891962289810181, 6.762506008148193]\n",
            "test_accuracy  \t [0.9884532419743688, 0.9885786802030457, 0.9862944162436548, 0.98997461928934]\n",
            "test_f1_macro  \t [0.9886298254910706, 0.9885760908191492, 0.9864342655085306, 0.9898799942990018]\n",
            "@@@@@@@@@@@@@@@ \n",
            "@@@ Round 8 @@@\n",
            "@@@@@@@@@@@@@@@\n",
            "fit_time  \t [2797.9537930488586, 2780.3243157863617, 2799.031191110611, 2836.1921730041504]\n",
            "score_time  \t [6.8698036670684814, 6.825961589813232, 6.896572113037109, 6.948557138442993]\n",
            "test_accuracy  \t [0.987565029818551, 0.9884517766497461, 0.9884517766497461, 0.9887055837563452]\n",
            "test_f1_macro  \t [0.9875351708470765, 0.988539246865983, 0.9886221962501263, 0.9888823429350329]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QJ-qe0MIhM-z",
        "colab_type": "text"
      },
      "source": [
        "##Save results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qrp8uvOonKpd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 629
        },
        "outputId": "e1b8b9af-97da-48ab-c064-6a859ace8d6e"
      },
      "source": [
        "from pathlib import Path\n",
        "\n",
        "clf = {}\n",
        "val = {}\n",
        "src = {}\n",
        "for c, clf_name in enumerate(scores.keys()):\n",
        "  if c not in clf:\n",
        "    clf[c] = clf_name\n",
        "  for v, val_name in enumerate(scores[clf_name].keys()):\n",
        "    if v not in val:\n",
        "      val[v] = val_name\n",
        "    for s, scr_name in enumerate(scores[clf_name][val_name].keys()):\n",
        "      scores[clf_name][val_name][scr_name] = np.array(scores[clf_name][val_name][scr_name])\n",
        "      if s not in src:\n",
        "        src[s] = scr_name\n",
        "      Path(dirres).mkdir(parents=True, exist_ok=True)\n",
        "      np.savetxt('{}/{}-{}-{}.txt'.format(dirres,clf_name,val_name,scr_name), \n",
        "                 scores[clf_name][val_name][scr_name], delimiter=',')\n",
        "      print('{}/{} - {} - {}\\n'.format(dirres,clf_name.split('-')[0],val_name,scr_name),\n",
        "            scores[clf_name][val_name][scr_name])\n"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cwru_rf/RF  - Usual K-Fold - fit_time\n",
            " [[3927.1396749  3926.90824914 3922.44255424 3890.11768699]\n",
            " [3897.85718846 3916.16446495 3834.14393353 3819.24493814]\n",
            " [3814.76539874 3840.27956629 3819.40786886 3839.21976328]\n",
            " [3808.98344159 3969.99390173 4037.28494787 3986.39758134]\n",
            " [3947.06449676 3906.29696131 3875.45397711 3888.76482272]\n",
            " [3820.27006817 3681.80837488 3659.19937968 2798.54359794]\n",
            " [2804.36655974 2842.63476062 2831.73977423 2773.03854418]\n",
            " [2797.95379305 2780.32431579 2799.03119111 2836.192173  ]]\n",
            "cwru_rf/RF  - Usual K-Fold - score_time\n",
            " [[9.35384011 9.79570961 9.66633487 9.21599674]\n",
            " [9.5065124  9.43583012 9.09630847 9.46513581]\n",
            " [9.1514914  9.51699543 9.21240544 9.51669049]\n",
            " [9.40370369 9.97431254 9.71101403 9.43590856]\n",
            " [9.56975007 9.35050392 9.44455123 9.44234776]\n",
            " [9.1400547  9.14913154 7.13413072 6.64826965]\n",
            " [6.89622641 6.89943337 6.89196229 6.76250601]\n",
            " [6.86980367 6.82596159 6.89657211 6.94855714]]\n",
            "cwru_rf/RF  - Usual K-Fold - test_accuracy\n",
            " [[0.98870702 0.98997462 0.98667513 0.98819797]\n",
            " [0.98832635 0.98718274 0.98718274 0.99010152]\n",
            " [0.98832635 0.98756345 0.98845178 0.98857868]\n",
            " [0.98769192 0.99022843 0.98692893 0.98870558]\n",
            " [0.98921457 0.98604061 0.99022843 0.98819797]\n",
            " [0.98756503 0.98997462 0.98692893 0.98743655]\n",
            " [0.98845324 0.98857868 0.98629442 0.98997462]\n",
            " [0.98756503 0.98845178 0.98845178 0.98870558]]\n",
            "cwru_rf/RF  - Usual K-Fold - test_f1_macro\n",
            " [[0.98861361 0.99019477 0.98695352 0.98833608]\n",
            " [0.98836083 0.98744136 0.98734254 0.98999661]\n",
            " [0.98834402 0.98771018 0.9888443  0.98861921]\n",
            " [0.98760787 0.99051319 0.98705717 0.98886268]\n",
            " [0.9892791  0.98621193 0.99029121 0.98839505]\n",
            " [0.9876976  0.98984178 0.98698446 0.98781412]\n",
            " [0.98862983 0.98857609 0.98643427 0.98987999]\n",
            " [0.98753517 0.98853925 0.9886222  0.98888234]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkb8XMN-Ht58",
        "colab_type": "text"
      },
      "source": [
        "##Average & Standard Deviation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hJxdjboqtuNb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "38f75cc3-b7c3-48ae-e9e9-9828492b6546"
      },
      "source": [
        "c,v,s = len(clf),len(val),len(src)\n",
        "for i in range(s):\n",
        "  print(src[i])\n",
        "  for k in range(v):\n",
        "    print('\\t'+val[k]+' ', end='')\n",
        "  print()\n",
        "  for j in range(c):\n",
        "    print(clf[j].split('-')[0], end='\\t')\n",
        "    for k in range(v):\n",
        "      print(\"{0:.3f} ({1:.3f})\".format(\n",
        "          scores[clf[j]][val[k]][src[i]].mean(),\n",
        "          scores[clf[j]][val[k]][src[i]].std()), end='\\t')\n",
        "    print()\n",
        "  print()"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fit_time\n",
            "\tUsual K-Fold \n",
            "RF \t3571.657 (483.917)\t\n",
            "\n",
            "score_time\n",
            "\tUsual K-Fold \n",
            "RF \t8.635 (1.201)\t\n",
            "\n",
            "test_accuracy\n",
            "\tUsual K-Fold \n",
            "RF \t0.988 (0.001)\t\n",
            "\n",
            "test_f1_macro\n",
            "\tUsual K-Fold \n",
            "RF \t0.988 (0.001)\t\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qZG2ZnLQAeJL",
        "colab_type": "text"
      },
      "source": [
        "## Experiment results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "manFfTfh_9Ta",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    }
  ]
}