Main_ResNet.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Main-ResNet.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GUbHaZ4GUQjs"
      },
      "source": [
        "# Install *treelite* to accelerate the tree model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "weF3SfTlUOpV"
      },
      "source": [
        "!pip --quiet install ../input/treelite/treelite-0.93-py3-none-manylinux2010_x86_64.whl"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UPX0WiPVUV6P"
      },
      "source": [
        "!pip --quiet install ../input/treelite/treelite_runtime-0.93-py3-none-manylinux2010_x86_64.whl"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJL9c6MjUbYn"
      },
      "source": [
        "# Import the required libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gXat-fQ1UdNR"
      },
      "source": [
        "import os\n",
        "import time\n",
        "import pickle\n",
        "import random\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from tqdm import tqdm\n",
        "from sklearn.metrics import log_loss, roc_auc_score\n",
        "import gc\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.autograd import Variable\n",
        "from torch.utils.data import DataLoader\n",
        "from torch.nn import CrossEntropyLoss, MSELoss\n",
        "from torch.nn.modules.loss import _WeightedLoss\n",
        "import torch.nn.functional as F\n",
        "from sklearn.metrics import roc_auc_score, accuracy_score\n",
        "import treelite\n",
        "import treelite_runtime \n",
        "import warnings"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "awSScv4aVFEe"
      },
      "source": [
        "# Display settings for note rows and warnings\n",
        "1. By default, Note will replace the out-of-range rows with ... . Sometimes you need to export the data set for observation, so you can make this part visible after changing this setting.\n",
        "2. Some of the output of the code will be alerted (actually does not affect the results), set filter alerts can make the output much more readable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dMBmnreEVHeD"
      },
      "source": [
        "# Set the display range of the note row to 100\n",
        "pd.set_option('display.max_columns', 100)\n",
        "pd.set_option('display.max_rows', 100)\n",
        "\n",
        "# Output set to ignore alarms\n",
        "warnings.filterwarnings(\"ignore\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U0kRSDNRVKKs"
      },
      "source": [
        "# Hyperparameters definition"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lMv0UKT3VOUC"
      },
      "source": [
        "DATA_PATH = '../input/jane-street-market-prediction/'\n",
        "NFOLDS = 5\n",
        "GPU_FLAG = torch.cuda.is_available()\n",
        "TRAIN = False\n",
        "CACHE_PATH = '../input/mlp012003weights'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZQOsDdpNVQ41"
      },
      "source": [
        "# Define methods to be used in the training process\n",
        "\n",
        "## Method of model preservation\n",
        "\n",
        ">Save model is a common method in the competition, some models have a very long training process, if you do not want to kick to power or hand shake to turn off the page overnight back to the original phase, you should save the model timely. The trained model can be fine_tune, which can be loaded directly when the model is fused, saving the time of retraining.\n",
        "\n",
        "1. The parameter dic of save_pickle is the weight of the model. Note that it is a weight. The model needs to be instantiated first when loading. save_path is the relative path to save.\n",
        "2. with *open(save_path, 'wb') as f*, open the file with the path of save_path in wb mode and assign it to f. This way, you don't need to write a statement to close the file after saving.\n",
        "3. The pickle module implements binary serialization and deserialization of a Python object structure. dic is the incoming model weight, and the model weight is saved by writing it to f using pickle_dump method.\n",
        "4. Loading the model weights is done by deserializing the file contents with pikcle_load method and returning it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8AgxTp9NVTes"
      },
      "source": [
        "def save_pickle(dic, save_path):\n",
        "    with open(save_path, 'wb') as f:\n",
        "    # with gzip.open(save_path, 'wb') as f:\n",
        "        pickle.dump(dic, f)\n",
        "\n",
        "def load_pickle(load_path):\n",
        "    with open(load_path, 'rb') as f:\n",
        "    # with gzip.open(load_path, 'rb') as f:\n",
        "        message_dict = pickle.load(f)\n",
        "    return message_dict"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4F_kyzCmWEl-"
      },
      "source": [
        "## Set seeds in batch\n",
        ">During model development, it is sometimes useful to be able to obtain reproducible results in run after run to determine whether the change in performance comes from a change in the model or the dataset, or is simply the result of some new random sample points. To ensure that the training process is reproducible, define a seed_everything function which sets the seed for generating random numbers is necessary, which can follow the following steps:\n",
        "\n",
        "1. **random.seed( )** -> Set the random seeds.\n",
        "2. **os.environ['PYTHONHASHSEED']** -> Set hash seeds.\n",
        "3. **np.random.seed( )** -> Set *numpy* seeds.\n",
        "4. **torch.manual_seed( )** -> Set seeds of the random numbers generated by CPU.\n",
        "5. **torch.cuda.manual_seed(seed)** -> Set seeds of the current random numbers generated by GPU.\n",
        "6. **torch.cuda.manual_seed_all( )** -> Set seeds of random numbers if having multi-GPUs.\n",
        "7. **torch.backends.cudnn.deterministic** -> When the flag is True, the algorithm of the neural network calculation process is consistent, otherwise, the same network structure and superparameter may run out of different weights."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pszbuNGNWFZh"
      },
      "source": [
        "def seed_everything(seed=42):\n",
        "    random.seed(seed)\n",
        "    os.environ['PYTHONHASHSEED'] = str(seed)\n",
        "    np.random.seed(seed)\n",
        "    torch.manual_seed(seed)\n",
        "    torch.cuda.manual_seed(seed)\n",
        "    torch.backends.cudnn.deterministic = True\n",
        "seed_everything(seed=42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SYN7Er0sWIHY"
      },
      "source": [
        "# Dataset loading\n",
        ">The data needs to be loaded during training and evaluation which can be controlled without loading data during the model fusion phase with the flag 'TRAIN'.\n",
        "\n",
        "The training set for this competition has 5.77G data, which is slow to load with the general read_csv method of pandas. There are two ways to speed it up.\n",
        "\n",
        "1. Using datatable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dSd7HoWlWKj5"
      },
      "source": [
        "import datatable as dt\n",
        "train = dt.fread('../input/jane-street-market-prediction/train.csv').to_pandas()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q197AeyRWOvl"
      },
      "source": [
        "2. Using parquet from pandas"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NSb-bH93WRoY"
      },
      "source": [
        "import pandas as pd\n",
        "train.to_parquet('train.parquet')\n",
        "train = pd.read_parquet('./train.parquet')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c5hmfhpDWSQf"
      },
      "source": [
        "# EDA (Exploratory Data Analysis)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SvTHFn6KWZi4"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTpx9ohFWbub"
      },
      "source": [
        "# Data pre-processing\n",
        "\n",
        "1. We already know that the training data set features 130 dimensions, the law is the beginning of feature_, followed by the number from 0 to 129. So first we use the row expression to extract the column names, which will be used in the submission."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XknZw5TFWye1"
      },
      "source": [
        "feat_cols = [f'feature_{i}' for i in range(130)]\n",
        "# Which can also be code as below\n",
        "feat_cols = [f for f in train.columns if 'feature_' in f]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bTWSqEdfW9x8"
      },
      "source": [
        "2. The cumulative return of the first 85 days is found to be inconsistent with the cumulative return curve from 86 to 500 days by EDA. It can be assumed that the features of the first 85 days will be unfavorable to the generalization of the model. Also, of course it may be favorable to the generalization of the model. It mainly depends on the similarity of the test set features to the features of the first 85 days. From the submission of the public list removing the features of the first 85 days can improve the score, indicating that the validation set of the public list is not similar to the features of the first 85 day features are not similar. However, removing the first 85 days of data for training is a controversial point. Some players believe that features that similar to the first 85 days may appear in the future. If the model is not trained, it will overfit the validation data of the public list and weaken the generalization ability of the model in the test data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KzhdUIScXvf6"
      },
      "source": [
        "train = train.loc[train.date > 85].reset_index(drop=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QIquqj6tXxwD"
      },
      "source": [
        "3. This step is used for label establishing. We construct a 5-dimensional label with the 5 resp given by the train_dataset. EDA can find out whether the median of resp and 5 resp is similarly greater than 0. That is, the generalization ability of the model can be improved by predicting median of 5 labels. Because the probability that multiple labels are predicted wrong is smaller than the probability that only one label is predicted wrong.\n",
        "\n",
        "* Convert the value of resp to [0,1]\n",
        "    1. **train['resp'] > 0** -> Get a bool sequence.\n",
        "    2. **( ).astype('int')** -> Convert a bool sequence to an integer. True denotes 1. False denotes 0.\n",
        "\n",
        "* The generation of y uses *stack*, row expressions and transpose\n",
        "    1. The function of the row expression *[train[c] for c in resp_cols]* is to generate an array of (5,n) from the 5 columns of target_cols using the row expression.\n",
        "    2. *np.stack* splices the data in the specified dimension to get an array of ( ,n), default to be 0 dimension. It can be refer from https://numpy.org/doc/stable/reference/generated/numpy.stack.html.\n",
        "    3. After transposing, we get an array of (n, )."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h_I7fH0Ec4j7"
      },
      "source": [
        "target_cols = ['action', 'action_1', 'action_2', 'action_3', 'action_4']\n",
        "\n",
        "train['action'] = (train['resp'] > 0).astype('int')\n",
        "train['action_1'] = (train['resp_1'] > 0).astype('int')\n",
        "train['action_2'] = (train['resp_2'] > 0).astype('int')\n",
        "train['action_3'] = (train['resp_3'] > 0).astype('int')\n",
        "train['action_4'] = (train['resp_4'] > 0).astype('int')\n",
        "\n",
        "y = np.stack([train[c] for c in resp_cols]).T"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ri_HqIOc6-5"
      },
      "source": [
        "4. Add the features obtained by feature engineering. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "71n84q1adPb2"
      },
      "source": [
        "train['cross_41_42_43'] = train['feature_41'] + train['feature_42'] + train['feature_43']\n",
        "train['cross_1_2'] = train['feature_1'] / (train['feature_2'] + 1e-5)\n",
        "\n",
        "all_feat_cols = feat_cols + ['cross_41_42_43', 'cross_1_2'] "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sJr2XJ-tdSQn"
      },
      "source": [
        "4. Load the mean value of the features computed from the train_dataset and use it to fill in the null values when submit.\n",
        "\n",
        "  >When the data features are analyzed, some of the features are found to have null values. The neural network model needs to process the null values. Here the mean value is used to fill the test set with the same mean value as the training set to ensure consistency.\n",
        "\n",
        "* Note several scenarios for calculating the mean value\n",
        "  1. Calculate the mean value of the full amount of train_dataset\n",
        "  2. Calculate the mean of the actual training set + validation set with 85 days of dataset removed.\n",
        "  3. Calculate the mean of the training set with the validation set data removed.\n",
        "    \n",
        "  >If the difference in effect is not significant, just be careful to ensure that the filling at training is consistent with the filling at prediction.\n",
        "\n",
        "  >The parameters of *np.save* are path and array type object, so use *.values* to get the array object of DataFrame. It can be refer from https://numpy.org/doc/stable/reference/generated/numpy.save.html."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tSe2XNQhfauN"
      },
      "source": [
        "if TRAIN:\n",
        "    # Calculate the mean value of the train_dataset and save it, so that there is no need to load the train_dataset and recalculate it during inference.\n",
        "    f_mean = train.mean()\n",
        "    np.save(f'{CACHE_PATH}/f_mean_online.npy', f_mean.values)\n",
        "else:\n",
        "    f_mean = np.load(f'{CACHE_PATH}/f_mean_online.npy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kov-axWFfnJe"
      },
      "source": [
        "# Construct the training and validation sets\n",
        "\n",
        "Since the official validation set is not labeled (it can be treated as a test set), you need to construct a validation set by yourself, using 450 to 500 days of data as validation, or you can choose another time, just be careful not to leak the label.\n",
        "\n",
        ">***Personal opinion***: although the data provided by the competition has date as a time sequence (not a feature of training), the returns of each transaction are independent of each other (not relate to date), so there is no need to ensure the time sequence of the training data and it is possible to disrupt the division of the training and validation sets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sEybnL7Dgbnb"
      },
      "source": [
        "train.fillna(f_mean, inplace=True)\n",
        "valid = train.loc[(train.date >= 450) & (train.date < 500)].reset_index(drop=True)\n",
        "train = train.loc[train.date < 450].reset_index(drop=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LLkG9LKTfv_s"
      },
      "source": [
        "# ***Define the network structure of ResNet***\n",
        "1. Inheritance of *nn.Module*\n",
        "2. Implement *init* and *forward*\n",
        "  - The *init* defines the layers that the model needs to use\n",
        "  - The *forward* defines the process of turning features into labels\n",
        "\n",
        "## *init* defines layers:\n",
        "1. *super* calls Model's *init* to do the initialization. \n",
        "2. *batch_norm* -> *nn.BatchNorm1d* is used to create a batch normalization layer. The role of *nn.BatchNorm1d* is for data normalization, in which the deep neural network training process in each layer of the neural network input can maintain the same distribution. Because the distribution of the data in the neural network changes after each layer is activated, which is called Internal Covariate Shift,. The input distribution of the middle layer always changes, which will increase the difficulty of fitting the model. Also, the input distribution of the middle layer will make the output gradually close to the place where the gradient of the activation function is smaller, leading to the disappearance of the gradient, so it is necessary to do data normalization for the output of each layer. It is important to note that the BN layer is placed between the hidden layer and the activation layer.\n",
        "3. *dropout* -> Dropout layer is created using *nn.Dropout* which serves to randomly extinguish some neurons (changing the output of some neurons to 0) as well as reducing overfitting.\n",
        "4. *dense1* -> Create a fully connected layer using *nn.Linea*. The first parameter is the input dimension and the second parameter is the output dimension\n",
        "5. *LeakyReLU* -> Create an activation layer with the activation function ReLU provided by *nn*\n",
        "\n",
        "## *forward* definition forward propagation:\n",
        "\n",
        ">Instead of using the classic block structure of ResNet, I just borrow the idea of ResNet, where each block contains only one hidden layer, and the input of each layer is obtained by splicing the input of the previous layer with the output of the previous layer. By stacking the layers defined in the *init* according to the designed model structure, the input dimension is n * 130 dimensions and the feature x is transformed by each layer to obtain the output in n * 5 dimensions.\n",
        "\n",
        "1. **input layer** -> Contains *batch_norm* layer and dropout layer\n",
        "2. **4 blocks** -> Each contains a fully-connected layer, a *batch_norm* operation layer, an activation layer, and a dropout layer\n",
        "3. **output layer** -> Contains a fully-connected layer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jt7iIuEkOC1P"
      },
      "source": [
        "class Model(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Model, self).__init__()\n",
        "        self.batch_norm0 = nn.BatchNorm1d(len(all_feat_cols))\n",
        "        self.dropout0 = nn.Dropout(0.2)\n",
        "\n",
        "        dropout_rate = 0.2\n",
        "        hidden_size = 256\n",
        "        self.dense1 = nn.Linear(len(all_feat_cols), hidden_size)\n",
        "        self.batch_norm1 = nn.BatchNorm1d(hidden_size)\n",
        "        self.dropout1 = nn.Dropout(dropout_rate)\n",
        "\n",
        "        self.dense2 = nn.Linear(hidden_size+len(all_feat_cols), hidden_size)\n",
        "        self.batch_norm2 = nn.BatchNorm1d(hidden_size)\n",
        "        self.dropout2 = nn.Dropout(dropout_rate)\n",
        "\n",
        "        self.dense3 = nn.Linear(hidden_size+hidden_size, hidden_size)\n",
        "        self.batch_norm3 = nn.BatchNorm1d(hidden_size)\n",
        "        self.dropout3 = nn.Dropout(dropout_rate)\n",
        "\n",
        "        self.dense4 = nn.Linear(hidden_size+hidden_size, hidden_size)\n",
        "        self.batch_norm4 = nn.BatchNorm1d(hidden_size)\n",
        "        self.dropout4 = nn.Dropout(dropout_rate)\n",
        "\n",
        "        self.dense5 = nn.Linear(hidden_size+hidden_size, len(target_cols))\n",
        "\n",
        "        self.Relu = nn.ReLU(inplace=True)\n",
        "        self.PReLU = nn.PReLU()\n",
        "        self.LeakyReLU = nn.LeakyReLU(negative_slope=0.01, inplace=True)\n",
        "        # self.GeLU = nn.GELU()\n",
        "        self.RReLU = nn.RReLU()\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.batch_norm0(x)\n",
        "        x = self.dropout0(x)\n",
        "\n",
        "        x1 = self.dense1(x)\n",
        "        x1 = self.batch_norm1(x1)\n",
        "        x1 = self.LeakyReLU(x1)\n",
        "        x1 = self.dropout1(x1)\n",
        "\n",
        "        x = torch.cat([x, x1], 1)\n",
        "\n",
        "        x2 = self.dense2(x)\n",
        "        x2 = self.batch_norm2(x2)\n",
        "        x2 = self.LeakyReLU(x2)\n",
        "        x2 = self.dropout2(x2)\n",
        "\n",
        "        x = torch.cat([x1, x2], 1)\n",
        "\n",
        "        x3 = self.dense3(x)\n",
        "        x3 = self.batch_norm3(x3)\n",
        "        x3 = self.LeakyReLU(x3)\n",
        "        x3 = self.dropout3(x3)\n",
        "\n",
        "        x = torch.cat([x2, x3], 1)\n",
        "\n",
        "        x4 = self.dense4(x)\n",
        "        x4 = self.batch_norm4(x4)\n",
        "        x4 = self.LeakyReLU(x4)\n",
        "        x4 = self.dropout4(x4)\n",
        "\n",
        "        x = torch.cat([x3, x4], 1)\n",
        "\n",
        "        x = self.dense5(x)\n",
        "\n",
        "        return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xmfYxfcOL3X"
      },
      "source": [
        "# Define the early-stopping function\n",
        "\n",
        ">In order to prevent the model from overtraining and overfitting the training set, the training is stopped in time using this function when the evaluation index of the validation set does not improve anymore or even decreases.Define the early-stopping class, you can also call the package *from pytorchtools import EarlyStopping* to use directly. The implementation of the code is more or less the same.\n",
        "\n",
        "Explanation of early-stopping class:\n",
        "1. For several parameters of the *init* function, patience is the number of times the indicator is tolerated not to improve, mode is the measure to evaluate the improvement of the indicator, and delta is the floating coefficient of the compared indicator (maybe unnecessary to use).\n",
        "2. To turn a python class instance into a callable object, all you need to do is implement a special method *__call__( )* which is similar to overloading the *( )* operator in a class, allowing the class instance object to be used as an 'object name()' as if it were a normal function\n",
        "3. The logic of early-stopping is to use the early-stopping class to calculate whether the following evaluation metrics (AUC is used later) are elevated after each training with the validation set, if not, the counter = counter + 1, otherwise, the model is saved. When the counter reaches the upper limit of tolerance (which is denoted as patience), the flag of early-stopping is set to True. Then the early-stopping flag of the early-stopping class can be used to decide whether to stop the training process.\n",
        "4. *save_checkpoint* function is used to save the model weights. *model.state_dict( )* is used to get the model weights. We can also use the *torch.save* method to save in the specified *model_path* path. Note that when loading the model later, you need to create the model before loading.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d6RFkMQHP-Jr"
      },
      "source": [
        "class EarlyStopping:\n",
        "    def __init__(self, patience=7, mode=\"max\", delta=0.001):\n",
        "        self.patience = patience\n",
        "        self.counter = 0\n",
        "        self.mode = mode\n",
        "        self.best_score = None\n",
        "        self.early_stop = False\n",
        "        self.delta = delta\n",
        "        if self.mode == \"min\":\n",
        "            self.val_score = np.Inf\n",
        "        else:\n",
        "            self.val_score = -np.Inf\n",
        "\n",
        "    def __call__(self, epoch_score, model, model_path):\n",
        "\n",
        "        if self.mode == \"min\":\n",
        "            score = -1.0 * epoch_score\n",
        "        else:\n",
        "            score = np.copy(epoch_score)\n",
        "\n",
        "        if self.best_score is None:\n",
        "            self.best_score = score\n",
        "            self.save_checkpoint(epoch_score, model, model_path)\n",
        "        elif score < self.best_score: #  + self.delta\n",
        "            self.counter += 1\n",
        "            # print('EarlyStopping counter: {} out of {}'.format(self.counter, self.patience))\n",
        "            if self.counter >= self.patience:\n",
        "                self.early_stop = True\n",
        "        else:\n",
        "            self.best_score = score\n",
        "            # ema.apply_shadow()\n",
        "            self.save_checkpoint(epoch_score, model, model_path)\n",
        "            # ema.restore()\n",
        "            self.counter = 0\n",
        "\n",
        "    def save_checkpoint(self, epoch_score, model, model_path):\n",
        "        if epoch_score not in [-np.inf, np.inf, -np.nan, np.nan]:\n",
        "            # print('Validation score improved ({} --> {}). Saving model!'.format(self.val_score, epoch_score))\n",
        "            # if not DEBUG:\n",
        "            torch.save(model.state_dict(), model_path)\n",
        "        self.val_score = epoch_score"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94E7DCy0SAwH"
      },
      "source": [
        "# Define the loss function\n",
        "\n",
        ">Generally you can use the loss function that comes with PyTorch. PyTorch provides a rich set of loss functions, such as MSELoss, L1Loss for regression, CrossEntropyLoss for classification. Of course you can also customize the loss function. The custom loss function should be inherited from *torch.nn.Module* so that the backward method will be implemented automatically as soon as the forward method is set. The inheritance method of the loss function can be found in the follow URL: https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html.\n",
        "\n",
        "Our custom loss function incorporates a label smoothing mechanism (*label_smoothing*). The role of label smooth is saying that the neural network will prompt itself to learn in the direction of the largest difference between the correct label and the wrong label. In the case of less training data, not enough to characterize all the sample features, it will lead to network overfitting. Label smoothing implements a improvement, which is a regularization strategy, mainly through *soft one-hot* to add noise (generally, *one-hot* is labeled as 1 and other as 0, *soft one-hot* is labeled as a number less than 1 and other as number slightly greater than 0, so that the label is not so absolute). It can reduce the weight of the category of the real sample label in the calculation of the loss function and finally to suppress the effect of overfitting.\n",
        "\n",
        "Define a binary cross entropy loss function with *label_smoothing* -> A custom class inherits from *_WeightedLoss* (class *_WeightedLoss* inherits from *_Loss*, class *_Loss* inherits from *Model*). Add the *label_smoothing* methodv and redefine the *forward* method.\n",
        "\n",
        "Explanation of *_smooth* method:\n",
        "1. *@staticmethod* does not need to represent self parameter of its own object and cls parameter of its own class. It can be used like a function.\n",
        "2. *assert* is python's assertion method which is used while debugging and throws an exception based on the expression after assert. In that case you don't need to frame a bunch of code with if.\n",
        "3. Scaled down by target = 1.0 - smoothing (balance factor) when target is 1. Scaled up by target = 0.5 * smoothing when target is 0.\n",
        "\n",
        "Explanation of *forward* method:\n",
        "1. Before calculating the loss, call the *_smooth* method to smoothing the label.\n",
        "2. Call *torch.nn.functional.binary_cross_entropy_with_logits* to compute the binary cross-information entropy loss of inputs and targets.\n",
        "3. Returns the sum or mean of the losses based on the *reduction*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "76iL9JoVSVlg"
      },
      "source": [
        "##### Model&Data fnc\n",
        "class SmoothBCEwLogits(_WeightedLoss):\n",
        "    def __init__(self, weight=None, reduction='mean', smoothing=0.0):\n",
        "        super().__init__(weight=weight, reduction=reduction)\n",
        "        self.smoothing = smoothing\n",
        "        self.weight = weight\n",
        "        self.reduction = reduction\n",
        "\n",
        "    @staticmethod\n",
        "    def _smooth(targets:torch.Tensor, n_labels:int, smoothing=0.0):\n",
        "        assert 0 <= smoothing < 1\n",
        "        with torch.no_grad():\n",
        "            targets = targets * (1.0 - smoothing) + 0.5 * smoothing\n",
        "        return targets\n",
        "\n",
        "    def forward(self, inputs, targets):\n",
        "        targets = SmoothBCEwLogits._smooth(targets, inputs.size(-1),\n",
        "            self.smoothing)\n",
        "        loss = F.binary_cross_entropy_with_logits(inputs, targets,self.weight)\n",
        "\n",
        "        if  self.reduction == 'sum':\n",
        "            loss = loss.sum()\n",
        "        elif  self.reduction == 'mean':\n",
        "            loss = loss.mean()\n",
        "\n",
        "        return loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EO1kgE5KYcMx"
      },
      "source": [
        "# Data loading for PyTorch\n",
        "\n",
        "The sequence of operations for loading data into the model using PyTorch is as follows:\n",
        "1. Create a *Dataset* object, in this case a custom *MarketDataset* type.\n",
        "2. Create a *DataLoader* object.\n",
        "3. Loop this *DataLoader* object to load features and labels into the model for training.\n",
        "\n",
        "Creating a *Dataset* class requires the inclusion of at least 3 function:\n",
        "1. **__init__** -> Initialization. Pass in data via parameters, or load data in methods.\n",
        "2. **__len__** -> Return the total number of items in this dataset.\n",
        "3. **__getitem__** -> Remove the data in dataset specified by the idx parameter. Convert it to a tensor and return it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2YZhuo8XYacX"
      },
      "source": [
        "class MarketDataset:\n",
        "    def __init__(self, df):\n",
        "        self.features = df[all_feat_cols].values\n",
        "        self.label = df[target_cols].values.reshape(-1, len(target_cols))\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.label)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        return {\n",
        "            'features': torch.tensor(self.features[idx], dtype=torch.float),\n",
        "            'label': torch.tensor(self.label[idx], dtype=torch.float)\n",
        "        }"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQoG-vHoacOR"
      },
      "source": [
        "# Define the training function\n",
        "\n",
        "Parameter description:\n",
        "1. **model** -> Used for passing in the ResNet model defined by PyTorch.\n",
        "2. **optimizer** -> Used for passing in optimizers.\n",
        "3. **scheduler** -> Used for passing in learning rate to adjust objects.\n",
        "4. **loss_fn** -> Used for passing in loss function.\n",
        "5. **dataloader** -> Used for passing in the data loading object.\n",
        "6. **device** -> Used for passing in device objects.\n",
        "\n",
        "Training process:\n",
        "1. **model.train( )** -> Set the model to training mode. The effect is to enable *batch_normalization* and *drop_out*.\n",
        "2. **final_loss** -> Used to calculate the average loss of the whole training process.\n",
        "3. **for loop** -> Take one batch data at a time from the dataloader.\n",
        "4. **zero_grad( )** -> Clear the gradient of the previous batch training. This step must be performed for each batch.\n",
        "5. **to(device)** -> Copy data to GPU.\n",
        "6. **model(features)** -> Call the *forward* method of the model to get the results of the ResNet calculation.\n",
        "7. **loss_fn(outputs, label)** -> Use loss function to calculate the model prediction result and the loss of label, returning the *SmoothBCEwLogits* object.\n",
        "8. **loss.backward( )** -> Back propagation calculates the gradients.\n",
        "9. **optimizer.step( )** -> Optimizer updates model parameters.\n",
        "10. **scheduler.step( )** -> If a *scheduler* object is passed in, then aejust learning rate.\n",
        "11. **final_loss += loss.item( )** -> Record the sum of the loss values of all the batches, and use it to calculate the average of the loss of the whole training process.\n",
        "12. **final_loss /= len(dataloader)** -> Calculate the average of the loss of all the batches throughout the training process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DNKteEv6dUmr"
      },
      "source": [
        "def train_fn(model, optimizer, scheduler, loss_fn, dataloader, device):\n",
        "    model.train()\n",
        "    final_loss = 0\n",
        "\n",
        "    for data in dataloader:\n",
        "        optimizer.zero_grad()\n",
        "        features = data['features'].to(device)\n",
        "        label = data['label'].to(device)\n",
        "        outputs = model(features)\n",
        "        loss = loss_fn(outputs, label)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        if scheduler:\n",
        "            scheduler.step()\n",
        "\n",
        "        final_loss += loss.item()\n",
        "\n",
        "    final_loss /= len(dataloader)\n",
        "\n",
        "    return final_loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32ncrK4CdXOR"
      },
      "source": [
        "# Define the evaluation (inference) function\n",
        "\n",
        ">The difference between inference and training is that the gradient is not calculated and the model parameters aren't updated. The parameter passing is consistent with the above training function.\n",
        "\n",
        "The process of inference:\n",
        "1. **model.eval()** -> Set the model to evaluation mode where the model will not drop out.\n",
        "2. **preds** -> Used to record the prediction results for each validator batch.\n",
        "3. **for loop** -> Similar to training, the validation data is fetched by batch from the *Dataloader* through a for loop.\n",
        "4. **to(device)** -> Copy verification data to the GPU.\n",
        "5. **with torch.no_grad()** -> The role of the model is to let the model does not calculate the gradient and speed up the model's calculation speed. Otherwise it is easy to fully occupy the memory. \n",
        "6. **model(features)** -> Predicting results with models.\n",
        "7. **outputs.sigmoid()** -> The sigmoid function is used for activate prediction result because it is dichotomous, while softmax is used for multiclassification.\n",
        "8. **.detach()** -> Cutting down back propagation and returning a new variable which is separated from the currently computed graph. Getting this variable never needs to compute its gradient.\n",
        "9. **.cpu()** -> Copy the result from the GPU and convert it to Numpy type.\n",
        "10. **np.concatenate()** -> Splice the prediction results of all the batches and then reshape them into the dimension of *(n, len(target_cols))*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zTZT2zPYlmmc"
      },
      "source": [
        "def inference_fn(model, dataloader, device):\n",
        "    model.eval()\n",
        "    preds = []\n",
        "\n",
        "    for data in dataloader:\n",
        "        features = data['features'].to(device)\n",
        "\n",
        "        with torch.no_grad():\n",
        "            outputs = model(features)\n",
        "\n",
        "        preds.append(outputs.sigmoid().detach().cpu().numpy())\n",
        "\n",
        "    preds = np.concatenate(preds).reshape(-1, len(target_cols))\n",
        "\n",
        "    return preds"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gjoPd_76lq76"
      },
      "source": [
        "# Define the scoring function\n",
        "\n",
        ">Define the scoring function based on the officially given scoring index.\n",
        "\n",
        "$p_i = \\sum_j(weight_{ij} * resp_{ij} * action_{ij})$\n",
        "\n",
        "$t = \\frac{\\sum p_i }{\\sqrt{\\sum p_i^2}} * \\sqrt{\\frac{250}{|i|}}$\n",
        "\n",
        "$u = min(max(t,0),6) \\sum p_i$\n",
        "\n",
        "Explanation of *utility_score_bincount*:\n",
        "1. The parameters correspond to the date, weight, resp and action columns in the dataset.\n",
        "2. **np.unique(date)** -> Get the sequence without duplicate dates.\n",
        "3. **np.bincount( )** -> Obtain a sequence of cumulative returns by day. *bincount* can be found at the following URL: https://blog.csdn.net/xlinsist/article/details/51346523.\n",
        "4. **t,u** -> It is to calculate the *t* and *utility_score* according to the official formula."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i3VcQcgTmvmd"
      },
      "source": [
        "def utility_score_bincount(date, weight, resp, action):\n",
        "    count_i = len(np.unique(date))\n",
        "    Pi = np.bincount(date, weight * resp * action)\n",
        "    t = np.sum(Pi) / np.sqrt(np.sum(Pi ** 2)) * np.sqrt(250 / count_i)\n",
        "    u = np.clip(t, 0, 6) * np.sum(Pi)\n",
        "    return u"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VlNd_gYfmxhc"
      },
      "source": [
        "# Model training:\n",
        "1. Use TRAIN to mark whether to train or not. Skip this step if TRAIN = False when inferring.\n",
        "2. Create Dataset objects for training set *train* and validator *valid* respectively.\n",
        "3. Create *DataLoader* objects separately. *shuffle* marks whether to upset the order. *num_workers* specifies the number of parallel threads. *DataLoader* of the validation set does not upset the order in order to ensure the consistency of each validation data. Otherwise there will be interference when comparing the model effect.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yg7DSKpWm1Fn"
      },
      "source": [
        "if TRAIN:\n",
        "    train_set = MarketDataset(train)\n",
        "    train_loader = DataLoader(train_set, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)\n",
        "    valid_set = MarketDataset(valid)\n",
        "    valid_loader = DataLoader(valid_set, batch_size=BATCH_SIZE, shuffle=False, num_workers=4)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KOCTTrwRp8kl"
      },
      "source": [
        "1. The model is trained NFOLDS times. The *DataLoader* of the training set is disrupted each time so that multiple models are obtained for fusion and better generalization ability.\n",
        "2. **torch.cuda.empty_cache( )** -> Release memory that is no longer in use.\n",
        "3. **torch.device('cuda:0')** -> *torch.device* represents the object to which the *torch.Tensor* is assigned. If the GPU is enabled then it will be *cuda:0*, otherwise *cpu*. The number X of *cuda:X* can be obtained with *torch.cuda.current_device( )*.\n",
        "4. Create the *model* object of ResNet and copy the model into the previously obtained device object using *.to(device)*.\n",
        "5. Instantiate an optimizer which is used to update model parameters. The so-called optimization can be Adam algorithm. Instantiation provides ResNet model parameters. Learning rate and weight decay are the parameters of the optimizer. More information about PyTorch optimzer may refer to URL: https://pytorch-cn.readthedocs.io/zh/latest/package_references/torch-optim/.\n",
        "6. We can choose to instantiate a dynamic learning rate object. *torch.optim.lr_scheduler* module provides some methods to adjust the learning rate (*learning_rate*) according to the epoch of training. If the initial learning rate is set too small as the convergence speed is slow and training efficiency is low. However, setting a larger rate may be too easy to jitter. The role of this module is to adjust the learning rate bigger and then smaller. You can refer to this blog to deepen the understanding of the module: https://blog.csdn.net/qyhaill/article/details/103043637.\n",
        "7. Create the object of the loss function.\n",
        "8. Create early-stopping objects. \n",
        "9. *model_weights* is the path where the model weights are saved. *f\"{}...\"* indicates that python expressions within curly brackets are supported within strings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G0LyanAKp_aU"
      },
      "source": [
        "    start_time = time.time()\n",
        "    for _fold in range(NFOLDS):\n",
        "        print(f'Fold{_fold}:')\n",
        "        seed_everything(seed=42+_fold)\n",
        "        torch.cuda.empty_cache()\n",
        "        device = torch.device(\"cuda:0\")\n",
        "        model = Model()\n",
        "        model.to(device)\n",
        "        # model = nn.DataParallel(model)\n",
        "\n",
        "        optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)\n",
        "        # optimizer = Nadam(model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)\n",
        "        # optimizer = Lookahead(optimizer=optimizer, k=10, alpha=0.5)\n",
        "        scheduler = None\n",
        "        # scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer=optimizer, pct_start=0.1, div_factor=1e3,\n",
        "        #                                                 max_lr=1e-2, epochs=EPOCHS, steps_per_epoch=len(train_loader))\n",
        "        # loss_fn = nn.BCEWithLogitsLoss()\n",
        "        loss_fn = SmoothBCEwLogits(smoothing=0.005)\n",
        "\n",
        "        es = EarlyStopping(patience=EARLYSTOP_NUM, mode=\"max\")\n",
        "        \n",
        "        model_weights = f\"{CACHE_PATH}/online_model{_fold}.pth\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PxYWPvcoMHfi"
      },
      "source": [
        "# Training process\n",
        "\n",
        ">The above processes is in preparation for training, getting the module ready. The we start to train the model. We will train the model EPOCHS times, which is a previously defined hyperparameter.\n",
        "\n",
        "1. The module defined above is passed into *train_fn*. The training function is defined before, here only one statement is needed to complete the training. What a surprise!\n",
        "2. What is the latter code used for? It is to evaluate the effect of the model in the validation set and to stop early based on the evaluation,\n",
        "3. First we use *inferance_fn* to make predictions on the validation set with the trained model and get the prediction result *valid_pred*.\n",
        "4. Compute the AUC and logloss of the model on the validation dataset.\n",
        "5. The results of model prediction is 5 resp. To calculate the *utility_score*, we need to change 5 resp into 1 resp. We use *np.median* to take the median of the 5 resp and *utility_score_bincount* method to calculate *utility_score*.\n",
        "6. Pass the validation set's AUC, model and save path to the early-stopping module. If the early-stopping flag *early_stop* is True, the early-stopping condition is met and the training loop stops."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DuIalSLpMJXN"
      },
      "source": [
        "        for epoch in range(EPOCHS):\n",
        "            train_loss = train_fn(model, optimizer, scheduler, loss_fn, train_loader, device)\n",
        "\n",
        "            valid_pred = inference_fn(model, valid_loader, device)\n",
        "            valid_auc = roc_auc_score(valid[target_cols].values, valid_pred)\n",
        "            valid_logloss = log_loss(valid[target_cols].values, valid_pred)\n",
        "            valid_pred = np.median(valid_pred, axis=1)\n",
        "            valid_pred = np.where(valid_pred >= 0.5, 1, 0).astype(int)\n",
        "            valid_u_score = utility_score_bincount(date=valid.date.values, weight=valid.weight.values,\n",
        "                                                   resp=valid.resp.values, action=valid_pred)\n",
        "            print(f\"FOLD{_fold} EPOCH:{epoch:3} train_loss={train_loss:.5f} \"\n",
        "                      f\"valid_u_score={valid_u_score:.5f} valid_auc={valid_auc:.5f} \"\n",
        "                      f\"time: {(time.time() - start_time) / 60:.2f}min\")\n",
        "            es(valid_auc, model, model_path=model_weights)\n",
        "            if es.early_stop:\n",
        "                print(\"Early stopping\")\n",
        "                break\n",
        "        torch.save(model.state_dict(), model_weights)\n",
        "        "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6qVVl-KQeO4"
      },
      "source": [
        "# ResNet model loading\n",
        "\n",
        ">The model is trained and can be loaded directly if we want to use it later. We can use the following code to load the model during model fusion.\n",
        "\n",
        "Since we are saving the weights of the model, we need to create the model object first. The number of folds that training is divided into here is the number of models needed to loaded.\n",
        "\n",
        "1. Get the weights of the model with *torch.load(path)*.\n",
        "2. Use *model.load_state_dict( )* to put the weights into the model.\n",
        "3. Shift the model's mode to *eval*.\n",
        "4. Put in *model_list*, later we can directly use the model in *model_list* to make predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lspbW48SQg05"
      },
      "source": [
        "model_list = []\n",
        "for _fold in range(NFOLDS):\n",
        "    torch.cuda.empty_cache()\n",
        "    device = torch.device(\"cuda:0\")\n",
        "    model = Model()\n",
        "    model.to(device)\n",
        "    model_weights = f\"{CACHE_PATH}/online_model{_fold}.pth\"\n",
        "    model.load_state_dict(torch.load(model_weights))\n",
        "\n",
        "    model.eval()\n",
        "    model_list.append(model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JTl83XlU3Aqv"
      },
      "source": [
        "# ResNet Summary\n",
        "## Review the main steps\n",
        ">At this point we have completed the whole process from building -> training -> early-stopping and saving -> loading from the PyTorch version of the ResNet model. Here recall the main steps.\n",
        "\n",
        "1. Prepare module. ResNet model, loss function, early-stopping module, training process function, evaluation (inference) process function, scoring function, etc.\n",
        "2. Building blocks for model training.\n",
        "\n",
        "    Outermost loop: training NFOLDS times:\n",
        "        Prepare modules for each model: ResNet model, loss function, optimizer, early-stopping module, learning rate adjustment module.\n",
        "\n",
        "    innermost loop: Each model is trained EPOCHS times. For each train:\n",
        "        1) Pass the module to the training process function to get the trained model.\n",
        "        2) Give the trained model and validation set to the evaluation (inference) process function and get the evaluation result.\n",
        "        3) Give the model parameters and evaluation results to the early-stopping module. Save the model and update the early-stopping marker according to the evaluation.\n",
        "        4) Determine whether to stop model training based on early-stopping markers.\n",
        "           \n",
        "## Review of the main modules\n",
        "1. **ResNet model** -> The structure of the neural network. The goal of training is to get the weights of the network structure.\n",
        "2. **Loss function** -> The role is to calculate the Differences between the model and the label.\n",
        "3. **Optimizer** -> Calculate the gradients and modify model weights by policy.\n",
        "4. **Early-stopping Module** -> A module to prevent over-training of the model, i.e. It can prevent over-fitting and save training time.\n",
        "5. **Training process function** -> Feed the training data to the model by batch. Calculate the loss and update the weights until all batches are completed.\n",
        "       \n",
        "## Review of the main hyperparameters\n",
        "\n",
        "The tuning of the model is done mainly for the following hyperparameters:\n",
        "1. **layer_size** -> Numbers of layers of ResNet structure.\n",
        "2. **hidden_size** -> Number of neurons per hidden layer.\n",
        "3. **dropout_rate** -> The proportion of neurons randomly extinguished.\n",
        "4. **NFOLDS** -> Number of ResNet models trained.\n",
        "5. **EPOCHS** -> Number of training sessions per model.\n",
        "6. **learning_rate** -> Learning rete.\n",
        "7. **label_smoothing** -> Label balance factor.\n",
        "8. **patience** -> Number of early stopping tolerance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h7BWRpci3Goo"
      },
      "source": [
        "# TensorFlow model\n",
        "\n",
        ">The above part uses PyTorch to implement a ResNet idea model. The following uses TensorFlow to implement a simpleNN that I did not expect that simple network in this dataset has good performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DwxpHgGC3M_q"
      },
      "source": [
        "## Import the required libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BU8EmDNQ3EXf"
      },
      "source": [
        "from tensorflow.keras.layers import Input, Dense, BatchNormalization, Dropout, Concatenate, Lambda, GaussianNoise, Activation\n",
        "from tensorflow.keras.models import Model, Sequential\n",
        "from tensorflow.keras.losses import BinaryCrossentropy\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "from tensorflow.keras.layers.experimental.preprocessing import Normalization\n",
        "import tensorflow as tf\n",
        "import tensorflow_addons as tfa\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from tqdm import tqdm\n",
        "from random import choices"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VpOizCMH3O-w"
      },
      "source": [
        "## Define hyperparameter \n",
        "\n",
        ">The role of the hyperparameter is similar to that in the PyTorch model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dwtPfi2S3RgI"
      },
      "source": [
        "np.random.seed(SEED)\n",
        "\n",
        "SEED = 1111\n",
        "epochs = 205\n",
        "batch_size = 4096\n",
        "hidden_units = [160, 160, 160]\n",
        "dropout_rates = [0.2, 0.2, 0.2, 0.2]\n",
        "label_smoothing = 1e-2\n",
        "learning_rate = 1e-2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xYqkzACWMiha"
      },
      "source": [
        "## SimpleNN model\n",
        "\n",
        ">We build a simple 3-layer neural network model using Keras, an advanced neural network API written in Python and running with TensorFlow as the backend by default. The core data structure of Keras is the model, a way of organizing the network layers. The simplest model is the Sequential sequential model, which consists of multiple network layers stacked linearly. For more complex structures, use the Keras functional API, which allows the construction of arbitrary neural network graphs.\n",
        "\n",
        "Build models using the Keras functional API:\n",
        "1. Create an input layer with *tf.keras.layers.Input*. The parameters are the dimensions of the training set features.\n",
        "2. Add a BN operation layer and a dropout layer to the output of the input layer, calling the BatchNormaliztion and Dropout APIs provided by *tf.keras.layers*.\n",
        "3. *hidden_units* is a list of the number of incoming hidden layer neurons. The superparameter section has defined that there are 160 neurons in each of the 3 layers.\n",
        "4. Each layer has structured like a sandwich: fully connected layer + BatchNorm layer + activation layer + dropout layer.\n",
        "5. The activation function used is the switch activation function proposed by Google in October 2017: $f(x) = x - \\text{sigmoid}(βx)$. The switch has the properties of no upper bound with lower bound, smooth, non-monotonic. Also, we can use other activation functions such as ReLU.\n",
        "6. **output layer** -> a fully connected layer with an output of 5 (dimensions of the labels) and labels in each dimension is a binary classfication problem (action is 1 or 0). Then we use the sigmoid activation function to do activation on the output of the full connected layer.\n",
        "7. Pass the defined input *inp* and output *out* objects to the model *tf.keras.models.Model* to complete the creation of the model. We can use *keras.utils.plot_model(model, \"model_with_shape.png\", show_shapes=True)* to generate pictures of the model structure.\n",
        "8. Use *model.compile* to configure the training process of the model. We need to specify the optimizer, loss function, and evaluation matrix, which can refer to: https://keras.io/zh/models/model/#compile.\n",
        "9. The optimizer *optimizer* uses RectifiedAdam, which claims to provide fully automatic and dynamic self-tuning of the learning rate, eliminating the \"warm-up\" required by using Adam, ensuring the learning rate and convergence speed, while effectively avoiding the model from falling into the trap of \"local optimum\" trap."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "whQuxqq2MkBN"
      },
      "source": [
        "# fit\n",
        "def create_mlp(\n",
        "    num_columns, num_labels, hidden_units, dropout_rates, label_smoothing, learning_rate\n",
        "):\n",
        "\n",
        "    inp = tf.keras.layers.Input(shape=(num_columns,))\n",
        "    x = tf.keras.layers.BatchNormalization()(inp)\n",
        "    x = tf.keras.layers.Dropout(dropout_rates[0])(x)\n",
        "    for i in range(len(hidden_units)):\n",
        "        x = tf.keras.layers.Dense(hidden_units[i])(x)\n",
        "        x = tf.keras.layers.BatchNormalization()(x)\n",
        "        x = tf.keras.layers.Activation(tf.keras.activations.swish)(x)\n",
        "        x = tf.keras.layers.Dropout(dropout_rates[i + 1])(x)\n",
        "    \n",
        "    x = tf.keras.layers.Dense(num_labels)(x)\n",
        "    out = tf.keras.layers.Activation(\"sigmoid\")(x)\n",
        "\n",
        "    model = tf.keras.models.Model(inputs=inp, outputs=out)\n",
        "    model.compile(\n",
        "        optimizer=tfa.optimizers.RectifiedAdam(learning_rate=learning_rate),\n",
        "        loss=tf.keras.losses.BinaryCrossentropy(label_smoothing=label_smoothing),\n",
        "        metrics=tf.keras.metrics.AUC(name=\"AUC\"),\n",
        "    )\n",
        "\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rP6hjddIMnHU"
      },
      "source": [
        "# TensorFlow training model\n",
        ">The *create_mlp* function encapsulates the model building and training configuration. We can call this method to get a configured TensorFlow model, using the *fit* method to complete the training.\n",
        "\n",
        "1. If we train the model with k-fold cross-validation method, we should use *clear_session* to clear the training content to be given to the session. Otherwise the label may be leaked and it also tends to lead to high memory usage.\n",
        "2. The *fit* method provides the training set and labels. *epochs* is the number of times the model is trained. *batch_size* is the amount of data per batch, *verbose* is the configuration for printing information about the training process. *validation_data* is the data used for validation.\n",
        "3. There are two ways to save models in TensorFlow. The *save* method saves the model structure and weights together. The *save_weights* method saves only the model weights. The advantage of saving the full amount is that it can be used directly when loading without instantiating a model object and the advantage of saving only the weights is that the file is smaller.\n",
        "4. The trained model should be loaded when doing model fusion using the same method as the saved one. If it is a fully saved model then we should use *tensorflow.keras.models.load_model* method. If you use weights-saved method, you have to instantiate a model object first and then load the model weights with *load_weights*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "20u6OxNJMoEV"
      },
      "source": [
        "tf.keras.backend.clear_session()\n",
        "tf.random.set_seed(SEED)\n",
        "clf = create_mlp(\n",
        "    len(feat_cols), 5, hidden_units, dropout_rates, label_smoothing, learning_rate\n",
        "    )\n",
        "history = clf.fit(X_train,y_train\n",
        "                  , epochs=epochs\n",
        "                  , batch_size=batch_size\n",
        "                  , verbose=2\n",
        "                  , validation_data=(X_valid,y_valid)\n",
        "                 )\n",
        "                \n",
        "clf.save_weights('model.h5')\n",
        "clf.load_weights('../input/jane-street-with-keras-nn-overfit/model.h5')\n",
        "\n",
        "tf_models = [clf]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x158Gi21Mpjm"
      },
      "source": [
        "# XGBoost model\n",
        "\n",
        "## Loading the treelite type XGBoost model\n",
        "\n",
        ">After the XGBoost model is trained and converted to treelite type, the treelite type model can be loaded and used when doing model fusion.\n",
        "\n",
        ">treelite is a tree model deployment acceleration tool that can compile and optimize tree models into separate libraries which can be easily used for model deployment. After optimization it can increase the prediction speed of XGBoost model by 2-6 times. Project address: https://treelite.readthedocs.io/."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_IU2LMT-MxtZ"
      },
      "source": [
        "treelite_model = treelite_runtime.Predictor('../input/model-tree/model_xgb.so', verbose=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELk0hdj_Mzle"
      },
      "source": [
        "## Steps to convert XGBoost model to treelite type.\n",
        "1. Load the tree model into treelite with the *treelite.Model.load* method.\n",
        "2. Archive the deployment source with the *export_srcpkg* method.\n",
        "3. Deploy the shared library and get a *.so* type file. Later we can use *treelite_runtime* to load the separate treelite library of *.so*, which can be used for prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6TPkS0y3M2yS"
      },
      "source": [
        "import treelite\n",
        "model = treelite.Model.load('my_model.model', model_format='xgboost')\n",
        "\n",
        "# Produce a zipped source directory, containing all model information\n",
        "# Run `make` on the target machine\n",
        "model.export_srcpkg(platform='unix', toolchain='gcc',\n",
        "                    pkgpath='./mymodel.zip', libname='mymodel.so',\n",
        "                    verbose=True)\n",
        "\n",
        "# Like export_srcpkg, but generates a shared library immediately\n",
        "# Use this only when the host and target machines are compatible\n",
        "model.export_lib(toolchain='gcc', libpath='./mymodel.so', verbose=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9eXet1AWM4NR"
      },
      "source": [
        "## Steps to load and use treelite:\n",
        "1. Load the *.so* file with *treelite_runtime.Predictor*.\n",
        "2. Use treelite_runtime.Batch.from_npy2d to convert data into treelite-specific data structures.\n",
        "3. Predict using the *predict* method of the *Predictor* object to get the probability value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6oUQpqfCM5ww"
      },
      "source": [
        "import treelite_runtime\n",
        "predictor = treelite_runtime.Predictor('./mymodel.so', verbose=True)\n",
        "batch = treelite_runtime.Batch.from_npy2d(X)\n",
        "out_pred = predictor.predict(batch)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CniqkxTZM8aW"
      },
      "source": [
        "# Submit test results\n",
        "## Strategies for model fusion\n",
        "\n",
        "The models to be used for model fusion are loaded, and we start to do model fusion. The strategy used is weighted average.\n",
        "1. Each model has a weight, and the sum of the weights is 1. If the three models have equal weights, the averaging strategy is adopted.\n",
        "2. The results obtained from all three models are probability values (between 0 and 1), which are multiplied by the weights of the corresponding models to obtain the predicted probabilities with weights.\n",
        "3. The prediction results of the three models with weights are added up and compared with *th* (the threshold to determine whether action is 1). Those smaller then *th* are marked as 0, otherwise is marked as 1.\n",
        "\n",
        "## Features of this competition submission\n",
        "According to the official submission instructions given:\n",
        ">You must submit to this competition using the provided python time-series API, which ensures that models do not peek forward in time. To use the API, follow the following template in Kaggle Notebooks:\n",
        "\n",
        " ```\n",
        " import janestreet\n",
        " env = janestreet.make_env() # initialize the environment\n",
        " iter_test = env.iter_test() # an iterator which loops over the test set\n",
        "\n",
        " for (test_df, sample_prediction_df) in iter_test:\n",
        "    sample_prediction_df.action = 0 #make your 0/1 prediction here\n",
        "    env.predict(sample_prediction_df)\n",
        " ```\n",
        "\n",
        "According to the official instructions, we need to use the official data interface provided by *janestreet* to get a runtime environment *env*. Also we need to get an iterator *iter_test* from the environment. We can get a transaction data *test_df* and results saved *sample_prediction_df* at a time. After feeding the data *test_df* to the model to do predictions we should mark the result with *sample_prediction_df.action* as 0 or 1. Finally we should use the official method *predict* to submit the predicted results of this transaction data.\n",
        " \n",
        "## Steps for model fusion and predictions:\n",
        "The ResNet model, simpleNN model and XGBoost model are used to make predictions for *test_df* respectively. Then a probability is calculated according to the weighted average strategy. The value of *action* is obtained after comparing with *th* (judgment threshold).\n",
        "\n",
        "### XGBoost model prediction\n",
        "1. The XGBoost model is trained with a null substitution value of -999, and the *fillna* operation is not required when feed the data to the XGBoost model.\n",
        "2. The XGBoost model has been converted to treelite type, so the data needs to be converted to the format required by treelite.\n",
        "3. Complete the prediction with *predict* method. As already mentioned, treelite's *predict* is a probability value.\n",
        " \n",
        "### ResNet model prediction\n",
        "1. **Data pre-processing** -> We use the previously calculated and saved *f_mean* to fill in the null values of the *test_df*.\n",
        "2. **Feature engineering** -> ResNet's model does feature engineering. So here we also need to do the same feature engineering on test's data and add the results to the end of test's columns with *np.concatenate* (2 more columns of features).\n",
        "3. The ResNet models are trained with NFOLDS models. The five resnet models are fused with the average strategy to obtain the prediction results of the ResNet models.\n",
        "4. The prediction result is taken as the median one from 5 using *np.median*. \n",
        "5. The prediction uses the GPU speedup, note that there is a *to(device)* method. We need to copy the data back to the cpu to do the calculation when doing the model fusion.\n",
        " \n",
        "### TensorFlow model prediction\n",
        "Considering that TensorFlow model can also be trained with cross-validation, we set aside multiple TensorFlow models that do fusion with average strategies.\n",
        "\n",
        "1. Training data of simpleNN that implemented by TensorFlow does not do feature engineering but use the original test data with filled null values for prediction.\n",
        "2. The prediction results are obtained by the innermost determinant, using *np.mean* to get average and taking the median one from the 5 labels as the result with *np.median*.\n",
        " \n",
        "### Integration\n",
        "The predictions of the three models are all 1 dimension of *np.array* type. Their respective weights are multiplied and summed to obtain the fused prediction.\n",
        "\n",
        "### Minor optimization\n",
        "We know that even if the weight is predicted as 1 rather original 0, it does not contribute to the *utility_score* (may even be a drag on it) according to the information of the topic. That case, we do not need to use the model to make prediction for the transaction data if weight is 0, but we can directly mark the *action* as 0, which can reduce the calculation and improve the speed of prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VZK2ATRbM9yW"
      },
      "source": [
        "if Not TRAIN:\n",
        "    import janestreet\n",
        "    env = janestreet.make_env()\n",
        "    env_iter = env.iter_test()\n",
        "\n",
        "    for (test_df, pred_df) in tqdm(env_iter):\n",
        "        if test_df['weight'].item() > 0:\n",
        "            x_tt = test_df.loc[:, feat_cols].values\n",
        "            batch = treelite_runtime.Batch.from_npy2d(x_tt)    \n",
        "            tree_pred = treelite_model.predict(batch)\n",
        "            if np.isnan(x_tt.sum()):\n",
        "                x_tt = np.nan_to_num(x_tt) + np.isnan(x_tt) * f_mean\n",
        "\n",
        "            cross_41_42_43 = x_tt[:, 41] + x_tt[:, 42] + x_tt[:, 43]\n",
        "            cross_1_2 = x_tt[:, 1] / (x_tt[:, 2] + 1e-5)\n",
        "            feature_inp = np.concatenate((\n",
        "                x_tt,\n",
        "                np.array(cross_41_42_43).reshape(x_tt.shape[0], 1),\n",
        "                np.array(cross_1_2).reshape(x_tt.shape[0], 1),\n",
        "            ), axis=1)\n",
        "\n",
        "            # torch_pred\n",
        "            torch_pred = np.zeros((1, len(target_cols)))\n",
        "            for model in model_list:\n",
        "                torch_pred += model(torch.tensor(feature_inp, dtype=torch.float).to(device)).sigmoid().detach().cpu().numpy() / NFOLDS\n",
        "            torch_pred = np.median(torch_pred)\n",
        "            \n",
        "            # tf_pred\n",
        "            tf_pred = np.median(np.mean([model(x_tt, training = False).numpy() for model in tf_models],axis=0))\n",
        "            \n",
        "            # avg\n",
        "            pred = torch_pred*0.46 + tf_pred*0.51 + tree_pred*0.03\n",
        "            \n",
        "            pred_df.action = np.where(pred >= 0.493, 1, 0).astype(int)\n",
        "        else:\n",
        "            pred_df.action = 0\n",
        "        env.predict(pred_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rQIdybH3M_2n"
      },
      "source": [
        "print('done')"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}