Day_12_Data_PreProcessing_Part1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Day_12_Data_PreProcessing_Part1.ipynb",
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyP4TdmTBKxrs+efgIapaibi"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**Data preprocessing , one of the first and crucial step — the process in which we prepare the raw data and make it suitable for a ML model to increase its accuracy and efficiency.**"
      ],
      "metadata": {
        "id": "fNUjZjqFN1ax"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Importing Libraries\n",
        "\n",
        "    import lib_name as alias_name"
      ],
      "metadata": {
        "id": "972s1RlAN1T7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from matplotlib import pyplot as plt\n",
        "from sklearn import datasets"
      ],
      "metadata": {
        "id": "NOJ9XZabRO-1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Importing Datasets\n",
        "You can import data as simple\n",
        "\n",
        "    import pandas as pd\n",
        "    dataset = pd.read_csv('filename.csv')\n",
        "\n",
        "Or directly load from seaborn or sklearn\n",
        "\n",
        "    sns.load_dataset('iris') # sns is alias for seaborn\n",
        "\n",
        "For Scikit Learn\n",
        "\n",
        "    from sklearn import datasets\n",
        "    digits = datasets.load_digits()"
      ],
      "metadata": {
        "id": "tS90wMRiN1RI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Handling the missing data values\n",
        "Missing values, incompleteness, unknown data etc is one of the biggest issues while building machine learning model as it impacts the accuracy.\n",
        "\n",
        "<div style=\"text-align:center\"><img alt=\"Handling Missing Data\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/missing_data.png?raw=true\" /></div>\n",
        "\n",
        "To handle missing values, we can use Scikit-learn Imputer class of sklearn.preprocessing library.\n",
        "\n",
        "    from sklearn.preprocessing import Imputer\n",
        "    imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0)\n",
        "    imputer = imputer.fit(X[c:d, a:b])\n",
        "    X[c:d, a:b] = imputer.transform(X[c:d, a:b])\n"
      ],
      "metadata": {
        "id": "1VtgAbvLN1OQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Encoding categorical data\n",
        "Since ML model works on maths and numbers, so it’s necessary we encode these categorical variables into numbers.\n",
        "\n",
        "<div style=\"text-align:center\"><img alt=\"Encoding Categorical Data\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/encoding.png?raw=true\" /></div>\n",
        "\n",
        "We will use label encoder and One hot encoder ( For Dummy variable Encoding) to accomplish this task.\n",
        "\n",
        "    from sklearn.preprocessing import LabelEncoder, OneHotEncoder\n",
        "    labelencoder_X = LabelEncoder()\n",
        "    X[:, 0] = labelencoder_X.fit_transform(X[:, 0]) \n",
        "    onehotencoder = OneHotEncoder(categorical_features = [0])\n",
        "    X = onehotencoder.fit_transform(X).toarray()\n",
        "    labelencoder_y = LabelEncoder()\n",
        "    y = labelencoder_y.fit_transform(y)\n"
      ],
      "metadata": {
        "id": "LTAGCpvhN1L4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Split Data into Train and Test data\n",
        "\n",
        "\n",
        "*   **Training data**. Used to train the ML model — Feed the algorithm with input data, to give an expected output as the algorithm evaluates the data repeatedly to learn and train with the data and it’s behaviour.\n",
        "*   **Validation data**. Part of training process in which the validation data i.e new data is needed into the model that it hasn’t evaluated before. This data provides the first test against unseen data which helps in evaluating how well the model makes predictions based on the new data and hyperparameter optimization.\n",
        "*   **Test data**. After building the ML model, testing data validates to check if the model makes accurate predictions as well as if it’s trained effectively.\n",
        "\n",
        "<div style=\"text-align:center\"><img alt=\"Split Data\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/split_data.png?raw=true\" /></div>\n",
        "\n",
        "\n",
        "    from sklearn.cross_validation import train_test_split\n",
        "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.4, random_state = 42)"
      ],
      "metadata": {
        "id": "sGzmfE-tN1Jw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Feature Scaling\n",
        "Feature scaling is a technique to standardize the independent variables in the data in a specified range by putting our variables in the same range and scale so that variables don’t dominate each other. It’s important because it always converges and gives results faster.\n",
        "\n",
        "*   **Normalization** also known as Min-Max scaling is a technique in which values in the data are scaled so that they end up ranging between 0 and 1.\n",
        "*   **Standardization** is a technique in which the values are centered around the mean with a unit standard deviation.\n",
        "\n",
        "<div style=\"text-align:center\"><img alt=\"Feature Scaling\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/feature_scaling.jpeg?raw=true\" /></div>\n",
        "\n",
        "    from sklearn.preprocessing import StandardScaler \n",
        "    ss_X = StandardScaler() \n",
        "    X_train = ss_X.fit_transform(X_train) \n",
        "    X_test = ss_X.transform(X_test) "
      ],
      "metadata": {
        "id": "MQDZC3KwZb1W"
      }
    }
  ]
}