Day_13_Data_PreProcessing_Part2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Day_13_Data_PreProcessing_Part2.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyMpVn+KWvWdBB23jhXqkeko"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Data Cleaning\n",
        "Data cleaning is the technique of eliminating garbage, incorrect, duplicate, corrupted, or incomplete data in a dataset as the part of the data preparation process with a motive to build reliable, uniform and standardized data sets.\n",
        "\n",
        "THere are four ways you can perform data cleaning:\n",
        "\n",
        "1.   Drop the missing values\n",
        "    ```\n",
        "    df.dropna()\n",
        "    ```\n",
        "2.   Replace the missing values\n",
        "    ```\n",
        "    from numpy import NaN\n",
        "\n",
        "    df.replace({NaN:1.00})\n",
        "    ```\n",
        "3.   Replace each NaN with a scalar value\n",
        "    ```\n",
        "    df.fillna(12)\n",
        "    ```\n",
        "4.   Fill the missing values forward or backward\n",
        "    ```\n",
        "    df.fillna(method='backfill')\n",
        "    ```\n",
        "\n"
      ],
      "metadata": {
        "id": "WhSqSMxzfRe9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Mean/mode/median imputation\n",
        "We can also do mean/median/mode imputation. For numerical data, we can compute it’s mean or median and use the result to replace missing values and for categorical (non-numerical) data, we can compute its mode to replace the missing value\n",
        "\n",
        "```\n",
        "df.salary.fillna(salary_mean,inplace=True)\n",
        "```\n",
        "\n",
        "**Hot Deck Imputation** — With this, we can replace the missing value of the observation with a randomly selected value from all the observations in the sample referencing the variables with similar value.\n",
        "\n",
        "**Rescale Data** — In order to uniformly scale the attributes with varying scales, rescaling is a useful technique to all have the attributes on the same scale using scikit-learn using the MinMaxScaler class.\n",
        "```\n",
        "s_m = MinMaxScaler(feature_range=(0, 2))\n",
        "rescaledX = s_m.fit_transform(X)\n",
        "```\n",
        "\n",
        "**Binarize Data** — It’s a very useful process which is generally used during feature engineering to manipulate our data using a binary reference threshold using scikit-learn with the Binarizer class.\n",
        "```\n",
        "b_n = Binarizer(threshold = 1.0).fit(X)\n",
        "b_X = b_n.transform(X)\n",
        "```\n",
        "\n",
        "**Regression Imputation** — In order to preserve the relationships between features, we can use regression imputation, basically a technique in which we fit a regression model on a feature with missing data and then using this model predict the values which is used to replace the missing values.\n",
        "\n",
        "**Stochastic regression imputation** — In this technique, in order to reproduce the correlation of features and labels, we add a random variation to the predicted value."
      ],
      "metadata": {
        "id": "Xzn1WWFSfRcf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data Augmentation\n",
        "It’s the technique of increasing the amount and diversity of your training set by applying random transformations.\n",
        "\n",
        "<div style=\"text-align:center\"><img alt=\"Data Augmentation\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/data_augmentation.png?raw=true\" /></div>"
      ],
      "metadata": {
        "id": "RXaEcDvzfRZ2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Width and Height Shifts\n",
        "```\n",
        "generator = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "    width_shift_range=[-90,-40,0,40,90],\n",
        "    height_shift_range=[-40,0,40]\n",
        ")\n",
        "\n",
        "x, y = next(generator.flow_from_directory('images', batch_size=1))\n",
        "plt.imshow(x[0].astype('uint8'));\n",
        "```"
      ],
      "metadata": {
        "id": "Y6QCigzZfRXU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Brightness\n",
        "```\n",
        "generator = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "   \n",
        "    brightness_range=(0.8,4.2)\n",
        ")\n",
        "\n",
        "x, y = next(generator.flow_from_directory('images', batch_size=1))\n",
        "plt.imshow(x[0].astype('uint8'));\n",
        "\n",
        "```"
      ],
      "metadata": {
        "id": "C1vURlBFfRUt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Shear Transformation\n",
        "```\n",
        "generator = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "    shear_range=46\n",
        ")\n",
        "\n",
        "x, y = next(generator.flow_from_directory('images', batch_size=1))\n",
        "plt.imshow(x[0].astype('uint8'));\n",
        "```"
      ],
      "metadata": {
        "id": "p0EsiKi0fRSN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Zoom\n",
        "```\n",
        "generator = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "    zoom_range=[0.2,3.0]\n",
        ")\n",
        "\n",
        "x, y = next(generator.flow_from_directory('images', batch_size=1))\n",
        "plt.imshow(x[0].astype('uint8'));\n",
        "```"
      ],
      "metadata": {
        "id": "YIR3Soh7fRP1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Channel Shift\n",
        "```\n",
        "generator = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "    channel_shift_range=180\n",
        ")\n",
        "\n",
        "x, y = next(generator.flow_from_directory('images', batch_size=1))\n",
        "plt.imshow(x[0].astype('uint8'));\n",
        "```"
      ],
      "metadata": {
        "id": "0f7iSKFelyXT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Flips\n",
        "```\n",
        "generator = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "    horizontal_flip=True,\n",
        "    vertical_flip=True\n",
        ")\n",
        "\n",
        "x, y = next(generator.flow_from_directory('images', batch_size=1))\n",
        "plt.imshow(x[0].astype('uint8'));\n",
        "```"
      ],
      "metadata": {
        "id": "5aTEfBY3lx2y"
      }
    }
  ]
}