Day_07_Statistics_for_Data_Science_and_Machine_Learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Day_7_Statistics_for_Data_Science_and_Machine_Learning.ipynb",
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMBxNaTjgD7S2Sn+rSv4UTt"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Branches of statistics\n",
        "Knowing the type of statistics you need to answer your question will help you choose the appropriate methods to get the most accurate answer possible. There are two branches of statistics \n",
        "\n",
        "*   Descriptive statistics — It describes and summarize the data\n",
        "*   Inferential statistics — It takes sample data in order to make inferences with respect to a larger population.\n",
        "\n"
      ],
      "metadata": {
        "id": "fDiDWZPZkXgk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "RwH2Q5-hkXbV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Two types of data\n",
        "\n",
        "*   Numeric, or quantitative data\n",
        "\n",
        "    It’s made up of numeric values which is further divided into Discrete ( counted items) and Continuous( measured variables). For this type of data we can use summary statistics like mean, and plots like scatter plots etc\n",
        "\n",
        "*   Categorical, or qualitative data\n",
        "\n",
        "    It’s made up of values that belong to distinct groups which is further divided into Nominal and Ordinal. For this type of data we use summary statistics such as counts and plots like barplots etc"
      ],
      "metadata": {
        "id": "_HtLAJlvkXYu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"text-align:center\"><img alt=\"Python Data Type\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/data.png?raw=true\"/></div>"
      ],
      "metadata": {
        "id": "HYVF_Mf1pVic"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Measure of Centre\n",
        "It's the value of the center or middle of the data set.\n",
        "\n",
        "*   Mean\n",
        "*   Median\n",
        "*   Mode\n",
        "*   Range\n",
        "\n"
      ],
      "metadata": {
        "id": "xphB_EMnpVb7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"text-align:center\"><img alt=\"Python Data Type\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/measure_centre.png?raw=true\"/></div>"
      ],
      "metadata": {
        "id": "08py9ESBtpQw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"text-align:center\"><img alt=\"Python Data Type\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/measure_centre_diagram.png?raw=true\"/></div>"
      ],
      "metadata": {
        "id": "DVO5c278tpOG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Measure of Spread / Measure of Variation\n",
        "Variability is the key to the statistics. So, when you are describing the data, never rely on the center alone. Measure of spread identified the spread of the values.\n",
        "\n",
        "*   Standard Deviation\n",
        "*   Inter-Quartile Range (IQR)\n",
        "*   Variance\n",
        "*   Range\n",
        "\n"
      ],
      "metadata": {
        "id": "q_ONltq2t2F0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"text-align:center\"><img alt=\"Python Data Type\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/measure_frequency1.png?raw=true\"/></div>"
      ],
      "metadata": {
        "id": "O7KZS2-lu7x8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"text-align:center\"><img alt=\"Python Data Type\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/measure_frequency2.png?raw=true\"/></div>"
      ],
      "metadata": {
        "id": "JwV0OeHru7p8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Which measure to use?\n",
        "\n",
        "*   If the distribution is normal or symmetrical, use mean and standard deviation. Mean works better for symmetrical data and is more sensitive to extreme values.\n",
        "*   If the distribution is skewed or has large outliers, then use Median, Range or IQR. Median is usually better to use when your data is skewed i.e not symmetrical.\n",
        "*   If the distribution is bimodal, Use mode to figure out if the two modes represent different groups , or range\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "GyLI83HnvGGX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Numpy Statistical Functions\n",
        "\n",
        "*   np.mean() - To determines the mean value\n",
        "*   np.median() - To determines the median value\n",
        "*   np.std() - To determines the standard deviation\n",
        "*   np.var() - To determines the variance\n",
        "*   np.average() - To determines the weighted average\n",
        "*   np.pecentile() - To determines the nth percentile\n",
        "*   np.amin() - To determines the minimum value\n",
        "*   np.amax() - To determines the maximum value\n"
      ],
      "metadata": {
        "id": "PEbDeWcQvGBz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "arr1= np.array([[12,43,56],[78,88,95],[79,89,43], [101,34,67]]) \n",
        "arr2 = np.array([5,6,7,12,34,67,89]) \n",
        "\n",
        "# Mean function\n",
        "print(\"Mean:\", np.mean(arr2))\n",
        "\n",
        "# Median function\n",
        "print(\"Median:\",np.median(arr2))\n",
        "\n",
        "# Standard Deviation Function\n",
        "print(\"Standard Deviation:\", np.std(arr2))\n",
        "\n",
        "# Variance Function\n",
        "print(\"Variance:\",np.var(arr2))\n",
        "# Average Function\n",
        "print(\"Average:\",np.average(arr2))\n",
        "\n",
        "# Percentile Function\n",
        "print(\"Percentile:\",np.percentile(arr2,5,0))\n",
        "\n",
        "# Minimum Function\n",
        "print(\"Minimum element:\",np.amin(arr1))\n",
        "\n",
        "# Maximum Function\n",
        "print(\"Maximum element:\",np.amax(arr1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vHn9jClTkXRs",
        "outputId": "09137d3f-697e-459c-a90f-dc3341786a86"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mean: 31.428571428571427\n",
            "Median: 12.0\n",
            "Standard Deviation: 31.409084867994768\n",
            "Variance: 986.530612244898\n",
            "Average: 31.428571428571427\n",
            "Percentile: 5.3\n",
            "Minimum element: 12\n",
            "Maximum element: 101\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Groupby and Aggregate Functoin\n",
        "When you want to compare summary statistics between groups, it’s much easier to use .groupby() and .agg(). The groupby function can be combined with one or more aggregation functions to easily summarize data."
      ],
      "metadata": {
        "id": "s0PXN31mvF_e"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "id": "nNOvNUaTWJ9P",
        "outputId": "9e58b2fe-ec2d-488f-db33-7c51d0f06a13"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     A             C          \n",
              "   min max       min       max\n",
              "B                             \n",
              "1    1   1  0.067527  0.067527\n",
              "2   10  10 -0.329537 -0.329537\n",
              "30  20  20 -0.122889 -0.122889\n",
              "40   2   2  0.114895  0.114895"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ce36ed14-236c-4c42-8043-82899777692b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr:last-of-type th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"2\" halign=\"left\">A</th>\n",
              "      <th colspan=\"2\" halign=\"left\">C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>min</th>\n",
              "      <th>max</th>\n",
              "      <th>min</th>\n",
              "      <th>max</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>B</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.067527</td>\n",
              "      <td>0.067527</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>-0.329537</td>\n",
              "      <td>-0.329537</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>20</td>\n",
              "      <td>20</td>\n",
              "      <td>-0.122889</td>\n",
              "      <td>-0.122889</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0.114895</td>\n",
              "      <td>0.114895</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ce36ed14-236c-4c42-8043-82899777692b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ce36ed14-236c-4c42-8043-82899777692b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ce36ed14-236c-4c42-8043-82899777692b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "import pandas as pd\n",
        "df = pd.DataFrame({'A': [1, 10, 20, 2],\n",
        "               'B': [1, 2, 30, 40],\n",
        "               'C': np.random.randn(4)})\n",
        "df.groupby('B').agg(['min', 'max'])"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Measure of Spread\n",
        "Quantiles are a great way of summarizing numerical data since they can be used to measure center and spread, as well as to get a sense of where a data point stands in relation to the rest of the data set."
      ],
      "metadata": {
        "id": "ySyWyvyfwx1o"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"text-align:center\"><img alt=\"Python Data Type\" src=\"https://github.com/thunderstroke325/60-Days-of-Data-Science-and-ML/blob/main/assets/measure_spread_figure.png?raw=true\"/></div>"
      ],
      "metadata": {
        "id": "oi85XUUXwxJd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "arr2 = np.array([5,6,7,12,34,67,89]) \n",
        "\n",
        "print(\"Q2 quantile of arr : \", np.quantile(arr2, .50))\n",
        "print(\"Q1 quantile of arr : \", np.quantile(arr2, .25))\n",
        "print(\"Q3 quantile of arr : \", np.quantile(arr2, .75))\n",
        "print(\"100th quantile of arr : \", np.quantile(arr2, .1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DFt2boCSxDZd",
        "outputId": "35490a72-f7b6-454b-9a50-79d1a205e424"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Q2 quantile of arr :  12.0\n",
            "Q1 quantile of arr :  6.5\n",
            "Q3 quantile of arr :  50.5\n",
            "100th quantile of arr :  5.6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "arr = [31, 35, 45, 49, 59, 69, 74, 79, 80, 81, 89, 94, 96, 99, 101, 104, 112, 117,119,127,134]\n",
        "  \n",
        "# First quartile (Q1)\n",
        "Q1 = np.median(arr[:12])\n",
        "  \n",
        "# Third quartile (Q3)\n",
        "Q3 = np.median(arr[12:])\n",
        "  \n",
        "# Interquartile range (IQR)\n",
        "IQR = Q3 - Q1\n",
        "  \n",
        "print(IQR)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ti0GnJjSxb4u",
        "outputId": "9050916f-9aa8-4267-d614-0f0b92104351"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "40.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Continuous distribution\n",
        "Distribution with location (loc) and Scale (scale) parameters.Continuous distributions can be uniform or can take forms where some values have a higher probability than others."
      ],
      "metadata": {
        "id": "6_JqgaZPxkmS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from scipy.stats import uniform\n",
        "arr2 = np.array([5,6,7,12,34,67,89]) \n",
        "print (uniform.cdf(arr2, loc =4 , scale = 5))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MUpTzAxyyCyj",
        "outputId": "479b3a68-6d9f-4c7e-9ad8-8637606c2a9e"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0.2 0.4 0.6 1.  1.  1.  1. ]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Normal Distribution"
      ],
      "metadata": {
        "id": "PuHiB3HGxkie"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from scipy.stats import norm\n",
        "import numpy as np\n",
        "arr2 = np.array([0.91,0.17,0.99996833, 0.81, 0.97,0.54])\n",
        "print(norm.ppf(arr2))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lzuXYA9gyciA",
        "outputId": "a49d62ec-7bfe-4faa-df99-bbdd42a21726"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[ 1.34075503 -0.95416525  4.00000928  0.8778963   1.88079361  0.10043372]\n"
          ]
        }
      ]
    }
  ]
}