Linear_Regression_in_Python_using_Statsmodels_DataCourses.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "74fd5938-f7aa-4a18-9f81-cd37e1c85be8",
   "metadata": {},
   "source": [
    "# Linear Regression in Python using Statsmodels\n",
    "https://www.datacourses.com/linear-regression-in-python-using-statsmodels-911/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be71e438-8e79-4b56-865b-cde5d8f5cbf2",
   "metadata": {},
   "source": [
    "## What is Regression?\n",
    "In the simplest terms, regression is the method of finding relationships between different phenomena. It is a statistical technique which is now widely being used in varios areas of machine learning.\n",
    "\n",
    "## What is Linear Regression?\n",
    "Linear regression is the simplest of regression analysis methods. It determines the linear function or the straight line that best represents your data's distribution. As such, linear regression is often called *line of best fit*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be8e7b1d-20c1-4d21-8ce0-053e3183790b",
   "metadata": {},
   "source": [
    "## Simple Linear Regression\n",
    "When you have to find the relationship between just two variables (one dependent and one independent), then simple regression is used.\\\n",
    "The indenpendent variable is usually denoted as X, while the dependent variable is denoted by Y."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c33ec245-2965-4985-95d1-8c40a8448a50",
   "metadata": {},
   "source": [
    "## Using Statsmodels to perform Simple Linear Regression in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "aeecd83c-ee46-4cd2-911d-48abb7ae8884",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sell</th>\n",
       "      <th>List</th>\n",
       "      <th>Living</th>\n",
       "      <th>Rooms</th>\n",
       "      <th>Beds</th>\n",
       "      <th>Baths</th>\n",
       "      <th>Age</th>\n",
       "      <th>Acres</th>\n",
       "      <th>Taxes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>142</td>\n",
       "      <td>160</td>\n",
       "      <td>28</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>0.28</td>\n",
       "      <td>3167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>175</td>\n",
       "      <td>180</td>\n",
       "      <td>18</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>0.43</td>\n",
       "      <td>4033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>129</td>\n",
       "      <td>132</td>\n",
       "      <td>13</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>41</td>\n",
       "      <td>0.33</td>\n",
       "      <td>1471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>138</td>\n",
       "      <td>140</td>\n",
       "      <td>17</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>0.46</td>\n",
       "      <td>3204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>232</td>\n",
       "      <td>240</td>\n",
       "      <td>25</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2.05</td>\n",
       "      <td>3613</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sell  List  Living  Rooms  Beds  Baths  Age  Acres  Taxes\n",
       "0   142   160      28     10     5      3   60   0.28   3167\n",
       "1   175   180      18      8     4      1   12   0.43   4033\n",
       "2   129   132      13      6     3      1   41   0.33   1471\n",
       "3   138   140      17      7     3      1   22   0.46   3204\n",
       "4   232   240      25      8     4      3    5   2.05   3613"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import necessary packages/libraries\n",
    "import pandas as pd\n",
    "from pandas import DataFrame\n",
    "import numpy as np\n",
    "import statsmodels.api as sm\n",
    "\n",
    "# read the csv file\n",
    "myurl = 'https://people.sc.fsu.edu/~jburkardt/data/csv/homes.csv'\n",
    "df = pd.read_csv(myurl)\n",
    "\n",
    "# modify the headers\n",
    "df.columns = ['Sell', 'List', 'Living', 'Rooms', 'Beds', 'Baths', 'Age', 'Acres', 'Taxes']\n",
    "\n",
    "# check the first few rows\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73de1467-524a-4583-b2da-8c217d295b8d",
   "metadata": {},
   "source": [
    "### Check how dependent the *selling price* of house is on *Taxes*. Let's assign 'Taxes' to the variable X. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a0a86f1e-ca5a-45ea-8207-06b11fe62799",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                 OLS Regression Results                                \n",
      "=======================================================================================\n",
      "Dep. Variable:                   Sell   R-squared (uncentered):                   0.970\n",
      "Model:                            OLS   Adj. R-squared (uncentered):              0.969\n",
      "Method:                 Least Squares   F-statistic:                              1571.\n",
      "Date:                Sat, 08 Jul 2023   Prob (F-statistic):                    6.90e-39\n",
      "Time:                        16:31:22   Log-Likelihood:                         -244.50\n",
      "No. Observations:                  50   AIC:                                      491.0\n",
      "Df Residuals:                      49   BIC:                                      492.9\n",
      "Df Model:                           1                                                  \n",
      "Covariance Type:            nonrobust                                                  \n",
      "==============================================================================\n",
      "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
      "------------------------------------------------------------------------------\n",
      "Taxes          0.0453      0.001     39.634      0.000       0.043       0.048\n",
      "==============================================================================\n",
      "Omnibus:                       32.344   Durbin-Watson:                   1.995\n",
      "Prob(Omnibus):                  0.000   Jarque-Bera (JB):               78.071\n",
      "Skew:                           1.829   Prob(JB):                     1.11e-17\n",
      "Kurtosis:                       7.908   Cond. No.                         1.00\n",
      "==============================================================================\n",
      "\n",
      "Notes:\n",
      "[1] R² is computed without centering (uncentered) since the model does not contain a constant.\n",
      "[2] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n"
     ]
    }
   ],
   "source": [
    "X = df[['Taxes']]\n",
    "Y = df[['Sell']]\n",
    "\n",
    "# convert the two columns to float\n",
    "X = X.astype(float)\n",
    "Y = Y.astype(float)\n",
    "\n",
    "# Create OLS model named 'model' and assigned to it the variables X and Y.\n",
    "# Once created, you can apply the fit() function to find the idean regression line\n",
    "# that fits the distribution of X and Y\n",
    "model = sm.OLS(Y, X).fit()\n",
    "\n",
    "# the variable model now holds the detailed information about our fitted regression model.\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b349e5d-ec3e-4410-8c4e-a1aec2cac40b",
   "metadata": {},
   "source": [
    "### Understanding the Results\n",
    "\n",
    "**R-squared value:** This is a statistical measure of how well the regression line fits with the real data points. The higher the value, the better the fit.\n",
    "\\\n",
    "**Adj, R-squared:** This is the corrected R-squared value according to the number of input features. Ideally, it should be close to the R-squareds value.\n",
    "\\\n",
    "**Coefficient:** This gives the ‘M’ value for the regression line. It tells how much the Selling price changes with a unit change in Taxes. A positive value means that the two variables are directly proportional. A negative value, however, would have meant that the two variables are inversely proportional to each other.\n",
    "\\\n",
    "**Std error:**  This tells us how accurate our coefficient value is. The lower the standard error, the higher the accuracy.\n",
    "\\\n",
    "**P >|t| :** This is the p-value. It tells us how statistically significant Tax values are to the Selling price. A value less than 0.05 usually means that it is quite significant."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12dece6b-159a-4d2a-8a50-0ce253a94d61",
   "metadata": {},
   "source": [
    "### Making Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "ad366c3f-5102-456f-85bb-86a695a2f3e0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    1448.935804\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "predictions = model.predict(X)\n",
    "#print(predictions)\n",
    "\n",
    "# if we reply on the model, let's see what is the selling pricess would be if taxes where 3200.0\n",
    "# first we need to make a dataframe with the value 3200.00\n",
    "test_tax = pd.DataFrame([32000])\n",
    "\n",
    "predictions = model.predict(test_tax)\n",
    "print(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d45fa58-958d-49ae-bf1b-36aa3f0b2b72",
   "metadata": {},
   "source": [
    "## Using Statsmodels to Perform Multiple Linear Regression in Python\n",
    "\n",
    "Let us see if we get a better prediction by considering a combination of more than one input variables. Let's try using a combination of 'Taxes', 'Living' and 'List'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6024a498-12bc-4c1a-ab6e-125ed6d5e874",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_X = df[['Taxes', 'Living', 'List']]\n",
    "\n",
    "new_X = new_X.astype(float)\n",
    "\n",
    "new_X = sm.add_constant(new_X)\n",
    "\n",
    "new_model = sm.OLS(Y, new_X).fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "4d38504a-8381-4cea-bf43-b9ffd3f0547f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            OLS Regression Results                            \n",
      "==============================================================================\n",
      "Dep. Variable:                   Sell   R-squared:                       0.995\n",
      "Model:                            OLS   Adj. R-squared:                  0.995\n",
      "Method:                 Least Squares   F-statistic:                     3383.\n",
      "Date:                Sat, 08 Jul 2023   Prob (F-statistic):           6.16e-54\n",
      "Time:                        16:31:22   Log-Likelihood:                -149.77\n",
      "No. Observations:                  50   AIC:                             307.5\n",
      "Df Residuals:                      46   BIC:                             315.2\n",
      "Df Model:                           3                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "==============================================================================\n",
      "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
      "------------------------------------------------------------------------------\n",
      "const          9.7904      2.025      4.835      0.000       5.715      13.866\n",
      "Taxes         -0.0013      0.001     -1.168      0.249      -0.003       0.001\n",
      "Living        -0.4205      0.191     -2.207      0.032      -0.804      -0.037\n",
      "List           0.9764      0.028     35.060      0.000       0.920       1.032\n",
      "==============================================================================\n",
      "Omnibus:                        1.080   Durbin-Watson:                   1.985\n",
      "Prob(Omnibus):                  0.583   Jarque-Bera (JB):                0.389\n",
      "Skew:                           0.022   Prob(JB):                        0.823\n",
      "Kurtosis:                       3.430   Cond. No.                     1.14e+04\n",
      "==============================================================================\n",
      "\n",
      "Notes:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
      "[2] The condition number is large, 1.14e+04. This might indicate that there are\n",
      "strong multicollinearity or other numerical problems.\n"
     ]
    }
   ],
   "source": [
    "print(new_model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd63241-b274-4719-aab0-778265dc4a53",
   "metadata": {},
   "source": [
    "There are 4 different coefficient values instead of one. These are coefficients (or M values) corresponding to *Taxes*, *Age* and *List*. There's also an additional coefficient called *constant coefficient*, which basically the *C*-value in our regression equation. \\\n",
    "\\\n",
    "The R-squared value is 0.995. It's a high value which means the regression plane fits quite well with real data points. \\\n",
    "Adj, R-squared is equal to the R-squared value, which is a good sign. \\\n",
    "The constant coefficient  value (C) is 9.9709 \\\n",
    "\\\n",
    "Let's take a look at each of the independent variables and how they affect the selling price. \\\n",
    "\\\n",
    "List price (or asking price) coefficient is 0.9764, which means the selling price is positively correlated to the asking price. The corresponding p-value is also less than 0.05, further reinforcing the relationship. \\\n",
    "\\\n",
    "Living Area coeficient is -0.4205. This means that selling price decreases with increasing living area. The p-value for this is also less than 0.05, meaning that this variable has a significant effect on the selling price. \\\n",
    "\\\n",
    "Negative coefficient for Taxes (0.0013), this is just the opposite of what we found out from earlier model. The p-value for Taxes 0.249 tells us that of the three variables that we considered, Taxes has a low significance to the selling price.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71f60498-fec7-43dd-9b93-01c7b092d990",
   "metadata": {},
   "source": [
    "### Making predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "13704e61-9fca-4568-831e-f152afa0c818",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    160.967981\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "predictions = new_model.predict(new_X)\n",
    "#print(predictions)\n",
    "\n",
    "test_input = pd.DataFrame({'const': [1.0], 'Taxes': [3200.0], 'Living': [14.0], 'List':[165.0]})\n",
    "\n",
    "predictions = new_model.predict(test_input)\n",
    "print(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5049c261-82ad-46c3-8d6d-f1eb71b4434a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}