NNDL_MLP_classifier_L3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "ZBjNdXsfrF7W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#1. Explore MLP classifier"
      ],
      "metadata": {
        "id": "n4oFMIDUrcIS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "*   A multilayer perceptron is a neural network connecting multiple layers in a directed graph, which means that the signal path through the nodes only goes one way. Each node, apart from the input nodes, has a nonlinear activation function. An MLP uses backpropagation as a supervised learning technique.\n",
        "* MLPClassifier relies on an underlying Neural Network to perform the task of classification\n",
        "* The nodes of the layers are neurons with nonlinear activation functions, except for the nodes of the input layer\n",
        "* Between the input and the output layer there may be one or more nonlinear hidden layers\n",
        "* A multilayer perceptron is a neural network connecting multiple layers in a directed graph, which means that the signal path through the nodes only goes one way\n",
        "* Each node, apart from the input nodes, has a nonlinear activation function. An MLP uses backpropagation as a supervised learning technique.\n",
        "\n"
      ],
      "metadata": {
        "id": "NS6kAvMDsb7Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#2. Give a summary of all the parameters used in MLP classifier."
      ],
      "metadata": {
        "id": "n8jEV_eps6bJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "1.hidden_layer_sizes:tuple, length = n_layers - 2, default=(100,)\n",
        "The ith element represents the number of neurons in the ith hidden layer.\n",
        "\n",
        "2.activation{‘identity’, ‘logistic’, ‘tanh’, ‘relu’}, default=’relu’\n",
        "Activation function for the hidden layer.\n",
        "\n",
        "‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x\n",
        "\n",
        "‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)).\n",
        "\n",
        "‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x).\n",
        "\n",
        "‘relu’, the rectified linear unit function, returns f(x) = max(0, x)\n",
        "\n",
        "3.solver{‘lbfgs’, ‘sgd’, ‘adam’}, default=’adam’\n",
        "The solver for weight optimization.\n",
        "\n",
        "‘lbfgs’ is an optimizer in the family of quasi-Newton methods.\n",
        "\n",
        "‘sgd’ refers to stochastic gradient descent.\n",
        "\n",
        "‘adam’ refers to a stochastic gradient-based optimizer proposed by Kingma, Diederik, and Jimmy Ba\n",
        "\n",
        "Note: The default solver ‘adam’ works pretty well on relatively large datasets (with thousands of training samples or more) in terms of both training time and validation score. For small datasets, however, ‘lbfgs’ can converge faster and perform better.\n",
        "\n",
        "4.learning_rate{‘constant’, ‘invscaling’, ‘adaptive’}, default=’constant’\n",
        "Learning rate schedule for weight updates.\n",
        "\n",
        "‘constant’ is a constant learning rate given by ‘learning_rate_init’.\n",
        "\n",
        "‘invscaling’ gradually decreases the learning rate at each time step ‘t’ using an inverse scaling exponent of ‘power_t’. effective_learning_rate = learning_rate_init / pow(t, power_t)\n",
        "\n",
        "‘adaptive’ keeps the learning rate constant to ‘learning_rate_init’ as long as training loss keeps decreasing. Each time two consecutive epochs fail to decrease training loss by at least tol, or fail to increase validation score by at least tol if ‘early_stopping’ is on, the current learning rate is divided by 5.\n",
        "\n",
        "Only used when solver='sgd'."
      ],
      "metadata": {
        "id": "qiqaSs2e5ZPW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#3. Classify an XOR problem using the multilayer perceptron Network"
      ],
      "metadata": {
        "id": "kBev67vPs_3Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "A network with one hidden layer containing two neurons should be enough to separate the XOR problem. Follow these steps :-\n",
        "\n",
        "The first neuron acts as an OR gate and the second one as a NOT AND gate.\n",
        "\n",
        "Add both the neurons and if they pass the treshold it’s positive. You can just use linear decision neurons for this with adjusting the biases for the tresholds.\n",
        "\n",
        "The inputs of the NOT AND gate should be negative for the 0/1 inputs.\n",
        "\n"
      ],
      "metadata": {
        "id": "v3DV9RYA4ZId"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#what is activation function\n",
        " they are fundamentally used for determining the output of deep learning models, its accuracy, and performance efficiency of the training model that can design or divide a huge scale neural network.\n",
        "\n",
        " They basically decide to activate or deactivate neurons to get the desired output.\n",
        "\n",
        "\n",
        " why we need?\n",
        " Without activation function, weight and bias would only have a linear transformation, or neural network is just a linear regression model"
      ],
      "metadata": {
        "id": "_ZpBzMnb7Oli"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#bias vs variance\n",
        "\n",
        "Bias describes how well a model matches the training set. A model with high bias won’t match the data set closely, while a model with low bias will match the data set very closely. Bias comes from models that are overly simple and fail to capture the trends present in the data set.\n",
        "\n",
        "Variance describes how much a model changes when you train it using different portions of your data set. A model with high variance will have the flexibility to match any data set you provided it, which may result in dramatically different models each time. Variance comes from models that are highly complex and employ a significant number of features."
      ],
      "metadata": {
        "id": "YHq2LLIV8Jpe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import sklearn\n",
        "from sklearn import preprocessing\n",
        "from sklearn.neural_network import MLPClassifier"
      ],
      "metadata": {
        "id": "JfBJNvtqyLf-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "input = np.array([\n",
        "    0, 0,\n",
        "    0, 1,\n",
        "    1, 0,\n",
        "    1, 1\n",
        "]).reshape(4, 2)\n",
        "print(\" input values X  and Y :\\n\",input)\n",
        "\n",
        "actual_output = np.array([0, 1, 1, 0]).reshape(4,)\n",
        "print(\" the expected output  / xor output is \\n\",actual_output)\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "HlvEthFptV7p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "da43fd5f-7c25-42c4-ae42-8f7ec35f31de"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " input values X  and Y :\n",
            " [[0 0]\n",
            " [0 1]\n",
            " [1 0]\n",
            " [1 1]]\n",
            " the expected output  / xor output is \n",
            " [0 1 1 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " A number used to multiply a variable- coefficeint"
      ],
      "metadata": {
        "id": "eoK610eA647B"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#RELU ACTIVATION"
      ],
      "metadata": {
        "id": "_zkJEwMw5gE2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model0 = sklearn.neural_network.MLPClassifier(\n",
        "    activation='relu', max_iter=10000, hidden_layer_sizes=(4,2))\n",
        "model0.fit(input, actual_output)\n",
        "\n",
        "print('model score:', model0.score(input, actual_output)) \n",
        "print('predicted output:', model0.predict(input)) \n",
        "print('expected output /  xor output is:', np.array([0, 1, 1, 0]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eaH2OWQm1vs0",
        "outputId": "74183b07-0cc3-49a6-c9ef-ddefa692d896"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "model score: 0.5\n",
            "predicted output: [0 0 0 0]\n",
            "expected output /  xor output is: [0 1 1 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(model0.coefs_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C7s9M1nb06FE",
        "outputId": "723d48aa-1418-4c5f-a8ff-79ce2b21c9a1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[array([[-3.17074141e-01, -8.03317020e-01, -5.47007932e-03,\n",
            "        -5.62769413e-01],\n",
            "       [-7.57034394e-02, -4.47107361e-01, -6.93338393e-05,\n",
            "         3.05850065e-02]]), array([[ 0.71112174,  0.05352076],\n",
            "       [ 0.59703831, -0.17586897],\n",
            "       [ 0.11824736,  0.52739136],\n",
            "       [-0.59619941,  0.33593378]]), array([[ 1.15678724],\n",
            "       [-0.57941588]])]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model = sklearn.neural_network.MLPClassifier(\n",
        "    activation='relu', max_iter=10000, hidden_layer_sizes=(4,2),learning_rate= 'constant')\n",
        "model.fit(input, actual_output)\n",
        "\n",
        "print('model score:', model.score(input, actual_output)) \n",
        "print('predicted output:', model.predict(input)) \n",
        "print('expected output /  xor output is:', np.array([0, 1, 1, 0]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4jOUqc5qz89r",
        "outputId": "2a49040a-4d3d-4f8a-e894-ee843d504a2c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "model score: 0.5\n",
            "predicted output: [0 0 0 0]\n",
            "expected output /  xor output is: [0 1 1 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(model.coefs_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rIuv5bbm0Ycb",
        "outputId": "f3d4c1d0-4f81-4168-ae1e-1568aec8d698"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[array([[ 7.24369670e-01,  4.05427811e-01, -8.80768862e-01,\n",
            "        -1.91928842e-01],\n",
            "       [ 7.41330857e-01,  7.58547118e-05, -8.47571087e-01,\n",
            "         8.05397748e-01]]), array([[-0.80827678, -0.41162839],\n",
            "       [-0.49665806, -0.90341941],\n",
            "       [ 0.17051427, -0.51525818],\n",
            "       [ 0.03743042, -0.88850338]]), array([[-0.44136225],\n",
            "       [ 1.18755692]])]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#LOGISTIC ACTIVATION"
      ],
      "metadata": {
        "id": "2BJxY2P752cr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model1 = sklearn.neural_network.MLPClassifier(\n",
        "    activation='logistic',hidden_layer_sizes=(4,2),learning_rate= 'constant')\n",
        "model1.fit(input, actual_output)\n",
        "\n",
        "print('model score:', model1.score(input, actual_output)) \n",
        "print('predicted output:', model1.predict(input)) \n",
        "print('expected output /  xor output is:', np.array([0, 1, 1, 0]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "REONKMmI1b7k",
        "outputId": "40504264-1158-4210-ee54-2bcae197bafe"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "model score: 0.5\n",
            "predicted output: [0 0 0 0]\n",
            "expected output /  xor output is: [0 1 1 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(model1.coefs_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3f1SrY_E0dNR",
        "outputId": "e2ae37eb-3aed-41e7-ce81-0b9390c8118c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[array([[-0.5347392 , -0.17445224, -0.59556084, -0.22504884],\n",
            "       [-0.07431683,  0.39889443,  0.03103533,  0.01297653]]), array([[ 0.3061541 , -0.45649227],\n",
            "       [ 0.05406594, -0.19223827],\n",
            "       [ 0.29224802, -0.31830381],\n",
            "       [-0.01828924,  0.0230003 ]]), array([[0.30027242],\n",
            "       [0.64472547]])]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#TANH ACTIVATION"
      ],
      "metadata": {
        "id": "ns9o8LjO6C_9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model2 = sklearn.neural_network.MLPClassifier(\n",
        "    activation='tanh', max_iter=10000, hidden_layer_sizes=(4,2),learning_rate= 'constant')\n",
        "model2.fit(input, actual_output)\n",
        "\n",
        "print('model score:', model2.score(input, actual_output)) \n",
        "print('predicted output:', model2.predict(input)) \n",
        "print('expected output /  xor output is:', np.array([0, 1, 1, 0]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VJWVeJnx1hwk",
        "outputId": "2a931877-06a9-40a8-df26-bdc73dfd4e61"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "model score: 1.0\n",
            "predicted output: [0 1 1 0]\n",
            "expected output /  xor output is: [0 1 1 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "weight = weight + learning_rate * (expected - predicted) * x"
      ],
      "metadata": {
        "id": "aQwC4whYzjL4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(model2.coefs_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FYEkiNayzThi",
        "outputId": "359eef1b-52dd-483a-bd48-34f3ea8efb0c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[array([[-1.6271855 , -1.17334467,  1.89596558,  1.41938974],\n",
            "       [ 0.78513314,  2.04255279,  2.36125044, -1.1055587 ]]), array([[ 2.13537702, -0.45282096],\n",
            "       [ 1.50885508,  1.80073469],\n",
            "       [ 0.39938076, -1.27537289],\n",
            "       [-0.42340309,  1.3004324 ]]), array([[ 1.32986994],\n",
            "       [-2.60959536]])]\n"
          ]
        }
      ]
    }
  ]
}