Prediksi_Churn_Rate_Pada_Sebuah_Bank_Menggunakan_PySpark.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Prediksi Churn Rate Pada Sebuah Bank Menggunakan PySpark.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMC76b1/tW5bsW6sPKDzryL",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alfianhid/Prediksi-Churn-Rate-Pada-Sebuah-Bank-Menggunakan-PySpark/blob/main/Prediksi_Churn_Rate_Pada_Sebuah_Bank_Menggunakan_PySpark.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2m3K9KNrX70Z"
      },
      "source": [
        "**Pertama, kita perlu install JDK karena program/aplikasi Spark berjalan di JVM (berbasis Java)**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wgX7JSe3X6IJ",
        "outputId": "4965fbb7-dc50-45be-fe4f-e6c875733532"
      },
      "source": [
        "# Saya menggunakan JDK 8 karena alasan lebih stabil\n",
        "!apt-get install openjdk-8-jdk"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following additional packages will be installed:\n",
            "  fonts-dejavu-core fonts-dejavu-extra libatk-wrapper-java\n",
            "  libatk-wrapper-java-jni libgail-common libgail18 libgtk2.0-0 libgtk2.0-bin\n",
            "  libgtk2.0-common libxxf86dga1 openjdk-8-jdk-headless openjdk-8-jre\n",
            "  openjdk-8-jre-headless x11-utils\n",
            "Suggested packages:\n",
            "  gvfs openjdk-8-demo openjdk-8-source visualvm icedtea-8-plugin libnss-mdns\n",
            "  fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei\n",
            "  fonts-wqy-zenhei fonts-indic mesa-utils\n",
            "The following NEW packages will be installed:\n",
            "  fonts-dejavu-core fonts-dejavu-extra libatk-wrapper-java\n",
            "  libatk-wrapper-java-jni libgail-common libgail18 libgtk2.0-0 libgtk2.0-bin\n",
            "  libgtk2.0-common libxxf86dga1 openjdk-8-jdk openjdk-8-jdk-headless\n",
            "  openjdk-8-jre openjdk-8-jre-headless x11-utils\n",
            "0 upgraded, 15 newly installed, 0 to remove and 11 not upgraded.\n",
            "Need to get 43.5 MB of archives.\n",
            "After this operation, 163 MB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxxf86dga1 amd64 2:1.1.4-1 [13.7 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 fonts-dejavu-core all 2.37-1 [1,041 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 fonts-dejavu-extra all 2.37-1 [1,953 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11-utils amd64 7.7+3build1 [196 kB]\n",
            "Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 libatk-wrapper-java all 0.33.3-20ubuntu0.1 [34.7 kB]\n",
            "Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libatk-wrapper-java-jni amd64 0.33.3-20ubuntu0.1 [28.3 kB]\n",
            "Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgtk2.0-common all 2.24.32-1ubuntu1 [125 kB]\n",
            "Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgtk2.0-0 amd64 2.24.32-1ubuntu1 [1,769 kB]\n",
            "Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgail18 amd64 2.24.32-1ubuntu1 [14.2 kB]\n",
            "Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgail-common amd64 2.24.32-1ubuntu1 [112 kB]\n",
            "Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgtk2.0-bin amd64 2.24.32-1ubuntu1 [7,536 B]\n",
            "Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jre-headless amd64 8u282-b08-0ubuntu1~18.04 [28.2 MB]\n",
            "Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jre amd64 8u282-b08-0ubuntu1~18.04 [69.7 kB]\n",
            "Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jdk-headless amd64 8u282-b08-0ubuntu1~18.04 [8,267 kB]\n",
            "Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jdk amd64 8u282-b08-0ubuntu1~18.04 [1,630 kB]\n",
            "Fetched 43.5 MB in 2s (19.4 MB/s)\n",
            "Selecting previously unselected package libxxf86dga1:amd64.\n",
            "(Reading database ... 149414 files and directories currently installed.)\n",
            "Preparing to unpack .../00-libxxf86dga1_2%3a1.1.4-1_amd64.deb ...\n",
            "Unpacking libxxf86dga1:amd64 (2:1.1.4-1) ...\n",
            "Selecting previously unselected package fonts-dejavu-core.\n",
            "Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ...\n",
            "Unpacking fonts-dejavu-core (2.37-1) ...\n",
            "Selecting previously unselected package fonts-dejavu-extra.\n",
            "Preparing to unpack .../02-fonts-dejavu-extra_2.37-1_all.deb ...\n",
            "Unpacking fonts-dejavu-extra (2.37-1) ...\n",
            "Selecting previously unselected package x11-utils.\n",
            "Preparing to unpack .../03-x11-utils_7.7+3build1_amd64.deb ...\n",
            "Unpacking x11-utils (7.7+3build1) ...\n",
            "Selecting previously unselected package libatk-wrapper-java.\n",
            "Preparing to unpack .../04-libatk-wrapper-java_0.33.3-20ubuntu0.1_all.deb ...\n",
            "Unpacking libatk-wrapper-java (0.33.3-20ubuntu0.1) ...\n",
            "Selecting previously unselected package libatk-wrapper-java-jni:amd64.\n",
            "Preparing to unpack .../05-libatk-wrapper-java-jni_0.33.3-20ubuntu0.1_amd64.deb ...\n",
            "Unpacking libatk-wrapper-java-jni:amd64 (0.33.3-20ubuntu0.1) ...\n",
            "Selecting previously unselected package libgtk2.0-common.\n",
            "Preparing to unpack .../06-libgtk2.0-common_2.24.32-1ubuntu1_all.deb ...\n",
            "Unpacking libgtk2.0-common (2.24.32-1ubuntu1) ...\n",
            "Selecting previously unselected package libgtk2.0-0:amd64.\n",
            "Preparing to unpack .../07-libgtk2.0-0_2.24.32-1ubuntu1_amd64.deb ...\n",
            "Unpacking libgtk2.0-0:amd64 (2.24.32-1ubuntu1) ...\n",
            "Selecting previously unselected package libgail18:amd64.\n",
            "Preparing to unpack .../08-libgail18_2.24.32-1ubuntu1_amd64.deb ...\n",
            "Unpacking libgail18:amd64 (2.24.32-1ubuntu1) ...\n",
            "Selecting previously unselected package libgail-common:amd64.\n",
            "Preparing to unpack .../09-libgail-common_2.24.32-1ubuntu1_amd64.deb ...\n",
            "Unpacking libgail-common:amd64 (2.24.32-1ubuntu1) ...\n",
            "Selecting previously unselected package libgtk2.0-bin.\n",
            "Preparing to unpack .../10-libgtk2.0-bin_2.24.32-1ubuntu1_amd64.deb ...\n",
            "Unpacking libgtk2.0-bin (2.24.32-1ubuntu1) ...\n",
            "Selecting previously unselected package openjdk-8-jre-headless:amd64.\n",
            "Preparing to unpack .../11-openjdk-8-jre-headless_8u282-b08-0ubuntu1~18.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jre-headless:amd64 (8u282-b08-0ubuntu1~18.04) ...\n",
            "Selecting previously unselected package openjdk-8-jre:amd64.\n",
            "Preparing to unpack .../12-openjdk-8-jre_8u282-b08-0ubuntu1~18.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jre:amd64 (8u282-b08-0ubuntu1~18.04) ...\n",
            "Selecting previously unselected package openjdk-8-jdk-headless:amd64.\n",
            "Preparing to unpack .../13-openjdk-8-jdk-headless_8u282-b08-0ubuntu1~18.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jdk-headless:amd64 (8u282-b08-0ubuntu1~18.04) ...\n",
            "Selecting previously unselected package openjdk-8-jdk:amd64.\n",
            "Preparing to unpack .../14-openjdk-8-jdk_8u282-b08-0ubuntu1~18.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jdk:amd64 (8u282-b08-0ubuntu1~18.04) ...\n",
            "Setting up libgtk2.0-common (2.24.32-1ubuntu1) ...\n",
            "Setting up fonts-dejavu-core (2.37-1) ...\n",
            "Setting up libxxf86dga1:amd64 (2:1.1.4-1) ...\n",
            "Setting up fonts-dejavu-extra (2.37-1) ...\n",
            "Setting up openjdk-8-jre-headless:amd64 (8u282-b08-0ubuntu1~18.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode\n",
            "Setting up libgtk2.0-0:amd64 (2.24.32-1ubuntu1) ...\n",
            "Setting up libgail18:amd64 (2.24.32-1ubuntu1) ...\n",
            "Setting up openjdk-8-jdk-headless:amd64 (8u282-b08-0ubuntu1~18.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javah to provide /usr/bin/javah (javah) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/hsdb to provide /usr/bin/hsdb (hsdb) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/clhsdb to provide /usr/bin/clhsdb (clhsdb) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode\n",
            "Setting up x11-utils (7.7+3build1) ...\n",
            "Setting up libgail-common:amd64 (2.24.32-1ubuntu1) ...\n",
            "Setting up libatk-wrapper-java (0.33.3-20ubuntu0.1) ...\n",
            "Setting up libgtk2.0-bin (2.24.32-1ubuntu1) ...\n",
            "Setting up libatk-wrapper-java-jni:amd64 (0.33.3-20ubuntu0.1) ...\n",
            "Setting up openjdk-8-jre:amd64 (8u282-b08-0ubuntu1~18.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/policytool to provide /usr/bin/policytool (policytool) in auto mode\n",
            "Setting up openjdk-8-jdk:amd64 (8u282-b08-0ubuntu1~18.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/appletviewer to provide /usr/bin/appletviewer (appletviewer) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n",
            "Processing triggers for hicolor-icon-theme (0.17-2) ...\n",
            "Processing triggers for fontconfig (2.12.6-0ubuntu2) ...\n",
            "Processing triggers for mime-support (3.60ubuntu1) ...\n",
            "Processing triggers for libc-bin (2.27-3ubuntu1.4) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JaKvvM4Sc3dk"
      },
      "source": [
        "**Setelah install JDK, kita perlu menge-set path environment Java supaya kodingan kita di Spark bisa berjalan**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GzmEhQKodnXL",
        "outputId": "d30b83e7-908e-4e54-d54e-3799d64f2e7b"
      },
      "source": [
        "import os # library OS berfungsi menjembatansi proses/tugas antara kodingan dengan sistem operasi\n",
        "os.environ[\"JAVA_HOME\"]=\"/usr/lib/jvm/java-8-openjdk-amd64\" # path folder JDK bisa kita dapatkan dari output saat install JDK\n",
        "!echo $JAVA_HOME # cek kembali apakah sudah di-set dengan benar"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/lib/jvm/java-8-openjdk-amd64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2cYdgfrce-f-"
      },
      "source": [
        "**Setelah JDK terinstall dengan benar, baru kita install Spark-nya. Di sini, saya menggunakan PySpark (ngoding Spark pake Python)**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jF87zk1efUw4",
        "outputId": "e4704761-14c7-4ee8-b223-1ffc10e6bab8"
      },
      "source": [
        "!pip install pyspark"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting pyspark\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/27/67/5158f846202d7f012d1c9ca21c3549a58fd3c6707ae8ee823adcaca6473c/pyspark-3.0.2.tar.gz (204.8MB)\n",
            "\u001b[K     |████████████████████████████████| 204.8MB 69kB/s \n",
            "\u001b[?25hCollecting py4j==0.10.9\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl (198kB)\n",
            "\u001b[K     |████████████████████████████████| 204kB 16.7MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.0.2-py2.py3-none-any.whl size=205186687 sha256=9284d8eb67b4e9bb4ceb561ede5646d9549157ac520bc19e6c52a0a69c258f7c\n",
            "  Stored in directory: /root/.cache/pip/wheels/8b/09/da/c1f2859bcc86375dc972c5b6af4881b3603269bcc4c9be5d16\n",
            "Successfully built pyspark\n",
            "Installing collected packages: py4j, pyspark\n",
            "Successfully installed py4j-0.10.9 pyspark-3.0.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BLNxLV3HgkUV"
      },
      "source": [
        "**Selanjutnya kita perlu mengintegrasikan Google Colab dengan Google Drive karena di sana tempat kita menyimpan dataset**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M32SSt13g_Lx",
        "outputId": "d4dac4a3-17b6-4640-ca90-6e2a7fb5a3ae"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive') # path folder default Google Drive di Google Colab"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eFuATTn8hm2L"
      },
      "source": [
        "**Lalu, kita wajib men-start session di Spark sebelum ngoding. Ini berfungsi agar kita bisa menjalankan library-library Spark dengan lancar.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I4t6CUqhi0kz"
      },
      "source": [
        "from pyspark.sql import SparkSession\n",
        "spark = SparkSession.builder.appName('Analisis Data Nasabah Bank').getOrCreate()\n",
        "# Sumber Dataset: https://archive.ics.uci.edu/ml/datasets/bank+marketing (UCI Machine Learning Repository)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xQ_K6P1AC3uL"
      },
      "source": [
        "# **Alur Kerja**\n",
        "# Real Data >> Data Preparation >> Data Cleaning >> Data Transformation >> Data Modeling >> Prediction Process with Machine Learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "frZwm0jKlBe6"
      },
      "source": [
        "**Kemudian tinggal kita lakukan \"read data\" terhadap dataset yang telah kita upload di Google Drive**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bF03v4ShlO9j"
      },
      "source": [
        "# \"inferSchema=True\" berfungsi agar kita bisa menampilkan informasi dari dataframe kita nanti\n",
        "# \"header=True\" berfungsi agar baris pertama pada dataset diubah menjadi header dalam dataframe\n",
        "df = spark.read.csv('/content/drive/MyDrive/Colab Notebooks/Datasets/dataset-nasabah-bank.csv',inferSchema=True,header=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kWLP6NjumxQx"
      },
      "source": [
        "**Setelah itu, kita bisa mencoba melihat dimensi data atau seberapa banyak data pada dataset yang kita miliki**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fpRapvwWmtsC",
        "outputId": "27529afb-e49a-4d80-eb01-d5eb13d26b37"
      },
      "source": [
        "print((df.count(),len(df.columns))) # baris x kolom"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(11162, 17)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tWhXMwCrpspx"
      },
      "source": [
        "**Dilanjutkan dengan mem-print Schema dari dataframe untuk mengetahui informasi dataframe kita**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LjJMKtpaotZ2",
        "outputId": "0d95f1eb-436e-4ca8-eebd-3297ecc05c09"
      },
      "source": [
        "df.printSchema() # nullable=true artinya nilai/value pada variabel bisa kosong atau tanpa nilai"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- age: integer (nullable = true)\n",
            " |-- job: string (nullable = true)\n",
            " |-- marital: string (nullable = true)\n",
            " |-- education: string (nullable = true)\n",
            " |-- default: string (nullable = true)\n",
            " |-- balance: integer (nullable = true)\n",
            " |-- housing: string (nullable = true)\n",
            " |-- loan: string (nullable = true)\n",
            " |-- contact: string (nullable = true)\n",
            " |-- day: integer (nullable = true)\n",
            " |-- month: string (nullable = true)\n",
            " |-- duration: integer (nullable = true)\n",
            " |-- campaign: integer (nullable = true)\n",
            " |-- pdays: integer (nullable = true)\n",
            " |-- previous: integer (nullable = true)\n",
            " |-- poutcome: string (nullable = true)\n",
            " |-- deposit: string (nullable = true)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jlM0nFAAqlZ5"
      },
      "source": [
        "**Next, seperti biasa, kita tampilkan lima baris data teratas pada dataframe kita**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B2UjHI9Dqxww",
        "outputId": "cee06cc3-b6f1-4128-a5de-823b6769aa60"
      },
      "source": [
        "df.show(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+---+----------+-------+---------+-------+-------+-------+----+-------+---+-----+--------+--------+-----+--------+--------+-------+\n",
            "|age|       job|marital|education|default|balance|housing|loan|contact|day|month|duration|campaign|pdays|previous|poutcome|deposit|\n",
            "+---+----------+-------+---------+-------+-------+-------+----+-------+---+-----+--------+--------+-----+--------+--------+-------+\n",
            "| 59|    admin.|married|secondary|     no|   2343|    yes|  no|unknown|  5|  may|    1042|       1|   -1|       0| unknown|    yes|\n",
            "| 56|    admin.|married|secondary|     no|     45|     no|  no|unknown|  5|  may|    1467|       1|   -1|       0| unknown|    yes|\n",
            "| 41|technician|married|secondary|     no|   1270|    yes|  no|unknown|  5|  may|    1389|       1|   -1|       0| unknown|    yes|\n",
            "| 55|  services|married|secondary|     no|   2476|    yes|  no|unknown|  5|  may|     579|       1|   -1|       0| unknown|    yes|\n",
            "| 54|    admin.|married| tertiary|     no|    184|     no|  no|unknown|  5|  may|     673|       2|   -1|       0| unknown|    yes|\n",
            "+---+----------+-------+---------+-------+-------+-------+----+-------+---+-----+--------+--------+-----+--------+--------+-------+\n",
            "only showing top 5 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-gF5dJbKrLuh"
      },
      "source": [
        "**Dari lima baris data tersebut, perlu dilakukan \"data cleaning\" pada kolom yang tak memiliki arti/makna supaya analisis data kita lebih efisien**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aaCaosMkrl5I",
        "outputId": "00198228-f45c-4ebf-8da9-b13ea5785e60"
      },
      "source": [
        "my_data = df.drop(*['default', 'contact', 'day','month']) # menghapus kolom default, contact, day, dan month\n",
        "my_data.columns # menampilkan sisa kolom setelah proses drop kolom dijalankan"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['age',\n",
              " 'job',\n",
              " 'marital',\n",
              " 'education',\n",
              " 'balance',\n",
              " 'housing',\n",
              " 'loan',\n",
              " 'duration',\n",
              " 'campaign',\n",
              " 'pdays',\n",
              " 'previous',\n",
              " 'poutcome',\n",
              " 'deposit']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYRzdP9_w_72"
      },
      "source": [
        "**Dan selanjutnya, kita bisa mencoba menampilkan informasi lain dari dataframe yang kita miliki untuk lebih mengenal data kita**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "agIKX95XxT2_",
        "outputId": "551dde50-11e6-483e-a22a-572df83a7597"
      },
      "source": [
        "my_data.describe().show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-------+------------------+-------+--------+---------+------------------+-------+-----+------------------+------------------+------------------+------------------+--------+-------+\n",
            "|summary|               age|    job| marital|education|           balance|housing| loan|          duration|          campaign|             pdays|          previous|poutcome|deposit|\n",
            "+-------+------------------+-------+--------+---------+------------------+-------+-----+------------------+------------------+------------------+------------------+--------+-------+\n",
            "|  count|             11162|  11162|   11162|    11162|             11162|  11162|11162|             11162|             11162|             11162|             11162|   11162|  11162|\n",
            "|   mean|41.231947679627304|   null|    null|     null|1528.5385235620856|   null| null|371.99381831213043| 2.508421429851281| 51.33040673714388|0.8325568894463358|    null|   null|\n",
            "| stddev|11.913369192215518|   null|    null|     null| 3225.413325946149|   null| null|347.12838571630687|2.7220771816614824|108.75828197197717| 2.292007218670508|    null|   null|\n",
            "|    min|                18| admin.|divorced|  primary|             -6847|     no|   no|                 2|                 1|                -1|                 0| failure|     no|\n",
            "|    max|                95|unknown|  single|  unknown|             81204|    yes|  yes|              3881|                63|               854|                58| unknown|    yes|\n",
            "+-------+------------------+-------+--------+---------+------------------+-------+-----+------------------+------------------+------------------+------------------+--------+-------+\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S0ikxZk4x0Xh"
      },
      "source": [
        "**Nah, dari summary di atas, kita bisa melihat bahwa masih banyak kolom yang memiliki nilai berupa null. Maka dari itu, kita perlu melakukan normalisasi data dengan menghapus data yang memiliki nilai null tersebut.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FhO5yROFzcJ0",
        "outputId": "e275fb44-0068-41ea-ec88-a87da826cd8d"
      },
      "source": [
        "df.na.drop(subset=[\"job\",\"marital\",\"education\",\"housing\",\"loan\",\"poutcome\",\"deposit\"]) \\ # menghapus baris data null hanya pada kolom terpilih\n",
        "   .show(truncate=False)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+---+-----------+--------+---------+-------+-------+-------+----+-------+---+-----+--------+--------+-----+--------+--------+-------+\n",
            "|age|job        |marital |education|default|balance|housing|loan|contact|day|month|duration|campaign|pdays|previous|poutcome|deposit|\n",
            "+---+-----------+--------+---------+-------+-------+-------+----+-------+---+-----+--------+--------+-----+--------+--------+-------+\n",
            "|59 |admin.     |married |secondary|no     |2343   |yes    |no  |unknown|5  |may  |1042    |1       |-1   |0       |unknown |yes    |\n",
            "|56 |admin.     |married |secondary|no     |45     |no     |no  |unknown|5  |may  |1467    |1       |-1   |0       |unknown |yes    |\n",
            "|41 |technician |married |secondary|no     |1270   |yes    |no  |unknown|5  |may  |1389    |1       |-1   |0       |unknown |yes    |\n",
            "|55 |services   |married |secondary|no     |2476   |yes    |no  |unknown|5  |may  |579     |1       |-1   |0       |unknown |yes    |\n",
            "|54 |admin.     |married |tertiary |no     |184    |no     |no  |unknown|5  |may  |673     |2       |-1   |0       |unknown |yes    |\n",
            "|42 |management |single  |tertiary |no     |0      |yes    |yes |unknown|5  |may  |562     |2       |-1   |0       |unknown |yes    |\n",
            "|56 |management |married |tertiary |no     |830    |yes    |yes |unknown|6  |may  |1201    |1       |-1   |0       |unknown |yes    |\n",
            "|60 |retired    |divorced|secondary|no     |545    |yes    |no  |unknown|6  |may  |1030    |1       |-1   |0       |unknown |yes    |\n",
            "|37 |technician |married |secondary|no     |1      |yes    |no  |unknown|6  |may  |608     |1       |-1   |0       |unknown |yes    |\n",
            "|28 |services   |single  |secondary|no     |5090   |yes    |no  |unknown|6  |may  |1297    |3       |-1   |0       |unknown |yes    |\n",
            "|38 |admin.     |single  |secondary|no     |100    |yes    |no  |unknown|7  |may  |786     |1       |-1   |0       |unknown |yes    |\n",
            "|30 |blue-collar|married |secondary|no     |309    |yes    |no  |unknown|7  |may  |1574    |2       |-1   |0       |unknown |yes    |\n",
            "|29 |management |married |tertiary |no     |199    |yes    |yes |unknown|7  |may  |1689    |4       |-1   |0       |unknown |yes    |\n",
            "|46 |blue-collar|single  |tertiary |no     |460    |yes    |no  |unknown|7  |may  |1102    |2       |-1   |0       |unknown |yes    |\n",
            "|31 |technician |single  |tertiary |no     |703    |yes    |no  |unknown|8  |may  |943     |2       |-1   |0       |unknown |yes    |\n",
            "|35 |management |divorced|tertiary |no     |3837   |yes    |no  |unknown|8  |may  |1084    |1       |-1   |0       |unknown |yes    |\n",
            "|32 |blue-collar|single  |primary  |no     |611    |yes    |no  |unknown|8  |may  |541     |3       |-1   |0       |unknown |yes    |\n",
            "|49 |services   |married |secondary|no     |-8     |yes    |no  |unknown|8  |may  |1119    |1       |-1   |0       |unknown |yes    |\n",
            "|41 |admin.     |married |secondary|no     |55     |yes    |no  |unknown|8  |may  |1120    |2       |-1   |0       |unknown |yes    |\n",
            "|49 |admin.     |divorced|secondary|no     |168    |yes    |yes |unknown|8  |may  |513     |1       |-1   |0       |unknown |yes    |\n",
            "+---+-----------+--------+---------+-------+-------+-------+----+-------+---+-----+--------+--------+-----+--------+--------+-------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j75vKNDP0Jbt"
      },
      "source": [
        "**Setelah berhasil menormalisasi data, kita lanjutkan lagi eksplorasi dataframe kita supaya hubungan terasa lebih mesra**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6kRv6JJt0aV2",
        "outputId": "a8e51ad5-585c-463a-a44f-f909f5689781"
      },
      "source": [
        "# Di sini, kita akan menghitung jumlah data (baris) pada setiap kategori dari sebuah variabel data\n",
        "my_data.groupBy('job').count().show()\n",
        "print()\n",
        "my_data.groupBy('marital').count().show()\n",
        "print()\n",
        "my_data.groupBy('education').count().show()\n",
        "print()\n",
        "my_data.groupBy('loan').count().show()\n",
        "print()\n",
        "my_data.groupBy('poutcome').count().show()\n",
        "print()\n",
        "my_data.groupBy('deposit').count().show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-------------+-----+\n",
            "|          job|count|\n",
            "+-------------+-----+\n",
            "|   management| 2566|\n",
            "|      retired|  778|\n",
            "|      unknown|   70|\n",
            "|self-employed|  405|\n",
            "|      student|  360|\n",
            "|  blue-collar| 1944|\n",
            "| entrepreneur|  328|\n",
            "|       admin.| 1334|\n",
            "|   technician| 1823|\n",
            "|     services|  923|\n",
            "|    housemaid|  274|\n",
            "|   unemployed|  357|\n",
            "+-------------+-----+\n",
            "\n",
            "\n",
            "+--------+-----+\n",
            "| marital|count|\n",
            "+--------+-----+\n",
            "|divorced| 1293|\n",
            "| married| 6351|\n",
            "|  single| 3518|\n",
            "+--------+-----+\n",
            "\n",
            "\n",
            "+---------+-----+\n",
            "|education|count|\n",
            "+---------+-----+\n",
            "|  unknown|  497|\n",
            "| tertiary| 3689|\n",
            "|secondary| 5476|\n",
            "|  primary| 1500|\n",
            "+---------+-----+\n",
            "\n",
            "\n",
            "+----+-----+\n",
            "|loan|count|\n",
            "+----+-----+\n",
            "|  no| 9702|\n",
            "| yes| 1460|\n",
            "+----+-----+\n",
            "\n",
            "\n",
            "+--------+-----+\n",
            "|poutcome|count|\n",
            "+--------+-----+\n",
            "| success| 1071|\n",
            "| unknown| 8326|\n",
            "|   other|  537|\n",
            "| failure| 1228|\n",
            "+--------+-----+\n",
            "\n",
            "\n",
            "+-------+-----+\n",
            "|deposit|count|\n",
            "+-------+-----+\n",
            "|     no| 5873|\n",
            "|    yes| 5289|\n",
            "+-------+-----+\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TQ2aDVDU1Yj3"
      },
      "source": [
        "**Akhirnya kita sampai di pintu gerbang akhir Data Pre-processing. Mulai dari sini, kita akan memfungsikan dataset kita untuk sebuah fungsi prediksi. Adapun yang akan kita prediksi adalah \"churn rate\" dari setiap nasabah bank. Churn rate adalah persentase nasabah yang berhenti menggunakan produk dan layanan bank. Ini sangat penting untuk dianalisis karena menjadi acuan dalam menentukan masa depan bank.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dzsp8sbv167f"
      },
      "source": [
        "from pyspark.ml.feature import StringIndexer, OneHotEncoder\n",
        "\n",
        "# Membuat objek dari StringIndexer class dan menge-set kolom input & output\n",
        "SI_job = StringIndexer(inputCol='job',outputCol='job_Index')\n",
        "SI_marital = StringIndexer(inputCol='marital',outputCol='marital_Index')\n",
        "SI_education = StringIndexer(inputCol='education',outputCol='education_Index')\n",
        "SI_housing = StringIndexer(inputCol='housing',outputCol='housing_Index')\n",
        "SI_loan = StringIndexer(inputCol='loan',outputCol='loan_Index')\n",
        "SI_poutcome = StringIndexer(inputCol='poutcome',outputCol='poutcome_Index')\n",
        "SI_deposit = StringIndexer(inputCol='deposit',outputCol='deposit_Index')\n",
        "\n",
        "\n",
        "# Mentransformasi / mengubah data ke dalam bentuk yang baru untuk mempermudah proses prediksi\n",
        "my_data = SI_job.fit(my_data).transform(my_data)\n",
        "my_data = SI_marital.fit(my_data).transform(my_data)\n",
        "my_data = SI_education.fit(my_data).transform(my_data)\n",
        "my_data = SI_housing.fit(my_data).transform(my_data)\n",
        "my_data = SI_loan.fit(my_data).transform(my_data)\n",
        "my_data = SI_poutcome.fit(my_data).transform(my_data)\n",
        "my_data = SI_deposit.fit(my_data).transform(my_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xrEel-MS3kjp"
      },
      "source": [
        "**Lalu, kita lihat hasil transformasi data tadi. Di sini, kita akan melihat 10 baris data teratas dari dataframe.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VvZ_rlVH3z0S",
        "outputId": "eab33917-0ed7-48c5-dc3e-f9f33fd8594a"
      },
      "source": [
        "my_data.select('job', 'job_Index', 'marital', 'marital_Index','housing','housing_Index','poutcome','poutcome_Index','deposit','deposit_Index').show(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+----------+---------+--------+-------------+-------+-------------+--------+--------------+-------+-------------+\n",
            "|       job|job_Index| marital|marital_Index|housing|housing_Index|poutcome|poutcome_Index|deposit|deposit_Index|\n",
            "+----------+---------+--------+-------------+-------+-------------+--------+--------------+-------+-------------+\n",
            "|    admin.|      3.0| married|          0.0|    yes|          1.0| unknown|           0.0|    yes|          1.0|\n",
            "|    admin.|      3.0| married|          0.0|     no|          0.0| unknown|           0.0|    yes|          1.0|\n",
            "|technician|      2.0| married|          0.0|    yes|          1.0| unknown|           0.0|    yes|          1.0|\n",
            "|  services|      4.0| married|          0.0|    yes|          1.0| unknown|           0.0|    yes|          1.0|\n",
            "|    admin.|      3.0| married|          0.0|     no|          0.0| unknown|           0.0|    yes|          1.0|\n",
            "|management|      0.0|  single|          1.0|    yes|          1.0| unknown|           0.0|    yes|          1.0|\n",
            "|management|      0.0| married|          0.0|    yes|          1.0| unknown|           0.0|    yes|          1.0|\n",
            "|   retired|      5.0|divorced|          2.0|    yes|          1.0| unknown|           0.0|    yes|          1.0|\n",
            "|technician|      2.0| married|          0.0|    yes|          1.0| unknown|           0.0|    yes|          1.0|\n",
            "|  services|      4.0|  single|          1.0|    yes|          1.0| unknown|           0.0|    yes|          1.0|\n",
            "+----------+---------+--------+-------------+-------+-------------+--------+--------------+-------+-------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UG6_vlzJ3342"
      },
      "source": [
        "**Setelah itu, kita akan mentransformasi data lagi ke dalam kolom OneHotEncoder sebelum seleksi fitur dari vektor biner**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fvPws86v33cE",
        "outputId": "3406c50f-6bc0-44a4-eb3e-9b273669d2ea"
      },
      "source": [
        "# Buat objek dan set kolom input & output\n",
        "OHE = OneHotEncoder(inputCols=['job_Index', 'marital_Index','education_Index','housing_Index','loan_Index','poutcome_Index','deposit_Index'],outputCols=['job_OHE', 'marital_OHE','education_OHE','housing_OHE','loan_OHE','poutcome_OHE','deposit_OHE'])\n",
        "\n",
        "# Proses transformasi data\n",
        "my_data = OHE.fit(my_data).transform(my_data)\n",
        "\n",
        "# Lihat hasil transformasi data\n",
        "my_data.select('job', 'job_Index', 'job_OHE','education','education_Index','education_OHE').show(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+----------+---------+--------------+---------+---------------+-------------+\n",
            "|       job|job_Index|       job_OHE|education|education_Index|education_OHE|\n",
            "+----------+---------+--------------+---------+---------------+-------------+\n",
            "|    admin.|      3.0|(11,[3],[1.0])|secondary|            0.0|(3,[0],[1.0])|\n",
            "|    admin.|      3.0|(11,[3],[1.0])|secondary|            0.0|(3,[0],[1.0])|\n",
            "|technician|      2.0|(11,[2],[1.0])|secondary|            0.0|(3,[0],[1.0])|\n",
            "|  services|      4.0|(11,[4],[1.0])|secondary|            0.0|(3,[0],[1.0])|\n",
            "|    admin.|      3.0|(11,[3],[1.0])| tertiary|            1.0|(3,[1],[1.0])|\n",
            "|management|      0.0|(11,[0],[1.0])| tertiary|            1.0|(3,[1],[1.0])|\n",
            "|management|      0.0|(11,[0],[1.0])| tertiary|            1.0|(3,[1],[1.0])|\n",
            "|   retired|      5.0|(11,[5],[1.0])|secondary|            0.0|(3,[0],[1.0])|\n",
            "|technician|      2.0|(11,[2],[1.0])|secondary|            0.0|(3,[0],[1.0])|\n",
            "|  services|      4.0|(11,[4],[1.0])|secondary|            0.0|(3,[0],[1.0])|\n",
            "+----------+---------+--------------+---------+---------------+-------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9XypqTMp5Q1a"
      },
      "source": [
        "**Lalu, kita transformasi lagi menggunakan VectorAssembler supaya kita bisa melakukan seleksi fitur**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Evp5zIgz5lVV"
      },
      "source": [
        "from pyspark.ml.feature import VectorAssembler # men-transformasi list menjadi vektor biner\n",
        "\n",
        "# menge-set kolom input & output dari transformasi data\n",
        "assembler = VectorAssembler(inputCols=['age',\n",
        "                                       'job_Index',\n",
        "                                       'marital_Index',\n",
        "                                       'education_Index',\n",
        "                                       'balance',\n",
        "                                       'housing_Index',\n",
        "                                       'loan_Index',\n",
        "                                       'duration',\n",
        "                                       'campaign',\n",
        "                                       'pdays',\n",
        "                                       'previous',\n",
        "                                       'poutcome_Index',\n",
        "                                       'job_OHE',\n",
        "                                       'marital_OHE',\n",
        "                                       'housing_OHE',\n",
        "                                       'education_OHE',\n",
        "                                       'loan_OHE',\n",
        "                                       'poutcome_OHE'],\n",
        "                           outputCol='features')\n",
        "\n",
        "# Proses transformasi data\n",
        "final_data = assembler.transform(my_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjfys8Uh7G2p"
      },
      "source": [
        "**Lanjut seperti biasa, menampilkan 10 baris data teratas dari dataframe**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "idEbKMlm7Pxl",
        "outputId": "fff1811a-16c5-4649-844c-e1d962fc67d1"
      },
      "source": [
        "final_data.select('features','deposit_Index').show(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+--------------------+-------------+\n",
            "|            features|deposit_Index|\n",
            "+--------------------+-------------+\n",
            "|(33,[0,1,4,5,7,8,...|          1.0|\n",
            "|(33,[0,1,4,7,8,9,...|          1.0|\n",
            "|(33,[0,1,4,5,7,8,...|          1.0|\n",
            "|(33,[0,1,4,5,7,8,...|          1.0|\n",
            "|(33,[0,1,3,4,7,8,...|          1.0|\n",
            "|(33,[0,2,3,5,6,7,...|          1.0|\n",
            "|(33,[0,3,4,5,6,7,...|          1.0|\n",
            "|(33,[0,1,2,4,5,7,...|          1.0|\n",
            "|(33,[0,1,4,5,7,8,...|          1.0|\n",
            "|(33,[0,1,2,4,5,7,...|          1.0|\n",
            "+--------------------+-------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtIxwI6g7Viq"
      },
      "source": [
        "**Nah, dari hasil seleksi fitur di atas, kemudian akan digunakan sebagai data model untuk memprediksi churn rate**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G5C8hrJo7jkG",
        "outputId": "70e3c931-0cea-45ae-f945-e80c0b3a41c5"
      },
      "source": [
        "model_df = final_data.select(['features','deposit_Index'])\n",
        "model_df = model_df.withColumnRenamed(\"deposit_Index\",\"label\") # mengubah nama kolom supaya lebih mudah dimengerti\n",
        "model_df.printSchema()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- features: vector (nullable = true)\n",
            " |-- label: double (nullable = false)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EtZzUgXi779M"
      },
      "source": [
        "**Kemudian data model yang telah dibuat tadi akan dilakukan training dan testing, sehingga nantinya bisa menghasilkan nilai churn rate**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mxr_cfn08FZI"
      },
      "source": [
        "training_df,test_df = model_df.randomSplit([0.75,0.25]) # 75% dari data untuk training dan 25% dari data untuk testing"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2n4LzCi8XLJ"
      },
      "source": [
        "**Yak lanjut aja dengan membuat model regresi logistik untuk melihat akurasi training dan testing kita**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L5NArawg88WN"
      },
      "source": [
        "from pyspark.ml.classification import LogisticRegression # Prediksi menggunakan Metode Logistic Regression\n",
        "log_reg = LogisticRegression().fit(training_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_RJTtdQu9V5r"
      },
      "source": [
        "**Akhirnya sampai di penghujung kodingan wkwkwk. Daripada tambah pusing, langsung aja print akurasi hasil training dan testing**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yr4aOoQS9W8o",
        "outputId": "a89c634b-04ce-4f3e-fa02-f8a5652fcc54"
      },
      "source": [
        "lr_summary = log_reg.summary\n",
        "lr_summary.accuracy # Akurasi secara keseluruhan = 79%. Hasil akurasi biasanya berkorelasi dengan konsep underfitting/overfitting.\n",
        "# Referensi bacaan mengenai konsep underfitting/overfitting: https://s.id/yhXPu "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7973215353342102"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GCvg4BGkHA7d"
      },
      "source": [
        "**Kita bandingkan dengan ambang batas area under ROC untuk mengetahui seberapa baik performa prediksi kita**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EV1I6LsJ--_w",
        "outputId": "3930b9af-eb0c-425d-e18c-f77398eab90b"
      },
      "source": [
        "# Referensi bacaan mengenai threshold ROC pada Logistic Regression: https://s.id/yheC6\n",
        "lr_summary.areaUnderROC # Threshold ROC = 87%. Karena akurasi < Threshold, maka hasil prediksi belum baik."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8792386115297004"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykHjb80N_JCX"
      },
      "source": [
        "**And finally, last but not least, kita tampilkan hasil prediksi churn rate dari dataframe yang telah kita train dan test tadi**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_9DAd06F_XKr",
        "outputId": "9dc12b6d-8d40-4e27-e205-bf0b63b3302b"
      },
      "source": [
        "predictions = log_reg.transform(test_df)\n",
        "predictions.select('label','prediction').show(10) # menampilkan 10 baris data teratas"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-----+----------+\n",
            "|label|prediction|\n",
            "+-----+----------+\n",
            "|  1.0|       1.0|\n",
            "|  1.0|       0.0|\n",
            "|  0.0|       0.0|\n",
            "|  0.0|       0.0|\n",
            "|  0.0|       0.0|\n",
            "|  1.0|       0.0|\n",
            "|  1.0|       1.0|\n",
            "|  1.0|       1.0|\n",
            "|  1.0|       0.0|\n",
            "|  0.0|       0.0|\n",
            "+-----+----------+\n",
            "only showing top 10 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JbmBIAug_iLn"
      },
      "source": [
        "**Yap! Dapat dilihat bahwa hasil prediksi kita kurang sempurna. Namun yang penting, kita sudah berhasil tahu dan belajar mengenai langkah-langkah dari awal sampai akhir untuk menganalisis data menggunakan PySpark. Atas perhatiannya, saya ucapkan terima kasih :)**"
      ]
    }
  ]
}