CENSUS_DATA_README.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0sGQE4jfN2BM"
      },
      "source": [
        "# General Description\n",
        "This data was extracted from the census bureau database found at\n",
        "https://www2.census.gov/programs-surveys/acs/data/pums/2019/1-Year/ (1 year census data), and https://www2.census.gov/programs-surveys/acs/data/pums/2019/5-Year/ (5 year census data).  \n",
        "Donor: Bargav Jayaraman and Zihao Su, University of Virginia. e-mail: zs3pv@virginia.edu for questions.  \n",
        "The data records were filtered with conditions that AGEP>16 (age is older than 16 years old) and WKHP>0 (usual hours worked per week in the past 12 months is more than 0).  \n",
        "The data has 1685316 row records for the 1 year census data and 8199834 row records for the 5 year census data. The data is split into features and labels. To see the data processing methods and the data dictionary, go to the `Features` and `Label` sections. To see how to import and use the data, go to the `Usage` section. To see benchmarks of the training models using the data, go to the `Benchmarks` section. To see how to reproduce the data, go to the `Steps to Reproduce Data` section. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EybCPcMfOTn_"
      },
      "source": [
        "# Features\n",
        "Found in `census_features.p` or `census_features.csv`, the features contain 13 columns, whose values are extracted from the raw census data and processed. Details of each column are listed below in order, along with the processing methods and data dictionary.\n",
        "- `AGEP`: age in years. Only records with AGEP>16 are included in the dataset. \n",
        "- `COW`: class of worker. The values are shifted down by 1, so they start at 0.\n",
        "\t* 0: 'Private For-Profit'\n",
        "\t* 1: 'Private Non-Profit'\n",
        "\t* 2: 'Local Govt'\n",
        "\t* 3: 'State Govt'\n",
        "\t* 4: 'Federal Govt'\n",
        "\t* 5: 'Self-Employed Other'\n",
        "\t* 6: 'Self-Employed Own'\n",
        "\t* 7: 'Unpaid Job'\n",
        "\t* 8: 'Unemployed'\n",
        "- `SCHL`: education attainment. The values are shifted down by 1, so they start at 0.\n",
        "\t* 0: 'No schooling completed'\n",
        "\t* 1: 'Nursery school, preschool'\n",
        "\t* 2: 'Kindergarten'\n",
        "\t* 3: 'Grade 1'\n",
        "\t* 4: 'Grade 2'\n",
        "\t* 5: 'Grade 3'\n",
        "\t* 6: 'Grade 4'\n",
        "\t* 7: 'Grade 5' \n",
        "\t* 8: 'Grade 6'\n",
        "\t* 9: 'Grade 7'\n",
        "\t* 10: 'Grade 8'\n",
        "\t* 11: 'Grade 9'\n",
        "\t* 12: 'Grade 10'\n",
        "\t* 13: 'Grade 11'\n",
        "\t* 14: '12th grade - no diploma'\n",
        "\t* 15: 'Regular high school diploma'\n",
        "\t* 16: 'GED or alternative credential'\n",
        "\t* 17: 'Some college, but less than 1 year'\n",
        "\t* 18: '1 or more years of college credit, no degree'\n",
        "\t* 19: 'Associates degree'\n",
        "\t* 20: 'Bachelors degree'\n",
        "\t* 21: 'Masters degree'\n",
        "\t* 22: 'Professional degree beyond a bachelors degree'\n",
        "\t* 23: 'Doctorate degree'\n",
        "- `MAR`: marital status. The values are shifted down by 1, so they start at 0.\n",
        "\t* 0: 'Married'\n",
        "\t* 1: 'Widowed'\n",
        "\t* 2: 'Divorced'\n",
        "\t* 3: 'Separated'\n",
        "\t* 4: 'Never married'\n",
        "- `RAC1P`: recoded detailed race code. Alaskan Native and American Indians are combined into a \"Native American\" class. The values are then replaced by their ranks of the sorted distinct values (in ascending order). Thus, the values start at 0 and are consecutive integers.\n",
        "\t* 0: 'White'\n",
        "\t* 1: 'Black'\n",
        "\t* 2: 'Native American'\n",
        "\t* 3: 'Asian'\n",
        "\t* 4: 'Pacific Islander'\n",
        "\t* 5: 'Some Other Race'\n",
        "\t* 6: 'Two or More Races'\n",
        "- `SEX`: sex. The values are shifted down by 1, so they start at 0.\n",
        "\t* 0: 'Male'\n",
        "\t* 1: 'Female'\n",
        "- `DREM`: cognitive difficulty. The 'no' value is changed from 2 to 0, so the values start at 0.\n",
        "\t* 0: 'No'\n",
        "\t* 1: 'Yes'\n",
        "- `DPHY`: ambulatory difficulty. The 'no' value is changed from 2 to 0, so the values start at 0.\n",
        "\t* 0: 'No'\n",
        "\t* 1: 'Yes'\n",
        "- `DEAR`: hearing difficulty. The 'no' value is changed from 2 to 0, so the values start at 0.\n",
        "\t* 0: 'No'\n",
        "\t* 1: 'Yes'\n",
        "- `DEYE`: vision difficulty. The 'no' value is changed from 2 to 0, so the values start at 0.\n",
        "\t* 0: 'No'\n",
        "\t* 1: 'Yes'\n",
        "- `WKHP`: usual hours worked per week in the past 12 months. Only Records with WKHP>0 are included in the dataset.\n",
        "- `WAOB`: world area of birth. The values are shifted down by 1, so they start at 0.\n",
        "\t* 0: 'US state'\n",
        "\t* 1: 'PR and US Island Areas'\n",
        "\t* 2: 'Latin America'\n",
        "\t* 3: 'Asia'\n",
        "\t* 4: 'Europe'\n",
        "\t* 5: 'Africa'\n",
        "\t* 6: 'Northern America'\n",
        "\t* 7: 'Oceania and at Sea'\n",
        "- `ST`: state code. The values are replaced by their ranks of the sorted distinct values (in ascending order). Thus, the values start at 0 and are consecutive integers.\n",
        "\t* 0:'Alabama/AL'\n",
        "    * 1:'Alaska/AK'\n",
        "    * 2:'Arizona/AZ'\n",
        "    * 3:'Arkansas/AR'\n",
        "    * 4:'California/CA'\n",
        "    * 5:'Colorado/CO'\n",
        "    * 6:'Connecticut/CT'\n",
        "    * 7:'Delaware/DE'\n",
        "    * 8:'District of Columbia/DC'\n",
        "    * 9:'Florida/FL'\n",
        "    * 10:'Georgia/GA'\n",
        "    * 11:'Hawaii/HI'\n",
        "    * 12:'Idaho/ID'\n",
        "    * 13:'Illinois/IL'\n",
        "    * 14:'Indiana/IN'\n",
        "    * 15:'Iowa/IA'\n",
        "    * 16:'Kansas/KS'\n",
        "    * 17:'Kentucky/KY'\n",
        "    * 18:'Louisiana/LA'\n",
        "    * 19:'Maine/ME'\n",
        "    * 20:'Maryland/MD'\n",
        "    * 21:'Massachusetts/MA'\n",
        "    * 22:'Michigan/MI'\n",
        "    * 23:'Minnesota/MN'\n",
        "    * 24:'Mississippi/MS'\n",
        "    * 25:'Missouri/MO'\n",
        "    * 26:'Montana/MT'\n",
        "    * 27:'Nebraska/NE'\n",
        "    * 28:'Nevada/NV'\n",
        "    * 29:'New Hampshire/NH'\n",
        "    * 30:'New Jersey/NJ'\n",
        "    * 31:'New Mexico/NM'\n",
        "    * 32:'New York/NY'\n",
        "    * 33:'North Carolina/NC'\n",
        "    * 34:'North Dakota/ND'\n",
        "    * 35:'Ohio/OH'\n",
        "    * 36:'Oklahoma/OK'\n",
        "    * 37:'Oregon/OR'\n",
        "    * 38:'Pennsylvania/PA'\n",
        "    * 39:'Rhode Island/RI'\n",
        "    * 40:'South Carolina/SC'\n",
        "    * 41:'South Dakota/SD'\n",
        "    * 42:'Tennessee/TN'\n",
        "    * 43:'Texas/TX'\n",
        "    * 44:'Utah/UT'\n",
        "    * 45:'Vermont/VT'\n",
        "    * 46:'Virginia/VA'\n",
        "    * 47:'Washington/WA'\n",
        "    * 48:'West Virginia/WV'\n",
        "    * 49:'Wisconsin/WI'\n",
        "    * 50:'Wyoming/WY'\n",
        "    * 51:'Puerto Rico/PR'\n",
        "\n",
        "After all the column values are processed as described above, all values are divided by the maximum value of their respective column, which normalizes the values to be between 0 and 1. Therefore, the actual values stored in the data files **DO NOT** correspond directly to the data dictionary above. \n",
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l3L5ytqlOyr-"
      },
      "source": [
        "# Label\n",
        "Found in `census_labels.p` or `census_labels.csv`, the labels contain 1 column of value that represents whether a person's total income is at least \\$50,000. The value is 1 for records with total income equal to or over \\$50,000 (obtained by filtering records through PINCP>=50,000, where PINCP is the total person's income). The value is 0 for records the remaining records."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GlntWiQRO3sx"
      },
      "source": [
        "# Usage\n",
        "## Pickle files\n",
        "Prerequisites: [Python 3.8](https://www.python.org/downloads/release/python-380/), [pickle](https://pypi.org/project/cloudpickle/), [numpy](https://pypi.org/project/numpy/).  \n",
        "To load a `.p` file, use: `data = pickle.load(open(<DATA_PATH>, 'rb'))`.  \n",
        "Sample usage code is seen below: "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PqOchg7IQeYi"
      },
      "source": [
        "Download the 1 year data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s_-QPW5tQMw_",
        "outputId": "b4432f7a-f74c-4c9b-c812-7aa7c48a42ec"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1oMaHqFJutp0RmAG8pHrmF1k_2H2vAegE\n",
            "To: /content/census_labels.p\n",
            "100% 13.5M/13.5M [00:00<00:00, 81.2MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1aDxPdt8iB4zUc7joWuieSQpRIhXcKPI2\n",
            "To: /content/census_feature_desc.p\n",
            "100% 2.52k/2.52k [00:00<00:00, 1.62MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=13UvEGw-I0Ylu5o-9uDLHCU-A3KAL7MiJ\n",
            "To: /content/census_features.p\n",
            "100% 175M/175M [00:01<00:00, 128MB/s]\n"
          ]
        }
      ],
      "source": [
        "!gdown --id 1oMaHqFJutp0RmAG8pHrmF1k_2H2vAegE\n",
        "!gdown --id 1aDxPdt8iB4zUc7joWuieSQpRIhXcKPI2\n",
        "!gdown --id 13UvEGw-I0Ylu5o-9uDLHCU-A3KAL7MiJ"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u4nZW9lKR7A9"
      },
      "source": [
        "Alternatively, download the 5 year data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JywRntsTR_FM"
      },
      "outputs": [],
      "source": [
        "!gdown --id 1jGw9TnsdC8nXxiCCK46mqNbIRkWOZpHU\n",
        "!gdown --id 1M7ms22gfdE1W1GecrWIaghWCgKOMY8lI\n",
        "!gdown --id 1L-X-nplPyISi85W8YuI-m6eB2pdpMeoG"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g3qPhbsmN1am"
      },
      "outputs": [],
      "source": [
        "import pickle,numpy as np"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 218
        },
        "id": "zNnno_eZNyX9",
        "outputId": "051bf6c2-0f66-4b7e-e900-325cfe7e194f"
      },
      "outputs": [
        {
          "ename": "NameError",
          "evalue": "ignored",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-ae490c791f8e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpickle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'./census_features.p'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'rb'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat32\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Dimension of X: {X.shape}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'pickle' is not defined"
          ]
        }
      ],
      "source": [
        "X = pickle.load(open('./census_features.p', 'rb'))\n",
        "X = np.array(X, dtype=np.float32)\n",
        "print(f\"Dimension of X: {X.shape}\")\n",
        "print(X[:3])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "juOsprqORi42"
      },
      "source": [
        "Note that the features matrix has 13 column values in each row, which correspond to columns in the `Features` in order."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qU88rPmNRmy2"
      },
      "source": [
        "Next, read the labels data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w0zysWitRrDg",
        "outputId": "bbd0601e-22f9-4e3e-afa3-b775955a760e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Length of y: 1685316\n",
            "[0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0]\n"
          ]
        }
      ],
      "source": [
        "y = pickle.load(open('./census_labels.p', 'rb'))\n",
        "y = np.array(y, dtype=np.int32)\n",
        "print(f\"Length of y: {len(y)}\")\n",
        "print(y[:20])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "neauFoEkShNn"
      },
      "source": [
        "Read other helpful information"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z34aPvBqRuMD",
        "outputId": "eeeb6e8b-ab5e-429a-ff53-fda5d7fd2498"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'DREM': 6, 'DPHY': 7, 'DEAR': 8, 'DEYE': 9, 'SEX': 5, 'COW': 1, 'MAR': 3, 'RAC1P': 4, 'WAOB': 11, 'SCHL': 2}\n"
          ]
        }
      ],
      "source": [
        "attribute_idx, attribute_dict, max_attr_vals = pickle.load(open('./census_feature_desc.p', 'rb'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RqzAtCqQStIE"
      },
      "source": [
        "`attribute_idx` contains the column number of features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NNjjj6xgSwaN",
        "outputId": "12baaaec-c105-4ff8-f1f5-e9586c43e1f7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'DREM': 6, 'DPHY': 7, 'DEAR': 8, 'DEYE': 9, 'SEX': 5, 'COW': 1, 'MAR': 3, 'RAC1P': 4, 'WAOB': 11, 'SCHL': 2}\n"
          ]
        }
      ],
      "source": [
        "print(attribute_idx)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9NGvkLrMN0QL"
      },
      "source": [
        "`attribute_dict` contains the data dictionary for each categorical feature. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "osg7zb7gS3hr",
        "outputId": "169da8bb-16af-47c9-fb2c-0a8bd5599c5d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{6: {0: 'No Cognitive Difficulty', 1: 'Cognitive Difficulty'}, 7: {0: 'No Ambulatory Difficulty', 1: 'Ambulatory Difficulty'}, 8: {0: 'No Hearing Difficulty', 1: 'Hearing Difficulty'}, 9: {0: 'No Vision Difficulty', 1: 'Vision Difficulty'}, 5: {0: 'Male', 1: 'Female'}, 1: {0: 'Private For-Profit', 1: 'Private Non-Profit', 2: 'Local Govt', 3: 'State Govt', 4: 'Federal Govt', 5: 'Self-Employed Other', 6: 'Self-Employed Own', 7: 'Unpaid Job', 8: 'Unemployed'}, 3: {0: 'Married', 1: 'Widowed', 2: 'Divorced', 3: 'Separated', 4: 'Never married'}, 4: {0: 'White', 1: 'Black', 2: 'Native American', 3: 'Asian', 4: 'Pacific Islander', 5: 'Some Other Race', 6: 'Two or More Races'}, 11: {0: 'US state', 1: 'PR and US Island Areas', 2: 'Latin America', 3: 'Asia', 4: 'Europe', 5: 'Africa', 6: 'Northern America', 7: 'Oceania and at Sea'}, 13: {0: 'Alabama/AL', 1: 'Alaska/AK', 2: 'Arizona/AZ', 3: 'Arkansas/AR', 4: 'California/CA', 5: 'Colorado/CO', 6: 'Connecticut/CT', 7: 'Delaware/DE', 8: 'District of Columbia/DC', 9: 'Florida/FL', 10: 'Georgia/GA', 11: 'Hawaii/HI', 12: 'Idaho/ID', 13: 'Illinois/IL', 14: 'Indiana/IN', 15: 'Iowa/IA', 16: 'Kansas/KS', 17: 'Kentucky/KY', 18: 'Louisiana/LA', 19: 'Maine/ME', 20: 'Maryland/MD', 21: 'Massachusetts/MA', 22: 'Michigan/MI', 23: 'Minnesota/MN', 24: 'Mississippi/MS', 25: 'Missouri/MO', 26: 'Montana/MT', 27: 'Nebraska/NE', 28: 'Nevada/NV', 29: 'New Hampshire/NH', 30: 'New Jersey/NJ', 31: 'New Mexico/NM', 32: 'New York/NY', 33: 'North Carolina/NC', 34: 'North Dakota/ND', 35: 'Ohio/OH', 36: 'Oklahoma/OK', 37: 'Oregon/OR', 38: 'Pennsylvania/PA', 39: 'Rhode Island/RI', 40: 'South Carolina/SC', 41: 'South Dakota/SD', 42: 'Tennessee/TN', 43: 'Texas/TX', 44: 'Utah/UT', 45: 'Vermont/VT', 46: 'Virginia/VA', 47: 'Washington/WA', 48: 'West Virginia/WV', 49: 'Wisconsin/WI', 50: 'Wyoming/WY', 51: 'Puerto Rico/PR'}, 2: {0: 'No schooling completed', 1: 'Nursery school, preschool', 2: 'Kindergarten', 3: 'Grade 1', 4: 'Grade 2', 5: 'Grade 3', 6: 'Grade 4', 7: 'Grade 5', 8: 'Grade 6', 9: 'Grade 7', 10: 'Grade 8', 11: 'Grade 9', 12: 'Grade 10', 13: 'Grade 11', 14: '12th grade - no diploma', 15: 'Regular high school diploma', 16: 'GED or alternative credential', 17: 'Some college, but less than 1 year', 18: '1 or more years of college credit, no degree', 19: 'Associates degree', 20: 'Bachelors degree', 21: 'Masters degree', 22: 'Professional degree beyond a bachelors degree', 23: 'Doctorate degree'}}\n"
          ]
        }
      ],
      "source": [
        "print(attribute_dict)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uWDsfd5-S9tt"
      },
      "source": [
        "`max_attr_vals` contains the maximum value for each column before normalization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "reCydA1cTAa9"
      },
      "outputs": [],
      "source": [
        "print(max_attr_vals)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zTIUpgJnTDvu"
      },
      "source": [
        "You can reverse the normalization and get values that correspond to the data dictionary"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yaBpX835TGhE",
        "outputId": "82f465af-d8b0-4c59-a5a5-0ad9810f226a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[[20.  0. 18. ... 30.  0.  7.]\n",
            " [30.  3. 15. ... 40.  0.  7.]\n",
            " [44.  3. 15. ... 42.  0.  7.]\n",
            " ...\n",
            " [46.  0. 20. ... 40.  0.  6.]\n",
            " [63.  1. 21. ... 45.  0.  6.]\n",
            " [61.  0. 17. ... 45.  0.  6.]]\n"
          ]
        }
      ],
      "source": [
        "reverse_normalization = np.around(np.array(X)*max_attr_vals)\n",
        "print(reverse_normalization)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sD_AIR04TWwM"
      },
      "source": [
        "## CSV files\n",
        "Prerequisites: [Python 3.8](https://www.python.org/downloads/release/python-380/), [numpy](https://pypi.org/project/numpy/).  \n",
        "To load a `.csv` file, use: `data = numpy.genfromtxt(<DATA_PATH>,delimiter=',')`.  \n",
        "Sample usage code is seen below: "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0IV23D7SThyH"
      },
      "source": [
        "Download the 1 year data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ivgJzzsJQ1xE",
        "outputId": "d75cded5-8ca4-4a2b-9fc8-3f8c74c1d15b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1Ih4bHHhe012KFAhwWCZkLgnpHCMEiqn5\n",
            "To: /content/census_features.csv\n",
            "100% 548M/548M [00:03<00:00, 139MB/s]\n"
          ]
        }
      ],
      "source": [
        "!gdown --id 1FS25Lwn-0qgV2sPzvkmL_HdUYZ_HJRgy\n",
        "!gdown --id 1d2dYbwK9CjRgh89ISCdcYLdUWG0lfDtc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yBwS_W5eTl4T"
      },
      "source": [
        "Alternatively, download the 5 year data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5Ony5lIDTpwO"
      },
      "outputs": [],
      "source": [
        "!gdown --id 1n7O0x2uRdWhWJY4GPhWBxS1osIk5npPQ\n",
        "!gdown --id 1s45dppmjCv56hM6aFX4CTDPRz7I3tIED"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wrbzzX7XTcOl"
      },
      "outputs": [],
      "source": [
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oq97inyMUGF_"
      },
      "source": [
        "Read the features data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e7eUfym5UJbA",
        "outputId": "01d18b9f-1ffa-488c-d842-6e60237ef78b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dimension of X: (1685316, 13)\n"
          ]
        }
      ],
      "source": [
        "X = np.genfromtxt('./census_features.csv',delimiter=',')\n",
        "print(f\"Dimension of X: {X.shape}\")\n",
        "print(X[:3])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kiKtZtjKUQfn"
      },
      "source": [
        "Note that the features matrix has 13 column values in each row, which correspond to columns in the `Features` in order."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cLZDnd5pUT9Q"
      },
      "source": [
        "Read the labels data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Jvk7GdkxUZx1",
        "outputId": "64365d1d-7749-4dea-e746-29b030a7c99f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Length of y: 1685316\n",
            "[0. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n"
          ]
        }
      ],
      "source": [
        "y = np.genfromtxt('./census_labels.csv',delimiter=',')\n",
        "print(f\"Length of y: {len(y)}\")\n",
        "print(y[:20])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LwOjtSoMUgYC"
      },
      "source": [
        "# Benchmarks\n",
        "We conducted benchmarks by training various classifiers, as demonstrated below for train and test accuracy. For the data, we split into train-test into 2/3 and 1/3 randomly.\n",
        "\n",
        "## 1 year data results  \n",
        "Percentage of records with > \\$50,000 total person's income: 58.1%   \n",
        "Percentage of records with <= \\$50,000 total person's income: 41.9%   \n",
        "\n",
        "### Membership Inference Benchmarks  \n",
        "We tested 4 membership inference attacks (Yeom, Shokri, Merlin and Morgan) against NN models  trained using the 1-year census data (with test-to-train ratio of 0.5). We performed tests in various differential privacy settings. We performed 5 repeated runs for each setting and the benchmark results below are the average values from the repeated runs.\n",
        "\n",
        "**Training and Test Accuracy overview**  \n",
        "For NN models without differential privacy\n",
        "\n",
        "|           | No privacy |\n",
        "|-----------|------------|\n",
        "| Train acc | 0.83162    |\n",
        "| Test acc  | 0.74172    |\n",
        "\n",
        "For NN models with Gaussian differential privacy (GDP) ('eps' stands for epsilon, the privacy budget)\n",
        "\n",
        "|           | eps=0.1 | eps=1.0 | eps=10.0 | eps=100.0 |\n",
        "|-----------|---------|---------|----------|-----------|\n",
        "| Train acc | 0.68624 | 0.74272 | 0.76174  | 0.77200   |\n",
        "| Test acc  | 0.68576 | 0.73864 | 0.75652  | 0.75576   |\n",
        "\n",
        "For NN models with Renyi differential privacy (RDP) ('eps' stands for epsilon, the privacy budget)\n",
        "\n",
        "|           | eps=0.1 | eps=1.0 | eps=10.0 | eps=100.0 |\n",
        "|-----------|---------|---------|----------|-----------|\n",
        "| Train acc | 0.66876 | 0.74158 | 0.75350  | 0.77472   |\n",
        "| Test acc  | 0.66596 | 0.73672 | 0.74584  | 0.75744   |\n",
        "\n",
        "**Attack results for NN with no privacy:**\n",
        "\n",
        "|           | Yeom           | Shokri         | Merlin         | Morgan         |\n",
        "|-----------|----------------|----------------|----------------|----------------|\n",
        "| PPV       | 0.6952 &plusmn; 0.0039 | 0.4084 &plusmn; 0.3335 | 0.6895 &plusmn; 0.0036 | 0.7284 &plusmn; 0.0106 |\n",
        "| Advantage | 0.0894 &plusmn; 0.0060 | 0.0152 &plusmn; 0.0159 | 0.0488 &plusmn; 0.0063 | 0.0113 &plusmn; 0.0014 |\n",
        "\n",
        "**Attack results for NN with GDP:**  \n",
        "For epsilon=0.1:\n",
        "\n",
        "|           | Yeom            | Shokri         | Merlin         | Morgan         |\n",
        "|-----------|-----------------|----------------|----------------|----------------|\n",
        "| PPV       | 0.6625 &plusmn; 0.0045  | 0.5353 &plusmn; 0.2677 | 0.6725 &plusmn; 0.0109 | 0.6960 &plusmn; 0.0275 |\n",
        "| Advantage | -0.0029 &plusmn; 0.0039 | 0.0022 &plusmn; 0.0050 | 0.0003 &plusmn; 0.0076 | 0.0010 &plusmn; 0.0006 |\n",
        "\n",
        "For epsilon=1.0:\n",
        "\n",
        "|           | Yeom            | Shokri         | Merlin          | Morgan          |\n",
        "|-----------|-----------------|----------------|-----------------|-----------------|\n",
        "| PPV       | 0.6683 &plusmn; 0.0011  | 0.5374 &plusmn; 0.2687 | 0.6689 &plusmn; 0.0331  | 0.6216 &plusmn; 0.0685  |\n",
        "| Advantage | 0.0057 &plusmn; 0.0040  | 0.0065 &plusmn; 0.0035 | -0.0043 &plusmn; 0.0056 | -0.0003 &plusmn; 0.0020 |\n",
        "\n",
        "For epsilon=10.0:\n",
        "\n",
        "|           | Yeom            | Shokri         | Merlin          | Morgan          |\n",
        "|-----------|-----------------|----------------|-----------------|-----------------|\n",
        "| PPV       | 0.6686 &plusmn; 0.0010  | 0.4037 &plusmn; 0.3296 | 0.6595 &plusmn; 0.0106  | 0.6581 &plusmn; 0.0159  |\n",
        "| Advantage | 0.0054 &plusmn; 0.0023  | 0.0061 &plusmn; 0.0052 | -0.0024 &plusmn; 0.0028 | -0.0009 &plusmn; 0.0024 |\n",
        "\n",
        "For epsilon=100.0:\n",
        "\n",
        "|           | Yeom            | Shokri         | Merlin         | Morgan         |\n",
        "|-----------|-----------------|----------------|----------------|----------------|\n",
        "| PPV       | 0.6708 &plusmn; 0.0013  | 0.6717 &plusmn; 0.0025 | 0.6646 &plusmn; 0.0086 | 0.6660 &plusmn; 0.0281 |\n",
        "| Advantage | 0.0139 &plusmn; 0.0013  | 0.0090 &plusmn; 0.0026 | 0.0029 &plusmn; 0.0069 | 0.0027 &plusmn; 0.0032 |\n",
        "\n",
        "\n",
        "**Attack results for NN with RDP:**  \n",
        "For epsilon=0.1\n",
        "\n",
        "|           | Yeom             | Shokri         | Merlin          | Morgan         |\n",
        "|-----------|------------------|----------------|-----------------|----------------|\n",
        "| PPV       | 0.6672 &plusmn; 0.0031   | 0.6722 &plusmn; 0.0033 | 0.6832 &plusmn; 0.0368  | 0.7322 &plusmn; 0.1341 |\n",
        "| Advantage | -0.0007 &plusmn; 0.0035  | 0.0083 &plusmn; 0.0021 | -0.0015 &plusmn; 0.0066 | 0.0006 &plusmn; 0.0009 |\n",
        "\n",
        "For epsilon=1.0\n",
        "\n",
        "|           | Yeom            | Shokri         | Merlin         | Morgan         |\n",
        "|-----------|-----------------|----------------|----------------|----------------|\n",
        "| PPV       | 0.6687 &plusmn; 0.0021  | 0.5368 &plusmn; 0.2684 | 0.6620 &plusmn; 0.0158 | 0.6434 &plusmn; 0.0605 |\n",
        "| Advantage | 0.0015 &plusmn; 0.0017  | 0.0060 &plusmn; 0.0030 | 0.0014 &plusmn; 0.0041 | 0.0005 &plusmn; 0.0023 |\n",
        "\n",
        "For epsilon=10.0\n",
        "\n",
        "|           | Yeom            | Shokri         | Merlin          | Morgan         |\n",
        "|-----------|-----------------|----------------|-----------------|----------------|\n",
        "| PPV       | 0.6706 &plusmn; 0.0019  | 0.4027 &plusmn; 0.3288 | 0.6658 &plusmn; 0.0015  | 0.6836 &plusmn; 0.0242 |\n",
        "| Advantage | 0.0103 &plusmn; 0.0028  | 0.0046 &plusmn; 0.0056 | -0.0029 &plusmn; 0.0045 | 0.0018 &plusmn; 0.0021 |\n",
        "\n",
        "For epsilon=100.0\n",
        "\n",
        "|           | Yeom            | Shokri         | Merlin          | Morgan         |\n",
        "|-----------|-----------------|----------------|-----------------|----------------|\n",
        "| PPV       | 0.6723 &plusmn; 0.0004  | 0.6705 &plusmn; 0.0027 | 0.6661 &plusmn; 0.0025  | 0.6799 &plusmn; 0.0126 |\n",
        "| Advantage | 0.0177 &plusmn; 0.0010  | 0.0095 &plusmn; 0.0031 | -0.0013 &plusmn; 0.0055 | 0.0029 &plusmn; 0.0029 |\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1qgpIF_5N4F"
      },
      "source": [
        "## 5 year data results\n",
        "Percentage of records with >\\$50,000 total person's income: 61.5%\n",
        "Percentage of records with <=\\$50,000 total person's income: 38.5%\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4lN2CExMhGN"
      },
      "source": [
        "First, download the 1 year data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l1nQRHEQMf0t"
      },
      "outputs": [],
      "source": [
        "!gdown --id 1oMaHqFJutp0RmAG8pHrmF1k_2H2vAegE\n",
        "!gdown --id 1aDxPdt8iB4zUc7joWuieSQpRIhXcKPI2\n",
        "!gdown --id 13UvEGw-I0Ylu5o-9uDLHCU-A3KAL7MiJ"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kSqf7-WtMrTU"
      },
      "source": [
        "Alternatively, download the 5 year data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VScsskRqMviq"
      },
      "outputs": [],
      "source": [
        "!gdown --id 1jGw9TnsdC8nXxiCCK46mqNbIRkWOZpHU\n",
        "!gdown --id 1M7ms22gfdE1W1GecrWIaghWCgKOMY8lI\n",
        "!gdown --id 1L-X-nplPyISi85W8YuI-m6eB2pdpMeoG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U45ga5SjM0k0"
      },
      "source": [
        "Next, split the data into test and train data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OKVrmOiO5Spx",
        "outputId": "99972f2f-6048-4a43-928e-f9cc78989015"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "(1685316, 13) 1685316\n"
          ]
        }
      ],
      "source": [
        "import pickle\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "x = pickle.load(open('./census_features.p', 'rb'))\n",
        "y = pickle.load(open('./census_labels.p', 'rb'))\n",
        "x = np.array(x, dtype=np.float32)\n",
        "y = np.array(y, dtype=np.int32)\n",
        "\n",
        "print(x.shape, len(y))\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.33)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mqeJHFxBM4Ew"
      },
      "source": [
        "Next, explore various classifier benchmarks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d96_L0TwA-ZJ"
      },
      "source": [
        "Mutinomial Naive Bayes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q9uji1Y5_nOl",
        "outputId": "462277ac-3a11-46cb-a633-35ea8e5fcedb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 0.672298281644513\n",
            "Test accuracy: 0.6727351188068075\n"
          ]
        }
      ],
      "source": [
        "# https://scikit-learn.org/stable/modules/naive_bayes.html\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "nbClassifier = MultinomialNB()\n",
        "nbClassifier.fit(X_train,y_train)\n",
        "\n",
        "y_pred = nbClassifier.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = nbClassifier.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nHay8icd-h-F"
      },
      "source": [
        "Gaussian Naive Bayes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "66Sn8JFk8dyv",
        "outputId": "5dbfeabc-58aa-468a-a423-4ebff276d040"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 0.6788969863464998\n",
            "Test accuracy: 0.6791362120272226\n"
          ]
        }
      ],
      "source": [
        "# https://scikit-learn.org/stable/modules/naive_bayes.html\n",
        "from sklearn.naive_bayes import GaussianNB\n",
        "gnb = GaussianNB().fit(X_train, y_train)\n",
        "\n",
        "y_pred = gnb.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = gnb.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3IGdQWIPBHOC"
      },
      "source": [
        "Logistic Regression"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2fc2RAG1BJWN",
        "outputId": "054468a9-c3fc-4562-e623-2a4d5544d58e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 0.7539899093220541\n",
            "Test accuracy: 0.7535399304150822\n"
          ]
        }
      ],
      "source": [
        "# https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "logistic = LogisticRegression().fit(X_train, y_train)\n",
        "\n",
        "y_pred = logistic.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = logistic.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VZeTjR4s_Dnb"
      },
      "source": [
        "K-Nearest Neighbors (1) (did not run to completion after 30 minutes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N-oemp0z_JNf"
      },
      "outputs": [],
      "source": [
        "# https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "neigh = KNeighborsClassifier(n_neighbors=1)\n",
        "neigh.fit(X_train,y_train)\n",
        "\n",
        "y_pred = neigh.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = neigh.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e2VNd_5vNJFS"
      },
      "source": [
        "K-Nearest Neighbors (3) (did not run to completion after 30 minutes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8r-P6YnSC7Ll"
      },
      "outputs": [],
      "source": [
        "# https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "neigh = KNeighborsClassifier(n_neighbors=3)\n",
        "neigh.fit(X_train,y_train)\n",
        "\n",
        "y_pred = neigh.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = neigh.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8w-O5HAzDLC7"
      },
      "source": [
        "Support Vector Machine (did not run to completion after 30 minutes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LS5fbZ4UD1h7"
      },
      "outputs": [],
      "source": [
        "# https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.pipeline import make_pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "svc = make_pipeline(StandardScaler(), SVC(gamma='auto'))\n",
        "svc.fit(X_train, y_train)\n",
        "\n",
        "y_pred = svc.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = svc.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IwG1w0HKEJP3"
      },
      "source": [
        "Decision Tree"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_cvNsPb-EQDS",
        "outputId": "27f3be9e-f74f-45b9-bf71-8cdf182b1aa0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 0.9406178569752232\n",
            "Test accuracy: 0.7165376558693171\n"
          ]
        }
      ],
      "source": [
        "# https://scikit-learn.org/stable/modules/tree.html\n",
        "from sklearn import tree\n",
        "decisionTree = tree.DecisionTreeClassifier()\n",
        "decisionTree = decisionTree.fit(X_train, y_train)\n",
        "\n",
        "y_pred = decisionTree.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = decisionTree.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0oXxXkfYFlmh"
      },
      "source": [
        "Random Forest"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W6MkZzELFm6F",
        "outputId": "d602db42-4c32-4d81-ea34-8fac1a779bd4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 0.7156649937431421\n",
            "Test accuracy: 0.717889796909135\n"
          ]
        }
      ],
      "source": [
        "# https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "rf = RandomForestClassifier(max_depth=2, random_state=0)\n",
        "rf.fit(X_train, y_train)\n",
        "\n",
        "y_pred = rf.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = rf.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l8kr5y8tF-QX"
      },
      "source": [
        "Multi-layer Perceptron"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TQXx7YZBHbCr",
        "outputId": "bed38b79-a767-4c38-bdbb-f5f297e89e3f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 0.7533983196373236\n",
            "Test accuracy: 0.7540811464429881\n"
          ]
        }
      ],
      "source": [
        "# https://scikit-learn.org/stable/modules/neural_networks_supervised.html\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "\n",
        "mlp = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1)\n",
        "mlp.fit(X_train, y_train)\n",
        "\n",
        "y_pred = mlp.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = mlp.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1o6UNB7IJWX3"
      },
      "source": [
        "Stochastic Gradient Descent Classifier"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "alaPgVFPJfQV",
        "outputId": "6d358e3d-0097-4051-c302-a259b80e4a42"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 0.7535462170585062\n",
            "Test accuracy: 0.7541782416772302\n"
          ]
        }
      ],
      "source": [
        "# https://scikit-learn.org/stable/modules/sgd.html\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "\n",
        "sgd = SGDClassifier(loss=\"hinge\", penalty=\"l2\", max_iter=100)\n",
        "sgd.fit(X_train, y_train)\n",
        "\n",
        "y_pred = sgd.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = sgd.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X7qzHUAaKWjW"
      },
      "source": [
        "Ridge Classifier"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "znxZ6LUZKaw4",
        "outputId": "d36ced53-acdc-4500-e2e4-c275bc4c77ea"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 0.7508938052235243\n",
            "Test accuracy: 0.751477555717381\n"
          ]
        }
      ],
      "source": [
        "# https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeClassifier.html#sklearn.linear_model.RidgeClassifier\n",
        "\n",
        "from sklearn.linear_model import RidgeClassifier\n",
        "rc = RidgeClassifier().fit(X_train, y_train)\n",
        "\n",
        "y_pred = rc.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = rc.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a68V3T_bLAVj"
      },
      "source": [
        "Gaussian Process Classifier (crashed due to limited RAM)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bIU_s3AILCjN"
      },
      "outputs": [],
      "source": [
        "\n",
        "from sklearn.gaussian_process import GaussianProcessClassifier\n",
        "from sklearn.gaussian_process.kernels import RBF\n",
        "\n",
        "kernel = 1.0 * RBF(1.0)\n",
        "gpc = GaussianProcessClassifier(kernel=kernel,random_state=0).fit(X_train, y_train)\n",
        "\n",
        "y_pred = gpc.predict(X_train)\n",
        "train_accuracy = (y_train == y_pred).sum() / X_train.shape[0]\n",
        "print(f\"Train accuracy: {train_accuracy}\")\n",
        "\n",
        "y_pred = gpc.predict(X_test)\n",
        "test_accuracy = (y_test == y_pred).sum() / X_test.shape[0]\n",
        "print(f\"Test accuracy: {test_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ui2Cl94U6HG"
      },
      "source": [
        "# Steps to Reproduce Data\n",
        "The code used to produce the dataset can be found in https://github.com/bargavj/EvaluatingDPML. Below are the steps to reproduce the dataset.\n",
        "1. Assuming python 3 is installed and the terminal is at the project directory, install prerequisites by running `pip3 install -r requirements.txt`.\n",
        "2. To crawl census data, first create a new folder named `dataset`, then `cd` to the `extra` folder and run `python3 crawl_census_data.py`. The data will be downloaded to the `dataset/census/` folder. By default, the 1 year census data from 2019 will be downloaded. To download the 5 year census data, run `python3 crawl_census_data.py --target_census_data='5year'` instead.\n",
        "3. Next, to obtain the preprocessed census data, run `python3 preprocess_dataset.py census --preprocess=1`. The preprocessed data will be stored in the `dataset` folder as well.\n",
        "4. To run a benchmark test of training NN models without performing attacks, go to `evaluating_dpml` folder and run `python3 main.py census --save_data=1` to prepare the data, then run `python3 main.py census --target_model='nn' --target_l2_ratio=1e-4 --benchmark=1`. \n",
        "5. To reproduce benchmark results of membership inference attacks, go to `improved_mi` folder and run `./run_experiments.sh census` (Note that this step may take over a day to finish). Then, run `python3 interpret_results.py census --gamma=0.5 --plot='benchmark'` to obtain the results. To explore benchmark results with other test-to-train ratios, try changing the `--gamma` argument value to 0.1, 1, 2, 10. "
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "CENSUS_DATA_README.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}