g3doc/tutorials/Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_E4uORykIpG4"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "aBT221yVIujn"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aalPefrUUplk"
      },
      "source": [
        "# Fairness Indicators TensorBoard Plugin Example Colab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fFTJpyFlI-uI"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/responsible_ai/fairness_indicators/tutorials/Fairness_indicators_TensorBoard_Plugin_Example_Colab\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/fairness-indicators/blob/master/g3doc/tutorials/Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/fairness-indicators/blob/master/g3doc/tutorials/Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/fairness-indicators/g3doc/tutorials/Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UZ48WFLwbCL6"
      },
      "source": [
        "##Overview\n",
        "\n",
        "In this activity, you'll use [Fairness Indicators for TensorBoard](https://github.com/tensorflow/tensorboard/tree/master/docs/fairness-indicators.md). With the plugin, you can visualize fairness evaluations for your runs and easily compare performance across groups.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u33JXdluZ2lG"
      },
      "source": [
        "# Importing\n",
        "\n",
        "Run the following code to install the required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EoRNffG599XP"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U pip==20.2\n",
        "\n",
        "!pip install fairness_indicators 'absl-py\u003c0.9,\u003e=0.7'\n",
        "!pip install google-api-python-client==1.8.3\n",
        "!pip install tensorboard-plugin-fairness-indicators\n",
        "!pip install tensorflow-serving-api==2.15.1\n",
        "!pip install tensorflow-model-analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mglfaM4_mtIk"
      },
      "source": [
        "**Restart the runtime.** After the runtime is restarted, continue with following cells without running previous cell again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sFZJ8f_M7mlc"
      },
      "outputs": [],
      "source": [
        "# %tf.disable_v2_behavior()\t# Uncomment this line if running in Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B8dlyTyiTe-9"
      },
      "outputs": [],
      "source": [
        "import datetime\n",
        "import os\n",
        "import tempfile\n",
        "from tensorboard_plugin_fairness_indicators import summary_v2\n",
        "import tensorflow.compat.v1 as tf\n",
        "import numpy as np\n",
        "from tensorflow import keras\n",
        "from google.protobuf import text_format\n",
        "\n",
        "# example_model.py is provided in fairness_indicators package to train and\n",
        "# evaluate an example model.\n",
        "from fairness_indicators import example_model\n",
        "import tensorflow_model_analysis as tfma\n",
        "\n",
        "tf.compat.v1.enable_eager_execution()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TsplOJGqWCf5"
      },
      "source": [
        "# Data and Constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NdLBi6tN5i7I"
      },
      "outputs": [],
      "source": [
        "# To know about dataset, check Fairness Indicators Example Colab at:\n",
        "# https://github.com/tensorflow/fairness-indicators/blob/master/g3doc/tutorials/Fairness_Indicators_Example_Colab.ipynb\n",
        "\n",
        "train_tf_file = tf.keras.utils.get_file('train.tf', 'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')\n",
        "validate_tf_file = tf.keras.utils.get_file('validate.tf', 'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')\n",
        "\n",
        "BASE_DIR = tempfile.gettempdir()\n",
        "TEXT_FEATURE = 'comment_text'\n",
        "LABEL = 'toxicity'\n",
        "FEATURE_MAP = {\n",
        "    # Label:\n",
        "    LABEL: tf.io.FixedLenFeature([], tf.float32),\n",
        "    # Text:\n",
        "    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),\n",
        "\n",
        "    # Identities:\n",
        "    'sexual_orientation': tf.io.VarLenFeature(tf.string),\n",
        "    'gender': tf.io.VarLenFeature(tf.string),\n",
        "    'religion': tf.io.VarLenFeature(tf.string),\n",
        "    'race': tf.io.VarLenFeature(tf.string),\n",
        "    'disability': tf.io.VarLenFeature(tf.string),\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mfbgerCsEOmN"
      },
      "source": [
        "# Train the Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YwoC-dzEDid3"
      },
      "outputs": [],
      "source": [
        "model_dir = os.path.join(BASE_DIR, 'train',\n",
        "                         datetime.datetime.now().strftime('%Y%m%d-%H%M%S'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VqjEYySbYaX5"
      },
      "outputs": [],
      "source": [
        "classifier = example_model.ExampleModel(example_model.TEXT_FEATURE)\n",
        "classifier.compile(optimizer=keras.optimizers.Adam(), loss='mse')\n",
        "\n",
        "# Read the data from the training file\n",
        "data = []\n",
        "dataset = tf.data.Dataset.list_files(train_tf_file, shuffle=False)\n",
        "dataset = dataset.flat_map(tf.data.TFRecordDataset)\n",
        "for raw_record in dataset.take(1):\n",
        "  example = tf.train.Example()\n",
        "  example.ParseFromString(raw_record.numpy())\n",
        "  data.append(example)\n",
        "\n",
        "classifier.fit(\n",
        "    tf.constant([e.SerializeToString() for e in data]),\n",
        "    np.array([\n",
        "        e.features.feature[example_model.LABEL].float_list.value[:][0]\n",
        "        for e in data\n",
        "    ]),\n",
        ")\n",
        "classifier.save(model_dir, save_format='tf')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jTPqije9Eg5b"
      },
      "source": [
        "# Run TensorFlow Model Analysis with Fairness Indicators\n",
        "This step might take 2 to 5 minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QLjiy5VCzlRw"
      },
      "outputs": [],
      "source": [
        "tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')\n",
        "\n",
        "eval_config = text_format.Parse(\n",
        "    \"\"\"\n",
        "    model_specs {\n",
        "      signature_name: \"serving_default\"\n",
        "      prediction_key: \"predictions\" # placeholder\n",
        "      label_key: \"toxicity\" # placeholder\n",
        "    }\n",
        "    slicing_specs {}\n",
        "    slicing_specs {\n",
        "      feature_keys: [\"gender\"]\n",
        "    }\n",
        "    metrics_specs {\n",
        "      metrics {\n",
        "        class_name: \"ExampleCount\"\n",
        "      }\n",
        "      metrics {\n",
        "        class_name: \"FairnessIndicators\"\n",
        "      }\n",
        "    }\n",
        "\"\"\",\n",
        "    tfma.EvalConfig(),\n",
        ")\n",
        "\n",
        "tfma_eval_result_path = os.path.join(model_dir, 'tfma_eval_result')\n",
        "example_model.evaluate_model(\n",
        "    model_dir,\n",
        "    validate_tf_file,\n",
        "    tfma_eval_result_path,\n",
        "    eval_config,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U1ROnulYc8Ub"
      },
      "source": [
        "# Visualize Fairness Indicators in TensorBoard\n",
        "\n",
        "\n",
        "Below you will visualize Fairness Indicators in Tensorboard and compare performance of each slice of the data on selected metrics. You can adjust the baseline comparison slice as well as the displayed threshold(s) using the drop down menus at the top of the visualization. You can also select different evaluation runs using the drop down menu at the top-left corner."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zCV-Jo0xda6g"
      },
      "source": [
        "## Write Fairness Indicators Summary\n",
        "Write summary file containing all required information to visualize Fairness Indicators in TensorBoard."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JNaNhTCTAMHm"
      },
      "outputs": [],
      "source": [
        "import tensorflow.compat.v2 as tf2\n",
        "\n",
        "writer = tf2.summary.create_file_writer(\n",
        "    os.path.join(model_dir, 'fairness_indicators'))\n",
        "with writer.as_default():\n",
        "  summary_v2.FairnessIndicators(tfma_eval_result_path, step=1)\n",
        "writer.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MB2Gfm9BdXVY"
      },
      "source": [
        "## Launch TensorBoard\n",
        "Navigate to \"Fairness Indicators\" tab to visualize Fairness Indicators."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UiHhDWu8tyEI"
      },
      "outputs": [],
      "source": [
        
        "%load_ext tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ix6d718udWsK"
      },
      "outputs": [],
      "source": [
        
        "%tensorboard --logdir=$model_dir"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}