notebooks/official/prediction/get_started_with_tf_serving.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic,gcp"
      },
      "source": [
        "# Get started with TensorFlow Serving with Vertex AI Prediction\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_tf_serving.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fprediction%2Fget_started_with_tf_serving.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/raw/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/prediction/get_started_with_tf_serving.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_tf_serving.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial demonstrates how to serve predictions from a `Vertex AI Endpoint` with `TensorFlow Serving` serving binary.\n",
        "\n",
        "Learn more about [getting predictions from a custom trained model](https://cloud.google.com/vertex-ai/docs/predictions/get-predictions)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9402cfbdc2d"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use `Vertex AI Prediction` on a `Vertex AI Endpoint` resource with `TensorFlow Serving` serving binary.\n",
        "\n",
        "This tutorial uses the following Vertex AI services and resources:\n",
        "\n",
        "- `Vertex AI Prediction`\n",
        "- `Vertex AI Batch Prediction`\n",
        "- `Vertex AI Models`\n",
        "- `Vertex AI Endpoints`\n",
        "\n",
        "You perform the following steps:\n",
        "\n",
        "- Download a pretrained image classification model from TensorFlow Hub.\n",
        "- Create a serving function to receive compressed image data, and output decomopressed preprocessed data for the model input.\n",
        "- Upload the TensorFlow Hub model and serving function as a Vertex AI model resource.\n",
        "- Create a Vertex AI endpoint resource.\n",
        "- Deploy the model resource to the endpoint resource with `TensorFlow Serving` serving binary.\n",
        "- Make an online prediction with the deployed model.\n",
        "- Make a batch prediction with the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:iris,lcn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "This tutorial uses a pre-trained image classification model from TensorFlow Hub, which is trained on ImageNet dataset.\n",
        "\n",
        "Learn more about [ResNet V2 pretained model](https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3b1ffd5ab768"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade google-cloud-aiplatform \\\n",
        "                        google-cloud-pipeline-components \\\n",
        "                        tensorflow==2.15.1 \\\n",
        "                        tensorflow-hub -q"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ff555b32bab8"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f09b4dff629a"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "54c5ef8a8f43"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92e68cfc3a90"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "46604f70e831"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from google.cloud import aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket.\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for training and prediction.\n",
        "\n",
        "Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa T4 GPUs allocated to each VM, you would specify:\n",
        "\n",
        "    (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)\n",
        "\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n",
        "\n",
        "**Note**: TF releases before 2.3 for GPU support fail to load the custom model in this tutorial. It's a known issue and fixed in TF 2.3. This is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "outputs": [],
      "source": [
        "DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training"
      },
      "source": [
        "#### Set machine type\n",
        "\n",
        "Next, set the machine type to use for prediction.\n",
        "\n",
        "- Set the variable `DEPLOY_COMPUTE` to configure  the compute resources for the VMs you use for for prediction.\n",
        " - `machine type`\n",
        "     - `n1-standard`: 3.75GB of memory per vCPU.\n",
        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "**Note**: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "machine:training"
      },
      "outputs": [],
      "source": [
        "MACHINE_TYPE = \"n1-standard\"\n",
        "\n",
        "VCPU = \"4\"\n",
        "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Train machine type\", DEPLOY_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gar_enable_api"
      },
      "source": [
        "### Enable Artifact Registry API\n",
        "\n",
        "You must enable the Artifact Registry API service for your project.\n",
        "\n",
        "Learn more about [enabling services](https://cloud.google.com/artifact-registry/docs/enable-service)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gar_enable_api"
      },
      "outputs": [],
      "source": [
        "! gcloud services enable artifactregistry.googleapis.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gar_create_repo"
      },
      "source": [
        "## Create a private Docker repository\n",
        "\n",
        "Your first step is to create your own Docker repository in Google Artifact Registry.\n",
        "\n",
        "1. Run the `gcloud artifacts repositories create` command to create a new Docker repository with your region with the description \"docker repository\".\n",
        "\n",
        "2. Run the `gcloud artifacts repositories list` command to verify that your repository was created."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gar_create_repo"
      },
      "outputs": [],
      "source": [
        "PRIVATE_REPO = \"my-docker-repo\"\n",
        "\n",
        "! gcloud artifacts repositories create {PRIVATE_REPO} --repository-format=docker --location={LOCATION} --description=\"Docker repository\"\n",
        "\n",
        "! gcloud artifacts repositories list"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gar_auth"
      },
      "source": [
        "### Configure authentication to your private repo\n",
        "\n",
        "Before you push or pull container images, configure Docker to use the `gcloud` command-line tool to authenticate requests to `Artifact Registry` for your region."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gar_auth"
      },
      "outputs": [],
      "source": [
        "! gcloud auth configure-docker {LOCATION}-docker.pkg.dev --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:tfserving"
      },
      "source": [
        "#### Container (Docker) image for serving\n",
        "\n",
        "Set the TensorFlow Serving Docker container image for serving prediction.\n",
        "\n",
        "    1. Pull the corresponding CPU or GPU Docker image for TF Serving from Docker Hub.\n",
        "    2. Create a tag for registering the image with Artifact Registry\n",
        "    3. Register the image with Artifact Registry.\n",
        "\n",
        "Learn more about [TensorFlow Serving](https://www.tensorflow.org/tfx/serving/docker)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:tfserving"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "IS_COLAB = \"google.colab\" in sys.modules\n",
        "\n",
        "# Executes in Vertex AI Workbench\n",
        "if DEPLOY_GPU:\n",
        "    DEPLOY_IMAGE = (\n",
        "        f\"{LOCATION}-docker.pkg.dev/\"\n",
        "        + PROJECT_ID\n",
        "        + f\"/{PRIVATE_REPO}\"\n",
        "        + \"/tf_serving:gpu\"\n",
        "    )\n",
        "    TF_IMAGE = \"tensorflow/serving:2.5.4-gpu\"\n",
        "else:\n",
        "    DEPLOY_IMAGE = (\n",
        "        f\"{LOCATION}-docker.pkg.dev/\"\n",
        "        + PROJECT_ID\n",
        "        + f\"/{PRIVATE_REPO}\"\n",
        "        + \"/tf_serving:cpu\"\n",
        "    )\n",
        "    TF_IMAGE = \"tensorflow/serving:2.5.4\"\n",
        "\n",
        "if not IS_COLAB:\n",
        "    if DEPLOY_GPU:\n",
        "        ! sudo docker pull tensorflow/serving:2.5.4-gpu\n",
        "    else:\n",
        "        ! sudo docker pull tensorflow/serving:2.5.4\n",
        "\n",
        "    ! docker tag $TF_IMAGE $DEPLOY_IMAGE\n",
        "    ! docker push $DEPLOY_IMAGE\n",
        "else:\n",
        "    # install docker daemon\n",
        "    ! apt-get -qq install docker.io\n",
        "\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6c14c96973c3"
      },
      "source": [
        "*Executes in Colab*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2450b47cb86f"
      },
      "outputs": [],
      "source": [
        "%%bash -s $IS_COLAB $DEPLOY_IMAGE $TF_IMAGE\n",
        "if [ $1 == \"False\" ]; then\n",
        "  exit 0\n",
        "fi\n",
        "set -x\n",
        "dockerd -b none --iptables=0 -l warn &\n",
        "for i in $(seq 5); do [ ! -S \"/var/run/docker.sock\" ] && sleep 2 || break; done\n",
        "docker pull $3\n",
        "docker tag $3 $2\n",
        "docker push $2\n",
        "kill $(jobs -p)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8128b8ff025"
      },
      "source": [
        "## Get pretrained model from TensorFlow Hub\n",
        "\n",
        "For demonstration purposes, this tutorial uses a pretrained model from TensorFlow Hub (TFHub), which is then uploaded to a Vertex AI model resource. Once you have a Vertex AI model resource created, the model can then be deployed to a Vertex AI endpoint resource.\n",
        "\n",
        "### Download the pretrained model\n",
        "\n",
        "First, you download the pretrained model from TensorFlow Hub. The model gets downloaded as a TF.Keras layer. To finalize the model, in this example, you create a `Sequential()` model with the downloaded TFHub model as a layer, and specify the input shape to the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c55fa4c826f7"
      },
      "outputs": [],
      "source": [
        "tfhub_model = tf.keras.Sequential(\n",
        "    [hub.KerasLayer(\"https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5\")]\n",
        ")\n",
        "\n",
        "tfhub_model.build([None, 224, 224, 3])\n",
        "\n",
        "tfhub_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "63de49055083"
      },
      "source": [
        "### Save the model artifacts\n",
        "\n",
        "At this point, the model is in memory. Next, you save the model artifacts to a Cloud Storage location.\n",
        "\n",
        "**Note:** For TF Serving, the MODEL_DIR must end in a subfolder that is a number, e.g., 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "64618c713db9"
      },
      "outputs": [],
      "source": [
        "MODEL_DIR = BUCKET_URI + \"/model/1\"\n",
        "tfhub_model.save(MODEL_DIR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "how_serving_function_works"
      },
      "source": [
        "## Upload the model for serving\n",
        "\n",
        "Next, you upload your TF.Keras model from the custom job to Vertex AI model service, which creates a model resource for your custom model. During upload, you need to define a serving function to convert data to the format your model expects. If you send encoded data to Vertex AI, your serving function ensures that the data is decoded on the model server before it's passed as input to your model.\n",
        "\n",
        "### How does the serving function work\n",
        "\n",
        "When you send a request to an online prediction server, the request is received by a HTTP server. The HTTP server extracts the prediction request from the HTTP request content body. The extracted prediction request is forwarded to the serving function. For Google pre-built prediction containers, the request content is passed to the serving function as a `tf.string`.\n",
        "\n",
        "The serving function consists of two parts:\n",
        "\n",
        "- `preprocessing function`:\n",
        "  - Converts the input (`tf.string`) to the input shape and data type of the underlying model (dynamic graph).\n",
        "  - Performs the same preprocessing of the data that was done during training the underlying model -- e.g., normalizing, scaling, etc.\n",
        "- `post-processing function`:\n",
        "  - Converts the model output to format expected by the receiving application -- e.q., compresses the output.\n",
        "  - Packages the output for the the receiving application -- e.g., add headings, make JSON object, etc.\n",
        "\n",
        "Both the preprocessing and post-processing functions are converted to static graphs which are fused to the model. The output from the underlying model is passed to the post-processing function. The post-processing function passes the converted/packaged output back to the HTTP server. The HTTP server returns the output as the HTTP response content.\n",
        "\n",
        "One consideration you need to consider when building serving functions for TF.Keras models is that they run as static graphs. That means, you can't use TF graph operations that require a dynamic graph. If you do, you get an error during the compile of the serving function which indicates that you're using an EagerTensor which isn't supported."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "serving_function_image:post"
      },
      "source": [
        "### Serving function for image data\n",
        "\n",
        "#### Preprocessing\n",
        "\n",
        "To pass images to the prediction service, you encode the compressed (e.g., JPEG) image bytes into base 64 -- which makes the content safe from modification while transmitting binary data over the network. Since this deployed model expects input data as raw (uncompressed) bytes, you need to ensure that the base 64 encoded data gets converted back to raw bytes, and then preprocessed to match the model input requirements, before it's passed as input to the deployed model.\n",
        "\n",
        "To resolve this, you define a serving function (`serving_fn`) and attach it to the model as a preprocessing step. Add a `@tf.function` decorator so the serving function is fused to the underlying model (instead of upstream on a CPU).\n",
        "\n",
        "When you send a prediction or explanation request, the content of the request is base 64 decoded into a Tensorflow string (`tf.string`), which is passed to the serving function (`serving_fn`). The serving function preprocesses the `tf.string` into raw (uncompressed) numpy bytes (`preprocess_fn`) to match the input requirements of the model:\n",
        "\n",
        "- `io.decode_jpeg`- Decompresses the JPG image which is returned as a Tensorflow tensor with three channels (RGB).\n",
        "- `image.convert_image_dtype` - Changes integer pixel values to float 32, and rescales pixel data between 0 and 1.\n",
        "- `image.resize` - Resizes the image to match the input shape for the model.\n",
        "\n",
        "At this point, the data can be passed to the model (`m_call`), via a concrete function. The serving function is a static graph, while the model is a dynamic graph. The concrete function performs the tasks of marshalling the input data from the serving function to the model, and marshalling the prediction result from the model back to the serving function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "serving_function_image"
      },
      "outputs": [],
      "source": [
        "CONCRETE_INPUT = \"numpy_inputs\"\n",
        "\n",
        "\n",
        "def _preprocess(bytes_input):\n",
        "    decoded = tf.io.decode_jpeg(bytes_input, channels=3)\n",
        "    decoded = tf.image.convert_image_dtype(decoded, tf.float32)\n",
        "    resized = tf.image.resize(decoded, size=(224, 224))\n",
        "    return resized\n",
        "\n",
        "\n",
        "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n",
        "def preprocess_fn(bytes_inputs):\n",
        "    decoded_images = tf.map_fn(\n",
        "        _preprocess, bytes_inputs, dtype=tf.float32, back_prop=False\n",
        "    )\n",
        "    return {\n",
        "        CONCRETE_INPUT: decoded_images\n",
        "    }  # User needs to make sure the key matches model's input\n",
        "\n",
        "\n",
        "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n",
        "def serving_fn(bytes_inputs):\n",
        "    images = preprocess_fn(bytes_inputs)\n",
        "    prob = m_call(**images)\n",
        "    return prob\n",
        "\n",
        "\n",
        "m_call = tf.function(tfhub_model.call).get_concrete_function(\n",
        "    [tf.TensorSpec(shape=[None, 224, 224, 3], dtype=tf.float32, name=CONCRETE_INPUT)]\n",
        ")\n",
        "\n",
        "tf.saved_model.save(tfhub_model, MODEL_DIR, signatures={\"serving_default\": serving_fn})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "serving_function_signature:image"
      },
      "source": [
        "## Get the serving function signature\n",
        "\n",
        "You can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.\n",
        "\n",
        "For your purpose, you need the signature of the serving function. Why? Well, when you send our data for prediction as a HTTP request packet, the image data is base64 encoded, and our TF.Keras model takes numpy input. Your serving function does the conversion from base64 to a numpy array.\n",
        "\n",
        "When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you use later when you make a prediction request."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "serving_function_signature:image"
      },
      "outputs": [],
      "source": [
        "loaded = tf.saved_model.load(MODEL_DIR)\n",
        "\n",
        "serving_input = list(\n",
        "    loaded.signatures[\"serving_default\"].structured_input_signature[1].keys()\n",
        ")[0]\n",
        "print(\"Serving function input:\", serving_input)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8ce91147c93"
      },
      "source": [
        "### Upload the TensorFlow Hub model to a Vertex AI model resource\n",
        "\n",
        "Finally, you upload the model artifacts from the TFHub model and serving function into a Vertex AI model resource. Since you're using a non Google pre-built serving binary -- i.e., TensorFlow Serving, you need to specify the following additional serving configuration settings:\n",
        "\n",
        "- `serving_container_command`: The serving binary (HTTP Server) to start up.\n",
        "- `serving_container_args`: The arguments to pass to the serving binary. For TensorFlow Serving, the required arguments are:\n",
        "    - `--model_name`: The human readable name to assign to the model.\n",
        "    - `--model_base_name`: Where to store the model artifacts in the container. The Vertex service sets the variable $(AIP_STORAGE_URI) to where the service installed the model artifacts in the container.\n",
        "    - `--rest_api_port`: The port to which to send REST based prediction requests. Can either be 8080 or 8501 (default for TensorFlow Serving).\n",
        "    - `--port`: The port to which to send gRPC based prediction requests. Should be 8500 for TensorFlow Serving.\n",
        "- `serving_container_health_route`: The URL for the service to periodically ping for a response to verify that the serving binary is running. For TensorFlow Serving, this is /v1/models/\\<model_name\\>.\n",
        "- `serving_container_predict_route`: The URL for the service to route REST-based prediction requests to. For TF Serving, this is /v1/models/[model_name]:predict.\n",
        "- `serving_container_ports`: A list of ports for the HTTP server to listen for requests.\n",
        "\n",
        "Uploading a model into a Vertex model resource returns a long running operation, since it may take a few moments. \n",
        "\n",
        "**Note:** You drop the ending number subfolder (e.g., /1) from the model path to upload. The Vertex service uploads the parent folder above the subfolder with the model artifacts -- which is what TensorFlow Serving binary expects.\n",
        "\n",
        "**Note:** When you upload the model artifacts to a Vertex AI model resource, you specify the corresponding deployment container image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ad61e1429512"
      },
      "outputs": [],
      "source": [
        "MODEL_NAME = \"example_\"\n",
        "\n",
        "model = aiplatform.Model.upload(\n",
        "    display_name=\"example_\",\n",
        "    artifact_uri=MODEL_DIR[:-2],\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    serving_container_health_route=\"/v1/models/\" + MODEL_NAME,\n",
        "    serving_container_predict_route=\"/v1/models/\" + MODEL_NAME + \":predict\",\n",
        "    serving_container_command=[\"/usr/bin/tensorflow_model_server\"],\n",
        "    serving_container_args=[\n",
        "        \"--model_name=\" + MODEL_NAME,\n",
        "        \"--model_base_path=\" + \"$(AIP_STORAGE_URI)\",\n",
        "        \"--rest_api_port=8080\",\n",
        "        \"--port=8500\",\n",
        "        \"--file_system_poll_wait_seconds=31540000\",\n",
        "    ],\n",
        "    serving_container_ports=[8080],\n",
        ")\n",
        "\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "628de0914ba1"
      },
      "source": [
        "## Create a Vertex AI endpoint resource\n",
        "\n",
        "You create a Vertex AI endpoint resource using the `Endpoint.create()` method. At a minimum, you specify the display name for the endpoint. Optionally, you can specify the project and location (region); otherwise the settings are inherited by the values you set when you initialized the Vertex AI SDK with the `init()` method.\n",
        "\n",
        "In this example, the following parameters are specified:\n",
        "\n",
        "- `display_name`: A human readable name for the endpoint resource.\n",
        "- `project`: Your project ID.\n",
        "- `location`: Your region.\n",
        "- `labels`: (optional) User defined metadata for the endpoint in the form of key/value pairs.\n",
        "\n",
        "This method returns a Vertex AI endpoint object.\n",
        "\n",
        "Learn more about [Vertex AI Endpoints](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ea443f9593b"
      },
      "outputs": [],
      "source": [
        "endpoint = aiplatform.Endpoint.create(\n",
        "    display_name=\"example_\",\n",
        "    project=PROJECT_ID,\n",
        "    location=LOCATION,\n",
        "    labels={\"your_key\": \"your_value\"},\n",
        ")\n",
        "\n",
        "print(endpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ca3fa3f6a894"
      },
      "source": [
        "## Deploy model to the endpoint\n",
        "\n",
        "You can deploy one or more Vertex AI model resource instances to the same endpoint. Each model resource that is deployed has its own deployment container for the serving binary. \n",
        "\n",
        "**Note:** For this example, you specified the deployment container for the TFHub model in the previous step of uploading the model artifacts to a Vertex AI model resource.\n",
        "\n",
        "In the next example, you deploy the Vertex AI model resource to a Vertex AI endpoint resource. The model resource already has a container image defined for deployment. To deploy, you specify the following additional configuration settings:\n",
        "\n",
        "- The machine type.\n",
        "- The (if any) type and number of GPUs.\n",
        "- Static, manual or auto-scaling of VM instances.\n",
        "\n",
        "In this example, you deploy the model with the minimal amount of specified parameters, as follows:\n",
        "\n",
        "- `model`: The Vertex AI model resource.\n",
        "- `deployed_model_displayed_name`: The human readable name for the deployed model instance.\n",
        "- `machine_type`: The machine type for each VM instance.\n",
        "\n",
        "Do to the requirements to provision the resource, this may take upto a few minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4e93b034a72f"
      },
      "outputs": [],
      "source": [
        "response = endpoint.deploy(\n",
        "    model=model,\n",
        "    deployed_model_display_name=\"example_\",\n",
        "    machine_type=DEPLOY_COMPUTE,\n",
        ")\n",
        "\n",
        "print(endpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "daf4494bd342"
      },
      "source": [
        "### Prepare test data for prediction\n",
        "\n",
        "Next, you load a compressed JPEG image into memory and then base64 encode it. For demonstration purposes, you use an image from the Flowers dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "486dc48ce249"
      },
      "outputs": [],
      "source": [
        "! gsutil cp gs://cloud-ml-data/img/flower_photos/daisy/100080576_f52e8ee070_n.jpg test.jpg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d9cde1862983"
      },
      "outputs": [],
      "source": [
        "import base64\n",
        "\n",
        "with open(\"test.jpg\", \"rb\") as f:\n",
        "    data = f.read()\n",
        "b64str = base64.b64encode(data).decode(\"utf-8\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "predict_request:mbsdk,custom,icn"
      },
      "source": [
        "### Make the prediction\n",
        "\n",
        "Now that your model resource is deployed to an endpoint resource, you can do online predictions by sending prediction requests to the Endpoint resource.\n",
        "\n",
        "#### Request\n",
        "\n",
        "Since in this example your test item is in a Cloud Storage bucket, you open and read the contents of the image using `tf.io.gfile.Gfile()`. To pass the test data to the prediction service, you encode the bytes into base64 -- which makes the content safe from modification while transmitting binary data over the network.\n",
        "\n",
        "The format of each instance is:\n",
        "\n",
        "    { serving_input: { 'b64': base64_encoded_bytes } }\n",
        "\n",
        "Since the `predict()` method can take multiple items (instances), send your single test item as a list of one test item.\n",
        "\n",
        "#### Response\n",
        "\n",
        "The response from the `predict()` call is a Python dictionary with the following entries:\n",
        "\n",
        "- `ids`: The internal assigned unique identifiers for each prediction request.\n",
        "- `predictions`: The predicted confidence, between 0 and 1, per class label.\n",
        "- `deployed_model_id`: The Vertex AI identifier for the deployed `Model` resource which did the predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "predict_request:mbsdk,custom,icn"
      },
      "outputs": [],
      "source": [
        "# The format of each instance should conform to the deployed model's prediction input schema.\n",
        "instances = [{serving_input: {\"b64\": b64str}}]\n",
        "\n",
        "prediction = endpoint.predict(instances=instances)\n",
        "\n",
        "print(prediction)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "11e16f54bc90"
      },
      "source": [
        "## Introduction to batch prediction\n",
        "\n",
        "Batch prediction provides the ability to do offline batch processing of large amounts of prediction requests. Resources are only provisioned during the batch process and then deprovisioned when the batch request is completed. The results are stored in Cloud Storage, in contrast to online prediction where the results are returned as a HTTP response packet.\n",
        "\n",
        "The input format for your batch job is dependent on the format supported by your model server. Foremost, the web server in your model server must support a JSONL format, which the web server converts to a format support either directly by the model input interface or a serving function interface. For batch prediction, this JSONL format is referred to as the `pivot` format.\n",
        "\n",
        "### Input format for batch prediction jobs\n",
        "\n",
        "The batch server accepts the following input formats:\n",
        "\n",
        "- JSONL\n",
        "- CSV\n",
        "- TFRecords\n",
        "- File-List\n",
        "\n",
        "### Pivot format\n",
        "\n",
        "The batch server converts the input format to the `pivot` (JSONL) format as follows:\n",
        "\n",
        "**JSONL**\n",
        "\n",
        "Each input line (request) should contain one and only one valid json value.\n",
        "\n",
        "    {\"values\": [1, 2, 3, 4], \"key\": 1}\n",
        "    {\"values\": [5, 6, 7, 8], \"key\": 2}\n",
        "\n",
        "The batch server generates the pivot data with the same format. The generated pivot data is then wrapped into a payload request:\n",
        "\n",
        "    {\"instances\": [\n",
        "      {\"values\": [1, 2, 3, 4], \"key\": 1},\n",
        "      {\"values\": [5, 6, 7, 8], \"key\": 2}\n",
        "    ]}\n",
        "\n",
        "**CSV**\n",
        "\n",
        "The csv header in the first line is always ignored. String fields are required to be double quoted explicitly, otherwise the row is discarded and parsing error messages are outputted to error files. Non-quoted values are always transferred as floats.\n",
        "\n",
        "    col1,col2,col3\n",
        "    1,3,\"cat1\"\n",
        "    2,4,\"cat2\"\n",
        "\n",
        "The batch server converts each input row (request) to a JSON array.\n",
        "\n",
        "    {\"instances\": [\n",
        "     [1.0,3.0,\"cat1\"],\n",
        "     [2.0,4.0,\"cat2\"]\n",
        "    ]}\n",
        "    \n",
        "**BigQuery**\n",
        "\n",
        "Each row is converted to a JSON array. For example:\n",
        "\n",
        "    [1.0,3.0,\"cat1\"]\n",
        "    [2.0,4.0,\"cat2\"]\n",
        "    \n",
        "The batch server generates the pivot data with the same format. The generated pivot data is then wrapped into a payload request:\n",
        "\n",
        "    {\"instances\": [\n",
        "     [1.0,3.0,\"cat1\"],\n",
        "     [2.0,4.0,\"cat2\"]\n",
        "    ]}\n",
        "\n",
        "**TFRecords**\n",
        "\n",
        "Instances in TFRecord files are read as binary by apache_beam.io.tfrecordio module. The binary objects are then serialized as ASCII strings. Predictor server is responsible to know the decoder to recover the instance.  \n",
        "\n",
        "    {\"instances\": [\n",
        "     {\"b64\",\"b64EncodedASCIIString\"},\n",
        "     {\"b64\",\"b64EncodedASCIIString\"}\n",
        "    ]}\n",
        "\n",
        "**FileList**\n",
        "\n",
        "The FileList format contains a list of files. Each line in a “FileList” file specifies a single file path, specified as a Cloud Storage location.\n",
        "\n",
        "    gs://my-bucket/file1.txt\n",
        "    gs://my-bucket/file2.txt\n",
        "\n",
        "The batch server reads the files as binaries. The binary objects are serialized as ASCII strings.\n",
        "\n",
        "    {\"instances\": [\n",
        "     {\"b64\",\"b64EncodedASCIIString\"},\n",
        "     {\"b64\",\"b64EncodedASCIIString\"}\n",
        "    ]}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ccd46a186da"
      },
      "source": [
        "## Make the batch input file\n",
        "\n",
        "Next, make a batch input file, which you store in your local Cloud Storage bucket. For custom models, you format the batch input file in JSONL format. Each JSON object entry in the JSONL file is specified in the same format as you specified for the online prediction request.\n",
        "\n",
        "In otherwords, both online and batch prediction use the same predict request format. The difference is that with online prediction, you pass the request as an in-memory dictionary object using the SDK method `predict()`. For batch prediction, you write each prediction request (dictionary entry) as a JSON object, one per line.\n",
        "\n",
        "The dictionary contains the key/value pairs:\n",
        "\n",
        "- `input_name`: the name of the input layer of the underlying model.\n",
        "- `'b64'`: A key that indicates the content is base64 encoded.\n",
        "- `content`: The compressed JPG image bytes as a base64 encoded string.\n",
        "\n",
        "Each instance in the prediction request is a dictionary entry of the form:\n",
        "\n",
        "                        {serving_input: {'b64': content}}\n",
        "\n",
        "To pass the image data to the prediction service you encode the bytes into base64 -- which makes the content safe from modification when transmitting binary data over the network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f88fdb54269b"
      },
      "outputs": [],
      "source": [
        "# For demonstration purposes, you write the same image (instance[0]) request twice to the JSONL file.\n",
        "# You receive back two predictions, one for each instance.\n",
        "\n",
        "with open(\"test.jsonl\", \"w\") as f:\n",
        "    json.dump(instances[0], f)\n",
        "    f.write(\"\\n\")\n",
        "    json.dump(instances[0], f)\n",
        "\n",
        "! gsutil cp test.jsonl {BUCKET_URI}/test.jsonl"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "batch_request:mbsdk,jsonl,custom"
      },
      "source": [
        "## Make the batch prediction request\n",
        "\n",
        "Now that your Model resource is trained, you can make a batch prediction by invoking the batch_predict() method, with the following parameters:\n",
        "\n",
        "- `job_display_name`: The human readable name for the batch prediction job.\n",
        "- `gcs_source`: A list of one or more batch request input files.\n",
        "- `gcs_destination_prefix`: The Cloud Storage location for storing the batch prediction resuls.\n",
        "- `instances_format`: The format for the input instances, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
        "- `predictions_format`: The format for the output predictions, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
        "- `machine_type`: The type of machine to use for training.\n",
        "- `accelerator_type`: The hardware accelerator type.\n",
        "- `accelerator_count`: The number of accelerators to attach to a worker replica.\n",
        "- `sync`: If set to True, the call gets blocked while waiting for the asynchronous batch job to complete."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f98ed28340ba"
      },
      "outputs": [],
      "source": [
        "MIN_NODES = 1\n",
        "MAX_NODES = 1\n",
        "\n",
        "batch_predict_job = model.batch_predict(\n",
        "    job_display_name=\"example_\",\n",
        "    instances_format=\"jsonl\",\n",
        "    predictions_format=\"jsonl\",\n",
        "    model_parameters=None,\n",
        "    gcs_source=f\"{BUCKET_URI}/test.jsonl\",\n",
        "    gcs_destination_prefix=f\"{BUCKET_URI}/results\",\n",
        "    machine_type=DEPLOY_COMPUTE,\n",
        "    accelerator_type=DEPLOY_GPU,\n",
        "    accelerator_count=DEPLOY_NGPU,\n",
        "    starting_replica_count=MIN_NODES,\n",
        "    max_replica_count=MAX_NODES,\n",
        "    sync=False,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "batch_request_wait:mbsdk"
      },
      "source": [
        "## Wait for completion of batch prediction job\n",
        "\n",
        "Next, wait for the batch job to complete. Alternatively, you can set the parameter `sync` to `True` in the `batch_predict()` method to block until the batch prediction job is completed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "batch_request_wait:mbsdk"
      },
      "outputs": [],
      "source": [
        "batch_predict_job.wait()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_batch_prediction:mbsdk,custom,icn"
      },
      "source": [
        "## Get the predictions\n",
        "\n",
        "Next, get the results from the completed batch prediction job.\n",
        "\n",
        "The results are written to the Cloud Storage output bucket you specified in the batch prediction request. You call the method iter_outputs() to get a list of each Cloud Storage file generated with the results. Each file contains one or more prediction requests in a JSON format:\n",
        "\n",
        "- `instance`: The prediction request.\n",
        "- `prediction`: The prediction response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_batch_prediction:mbsdk,custom,icn"
      },
      "outputs": [],
      "source": [
        "bp_iter_outputs = batch_predict_job.iter_outputs()\n",
        "\n",
        "prediction_results = list()\n",
        "for blob in bp_iter_outputs:\n",
        "    if blob.name.split(\"/\")[-1].startswith(\"prediction\"):\n",
        "        prediction_results.append(blob.name)\n",
        "\n",
        "tags = list()\n",
        "for prediction_result in prediction_results:\n",
        "    gfile_name = f\"gs://{bp_iter_outputs.bucket.name}/{prediction_result}\"\n",
        "    with tf.io.gfile.GFile(name=gfile_name, mode=\"r\") as gfile:\n",
        "        for line in gfile.readlines():\n",
        "            line = json.loads(line)\n",
        "            print(line)\n",
        "            break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "# set delete_bucket to True to delete your bucket\n",
        "delete_bucket = False\n",
        "\n",
        "# undeploy the model from the endpoint\n",
        "endpoint.undeploy_all()\n",
        "\n",
        "# delete the endpoint\n",
        "endpoint.delete()\n",
        "\n",
        "# delete the model\n",
        "model.delete()\n",
        "\n",
        "# delete the batch job\n",
        "batch_predict_job.delete()\n",
        "\n",
        "# delete the bucket\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -rf {BUCKET_URI}"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "get_started_with_tf_serving.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}