notebooks/official/prediction/get_started_with_raw_predict.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic,gcp"
      },
      "source": [
        "# Get started with TensorFlow serving functions with Vertex AI Raw Prediction\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_raw_predict.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fprediction%2Fget_started_with_raw_predict.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/raw/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/prediction/get_started_with_raw_predict.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_raw_predict.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial demonstrates how to use `Vertex AI Raw Prediction` to send raw HTTP content directly to a model deployed to a `Vertex AI Endpoint`. \n",
        "\n",
        "For example, the HTTP server for pre-built `Vertex AI` deployment containers does not support the HTTP request body for TensorFlow 1.x estimators. Using raw predict, one can send raw content through the HTTP server that is presented to the model input as-is -- no canonical processing.\n",
        "\n",
        "Learn more about [Raw Predict](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.endpoints/rawPredict)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9402cfbdc2d"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use `Vertex AI Raw Prediction` on a `Vertex AI Endpoint` resource.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- `Vertex AI Raw Prediction`\n",
        "- `Vertex AI Models`\n",
        "- `Vertex AI Endpoints`\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Download pretrained tabular classification model artifacts for a TensorFlow 1.x estimator.\n",
        "- Upload the TensorFlow estimator model as a `Vertex AI Model` resource.\n",
        "- Create an `Endpoint` resource.\n",
        "- Deploy the `Model` resource to an `Endpoint` resource.\n",
        "- Make an online raw prediction to the `Model` resource instance deployed to the `Endpoint` resource."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:iris,lcn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "This tutorial uses a pre-trained tabular classification model from a public Cloud Storage bucket, which is trained on the Penguins dataset (https://cloud.google.com/bigquery/public-data). The version of the dataset predicts the species."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3b1ffd5ab768"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aae9ca040eab"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ad8462a9eeb"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform \n",
        "! pip3 install tensorflow-hub"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ff555b32bab8"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f09b4dff629a"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "54c5ef8a8f43"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92e68cfc3a90"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "46604f70e831"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "107c51893a64"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "294fe4e5a671"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "#### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b7c2a8a91840"
      },
      "source": [
        "#### Set Google Cloud project"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a4f61991b160"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "\n",
        "aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for training and prediction.\n",
        "\n",
        "Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Tesla K80 GPUs allocated to each VM, you would specify:\n",
        "\n",
        "    (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
        "\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "outputs": [],
      "source": [
        "DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training,prediction"
      },
      "source": [
        "#### Set pre-built containers\n",
        "\n",
        "Set the pre-built Docker container image for prediction.\n",
        "\n",
        "\n",
        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:training,prediction"
      },
      "outputs": [],
      "source": [
        "TF = \"2-5\"\n",
        "\n",
        "if DEPLOY_GPU:\n",
        "    DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
        "else:\n",
        "    DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
        "\n",
        "\n",
        "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
        "    LOCATION.split(\"-\")[0], DEPLOY_VERSION\n",
        ")\n",
        "\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training"
      },
      "source": [
        "#### Set machine type\n",
        "\n",
        "Next, set the machine type to use for prediction.\n",
        "\n",
        "- Set the variable `DEPLOY_COMPUTE` to configure  the compute resources for the VMs you will use for prediction.\n",
        " - `machine type`\n",
        "     - `n1-standard`: 3.75GB of memory per vCPU.\n",
        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "machine:training"
      },
      "outputs": [],
      "source": [
        "MACHINE_TYPE = \"n1-standard\"\n",
        "\n",
        "VCPU = \"4\"\n",
        "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Train machine type\", DEPLOY_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8128b8ff025"
      },
      "source": [
        "## Get pretrained model from the public Cloud Storage location\n",
        "\n",
        "For demonstration purposes, this tutorial uses a pretrained model TensorFlow 1.x estimator tabular classification Model, which is then uploaded to a `Vertex AI Model` resource. Once you have a `Vertex AI Model` resource, the model can be deployed to a `Vertex AI Endpoint` resource.\n",
        "\n",
        "### Download the pretrained model\n",
        "\n",
        "Download the pretrained TensorFlow estimator model artifacts from the public Cloud Storage, and then upload the model artifacts to your own Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c55fa4c826f7"
      },
      "outputs": [],
      "source": [
        "MODEL_DIR = BUCKET_URI + \"/model\"\n",
        "\n",
        "! gsutil cp -r gs://cloud-samples-data/vertex-ai/google-cloud-aiplatform-ci-artifacts/models/penguins/estimator/ {MODEL_DIR}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8ce91147c93"
      },
      "source": [
        "## Upload the TensorFlow estimator model to a `Vertex AI Model` resource\n",
        "\n",
        "Finally, you upload the model artifacts from the TFHub model and serving function into a `Vertex AI Model` resource.\n",
        "\n",
        "*Note:* When you upload the model artifacts to a `Vertex AI Model` resource, you specify the corresponding deployment container image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ad61e1429512"
      },
      "outputs": [],
      "source": [
        "model = aiplatform.Model.upload(\n",
        "    display_name=\"example\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        ")\n",
        "\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "628de0914ba1"
      },
      "source": [
        "## Creating an `Endpoint` resource\n",
        "\n",
        "You create an `Endpoint` resource using the `Endpoint.create()` method. At a minimum, you specify the display name for the endpoint. Optionally, you can specify the project and location; otherwise the settings are inherited by the values you set when you initialized the Vertex AI SDK with the `init()` method.\n",
        "\n",
        "In this example, the following parameters are specified:\n",
        "\n",
        "- `display_name`: A human readable name for the `Endpoint` resource.\n",
        "- `project`: Your project ID.\n",
        "- `location`: Your location.\n",
        "- `labels`: (optional) User defined metadata for the `Endpoint` in the form of key/value pairs.\n",
        "\n",
        "This method returns an `Endpoint` object.\n",
        "\n",
        "Learn more about [Vertex AI Endpoints](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ea443f9593b"
      },
      "outputs": [],
      "source": [
        "endpoint = aiplatform.Endpoint.create(\n",
        "    display_name=\"example\",\n",
        "    project=PROJECT_ID,\n",
        "    location=LOCATION,\n",
        "    labels={\"your_key\": \"your_value\"},\n",
        ")\n",
        "\n",
        "print(endpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ca3fa3f6a894"
      },
      "source": [
        "## Deploying `Model` resources to an `Endpoint` resource.\n",
        "\n",
        "You can deploy one of more `Vertex AI Model` resource instances to the same endpoint. Each `Vertex AI Model` resource that is deployed will have its own deployment container for the serving binary. \n",
        "\n",
        "In the next example, you deploy the `Vertex AI Model` resource to a `Vertex AI Endpoint` resource. The `Vertex AI Model` resource already has defined the deployment container image. To deploy, you specify the following additional configuration settings:\n",
        "\n",
        "- The machine type.\n",
        "- The (if any) type and number of GPUs.\n",
        "- Static, manual or auto-scaling of VM instances.\n",
        "\n",
        "In this example, you deploy the model with the minimal amount of specified parameters, as follows:\n",
        "\n",
        "- `model`: The `Model` resource.\n",
        "- `deployed_model_displayed_name`: The human readable name for the deployed model instance.\n",
        "- `machine_type`: The machine type for each VM instance.\n",
        "\n",
        "Do to the requirements to provision the resource, this may take upto a few minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4e93b034a72f"
      },
      "outputs": [],
      "source": [
        "response = endpoint.deploy(\n",
        "    model=model,\n",
        "    deployed_model_display_name=\"example\",\n",
        "    machine_type=DEPLOY_COMPUTE,\n",
        ")\n",
        "\n",
        "print(endpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_test_items:bqml,penguins"
      },
      "source": [
        "#### Make prediction instances\n",
        "\n",
        "Next, you prepare a prediction request using a synthetic example. In this example, the model format is a TensorFlow 1.x estimator format. This model format takes a request signature not supported by the HTTP server in the `Vertex AI` prebuilt TensorFlow serving containers.\n",
        "\n",
        "For this model format, you use the `raw_predict()` to pass as-is a request that matches directly the serving interface of the model, with the following request format:\n",
        "\n",
        "    http_body -> {\n",
        "        'signature_name' : serving_signature,\n",
        "        'instances': [ {instance_1}, {instance_2}, ... ]\n",
        "    }\n",
        "\n",
        "    instance -> { 'feature_1': value_1, 'feature_2': value_2, ... }\n",
        "\n",
        "    serving_signature -> \"predict\"\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "make_test_items:bqml,penguins"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "from google.api import httpbody_pb2\n",
        "from google.cloud import aiplatform_v1\n",
        "\n",
        "DATA = {\n",
        "    \"signature_name\": \"predict\",\n",
        "    \"instances\": [\n",
        "        {\n",
        "            \"island\": \"DREAM\",\n",
        "            \"culmen_length_mm\": 36.6,\n",
        "            \"culmen_depth_mm\": 18.4,\n",
        "            \"flipper_length_mm\": 184.0,\n",
        "            \"body_mass_g\": 3475.0,\n",
        "            \"sex\": \"FEMALE\",\n",
        "        }\n",
        "    ],\n",
        "}\n",
        "\n",
        "http_body = httpbody_pb2.HttpBody(\n",
        "    data=json.dumps(DATA).encode(\"utf-8\"),\n",
        "    content_type=\"application/json\",\n",
        ")\n",
        "\n",
        "req = aiplatform_v1.RawPredictRequest(\n",
        "    http_body=http_body, endpoint=endpoint.resource_name\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "endpoint_predict:mbsdk"
      },
      "source": [
        "### Make a prediction\n",
        "\n",
        "Finally, you make the prediction request using Vertex AI Prediction service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "endpoint_predict:mbsdk"
      },
      "outputs": [],
      "source": [
        "API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(LOCATION)\n",
        "client_options = {\"api_endpoint\": API_ENDPOINT}\n",
        "\n",
        "pred_client = aiplatform.gapic.PredictionServiceClient(client_options=client_options)\n",
        "\n",
        "response = pred_client.raw_predict(req)\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "delete_bucket = False\n",
        "delete_model = True\n",
        "delete_endpoint = True\n",
        "\n",
        "if delete_endpoint:\n",
        "    try:\n",
        "        endpoint.undeploy_all()\n",
        "        endpoint.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "if delete_model:\n",
        "    try:\n",
        "        model.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -rf {BUCKET_URI}"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "get_started_with_raw_predict.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}