Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore, refactor (egen): refactors and edits model_monitoring.ipynb notebook #3195

Merged
merged 10 commits into from
Aug 13, 2024
Merged

chore, refactor (egen): refactors and edits model_monitoring.ipynb notebook #3195

Changes from 5 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
00a1d3e
chore: remove boiler plate, add colab enterprise, and format accordin…
k-root Jul 2, 2024
c45a353
chore, refactor: test end to end
k-root Jul 2, 2024
d4b502d
chore: lint test
k-root Jul 2, 2024
27ee52f
chore, fix: removes force protobuf for package compatibility issue an…
k-root Jul 3, 2024
5b57049
fix: changes import statement for execution
k-root Jul 4, 2024
2b5f772
adds protobuf in install to deal with build error
k-root Jul 15, 2024
58800de
fix: changes protobuf installation version
k-root Jul 29, 2024
96a4e4a
chore: lint run
k-root Jul 29, 2024
b68bc9e
Adds tensorflow in install, other fixes from template
Aug 6, 2024
014dfa3
Adds tensorflow in install, other fixes from template
Aug 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
221 changes: 89 additions & 132 deletions notebooks/official/model_monitoring/model_monitoring.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,21 +34,27 @@
"# Vertex AI Model Monitoring with Explainable AI Feature Attributions\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmodel_monitoring%2Fmodel_monitoring.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/raw/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
" <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br>\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/raw/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
" <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Vertex AI logo\">Open in Vertex AI Workbench\n",
" </a>\n",
"</table>"
]
},
Expand Down Expand Up @@ -93,21 +99,21 @@
"source": [
"### Objective\n",
"\n",
"In this notebook, you learn to use the `Vertex AI Model Monitoring` service to detect drift and anomalies in prediction requests from a deployed `Vertex AI Model` resource. \n",
"In this notebook, you learn to use the Vertex AI Model Monitoring service to detect drift and anomalies in prediction requests from a deployed Vertex AI model resource. \n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Vertex AI Model Monitoring`\n",
"- `Vertex AI Prediction`\n",
"- `Vertex AI Model` resource\n",
"- `Vertex AI Endpoint` resource\n",
"- Vertex AI Model Monitoring\n",
"- Vertex AI Prediction\n",
"- Vertex AI model resource\n",
"- Vertex AI endpoint resource\n",
"\n",
"The steps performed include:\n",
"\n",
"- Upload a pre-trained model as a `Vertex AI Model` resource.\n",
"- Create an `Vertex AI Endpoint` resource.\n",
"- Deploy the `Model` resource to the `Endpoint` resource.\n",
"- Configure the `Endpoint` resource for model monitoring.\n",
"- Upload a pre-trained model as a Vertex AI model resource.\n",
"- Create an Vertex AI endpoint resource.\n",
"- Deploy the model resource to the endpoint resource.\n",
"- Configure the endpoint resource for model monitoring.\n",
"- Initialize the baseline distribution for model monitoring.\n",
"- Generate synthetic prediction requests.\n",
"- Understand how to interpret the statistics, visualizations, other data reported by the model monitoring feature."
Expand Down Expand Up @@ -152,28 +158,34 @@
"id": "f3848df1e5b0"
},
"source": [
"## Installation\n",
"\n",
"Install the packages required for executing this notebook."
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a2c2cb2109a0"
},
"source": [
"### Install Vertex AI SDK for Python and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b24b232ee039"
"id": "ac30781c8ed0"
},
"outputs": [],
"source": [
"# Don't bother installing tensorflow or explainable_ai_sdk on Colab\n",
"\n",
"# Install required packages.\n",
"! pip3 install --upgrade --quiet \\\n",
" google-cloud-aiplatform \\\n",
" google-cloud-aiplatform[full] \\\n",
" google-cloud-bigquery \\\n",
" explainable_ai_sdk \n",
"#! pip3 install --upgrade --quiet tensorflow==2.11\n",
"! pip3 install --force-reinstall protobuf==4.23.3"
"! pip3 install --upgrade --quiet tensorflow==2.11"
]
},
{
Expand All @@ -182,7 +194,9 @@
"id": "restart"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel"
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
Expand All @@ -193,63 +207,81 @@
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "before_you_begin:nogpu"
"id": "ee775571c2b5"
},
"source": [
"## Before you begin\n",
"\n",
"### Set your project ID\n",
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "92e68cfc3a90"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "set_project_id"
"id": "46604f70e831"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "region"
"id": "4f872cd812d0"
},
"source": [
"#### Region\n",
"### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2dw8q9fdQEH5"
"id": "acbc2e1a15fe"
},
"outputs": [],
"source": [
"REGION = \"us-central1\""
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
"\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
Expand Down Expand Up @@ -279,64 +311,6 @@
" USER_EMAIL = \"noreply@google.com\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gcp_authenticate"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
"\n",
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated.\n",
"\n",
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ce6043da7b33"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0367eac06a10"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "21ad4dbb4a61"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down Expand Up @@ -365,7 +339,7 @@
"id": "autoset_bucket"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
"**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
]
},
{
Expand All @@ -376,7 +350,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
},
{
Expand Down Expand Up @@ -426,7 +400,6 @@
"import os\n",
"import pprint as pp\n",
"\n",
"import google.cloud.aiplatform as aiplatform\n",
"import matplotlib.pyplot as plt\n",
"from google.cloud import bigquery\n",
"from google.cloud.aiplatform import model_monitoring\n",
Expand All @@ -446,32 +419,10 @@
" ! gcloud --quiet components install beta\n",
" ! gcloud --quiet components update\n",
"\n",
"! gcloud config set ai/region $REGION\n",
"! gcloud config set ai/region $LOCATION\n",
"os.environ[\"GOOGLE_CLOUD_PROJECT\"] = PROJECT_ID"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "init_aip:mbsdk,region"
},
"source": [
"### Initialize Vertex AI SDK for Python\n",
"\n",
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wGa5T9eRR8Mz"
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION)"
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down Expand Up @@ -966,7 +917,7 @@
" alert_config=alerting_config,\n",
" objective_configs=objective_config,\n",
" project=PROJECT_ID,\n",
" location=REGION,\n",
" location=LOCATION,\n",
" endpoint=endpoint,\n",
")"
]
Expand Down Expand Up @@ -1254,7 +1205,13 @@
"rmtable = f\"bq rm -f model_deployment_monitoring_{ENDPOINT_ID}.serving_predict\"\n",
"! $rmtable\n",
"rmdataset = f\"bq rm -f model_deployment_monitoring_{ENDPOINT_ID}\"\n",
"! $rmdataset"
"! $rmdataset\n",
"\n",
"# Delete Cloud Storage bucket\n",
"delete_bucket = False\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -rf {BUCKET_URI}"
]
},
{
Expand Down