aws_marketplace/using_model_packages/auto_insurance/automating_auto_insurance_claim_processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Goal: Automate Auto Insurance Claim Processing Using Pre-trained Models \n",
    "Auto insurance claim process requires extracting metadata from images and performing validations to ensure that the claim is not fraudulent. This sample notebook shows how third party pre-trained machine learning models can be used to extract such metadata from images.\n",
    "\n",
    "This notebook uses [Vehicle Damage Inspection](https://aws.amazon.com/marketplace/pp/Persistent-Systems-Vehicle-Damage-Inspection/prodview-xhj66rbazm6oe) model to identify the type of damage and [Deep Vision vehicle recognition](https://aws.amazon.com/marketplace/pp/prodview-a7wgrolhu54ts?qid=1558356141251&sr=0-4&ref_=srh_res_product_title) to identify the make, model, year, and bounding box of the car. This notebook also shows how to use the bounding box to extract license information from the using [Amazon Rekognition](https://aws.amazon.com/rekognition/).\n",
    "\n",
    "## Pre-requisites\n",
    "This sample notebook requires subscription to following pre-trained machine learning model packages from AWS Marketplace:\n",
    "\n",
    "1. [Vehicle Damage Inspection](https://aws.amazon.com/marketplace/pp/Persistent-Systems-Vehicle-Damage-Inspection/prodview-xhj66rbazm6oe)\n",
    "2. [Deep Vision vehicle recognition](https://aws.amazon.com/marketplace/pp/prodview-a7wgrolhu54ts?qid=1558356141251&sr=0-4&ref_=srh_res_product_title)\n",
    "\n",
    "If your AWS account has not been subscribed to these listings, here is the process you can follow for each of the above mentioned listings:\n",
    "1. Open the listing from AWS Marketplace\n",
    "2. Read the **Highlights** section and then **product overview** section of the listing.\n",
    "3. View **usage information** and then **additional resources**.\n",
    "4. Note the supported instance types.\n",
    "5. Next, click on **Continue to subscribe**.\n",
    "6. Review **End user license agreement**, **support terms**, as well as **pricing information**.\n",
    "7. **\"Accept Offer\"** button needs to be clicked if your organization agrees with EULA, pricing information as well as support terms.\n",
    "\n",
    "**Notes**: \n",
    "1. If **Continue to configuration** button is active, it means your account already has a subscription to this listing.\n",
    "2. Once you click on **Continue to configuration** button and then choose region, you will see that a **Product Arn** will appear. This is the model package ARN that you need to specify while creating a deployable model. However, for this notebook, the algorithm ARN has been specified in **src/model_package_arns.py** file and you do not need to specify the same explicitly.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up environment and view a sample image\n",
    "\n",
    "In this section, we will import necessary libraries and define variables such as an S3 bucket, an IAM role, and SageMaker session to be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import json\n",
    "import uuid\n",
    "from sagemaker import ModelPackage\n",
    "from src.model_package_arns import ModelPackageArnProvider\n",
    "import sagemaker as sage\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker import ModelPackage\n",
    "from urllib.parse import urlparse\n",
    "import boto3\n",
    "from IPython.display import Image\n",
    "from PIL import Image as ImageEdit\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "sagemaker_session = sage.Session()\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "runtime = boto3.client(\"runtime.sagemaker\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For your convenience sample images which depict damage (manually added using a photo editor tool), have been provided with this notebook. Next, view the image to be processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vehicle_image_path = \"img/car_damage.jpg\"\n",
    "vehicle_image_damage_closeup_path = \"img/closeup.png\"\n",
    "\n",
    "# View the image\n",
    "Image(url=vehicle_image_path, width=400, height=800)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# View the close-up image of the damaged part\n",
    "Image(url=vehicle_image_damage_closeup_path, width=400, height=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Deploy Vehicle Damage Inspection model\n",
    "\n",
    "In this step, we will deploy the [Vehicle Damage Inspection](https://aws.amazon.com/marketplace/pp/Persistent-Systems-Vehicle-Damage-Inspection/prodview-xhj66rbazm6oe) model package. The model package can be used to detect following types of car damages:\n",
    "1. Normal image\n",
    "2. Broken headlight\n",
    "3. Broken windshield\n",
    "4. Full front damage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.1: Deploy the model for performing real-time inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name_1 = \"vehicle-damage-detection-endpoint\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the model_package_arn\n",
    "damage_detection_modelpackage_arn = (\n",
    "    ModelPackageArnProvider.get_vehicle_damage_detection_model_package_arn(\n",
    "        sagemaker_session.boto_region_name\n",
    "    )\n",
    ")\n",
    "\n",
    "# create a deployable model for damage inspection model package.\n",
    "damage_detection_model = ModelPackage(\n",
    "    role=role,\n",
    "    model_package_arn=damage_detection_modelpackage_arn,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "\n",
    "# Deploy the model\n",
    "predictor_damage_detection = damage_detection_model.deploy(\n",
    "    1, \"ml.m4.xlarge\", endpoint_name=endpoint_name_1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.2: Perform a prediction on Amazon Sagemaker Endpoint created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, we will prepare a payload and perform a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open the file and read the image into a bytearray.\n",
    "with open(vehicle_image_damage_closeup_path, \"rb\") as image:\n",
    "    b = bytearray(image.read())\n",
    "\n",
    "response = runtime.invoke_endpoint(EndpointName=endpoint_name_1, ContentType=\"image/jpeg\", Body=b)\n",
    "\n",
    "# Perform a prediction\n",
    "damage_detection_result = response[\"Body\"].read().decode()\n",
    "\n",
    "# View the prediction\n",
    "print(damage_detection_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Deploy the Vehicle recognition model.\n",
    "\n",
    "In this step, we will deploy the [Deep Vision vehicle recognition](https://aws.amazon.com/marketplace/pp/prodview-a7wgrolhu54ts?qid=1558356141251&sr=0-4&ref_=srh_res_product_title) model package.\n",
    "\n",
    "We will use it to detect year, make, model, and angle (such as front right, front left, front center, rear right, rear left, rear center, side left, side right) of the car in picture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.1:  Deploy the model for performing real-time inference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name_2 = \"vehicle-recognition-endpoint\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the model_package_arn\n",
    "vehicle_recognition_modelpackage_arn = (\n",
    "    ModelPackageArnProvider.get_vehicle_recognition_model_package_arn(\n",
    "        sagemaker_session.boto_region_name\n",
    "    )\n",
    ")\n",
    "\n",
    "# create a deployable model.\n",
    "vehicle_recognition_model = ModelPackage(\n",
    "    role=role,\n",
    "    model_package_arn=vehicle_recognition_modelpackage_arn,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "\n",
    "# Deploy the model\n",
    "predictor_vehicle_recognition = vehicle_recognition_model.deploy(\n",
    "    1, \"ml.p2.xlarge\", endpoint_name=endpoint_name_2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.2: Perform real-time inference on the model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the image and prepare the payload\n",
    "image = open(vehicle_image_path, \"rb\")\n",
    "image_64_encode = base64.b64encode(image.read()).decode(\"utf-8\")\n",
    "\n",
    "# Prepare payload for prediction\n",
    "payload = '{\"source\": \"' + str(image_64_encode) + '\"}'\n",
    "\n",
    "# Perform a prediction\n",
    "response = runtime.invoke_endpoint(\n",
    "    EndpointName=endpoint_name_2, ContentType=\"application/json\", Body=payload\n",
    ")\n",
    "\n",
    "# Perform a prediction\n",
    "result = response[\"Body\"].read().decode()\n",
    "\n",
    "vehicle_mmy_result = json.loads(result)\n",
    "# View the prediction\n",
    "print(json.dumps(vehicle_mmy_result, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.3: Store the precise car image for further processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the bounding box of the first result.\n",
    "left_top_x = int(vehicle_mmy_result[\"result\"][0][\"bbox\"][\"left\"])\n",
    "left_top_y = int(vehicle_mmy_result[\"result\"][0][\"bbox\"][\"top\"])\n",
    "\n",
    "\n",
    "right_bottom_x = int(vehicle_mmy_result[\"result\"][0][\"bbox\"][\"right\"])\n",
    "right_bottom_y = int(vehicle_mmy_result[\"result\"][0][\"bbox\"][\"bottom\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let us crop the image based on bounding box and use the same for extracting license information.\n",
    "vehicle_image = ImageEdit.open(vehicle_image_path)\n",
    "\n",
    "vehicle_image_bounding_box_path = \"vehicle_image_bounding_box_2.jpg\"\n",
    "\n",
    "vehicle_image_bounding_box = vehicle_image.crop(\n",
    "    (left_top_x, left_top_y, right_bottom_x, right_bottom_y)\n",
    ")\n",
    "vehicle_image_bounding_box.save(vehicle_image_bounding_box_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Extract labels from the picture (optional)\n",
    "\n",
    "Let us use the car image extracted from the original image for extracting license information using [Amazon Rekognition](https://aws.amazon.com/rekognition/).\n",
    "\n",
    "**Note**:\n",
    "\n",
    "This step requires the IAM role associated with this notebook to have *__rekognition:DetectText__* IAM permission.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = boto3.client(\"rekognition\")\n",
    "\n",
    "recognized_word = \"\"\n",
    "with open(vehicle_image_bounding_box_path, \"rb\") as image:\n",
    "    response = client.detect_text(Image={\"Bytes\": image.read()})\n",
    "\n",
    "for label in response[\"TextDetections\"]:\n",
    "    if label[\"Confidence\"] > 99 and label[\"Type\"] == \"WORD\":\n",
    "        print(label[\"DetectedText\"])\n",
    "        recognized_word = label[\"DetectedText\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: View all outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View the original image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(url=vehicle_image_path, width=400, height=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the metadata the metadata we have extracted so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Vehicle Make found: \" + vehicle_mmy_result[\"result\"][0][\"mmy\"][\"make\"])\n",
    "print(\"Vehicle Model found: \" + vehicle_mmy_result[\"result\"][0][\"mmy\"][\"model\"])\n",
    "print(\"Vehicle Year found: \" + vehicle_mmy_result[\"result\"][0][\"mmy\"][\"year\"])\n",
    "print(\"Damage detection probabilities: \" + json.loads(damage_detection_result)[\"Results\"])\n",
    "print(\"License detected: \" + recognized_word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how we were able extract information such as car's make, model, year, and damage-type using pre-trained machine learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Cleanup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "damage_detection_model.sagemaker_session.delete_endpoint(endpoint_name_1)\n",
    "damage_detection_model.sagemaker_session.delete_endpoint_config(endpoint_name_1)\n",
    "damage_detection_model.delete_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vehicle_recognition_model.sagemaker_session.delete_endpoint(endpoint_name_2)\n",
    "vehicle_recognition_model.sagemaker_session.delete_endpoint_config(endpoint_name_2)\n",
    "vehicle_recognition_model.delete_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, if the AWS Marketplace subscription was created just for the experiment and you would like to unsubscribe to the product, here are the steps that can be followed.\n",
    "Before you cancel the subscription, ensure that you do not have any [deployable model](https://console.aws.amazon.com/sagemaker/home#/models) created from the model package or using the algorithm. Note - You can find this information by looking at the container name associated with the model. \n",
    "\n",
    "**Steps to un-subscribe to product from AWS Marketplace**:\n",
    "1. Navigate to __Machine Learning__ tab on [__Your Software subscriptions page__](https://aws.amazon.com/marketplace/ai/library?productType=ml&ref_=lbr_tab_ml)\n",
    "2. Locate the listing that you would need to cancel subscription for, and then __Cancel Subscription__ can be clicked to cancel the subscription.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}