Triton GPT2 example

add to docs link and update the notebook
SeldonIO · Apr 17, 2021 · a71dfaf · a71dfaf
1 parent 0dbb97b
commit a71dfaf
Show file tree

Hide file tree

Showing 3 changed files with 91 additions and 36 deletions.
diff --git a/doc/source/examples/notebooks.rst b/doc/source/examples/notebooks.rst
@@ -37,6 +37,7 @@ Python Language Wrapper Examples
    TFserving MNIST <tfserving_mnist>
    Statsmodels Holt-Winter's time-series model <statsmodels>
    Runtime Metrics & Tags <runtime_metrics_tags>
+   Triton GPT2 Example <triton_gpt2_example>
 
 Specialised Framework Examples
 ------------------------------

diff --git a/doc/source/examples/triton_gpt2_example.nblink b/doc/source/examples/triton_gpt2_example.nblink
@@ -0,0 +1,3 @@
+{
+  "path": "../../../examples/triton_gpt2/README.ipynb"
+}
diff --git a/examples/triton_gpt2/README.ipynb b/examples/triton_gpt2/README.ipynb
@@ -1,14 +1,15 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
-   "id": "incorporate-present",
+   "id": "liked-toronto",
    "metadata": {},
    "source": [
     "# Pretrainned GPT2  Model Deployment Example\n",
     "\n",
-    "In this notebook we will run an example of text generation using GPT2 pretrained model exported from Hugging Face transformers library and deployed with Seldon-Core Triton ..... We will run a greedy algorithem implemantation for next token prediction.\n",
+    "In this notebook we will run an example of text generation using GPT2 pretrained model, which is exported from HuggingFace transformers library and deployed with Seldon's Triton pre-packed server.\n",
+    "the implemented example bellow is of the Greedy approach, implemantation for next token prediction.\n",
+    "\n",
     "more info: https://huggingface.co/transformers/model_doc/gpt2.html?highlight=gpt2\n",
     "\n",
     "## Steps:\n",
@@ -18,6 +19,7 @@
     "4. Setup seldon in your kubernetes cluster\n",
     "5. Deploy the ONNX model with Seldon’s prepackaged Triton server.\n",
     "6. Interact with the model, run a greedy alg example (generate sentance complition)\n",
+    "7. Cleanup\n",
     "\n",
     "## Basic requierments \n",
     "* Helm v3.0.0+\n",
@@ -29,7 +31,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "plain-purple",
+   "id": "korean-reporter",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -44,7 +46,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "dressed-paint",
+   "id": "assigned-diesel",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -53,7 +55,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "willing-greene",
+   "id": "completed-evaluation",
    "metadata": {},
    "source": [
     "### Export   🤗 TFGPT2LMHeadModel pre-trained model and save it locally"
@@ -62,7 +64,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "moving-monitoring",
+   "id": "iraqi-million",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -74,7 +76,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "expired-burns",
+   "id": "further-tribute",
    "metadata": {},
    "source": [
     "### Convert the TensorFlow saved model to ONNX"
@@ -83,7 +85,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "institutional-observation",
+   "id": "irish-mountain",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -92,7 +94,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "demographic-bottom",
+   "id": "sunset-pantyhose",
    "metadata": {},
    "source": [
     "### Copy your model to a local MinIo\n",
@@ -107,7 +109,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "offensive-astronomy",
+   "id": "lasting-performance",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -117,7 +119,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "three-recommendation",
+   "id": "convinced-syracuse",
    "metadata": {},
    "source": [
     "### Run Seldon in your kubernetes cluster\n",
@@ -127,26 +129,33 @@
   },
   {
    "cell_type": "markdown",
-   "id": "aware-advocate",
+   "id": "backed-outreach",
    "metadata": {},
    "source": [
     "### Deploy your model with Seldon pre-packaged Triton server"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "young-apparel",
+   "execution_count": 3,
+   "id": "declared-crown",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting secret.yaml\n"
+     ]
+    }
+   ],
    "source": [
     "%%writefile secret.yaml\n",
     "\n",
     "apiVersion: v1\n",
     "kind: Secret\n",
     "metadata:\n",
     "  name: seldon-init-container-secret\n",
-    "  namespace: seldon\n",
     "type: Opaque\n",
     "stringData:\n",
     "  AWS_ACCESS_KEY_ID: minioadmin\n",
@@ -157,17 +166,24 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "technological-treasure",
+   "execution_count": 4,
+   "id": "beneficial-anime",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting gpt2-deploy.yaml\n"
+     ]
+    }
+   ],
    "source": [
     "%%writefile gpt2-deploy.yaml\n",
     "apiVersion: machinelearning.seldon.io/v1alpha2\n",
     "kind: SeldonDeployment\n",
     "metadata:\n",
     "  name: gpt2\n",
-    "  namespace: seldon\n",
     "spec:\n",
     "  predictors:\n",
     "  - graph:\n",
@@ -176,7 +192,7 @@
     "        mode: all\n",
     "      modelUri: s3://onnx-gpt2\n",
     "      envSecretRefName: seldon-init-container-secret\n",
-    "      name: gpt2\n",
+    "      name: gpt23\n",
     "      type: MODEL\n",
     "    name: default\n",
     "    replicas: 1\n",
@@ -185,28 +201,45 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "stupid-pattern",
+   "execution_count": 5,
+   "id": "subjective-involvement",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "secret/seldon-init-container-secret configured\n",
+      "seldondeployment.machinelearning.seldon.io/gpt2 configured\n"
+     ]
+    }
+   ],
    "source": [
     "!kubectl apply -f secret.yaml\n",
     "!kubectl apply -f gpt2-deploy.yaml"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "sonic-peace",
+   "execution_count": 6,
+   "id": "demanding-thesaurus",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "deployment \"gpt2-default-0-gpt2\" successfully rolled out\r\n"
+     ]
+    }
+   ],
    "source": [
     "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=gpt2 -o jsonpath='{.items[0].metadata.name}')"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "cellular-still",
+   "id": "digital-supervisor",
    "metadata": {},
    "source": [
     "#### Interact with the model: get model metadata ( a \"test\" request to make sure model is avilable)"
@@ -215,7 +248,7 @@
   {
    "cell_type": "code",
    "execution_count": 43,
-   "id": "complete-solomon",
+   "id": "married-roller",
    "metadata": {},
    "outputs": [
     {
@@ -271,24 +304,24 @@
   },
   {
    "cell_type": "markdown",
-   "id": "worldwide-tuition",
+   "id": "anonymous-resource",
    "metadata": {},
    "source": [
     "### Run prediction test: generate a sentense complition using GPT2 model  - Greedy approach\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 44,
-   "id": "express-czech",
+   "execution_count": 7,
+   "id": "modified-termination",
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Input: I enjoy spring in London, especially walking in the parks\n",
-      "Output: I enjoy spring in London, especially walking in the parks . I love the smell of the roses and the\n"
+      "Input: I enjoy working in Seldon\n",
+      "Output: I enjoy working in Seldon 's office , and I 'm glad to see that\n"
      ]
     }
    ],
@@ -299,7 +332,7 @@
     "from transformers import GPT2Tokenizer\n",
     "\n",
     "tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
-    "input_text = 'I enjoy spring in London, especially walking in the parks'\n",
+    "input_text = 'I enjoy working in Seldon'\n",
     "count = 0\n",
     "max_gen_len = 10\n",
     "gen_sentence = input_text\n",
@@ -341,6 +374,24 @@
     "\n",
     "print(f'Input: {input_text}\\nOutput: {gen_sentence}')"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "patient-suite",
+   "metadata": {},
+   "source": [
+    "### Cleanup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "pacific-collectible",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!kubectl delete -f gpt2-deploy.yaml"
+   ]
   }
  ],
  "metadata": {