neuralmagic · robertgshaw2-neuralmagic · Mar 16, 2023 · Mar 16, 2023 · Mar 16, 2023 · Mar 16, 2023
diff --git a/README.md b/README.md
diff --git a/docs/neural-magic-workflow.png b/docs/neural-magic-workflow.png
diff --git a/docs/_static/css/nm-theme-adjustment.css → docs/old/_static/css/nm-theme-adjustment.css b/docs/_static/css/nm-theme-adjustment.css → docs/old/_static/css/nm-theme-adjustment.css
diff --git a/docs/_templates/versions.html → docs/old/_templates/versions.html b/docs/_templates/versions.html → docs/old/_templates/versions.html
diff --git a/docs/api/.gitkeep → docs/old/api/.gitkeep b/docs/api/.gitkeep → docs/old/api/.gitkeep
diff --git a/docs/api/deepsparse.rst → docs/old/api/deepsparse.rst b/docs/api/deepsparse.rst → docs/old/api/deepsparse.rst
diff --git a/docs/api/deepsparse.transformers.rst → docs/old/api/deepsparse.transformers.rst b/docs/api/deepsparse.transformers.rst → docs/old/api/deepsparse.transformers.rst
diff --git a/docs/api/deepsparse.utils.rst → docs/old/api/deepsparse.utils.rst b/docs/api/deepsparse.utils.rst → docs/old/api/deepsparse.utils.rst
diff --git a/docs/api/modules.rst → docs/old/api/modules.rst b/docs/api/modules.rst → docs/old/api/modules.rst
diff --git a/docs/conf.py → docs/old/conf.py b/docs/conf.py → docs/old/conf.py
diff --git a/...gging-optimizing/diagnostics-debugging.md → ...gging-optimizing/diagnostics-debugging.md b/...gging-optimizing/diagnostics-debugging.md → ...gging-optimizing/diagnostics-debugging.md
diff --git a/docs/debugging-optimizing/example-log.md → docs/old/debugging-optimizing/example-log.md b/docs/debugging-optimizing/example-log.md → docs/old/debugging-optimizing/example-log.md
diff --git a/docs/debugging-optimizing/index.rst → docs/old/debugging-optimizing/index.rst b/docs/debugging-optimizing/index.rst → docs/old/debugging-optimizing/index.rst
diff --git a/docs/debugging-optimizing/numactl-utility.md → ...d/debugging-optimizing/numactl-utility.md b/docs/debugging-optimizing/numactl-utility.md → ...d/debugging-optimizing/numactl-utility.md
diff --git a/docs/favicon.ico → docs/old/favicon.ico b/docs/favicon.ico → docs/old/favicon.ico
diff --git a/docs/index.rst → docs/old/index.rst b/docs/index.rst → docs/old/index.rst
diff --git a/docs/source/c++api-overview.md → docs/old/source/c++api-overview.md b/docs/source/c++api-overview.md → docs/old/source/c++api-overview.md
diff --git a/docs/source/hardware.md → docs/old/source/hardware.md b/docs/source/hardware.md → docs/old/source/hardware.md
diff --git a/docs/source/icon-deepsparse.png → docs/old/source/icon-deepsparse.png b/docs/source/icon-deepsparse.png → docs/old/source/icon-deepsparse.png
diff --git a/docs/source/multi-stream.png → docs/old/source/multi-stream.png b/docs/source/multi-stream.png → docs/old/source/multi-stream.png
diff --git a/docs/source/scheduler.md → docs/old/source/scheduler.md b/docs/source/scheduler.md → docs/old/source/scheduler.md
diff --git a/docs/source/single-stream.png → docs/old/source/single-stream.png b/docs/source/single-stream.png → docs/old/source/single-stream.png
diff --git a/docs/use-cases/README.md b/docs/use-cases/README.md
@@ -0,0 +1,90 @@
+<!--
+Copyright (c) 2021 - present / Neuralmagic, Inc. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Use Cases
+
+There are three interfaces for interacting with DeepSparse:
+
+- **Engine** is the lowest-level API that enables you to compile a model and run inference on raw input tensors.
+
+- **Pipeline** is the default DeepSparse API. Similar to Hugging Face Pipelines, it wraps Engine with task-specific pre-processing and post-processing steps, allowing you to make requests on raw data and receive post-processed predictions.
+
+- **Server** is a REST API wrapper around Pipelines built on FastAPI and Uvicorn. It enables you to start a model serving endpoint running DeepSparse with a single CLI.
+
+This directory offers examples using each API in various supported tasks. 
+
+### Supported Tasks
+
+DeepSparse supports the following tasks out of the box:
+
+|          NLP          |            CV             |
+|-----------------------|---------------------------|
+| [Text Classification `"text-classification"`](nlp/text-classification.md)     | [Image Classification `"image_classification"`](cv/image-classification.md)     |
+| [Token Classification `"token-classification"`](nlp/token-classification.md)  | [Object Detection `"yolo"`](cv/object-detection-yolov5.md)    |
+| [Sentiment Analysis `"sentiment-analysis"`](nlp/sentiment-analysis.md)        | [Instance Segmentation `"yolact"`](cv/image-segmentation-yolact.md)        |
+| [Question Answering `"question-answering"`](nlp/question-answering.md)        |                                                                         |
+| [Zero-Shot Text Classification `"zero-shot-text-classification"`](nlp/zero-shot-text-classification.md) |                                               |
+| [Embedding Extraction `"transformers_embedding_extraction"`](nlp/transformers-embedding-extraction.md) |                                               |
+
+### Examples
+
+**Pipeline Example** | Sentiment Analysis
+
+Here's an example of how a task is used to create a Pipeline:
+
+```python
+from deepsparse import Pipeline
+
+pipeline = Pipeline.create(
+  task="sentiment_analysis",
+  model_path="zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none")
+
+print(pipeline("I love DeepSparse Pipelines!"))
+# labels=['positive'] scores=[0.998009443283081]
+```
+
+**Server Example** | Sentiment Analysis
+
+Here's an example of how a task is used to create a Server:
+
+```bash
+deepsparse.server \
+  --task sentiment_analysis \
+  --model_path zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none
+```
+
+Making a request:
+
+```python
+import requests
+
+# Uvicorn is running on this port
+url = 'http://0.0.0.0:5543/predict'
+
+# send the data
+obj = {"sequences": "Sending requests to DeepSparse Server is fast and easy!"}
+resp = requests.post(url=url, json=obj)
+
+# recieve the post-processed output
+print(resp.text)
+# >> {"labels":["positive"],"scores":[0.9330279231071472]}
+```
+
+### Additional Resources
+
+- [Custom Tasks](../user-guide/deepsparse-pipelines.md#custom-use-case)
+- [Pipeline User Guide](../user-guide/deepsparse-pipelines.md)
+- [Server User Guide](../user-guide/deepsparse-server.md)
diff --git a/docs/use-cases/cv/embedding-extraction.md b/docs/use-cases/cv/embedding-extraction.md
@@ -0,0 +1,130 @@
+<!--
+Copyright (c) 2021 - present / Neuralmagic, Inc. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Deploying Embedding Extraction Models With DeepSparse
+This page explains how to deploy an Embedding Extraction Pipeline with DeepSparse.
+
+## Installation Requirements
+This use case requires the installation of [DeepSparse Server](../../user-guide/installation.md).
+
+Confirm your machine is compatible with our [hardware requirements](../../user-guide/hardware-support.md).
+
+## Model Format
+The Embedding Extraction Pipeline enables you to generate embeddings in any domain, meaning you can use it with any ONNX model. It (optionally) removes the projection head from the model, such that you can re-use SparseZoo models and custom models you have trained in the embedding extraction scenario.
+
+There are two options for passing a model to the Embedding Extraction Pipeline:
+
+- Pass a Local ONNX File
+- Pass a SparseZoo Stub (which identifies an ONNX model in the SparseZoo)
+
+## DeepSparse Pipelines
+Pipeline is the default interface for interacting with DeepSparse.
+
+Like Hugging Face Pipelines, DeepSparse Pipelines wrap pre- and post-processing around the inference performed by the Engine. This creates a clean API that allows you to pass raw text and images to DeepSparse and receive the post-processed predictions, making it easy to add DeepSparse to your application.
+
+We will use the `Pipeline.create()` constructor to create an instance of an embedding extraction Pipeline with a 95% pruned-quantized version of ResNet-50 trained on `imagenet`. We can then pass images the `Pipeline` and receive the embeddings. All of the pre-processing is handled by the `Pipeline`.
+
+The Embedding Extraction Pipeline handles some useful actions around inference:
+
+- First, on initialization, the Pipeline (optionally) removes a projection head from a model. You can use the `emb_extraction_layer` argument to specify which layer to return. If your ONNX model has no projection head, you can set `emb_extraction_layer=None` (the default) to skip this step.
+
+- Second, as with all DeepSparse Pipelines, it handles pre-processing such that you can pass raw input. You will notice that in addition to the typical task argument used in `Pipeline.create()`, the Embedding Extraction Pipeline includes a `base_task` argument. This argument tells the Pipeline the domain of the model, such that the Pipeline can figure out what pre-processing to do.
+
+Download an image to use with the Pipeline.
+```bash
+wget https://huggingface.co/spaces/neuralmagic/image-classification/resolve/main/lion.jpeg
+```
+
+This is an example of extracting the last layer from ResNet-50:
+
+```python
+from deepsparse import Pipeline
+
+# this step removes the projection head before compiling the model
+rn50_embedding_pipeline = Pipeline.create(
+    task="embedding-extraction",
+    base_task="image-classification", # tells the pipeline to expect images and normalize input with ImageNet means/stds
+    model_path="zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none",
+    emb_extraction_layer=-3, # extracts last layer before projection head and softmax
+)
+
+# this step runs pre-processing, inference and returns an embedding
+embedding = rn50_embedding_pipeline(images="lion.jpeg")
+print(len(embedding.embeddings[0][0]))
+# 2048 << size of final layer>>
+```
+
+### Cross Use Case Functionality
+Check out the [Pipeline User Guide](../../user-guide/deepsparse-pipelines.md) for more details on configuring the Pipeline.
+
+## DeepSparse Server
+As an alternative to the Python API, DeepSparse Server allows you to serve an Embedding Extraction Pipeline over HTTP. Configuring the server uses the same parameters and schemas as the Pipelines. 
+
+Once launched, a `/docs` endpoint is created with full endpoint descriptions and support for making sample requests.
+
+This configuration file sets `emb_extraction_layer` to -3:
+```yaml
+# config.yaml
+endpoints:
+    - task: embedding_extraction
+      model: zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none
+      kwargs:
+        base_task: image_classification
+        emb_extraction_layer: -3
+```
+Spin up the server: 
+```bash 
+deepsparse.server --config_file config.yaml
+```
+
+Make requests to the server: 
+```python
+import requests, json
+url = "http://0.0.0.0:5543/predict/from_files"
+paths = ["lion.jpeg"]
+files = [("request", open(img, 'rb')) for img in paths]
+resp = requests.post(url=url, files=files)
+result = json.loads(resp.text)
+
+print(len(result["embeddings"][0][0]))
+
+# 2048 << size of final layer>>
+```
+## Using a Custom ONNX File 
+Apart from using models from the SparseZoo, DeepSparse allows you to define custom ONNX files for embedding extraction. 
+
+The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. 
+Click Download on the [ResNet-50 - ImageNet page](https://sparsezoo.neuralmagic.com/models/cv%2Fclassification%2Fresnet_v1-50%2Fpytorch%2Fsparseml%2Fimagenet%2Fpruned95_uniform_quant-none) to download a ONNX ResNet model for demonstration. 
+
+Extract the downloaded file and use the ResNet-50 ONNX model for embedding extraction:
+```python
+from deepsparse import Pipeline
+
+# this step removes the projection head before compiling the model
+rn50_embedding_pipeline = Pipeline.create(
+    task="embedding-extraction",
+    base_task="image-classification", # tells the pipeline to expect images and normalize input with ImageNet means/stds
+    model_path="resnet.onnx",
+    emb_extraction_layer=-3, # extracts last layer before projection head and softmax
+)
+
+# this step runs pre-processing, inference and returns an embedding
+embedding = rn50_embedding_pipeline(images="lion.jpeg")
+print(len(embedding.embeddings[0][0]))
+# 2048
+```
+### Cross Use Case Functionality
+Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server.