[CLIP]: Zeroshot Pipeline

[dsikka]

Summary

NOTE: Pipeline has been tested with MLR's Zershot models. However, these are not yet available on sparsezoo.

• This PR is based on the Pipeline refactor and uses the new BasePipeline class
• This PR introduces 3 new pipelines: CLIPVisualPipeline, CLIPTextPipeline and CLIPZeroShotPipeline to run zero-shot classification.
• The text pipeline uses a tokenizer provided by OpenCLIP

Testing

• Mock engine tests were written to test all 3 pipelines and can be found in test_clip.py

Tested with MLR's models (on the following images):

basilica.jpg

buddy.jpeg

import numpy as np

from deepsparse import BasePipeline
from deepsparse.clip import CLIPTextInput, CLIPVisualInput, CLIPZeroShotInput

possible_classes = ["ice cream", "an elephant", "a dog", "a building", "a church"]
images = ["buddy.jpeg", “basilica.jpg”]

model_path_text = "zeroshot_research/text/model.onnx"
model_path_visual = "zeroshot_research/visual/model.onnx"

kwargs = {
    "visual_model_path": model_path_visual,
    "text_model_path": model_path_text,
}
pipeline = BasePipeline.create(task="clip_zeroshot", **kwargs)

pipeline_input = CLIPZeroShotInput(
    image=CLIPVisualInput(images=images),
    text=CLIPTextInput(text=possible_classes),
)

output = pipeline(pipeline_input).text_scores
for i in range(len(output)):
    prediction = possible_classes[np.argmax(output[i])]
    print(f"Image {images[i]} is a picture of {prediction}")

Produces the output:

DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.6.0.20230727 COMMUNITY | (3cb4a3e5) (optimized) (system=avx2, binary=avx2)

Image basilica.jpg is a picture of a church
Image buddy.jpeg is a picture of a dog

[bfineran]

looks great pending comments and getting a test model / files up

[dsikka]

@bfineran Rebased and updated as per PR comments.

[dsikka]

FYI This has been tested/verified with models by MLR

[dsikka]

Updated with a README.md showing examples