From fa736312937bfa55fbeaeebb26396941606e12ca Mon Sep 17 00:00:00 2001 From: Robert Shaw <114415538+rsnm2@users.noreply.github.com> Date: Thu, 31 Aug 2023 21:19:32 -0400 Subject: [PATCH] Add custom ONNX guides for use-cases. Add bucketing and scheduler guides (#960) * added user guide * Delete qa_server_config.yaml * removed gatsby headers * update benchmarking * Update benchmarking.md * Update and rename benchmarking.md to deepsparse-benchmarking.md * Update deepsparse-pipelines.md * Update deepsparse-server.md * Update scheduler.md * Update user-guide/scheduler.md Co-authored-by: Michael Goin * Update user-guide/scheduler.md Co-authored-by: Michael Goin * Update user-guide/scheduler.md Co-authored-by: Michael Goin * Update user-guide/deepsparse-pipelines.md Co-authored-by: Michael Goin * Update user-guide/deepsparse-pipelines.md Co-authored-by: Michael Goin * Update user-guide/deepsparse-pipelines.md Co-authored-by: Michael Goin * added README * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * added sentiment-analysis * Update sentiment-analysis.md * added installation * Update installation.md * Update installation.md * Update README.md * Update deepsparse-pipelines.md * Update deepsparse-pipelines.md * add text classification doc * add text classification doc * add text classification doc * Use Engine * add question answering document * add token classification document * update benchmarks * add transformers extraction embedding doc * add general embedding doc * add image classification doc * add image classification doc * add yolo document * add YOLACT doc * update yolov5 doc * update yolov5 doc * Update yolov5-object-detection.md * Update image-classification.md * Update image-segmentation-yolact.md * Apply suggestions from code review Co-authored-by: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> * RS Edits to CV * updated embedding extraction example * updated sentiment analysis and text classification examples * added zero shot text classification * RS edited token classification * updated question answering example * updated embedding extraction case * updated directory structure * updated dir structure * updated dir structure * Update image-classification.md * Update image-classification.md * Update image-classification.md * Update object-detection-yolov5.md * Update object-detection-yolov5.md * Update object-detection-yolov5.md * Update image-segmentation-yolact.md * Update image-segmentation-yolact.md * Update embedding-extraction.md * Update sentiment-analysis.md * Update question-answering.md * Update text-classification.md * Update embedding-extraction.md * Create README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update deepsparse-pipelines.md * Update deepsparse-pipelines.md * Update deepsparse-pipelines.md * Update deepsparse-server.md * Update deepsparse-pipelines.md * Update deepsparse-server.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update image-segmentation-yolact.md * Update image-classification.md * Update object-detection-yolov5.md * Update question-answering.md * Update sentiment-analysis.md * Update text-classification.md * Update token-classification.md * Update transformers-embedding-extraction.md * Update zero-shot-text-classification.md * Update question-answering.md * Update question-answering.md * Update question-answering.md * Update sentiment-analysis.md * Update sentiment-analysis.md * Update sentiment-analysis.md * Update text-classification.md * Update text-classification.md * Update text-classification.md * Update text-classification.md * Update token-classification.md * Update token-classification.md * Update zero-shot-text-classification.md * Update zero-shot-text-classification.md * Update zero-shot-text-classification.md * Update transformers-embedding-extraction.md * Update embedding-extraction.md * Update image-classification.md * Update image-segmentation-yolact.md * Update object-detection-yolov5.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Add files via upload * Update README.md * Update README.md * Update README.md * added copyrights * Update README.md Co-authored-by: Michael Goin * Update README.md Co-authored-by: Michael Goin * Update README.md Co-authored-by: Michael Goin * Update README.md Co-authored-by: Michael Goin * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * How to use the scheduler across engine, pipeline, server * How to use the scheduler across engine, pipeline, server * Using custom ONNX file with YOLOv5 * YOLACT ONNX docs * RestNet ONNX docs * ONNX embedding extraction * custom ONNX question answering * custom ONNX sentiment analysis * update sentiment and QA docs * update sentiment and QA docs * text classification ONNX * token classification ONNX * transformer embedding extraction ONNX * zero shot text classification ONNX * Add copy right * bucketing docs * update bucketing * Download models * move ONNX docs * update model download section * Update qa docs * scheduler update * Fix merge with main --------- Co-authored-by: Michael Goin Co-authored-by: Derrick Mwiti Co-authored-by: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> --- docs/use-cases/cv/embedding-extraction.md | 28 +++ docs/use-cases/cv/image-classification.md | 25 +++ .../use-cases/cv/image-segmentation-yolact.md | 26 +++ docs/use-cases/cv/object-detection-yolov5.md | 36 ++++ docs/use-cases/general/bucketing.md | 135 ++++++++++++++ docs/use-cases/general/images/wnut.png | Bin 0 -> 63297 bytes docs/use-cases/general/scheduler.md | 170 ++++++++++++++++++ docs/use-cases/nlp/question-answering.md | 36 ++++ docs/use-cases/nlp/sentiment-analysis.md | 34 +++- docs/use-cases/nlp/text-classification.md | 35 ++++ docs/use-cases/nlp/token-classification.md | 29 +++ .../nlp/transformers-embedding-extraction.md | 34 ++++ .../nlp/zero-shot-text-classification.md | 33 ++++ 13 files changed, 620 insertions(+), 1 deletion(-) create mode 100644 docs/use-cases/general/bucketing.md create mode 100644 docs/use-cases/general/images/wnut.png create mode 100644 docs/use-cases/general/scheduler.md diff --git a/docs/use-cases/cv/embedding-extraction.md b/docs/use-cases/cv/embedding-extraction.md index 1cb8177fb7..9b1501896c 100644 --- a/docs/use-cases/cv/embedding-extraction.md +++ b/docs/use-cases/cv/embedding-extraction.md @@ -106,3 +106,31 @@ print(len(result["embeddings"][0][0])) ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. + +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to define custom ONNX files for embedding extraction. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download the [ResNet-50 - ImageNet](https://sparsezoo.neuralmagic.com/models/cv%2Fclassification%2Fresnet_v1-50%2Fpytorch%2Fsparseml%2Fimagenet%2Fpruned95_uniform_quant-none) ONNX model for demonstration: + +```bash +sparsezoo.download zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_uniform_quant-none --save-dir ./embedding-extraction +``` +Use the ResNet-50 ONNX model for embedding extraction: +```python +from deepsparse import Pipeline + +# this step removes the projection head before compiling the model +rn50_embedding_pipeline = Pipeline.create( + task="embedding-extraction", + base_task="image-classification", # tells the pipeline to expect images and normalize input with ImageNet means/stds + model_path="embedding-extraction/model.onnx", + emb_extraction_layer=-3, # extracts last layer before projection head and softmax +) + +# this step runs pre-processing, inference and returns an embedding +embedding = rn50_embedding_pipeline(images="lion.jpeg") +print(len(embedding.embeddings[0][0])) +# 2048 +``` diff --git a/docs/use-cases/cv/image-classification.md b/docs/use-cases/cv/image-classification.md index afdf2b337c..33c1a7dd5e 100644 --- a/docs/use-cases/cv/image-classification.md +++ b/docs/use-cases/cv/image-classification.md @@ -259,6 +259,31 @@ resp = requests.post(url=url, files=files) print(resp.text) # {"labels":[291,260,244],"scores":[24.185693740844727,18.982254028320312,16.390701293945312]} ``` + ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to define custom ONNX files when deploying a model. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download the [ResNet-50 - ImageNet](https://sparsezoo.neuralmagic.com/models/cv%2Fclassification%2Fresnet_v1-50%2Fpytorch%2Fsparseml%2Fimagenet%2Fpruned95_uniform_quant-none) ONNX model for demonstration: +```bash +sparsezoo.download zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_uniform_quant-none --save-dir ./image_classification +``` +Use the ResNet-50 ONNX model for inference: +```python +from deepsparse import Pipeline + +# download onnx from sparsezoo and compile with batch size 1 +pipeline = Pipeline.create( + task="image_classification", + model_path="image_classification/model.onnx", # sparsezoo stub or path to local ONNX +) + +# run inference on image file +prediction = pipeline(images=["lion.jpeg"]) +print(prediction.labels) +# [291] +``` diff --git a/docs/use-cases/cv/image-segmentation-yolact.md b/docs/use-cases/cv/image-segmentation-yolact.md index 81b0f0f911..f37cee9e0d 100644 --- a/docs/use-cases/cv/image-segmentation-yolact.md +++ b/docs/use-cases/cv/image-segmentation-yolact.md @@ -224,6 +224,32 @@ resp = requests.post(url=url, files=files) annotations = json.loads(resp.text) # dictionary of annotation results boxes, classes, masks, scores = annotations["boxes"], annotations["classes"], annotations["masks"], annotations["scores"] ``` + ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. + +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to define custom ONNX files when deploying a model. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download on the [YOLCAT](https://sparsezoo.neuralmagic.com/models/cv%2Fsegmentation%2Fyolact-darknet53%2Fpytorch%2Fdbolya%2Fcoco%2Fpruned82_quant-none) ONNX model for demonstration: +```bash +sparsezoo.download zoo:cv/segmentation/yolact-darknet53/pytorch/dbolya/coco/pruned82_quant-none --save-dir ./yolact +``` +Use the YOLACT ONNX model for inference: +```python +from deepsparse.pipeline import Pipeline + +yolact_pipeline = Pipeline.create( + task="yolact", + model_path="yolact/model.onnx", +) + +images = ["thailand.jpeg"] +predictions = yolact_pipeline(images=images) +# predictions has attributes `boxes`, `classes`, `masks` and `scores` +predictions.classes[0] +# [20,20, .......0, 0,24] +``` diff --git a/docs/use-cases/cv/object-detection-yolov5.md b/docs/use-cases/cv/object-detection-yolov5.md index 95af3ec6ca..12f0a1bde7 100644 --- a/docs/use-cases/cv/object-detection-yolov5.md +++ b/docs/use-cases/cv/object-detection-yolov5.md @@ -285,3 +285,39 @@ print(labels) ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring a Server. +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to define custom ONNX files when deploying a model. + +The first step is to obtain the YOLOv5 ONNX model. This could be a YOLOv5 model you have trained and converted to ONNX. +In this case, let's demonstrate by converting a YOLOv5 model to ONNX using the `ultralytics` package: +```python +from ultralytics import YOLO + +# Load a model +model = YOLO("yolov5nu.pt") # load a pretrained model +success = model.export(format="onnx") # export the model to ONNX format +``` +Download a sample image for detection: +```bash +wget -O basilica.jpg https://github.com/raw/neuralmagic/deepsparse/main/src/deepsparse/yolo/sample_images/basilica.jpg + +``` +Next, run the DeepSparse object detection pipeline with the custom ONNX file: + +```python +from deepsparse import Pipeline + +# download onnx from sparsezoo and compile with batch size 1 +yolo_pipeline = Pipeline.create( + task="yolo", + model_path="yolov5nu.onnx", # sparsezoo stub or path to local ONNX +) +images = ["basilica.jpg"] + +# run inference on image file +pipeline_outputs = yolo_pipeline(images=images) +print(pipeline_outputs.boxes) +print(pipeline_outputs.labels) +# [[[-0.8809833526611328, 5.1244752407073975, 27.885415077209473, 57.20366072654724], [-9.014896631240845, -2.4366320967674255, 21.488688468933105, 37.2245477437973], [14.241515636444092, 11.096746131777763, 30.164274215698242, 22.02291651070118], [7.107024908065796, 5.017698150128126, 15.09239387512207, 10.45704211294651]]] +# [['8367.0', '1274.0', '8192.0', '6344.0']] +``` diff --git a/docs/use-cases/general/bucketing.md b/docs/use-cases/general/bucketing.md new file mode 100644 index 0000000000..188f31a49c --- /dev/null +++ b/docs/use-cases/general/bucketing.md @@ -0,0 +1,135 @@ + +# How to Use Bucketing With DeepSparse +DeepSparse supports bucketing to lower latency and increase the throughput of deep learning pipelines. Bucketing sequences of different sizes increases inference speed. + +Input lengths in NLP problems can vary. We usually select a maximum length where sentences longer that the maximum length are truncated and shorter ones are padded to reach the maximum length. This solution can be inefficient for real-world applications leading to more memory utilization. + +Bucketing is a solution that places sequences of varying lengths in different buckets. It is more efficient because it reduces the amount of padding required. + +In this document, we will explore how to use bucketing with DeepSparse. + +## How Bucketing Works in DeepSparse +DeepSparse handles bucketing natively to reduce the time you would otherwise spend building this preprocessing pipeline. Bucketing with DeepSparse leads to a performance boost compared to a pipeline without bucketing. When buckets are provided, DeepSparse will create different models for the provided input sizes. + +For example, if your input data length ranges from 157 to 4063, with 700 being the median and you are using a model like BERT, whose maximum token length is 512, you can use these input shapes [256,320,384,448, 512]. This means that all tokens shorter than 256 will be padded to 256, while any tokens longer than 512 will be truncated to 512. Tokens longer than 256 will be padded to 320, and so on. + +At inference, each input is sent to the corresponding bucketed model. In this case, you’d have 5 models because you have defined 5 buckets. Bucketing reduces the amount of compute because you are no longer padding all the sequences to the maximum length in the dataset. You can decide on the bucket sizes by examining the distribution of the dataset and experimenting with different sizes. The best choice is the one that covers all the inputs in the range of the dataset. + +## Bucketing NLP Models with DeepSparse +DeepSparse makes it easy to set up bucketing. You pass the desired bucket sizes, and DeepSparse will automatically set up the buckets. You can determine the optimal size of the buckets by analyzing the lengths of the input data and selecting buckets where most of the data lies. + +For example, here's the distribution of the [wnut_17](https://huggingface.co/datasets/wnut_17) dataset: +![image](images/wnut.png) +Visualizing the data distribution enables you to choose the best bucket sizes to use. + +Define a token classification pipeline that uses no buckets, later you will compare it performance with one that uses buckets. The `deployment` folder contains the model configuration files for a token classification model obtained by: +```bash +sparsezoo.download zoo:nlp/token_classification/bert-large/pytorch/huggingface/conll2003/base-none --save-dir ./dense-model +``` +The folder contains: +- `config.json` +- `model.onnx` +- `tokenizer.json` + +```python +from deepsparse import Pipeline +import deepsparse.transformers +from datasets import load_dataset +from transformers import AutoTokenizer +from tqdm import tqdm +import time + +def run(model_path, batch_size, buckets): + ### SETUP DATASETS - in this case, we download WNUT_17 + print("Setting up the dataset:") + + INPUT_COL = "sentences" + dataset = load_dataset("wnut_17", split="train") + sentences = [] + for sentence in dataset["tokens"]: + string = "" + for elt in sentence: + string += elt + string += " " + sentences.append(string) + dataset = dataset.add_column(INPUT_COL, sentences) + + ### TOKENIZE DATASET - (used to comptue buckets) + tokenizer = AutoTokenizer.from_pretrained(model_path) + def pre_process_fn(examples): + return tokenizer(examples[INPUT_COL], add_special_tokens=True, return_tensors="np",padding=False,truncation=False) + + dataset = dataset.map(pre_process_fn, batched=True) + dataset = dataset.add_column("num_tokens", list(map(len, dataset["input_ids"]))) + dataset = dataset.sort("num_tokens") + max_token_len = dataset[-1]["num_tokens"] + + ### SPLIT DATA INTO BATCHES + num_pad_items = batch_size - (dataset.num_rows % batch_size) + inputs = ([""] * num_pad_items) + dataset[INPUT_COL] + batches = [] + for b_index_start in range(0, len(inputs), batch_size): + batches.append(inputs[b_index_start:b_index_start+batch_size]) + + ### RUN THROUPUT TESTING + print("\nCompiling models:") + + # compile model with buckets + buckets.append(max_token_len) + ds_pipeline = Pipeline.create( + "token_classification", + model_path=model_path, + batch_size=batch_size, + sequence_length=buckets, + ) + + print("\nRunning test:") + + # run inferences on the dataset + start = time.perf_counter() + + predictions = [] + for batch in tqdm(batches): + predictions.append(ds_pipeline(batch)) + + # flatten and remove padded predictions + predictions = [pred for sublist in predictions for pred in sublist.predictions] + predictions = predictions[num_pad_items:] + end = time.perf_counter() + + # compute throughput + total_time_executing = (end - start) * 1000.0 + items_per_sec = len(predictions) / total_time_executing + + print(f"Items Per Second: {items_per_sec}") + print(f"Program took: {total_time_executing} ms") + return predictions + +predictions = run("token_classification", 64, []) +# Items Per Second: 0.0060998544593741395 +# Program took: 556406.7179970443 ms +``` + +Run the same script with varying input lengths: +```python +batch_size = 64 +buckets = [15,35,55,75] +predictions = run("token_classification", batch_size, buckets) +# Items Per Second: 0.01046572543802951 +# Program took: 324296.67872493155 ms +``` +The pipeline using buckets achieves 1.7 more items per second compared to the one without. diff --git a/docs/use-cases/general/images/wnut.png b/docs/use-cases/general/images/wnut.png new file mode 100644 index 0000000000000000000000000000000000000000..9602a8d3b15ffce2940597b710960a6287c2c719 GIT binary patch literal 63297 zcmeFabySpZ*ES3YC<6$>(9+TkDu{Flf{LJ|gp{P@&<#>UiUKMrAt+sfw6r2!A|;?Q zfOPkJUJQDB|L$i!&-cgsz3W@wD{H}A%v^ECKKI_|K90R7^!6b(8otX2zp#EJZ&5>_-=**adL?k+j-(?Rc`a>Vxu;?9Ni$>_%p)KfQ`l({jUb zs#aNX5M!FPF15Cc2TPJ>hmJo+-e!yP>Mk1)q~ zeTQ8R`?l@vv1eNf{crLplTdRd2<47+Bwn8s%*C}YU>6sJ8S#Wjp4oBm2~kywyLeH0 zGKzY2b-3_>6onC+gfRbkBiWDu`FD@cDM^USQhxHI%qK>?DfA^JDxzz`$KH{)S4xS9 zxnA^9%Um|-QHR>9**g+9UG3bx-bCy;wIioE-}L_0{;=YyCk(HotX0`>gg?@vHEP{Y ztgy@MLZxv#e@l&$qvmR#z?#zXW4|EvGK%bhqQ7zMp{d(Wz*_U=2Y6Q=D^(XTa7o-% zrO%CMjh!Ky@z)~>(ZUN5_1`Z#7vMOFj}-uSY)!_K!?RaPBAW8|)oRvCWa^C2*qwgh zKT1qxi(6#nZJ6l5ovejHks(9hMNA>r%C*`QR{YMe2kE(-$$(WT&2S%s!+Q2UNgHlM z3 zgzh8(A%0D_jre=Wh3mcSZN;$Y4h$Vzo>$C#6|3DNO(qt_Bf=Xxqxa|@X<&0qO7Zzq zm8sr^#pJV+gIMMMFKk>f7EcYl6wD8!!8koi)?zs6&0kDoGW@aHRNZht%C0*p^N|fh<-)`}Hy@0zeS-Y_ zcH37e2C?e(iMF@tcRPpqC$nV9d`p5|)vmwvZ!|rP<*D5L^~o!uaK9^serxu)i}f_y z0Sv^rX;NZ}xTN*4&TKH+4hp6ABldaqN6X^lJbp0|+bqCfLjY!O0UU*)gV z6{j>+gk3Rr>#r_TNfVRE$fH6ejQq8jHbaCOW}BVWasmDXoYxlN!hTc*m~52FngR3)d5Y)PN)xq%SWc@&o# z)-zyESV{U&N+p$|+sd3`7i*yv;}vq&=+(1RElr=b;Bo$@_ereDHGJpbOH9cnMhe-TuJD&#!A=ZTX)&7&TQ1vnuEw-&Eci;aEU{1Hr^J8ZGimji^9@=u ztcI|cPj6rUtR$i+f|HVI({PpPakg?Pt9!kf!1HG}6L0Ct=_-ug6ub5Ff&|m%6U7T& zG2$PB;qp^4FvZM_OBqQSo49 zI1@ex_;Z!yYx*+dGB&IR7hIl5-nqyz2&Y#PV@VKaz*}~vRnF_H29lw4rS(+tQp%zQui)$&tscCSc= zy==O{ll&`6LP|m^j*oww?Qne2RjR0?p(Cy1nMab>p~F^2*CSAQCF$l&&rDBww@Ob{ zS?HI+FXtDm7aSJ!zR-U064epS7Y!A?Z4#=SFX4<8)%?q8{q>C)a9PDq9X)hFiG8qo}r? zb>H;7`G}S4hR%&-Ap|A&HD2RO=@C6Tzb=6Zf?5v1oUcTb8;%$TRCG;w{cEskM?V zTPb;NeaF~wAkMtKw4+GR%;Uq^GRKnk;+*0t^QbRXBlrtL!!D5w;mhY4sa1uhf&yNg zYJb`NvUAceJ>U3|aglNQbo*m+<~!GTY1|%s6J`_6(#_(z<#Np;0!E`Q_QtK>%IvCH zvSaL;I>j61!glL;)_BeQ&3vq(jeOxTl#TJ|kE*gtFQ+ZL_==&bEmtkq47a-;qxUU^ z(w~PtGZV*4#!M~|k8<^1ci7@w{xY)>UmoBRxpj8QEIm&F|C%k;L0jvm;Q5jdS?%$Y z2|ucO%G_!0@uC<)8$}p-F1Ype_w_)xM;!!K_?q}p_y#nG z{0+P^Bt2rdw4|3G>^!S_9QQb_t%aSqIaG(viSLW3yTke|KiI_kK>-b&`Q{r;NXAF^xEoYTai$!zT#y?a#rEYXf={M zZ(prP;Zv78=(@bQXK69Fu`)QF6)7qF{=UrOqjXXW(s!g=S?79VdYv=8Ca>9BRUFJF zuFtJY&?S^3xBQp74BjuD+djjIts>+p$SIg?x@*GQ_4C7O`pB|Vd3LhidIy9%50fnB4#m{ajU4N;_m3BVXM`TAG~7hJ`8$Z@bUD% z<9B)gv2FYpMW4qK?CU(aocsJN_8$*lzqliRXC!O*<~@-G3st)w+rE|A>e)P@8)l!9 zR_?P}{dlF_mbGUel&Eb zyuDy^WsPR7{7WA0z4mhNG1f1GW#Y~r_wp1TE}U(oRi5h>wb1M8>`#h!##yv{;1Ph-wuIV;I4G48>%)|5UNn&)yWQrxYK=T+YH-R;*U zSph03VvVAG4udm4A3N&LoJ}%Iwypl^{qp5l-oe);n>zZRc1z|JDEp$>n3>EUojpmd zynA{Q_AJXEXLAyBDDvg?=c|+L4t_-R)94BB2&asFtl5}zo?Pg1`7z2e%2SnPU$9y` zdGJEg`snl6_i?9n>8}Bs!S+N;VpHPoD(UL&T^3o4n43D?<3l#lXnOMX(aTvx)Q+au1 zAKDeLPH27%KeL5NbH$)c6!2U`lHsn+!e zH*aEag5xt7*qGE9IN%5q{7PWb{64;h$&PXA_>o`gQPn$JoKt)W*@=7NN@c;tY5L&rVU> z5d(vq1^UIju6AJ=+<(a8t`0u zIrvYU$sB>O6XoM`adF{w5#qIVFyj-ra^(sizaXEWAP+c$$I;COVd%I053n^?|Xww z#h`aZZ(F#UT5HQ$JOpC~o*^M3cv0;5{QvmmUr+qwO09ogDImaq`Oi!L_~@UPsymuG z$l5*x&qPT4Yrak{{`13=3&r@LuKzI<=y5)N7YwunOpNdMq)EW&mNFH=Jf63ZQ@IOX zfs{c%n2F#IJNgwmww>YD+GfVUkixhwCw|T2EII={rn1+ z#$Kv+j_lms+x7Skj}yM43Q{i3ToFI7LXqIO>5{Z0F@JEO)p2-F+aqttMe@S@^cRn< zy{}sZ_4t_B#8Mdl{wG!MD>MC$;$9c64@x1r_i;eVxqqI5;YWN%O$7fxF2(ReVqtTT zz5Go0@3(}Wz@RvJ{y*-6iOmTk&ba%?Nccb7;mZ){_>ZPR|7NpGMXQ}dj{WDuG5z74 z|M^_0)HDofC3$vd+5fy5=)~#&ywCr!PQ2+H8}~UnSgt$TtD6iUk~nj)yAft!X&|{K z^Ut>WCF7ab=Ec{R{}?~m+agk+b-LR{eRxoXr{;dc)r2tH_cRe5fydTjgyF|}zf%Fo z*A&tpA8Qv2W1V5}#rWfo9$u+GoX4p9yzbiKOW9!Dj`tp-Tdaz+N~0)SozT^BkEKB2 zaaZQ*$Kn-$+;lp6$E&yz+k{8; zQua7JTso4#<&X^`FqBuia|?fr{`;8A1R;^+zTNhW&*6+~UK~l$&e}|{bO3JieCZ28 z^4-usX9f(aucqJNzQ=5D<_eN{EMcAgaFV{k9|zxE?C62KQp=li*lT5v+MN#Dmibbh zP)cc~78yJCe_uC#5X_4Yr6^GpSKVWJOdh6#7g-v<00 zf&Z&UpcC79=_=)0>j@g?B}r78S>tIUf@uXJ!p2|%L=U~F-^o(@j;GaGhX>TF${$Pc z>y7(kKavbrK1LXh#U=5&Uv}DWv-_TS-Ei*FW8Pg^wi9)mDh8HkxkdEqV%1WR%eY(C zfz?E<*JRDsIQ~-Ik$0}q(cwX3j{-ey&{1rdS-u+IU_CJ)kU+-IS6mpKa!M>H2B8ZTNy#GO+_rcjK$_O!9&84xo z5AXf#Iy~5$o`@8k%l>?t*5g~-W5ab+kKyvy_w<(2(YRGRVWx~#!z|-LP(Qm*V<&%E z*4Qo&P9{>x^J#B3e~==|>bbZ&Yq4PzsG7*iz01kl}FC8o>17Y1W?$8MzH=#l*Ho9S!FP^8`|RYK>dM@$S63 zfmiaN12&WFVe^dNP=-KF^t2%D$j{EMxWJ#cd|ZhjEhses9H3bmClmOI&byuB`+a}p zqN1u@vE{&#UTLA3 z1l=K2j^oDTt6rATy_t4-y?2f@P#`POL8;=aKYjz1S>idL!8b93%7)?o6g8R$uB#u| z8e1Z0j3u#iB(e$fu05CVa)&IT5x?WlqSlt%uLf?>PVnGsD;$}hf;_0y=fI8Y$fc>d zcH)OzJ&~Lr3AQc7-^zyc!?;x8mMB3wy~^|yr`M4(gkBZ{A2O~#X<(CB3FSBIOsr6o zU-H>sX>I%%%hd?XFRg6UaEV!~Iy>xssh-b7nU$uldYU?qrk<0!$U+mn^^{QGW7JaXl`|qeU-~X|{)H;JCf2 zpuG{C6iM>v_WBV(maLCK9y3@5<~3w07=TNNj9)-i>FaQ;PKEIWu_e1QaqCx$j8A{7 zoX^K!`_U0*7>X5FwN!f*mjNbLYWW?>l^-G>Ce?BSpjrdpwZsD9!OC%4nIkB=*1<=yI8SpgVG*L^c%)~}-rijk=h`L(-qu63kie5Sx(oCIPUg&C|7YMzO#MG%qd#grJgM!I&v^K{-V z)$UDYion$1O^|PWS+PA8*7!OrnK}SImZ~vA@A}nmySBi4BS*lzr_T>QhsxY5EZwX+I^3J~7M}2Q zd?xAR9WPuUK3bs@4|f_@*$Prkp?CZC*ls?ryv6rN+&vpSd~)uP948&jJ2P7=SJBPo z!1RCm^z6tlbtva2L6K4Fj~mEPIeJaH75HdAE$Xb5r3zc3Jxau5<=^%uj=Zb0KE~JW zf2YhSrBlfzm5yIz5AU%g!Qq9;@mBpTu9%YBQd6~iIlxBV&X;+Q}tcnxc_`O6IJr3tjYa=#i&rX7~;|WJ;jA{`uR7 zCv3rdo|E`1*NSOXDDy~EQ|P?6Czls&eXizH@9fV=(rY3Sp64A~w-6g8i{-@+N8Y+F zQFsk4(Hiia78ge)dn|@>8JtlQg{=r3YdIHe;IOB+c>Ee-KKzRkiailV<+c-aw$ytD zM;itL7cBx|l`6>OG$F?ypDIY?<2O+DO+cT9#qaNbW0MS3`L^?AsFpn7Y5t;iTGlR# zjti#z>ShA@Qo6%zW_%07_xGpi!=vsGl6oU1Vh54&k!;}_$m}SX8sbBQOl+6;W{H5} ztSg5q)abRrynEA})Oi^_@!m_@mU&G*eMkhU{@8jPLuNX$Bs1jTA{2ZXGOt=zpFY_; z?iNc;2f*__-HwJdbE-!)Xr9i?G-7#!N`YLBm_(1ocLLZwZ=OJ=2AAWD!ARZVMnS7b z3PRmn?`^uW?`7-H#F{19dF_04Vz?ZVd&6BuzUj*{QSyhM%p^!pg-0%w@}5(RUX8Hz zWR{5A=}S385U9ax*-@`onG|2AHAx(cfD=A(Mn8{|$BDVH5M%3A>vZd52}oXWZ@--$w5m-mx6myrQmtC~3A z9d#PgTVXAAuP#c?Pv1*K#iVe8J~KzwttBn3?@mYEG48+IEy3KlMgZ5+r|qn9hZih5_*%}=U%ToD6tyM4UH!h zU#>s3@~X&S=c8uB{?6L;6F*!6N@9g{kGDWuLF(5y)V6l#N5w=mqpXZoUGmwn;%M;& znx*6Rn&*SQJ6b7p{hF9-V0-nD8fY4iTxA`&BFaJx#4j-+a~mSo9UO~X{g7oQIdzQ+p{V@)KT^a z8$-cvYd@OV4p0>ZwY!#{yHOg|CeYfwig}{N@Zkq!B z6t12QqLEOz$#`>VLXs{gvR8+^IP$Ou_js)f0sHkrTi89~#5HwB+g#K3I3DRJR^TYQ zh|Fd@`Rx)m!pE#P^UBAO(jl8}sF%V1GGSZ=)q=RRKXR&Js&qR#t=qIaRa!u#hcotjdwlTK#R_J2{vPJ&F46hi zimnR#>8S|Y4Ur$G={@=13OxAq%vHOmArOE1BPGsdry6d2ZB2TM<7J-e201u6RN}uJ zoO@^or`Y>ocUkgiP0~IyROab9=4(yLmXOa{K_bVcRB9v_B|2yvts*k{$Zlgmvo)5D zl=VhPl-!`@lUOdzpn)9l7GsH7SEO*-z(;bunDl7Vp?p1oj21ZBwydbONO6^%G=6K$rTB z?VSzc8QBUN+as#1r<8ELPbukcPAOrM8g}S9k1dTXM|RVnL;_F`Pkc2nv5!)Y_GbjJ zhl`9AJY}Mi`_p}8r9(7gQ4aJtG*4J>A~8e!V?s&UTG|nC2h?h^S&G-?@ze4PH6YSr z@1&GNC@!eYrOpeSO1l;+vnHB>xuCMV_{~1!xS_mzAXY-grP0)V?IFGf*Db_wPupYW z=@klYEn+b4=k_w{hIWd%O_B~WctFGcZGXc5Dw3WmKT(L|MiPIoyic{0KST39dDO9= z3}k8jE!s2gQ6d`ikdqGKU31%8D4))}Sg9F`1=stL-YuJS`%D^5lO@YXD3a4b=5M~N zQt+bAp z%5uynGbf>u6VVf4wlS{ZgJ9O(kUzLhkFYLF9}v*_uI3VwCUB2jZQ(fhH%D^cNI%{r z**RO2&MeaZWQld01M!grT#gRL&b~YCGC!y=6{Ioz)Q(%*`=704RV*qn*gMCKKkQBH z4aHt9iNRe@8MUxxYhJVd9lnVmMn_%5aDaPiPkA1Oi46lMj^{GnDR?jtHqL|&1Nqi~ zc}ZYsuJv9}DC9QTlitPz3OrldH`I+VgeoCbpQ`A}uZYh@8FZv*1rE+^F@(ZY>{8&w z66B4V7wUZOVJ98hJdQZQ_jb$Z-xo(OoSr-l?LKfZS$|w&cwp`CZrb$YSCFXt43KLm zxOAt!IGQT>JqRMEVt^AnI0ZK>HEK{iCl=&0{_zg^=lqa2cDu}5`bYH80-^?EH01lN z2yZU-T*9xwmJ~DOzJhp^3TC^^gY!^8%n$2$Fx&nP=CQy3cQF5NkW7!9p?*B(##l#I z9Ivfq>Oag&@FIxCpq=?P2B7~~6+hgrA7lg?MHF`exGjF7*wr^bnf(!+aUv+_Dx1vH zhjEMHP~2(VgE{Yx;EFwH6t3<0 z1`h>;_av+v4A2ip&7Xv~&7yuq7NszO8!*e{v37kcE)Sa{@f2FuKZX28&{bCn?O$+jm6PqRrNOD6Dbx< zTWP=Y&_dx9C3zlKHrpAM_nG%m=xaU18{{g!E>R1%0A5k>m5T|G z@G<|s>Pp%ac|}S2T2?&?zX@7ki_u~t7}xiQxv-Uf|A(aOl)IT}RXZ4WyXYSRQQ02~ zp+noc+vVs%ShQsMQo?OARcH-CC6QY7_o4+o+v@qf*Sk}f==6Wd>~3d!9fmaWh`L7> z{p>k3j?u4|C+86tc;kHBO)ygR=kQ(e)goOnsxke}0=c8*-#&M|i}I8CWrmpkSQcm0 zVC)XkvKJINvOg1$p-;SgVgma<{n01?r&vhcZelz&80TeK^oXV2JTbo)LF<2;UxrLy zCW?!4{!jgp9khQP^tXYi+`;PWGeq((u-izzqhqSbk${%pvqWT6F<`vlkbq81WOFV0 zn)@;UYdyMo#*x6s^;84_Vc84&FW&=~hLD#|?>Mw6m70K#27U*DV4@t&g0m1->H0lZ z)8AXO&;i8uC;?@S@~X4-Gf^i?5rC*Ra%3fv$P(Z%(Hij8ZX0bc9UU$eg82LAAph8W zeE_rRKD+!-M`S*?q(MP)w};<;s!7(l&z|hxU|E6G;VKu~pHU6*9Cuzl0)(RG!yC`| z8F4vaUURP6e|Vb=iWb+GYxvF(XRHTu3gV)DoX8vK_$zy%D~L}vKw(Es-=AN~xPFe5 z^O;Kl-c{6OphhZtp~%`NA!AuS4BsJ!mu=mFGl3I)QEcbi6T|722>aOZ-E=Yz9lx{zcFXA)WnMf8 zeQu@m_!+G679sZI@xHZo8}sbxJ;*T)CS_2(l1My~a6u}%H^Z_=^TgXci3)W|>{)LE z;?jA|^l%zsUi0P51@S5hoq=MaQ{MOlf8FmQvrN?9dTnrIiBF*mr@R~js80V z->w|(uQiUobDaM8bY;4g-XU@E&`7&&aWixaLN@jAh%*qG4HGXoL7V$O_SmA^TzL{r z1HmzbBniMxFfyE$NN{MN;Ivkekvst(+~&rMiyWhNjf9gRk{Qkz0?;5$F2x=e8TX^v ztQ0y3e*64DK?8@catT>m+AE)iyOB$}mW_m9+2I~Pi00^*4CsjKccZsFNEj4X?h@{zH=R;+=$LBS-eXiP zd!_J)?w7LDEP^_!y)Y3($q>)Iw~s*BJCJ_=>* z0Z4gGw0oa@9WBx4ZT4YW@K<81ZOn!}Y4Y~!A^uNs@X4F@62d73m@fCg)Ie-oWxHpN zJdlPh386+{0Ph4y&cnV;@w}{sQvWE36R0m!ikR&MfCo|2UXJHRjsYPAn=kdj$ZJFw zU4Rm5J+J{-2fxqZzU*e$)uYEprM%3&Tb+F30*WfUy{@)38-tYVqYmvn_SLkfUUe=_ zy8u~WCLX z+R+&`wKQv448L(85wkpQkyebLp)ZvZP8v!MH8eRbU4dpkhGy)khIF7mJP$s*eU~mz z?j4|nOtA>NuNN)4rA5mcz;2Yjkd4Q*$VA!@=VUFi-%#yNi(edAZ8ta+3}WZ2Syy`| z=H{^7Kv7Fam0?~MB=gi@ ze~Mxys01bgf5cvKo3eN1K~bg4hZq*+DX&|IQ#ydzq3l*Vm+o{#8pz}S0>Cc>ilXD; z8e1A}%kNK5-?ZAuNPT@LD+rJYcoV%ghl3OGbOh4W1py8_`x}Q> zEK1`$7+=X0n75kgOj^N{cK;?Mt)#%{4T2SXz>q%d0jrXU@100285Ehp%<#0XAI67@vKAt#J zADe@N_AR5Y)Dtj(FYfZ}x{CnXKqFoeEqV`L%ho;P4;Po70h*skMPkQ9_ub3;-;C+^ zv~)w$*j5l{xpjmy)!SYwyRB!oyO&nVj==Am%Oe*7f-D!MIN0aBPHCz)p6<({5Z#a| z?zPm|8bZvj4j}0Mk6Rt~q#5fSm4cjTC8>tx#j0q5LkJTGObCO(jrpNgbru~)hXHlX zV@7NK9^9>CV0GsttPu767eSXN6cd*#pvmUvx~C5tp=v(gwjuA4e4 zBd>Mx2a&OZ#!sDQyA2M%8?-RUcexB^N?)oxsoE^GfoP0UV@s|_39jd#PV>QeCiCs)&tX!{4A1#{6G$*<9VTSGDAppK zqL}HG*5LUnQ$`{e>$ebkufPNp-vynY^LoAv%^ra$dUS|s0}B3>-A~WateYsv-d>JD z=5LC9n&|%ouwCaiB{&W%g@LC%n6P&DNXf%(*NG?0dZE3drF5bDyYmID4BtH__Vjhh z(pqWedz0}7zFv!asP&VNqrGrWHsw@DTCfyV=e}sN?jeo6X&=3 zIv(EJD~t5V0K6)<3@<|;NY}AQMaT3S-s^)TOG986je2{gG^ zF9T}Ew7BXzA_-uSj{sv>J`*65&F${%dIm5iXYG2Ry*0 zdb!r2={T|&$h*ncaa(*h<8PysLw2#!pdGNRNXdD1DPO6i&;=wd#x4;p6?1>1k8Dr@ zdK!%I88vqsMSO2C@caG#$!J{|t&FygV!)l71Wu(Y!4>$UaOz8(y3WI<0?z=o++APu?y&Woc|@HU9)reB+0FN=xPcy7)T_NC`Q8RB-Xdk3=XNDBg2 zdXxWJD?rz?wot1~I2>FWxo2pEZNlRT@wDO>vT1QdK>xzFG@lC?Un9_7-xAdaG46Dq zQizmV3WijN#B&KBXmGEW$Z!Y~ghI0?LXo492Lt>?v+gaoAH5w8j8U}|9%mm_eQWmbSkF)0$-6Jn{BB+J*f)jA<`(G;3{IY^4Gp>T3agxHItF+`y!@aDuW)f6lc!g0((0<8Ee}Z-*SfLk(u@cl_#A% zyU@0e;j8A!gBSmJuxkQ}OsbQZ5T_akXI#Qy+><~B=fyD=!LEoteMJrk`LBpHfJO>1 z9D>Z*A#7!HXZ44%D`)pHwNOg>1gl|-`2(vNXLZMk7Lr4Q3tjQT^o@7?ULAY7Fp?7p z`|%%f?B&ZnT)v5n)G$|kShZJe^$uk+?BX$7<#zOOe4k}H)Z|wp!fjvQ@=$%uZv;KJ zA#h}2?R!V-^4(O6V`=vP75%>ONA$b;<~oeQEF|a{u}R^xl?ckhx_LY+@CzsS5!0Xe z(Ze)Nj4QGi;SkcX=jpa4AVy|R4UK;)!%0}5^ur%veOuBRtTw8&m&Xti82YHm*HV4Q zY5?aK2VC+0lSA((!C2@q@xIV>Z@kdsNSb1r>JLB>@Kq8}@1$OmMv(m)2aBJf*IJKf zQTL={abHfx0Y!>6e|d=@Ow)#7{_RhMK=SKpg&?k(8?ZZU;eZuyODaymjz}x|Jp&Qq zJtz^#1MuZK8B(=hnJ8W|zM(fN!^b^)YfQPYEROLhAVMjKOT)+V>??H%eMuR0e=#t6 zRJ^--wN>fXN>0}k0H9~`m zXEJ`Th28-ZF0^+tfEK3#%iHacCd~}E=nn+`hDmSp=8t2DP!S+SXpzmx`Je%bASU)3 z^x(W1NR5M-OvJh8pS5L#Ep)Jr-FwRvP5x^{uWLRzf%ER7@twOl(mWm@{t|GMgbUG=|vs=prK|M|@?Nb>+0PxMgq)Ol%~ z76K#2y|%{eK)U)esD~8{V)22Pr*NUbe4ugU71nS%aZv)A%iIL#MflG<^{djy?EH}| zZ=7Jc1d6gq`)FPT)j{x>uZ5q1_zc!rK1YWul(aOx_7fgUHJ{@_c4Z)lm9k|=?IuJ* znNPu!jlsc>nH5?f>czI~LxlB`qR(6IZ;v+v56-dcfqn8W!SeqQF#rk00FV~X)vpnS zin>h3YU<b*x|5eL%4cv>Q9X`saTPVuc#yN?`MO~kU3O0Dq?B#hO&b2*01NHW8Y z7nde8b78;g%N0NsXatOrV-|)TNGxx?FW-vV?zJbY{~r?Mkq6I@v*Y2r|B7@g zL1Bgb|0%2ha^n8rWRzl=fUGJ*P`W^1zZ-cjkOGh=>Zd^ZbvS0rr{_YA0GIPo^nJNJ z9U(k^2I53$6`Ncqcu@L6j!So4?VgT!8p98NeBLu=iT2m_wR3cJ_yPq$e*YZ%sN1hR zc^a%iB)sh90DE7S3Q@eS1Gqd=eDgEq$I>seA6(jHHZu&WDf~!qK=dVN@qdWE0AvIv z;k`di-#Wk!2p?hbW9-@8NKzackQn$tR}Un3Td=gXN&-oE#BO4tH+7ux4u~YdP)Zq; zk$LC^{k`izG`i?-nIm>JM;LKB2C_H2TKZIvMjRFgh zAZiKEBk%bx3=>62g0|-=JHhqY&pf|qp;j>&MAHyRp-;)T2hjJiAvrFcMUbdi5rsvh zSLxVHrrDG#w{^)4q$oD@W-1%Amw-G(SGP{U3!)Do4!mqB`#zL+=G}DU)dr9-dwwEE zoga1szG|Ks?w`^eM$x)i%J~CS1j56o;0@=xJ6BQNS{yxai)9m}>coNIT%6p{M9mIj zO!*KA0(cNd9SZ7Ms+;nmqh`Hm8j^hcx8KAKW8bv%y!4O*25%m0R8e_4xn$U zUjOu>hX9W&kI zD>PUNtiNJ%jwS=9zX&rBr5wnT9?=Yuk9vI?|M!zpNXQ%+xG%R>A;XPC5dnRyjCQBe z1Bfn>qm*B>`Nj2!B+t%b?3DbLlIkD(%S<7 zl}1ZvZwyqYvad_9t-bOoMN%?IDDU8q-=FBHzcO5L;%Xfnm0dvH1yVYxmrgmh^OQHO zI1@Y*t&0MQA*tRV25NEVzY{H~6p%~zJQ6E9nBH3ip!C$W+s%9}_a`5b-U^97LKV~E z5KSB%ZcRs6fb@I5Rt9-SB*6INdB=J_lI5|geod45R@}}T22v0C$jbKq3#!LTadKqT zb(_5Co)rXM^xHAl>4+8A$|$79K!!d{1P$nLTOR?-!i62gG=p=RMqBmE)2gsQ^(faL6Sk93gh zZ7IOX-h+gTM<{Fw_Z+AIGFTdA#{O!2T{t6>Eb-5 zaq-}^9)aUYSs3XF@#^ZECIGPz>kpB*sOj6KS?PQdVLgTv2C#QcqSFSbVb{G^!VDfk ziJcyRNJWc7QAvcfxtJp*BkZdl4eAW+Ul`Bcv@+P!Ay6xQk-jFbynZ)j$!p1@C=V3s zXx_`#1(j3;6d{I4;~VvyDN=99y5v_aR&FCwB3XdC&Wiz+A(U<5Z#*|kQX-_4T9rdZ zYqO@iAdbkEW{BYpisiU2f|quxc*W-_<_dEnXK(stS68=$ zHh`#!aG_-NusmyfWB$<{-RQJfR*(&Ue|kpp$c=yuG5bl|EQt)Now3*me)b@buBg@ z!WuyA9Ky~3)&P~BK;enHH*`Kxk)YDFLA}00zxTOyN(nx8{wlfm=44oC+W(vvP2pRj?o2^zCJ3NHFyOF_JgH zNU~Ql<}RA1d4!-|U#+zN35ZQJU_hS<7*7$J0D@4Bj&SQZ1mGb z1h>+=3{Gy<@`D+gN;R`|8Bbtpf4G3gd~Y-QnVBJ=&-`#;@0*j!^%%9RVZ@`kgj~%Aaz-I3c_) zH;+_1*a-j?MLvqR7AXH|KtUOF1n7D&?tP@3?~EF+c-+R2`KWi!oBuu}Y_vm4+v5Dv?`qSEj>-8lXx{fC!xSh|X-( zJ-5i)MqFM&_*Kop=7=$aBSax=rJi?4B~=j{2-RUI()&lP0ASMrowuRbw0+7{0ZG4v9$LQL< z$C>(J)icQZDr1PG^Ya9Bo7PqetBp`}&49Nry~?KK8ACVw zx++Z+r6CXZqNVK1&eiYjjkGSK=itxGS;FbcbG?ph^z8IFc8OEac`m>I9JDx;*2>1G zTfu)LMWUva5vlZ0yUj9-#DlTh16s+O|C3m~l1BPRKoz4_VqOr^Tz&*qL~Q4+;>IOq zR*1AV2<A3G-sibDo;P*@nj z(i(a551C|ZdN+)$O0c&*Gv_ZkH5``(gRDUi5glbq6diRC(N0~Kh&zhZ>sYePlhM2c zV9%hMG+VsGpo~NH^IVOW?EDsaqO)3OA%2OfVp`TVJG{rLD_ML6x42SHaZB25GLXXG zMlNSr7BjQ~9>}ZnI*p7E%tJg;1f6(gjoU&w*xsbJhrugzwThUmM=J#0GTzry%24)) ze`WX8F1#-dw8_7-OMEkd$dvMu)|3+6I|mAdq!g4UW8tg3Z|+~I^-n6LL&KHT5O)EH zje3qoAXL`_kUEB-LR9h;dT`CBsMYg%ls(FfA;gH+(OAS|#2R1|;A#R0#~z91EbYLv z1H3<)9%m;J9c)o(_|V`^|m< z?UPp0i4WEP5}bDi(CNA~Rn}MQhVIp@;@im`^9DpAp!b#Z$4FYVce`Hm(Fzm_5ct&1 zA_Y@eN2GufUC2epVo)t$`w<-u6YLt`LjwkLOPNlyAlOcAdgPXl7c9Z_5YyT@avV5$huc9}O``e$4LGwZr$>qwKId**!e-~+tnoizOp zl2mjJ1E(|Skpii^@n1k*zJY@gx7Qs=apvDx@0GBYPh|tD#yu_msAhaF*1(tf7}@P- zJSH=|PM0AY0T(s%7SdqVumU36tF6oSslPQ;wq^-7ItO2h7LAFEa^5W>Y?smaiB-ke zZ3Qi)^}mn#J7|9;?eDtzt8D(NPPDxNL@ol&&gby;u|})kn^iY2t@@F=VERQypJTeD zx&H}WkLvyjT@Rt`%fQm{bQRO~i0(_o)3{^W8#lTta$M|u z|65bWtF%aib9QXdrisLs^z`wr@0?WqEoelXnKMm~<66qcTs5TWchNH7tKJHp(!?Ic z;}=kgZ~EI6uxI)^D}M#-?^^lauEAd2qCBqKhN=o4s_eMhI2B71P_q+m;%R0t}CY?ReJ-?qGe^7uT*W9arTlsMEcT$?|_gjOJxJ?@HGR+T>~#|p*A_5 z5V*Teoe;QF?}2=UF{5dyFpHEuIBt$j#DvJ(tjC)d_>&l?v20M1T7t<(zCH>RcS+Id zCWS0Q2{@ja>xRCsBJ~P-qZ2;kcS2I{<^icS|{a~#-Q9wavEEx3J($_L38APBcKHB=7%z$ zNi&{3&7Hr-gFn1+Zifr&&m5m*AXs9W!L^W(9oNnS-z^|U^ru)N!XkvBPn&^=!ZjUm zp@L4&pnWo)+6|>ar*vS#cSH8cFeQPAcw0)VK#{60aGabYI{^vg5A?Z$$*2E{>o3y& zUvd54C9Y<8Y9M>8rVtPX3eCEb3qhE5`v6c0CP6ZibO4Q@i6T@&f%1~&T}n}eIi-l* zSH=OT0)i_jgWUiHofgNc0>(!Ml$Y+@voss6u!p`^CHUtisG$64Yr;p8?r5QWBGOBB zWG+SDy>48{s42t_DhOY@K0m~V%K0Ol_&YgUY67ZFeYMQw0;D-@A#GiP%>k3t_c_>D z?p5XsG)s14g>vr=K^2Jw>6WHY(ndfZpoR+8@>}#~@H>2KldTrE8KtE2TJM1>JQM;p zMJjPd&Ja}06UyfL-1$A;fR0rq*|W`tr%hAejRm|(dR+A$3Y;F?)&dDw3_@m|1fT@7 z_w71(KE$~QF6|pL&}sGAo(ygTmAE@Tf8r@0B_Rj(NO1|N;k#Q)6JxL3=&=@oN*Ij~ zISa+Mc!4$1ed>1$N)C&l0ICFxH%N75{0N1HHS%?-0Ib6WWmVqJwg=gYfhyjsPXum1 z1I60NYPOdBM*>?sRL1AxL@_ z5K(=W{uauA9O?Z?Sp$6sO$C$uQUa)E{_K*K2BoC8=j|$|YLMP>T!~*9STJS1FOad@ zjRbms;x!91q%7T`q^~3v#U2<*-UYXnW{T-}nWqp}WJuqGM{K=Yc7@~xr}W3;uiiPX z*#A8(j2D@sDN24#I^>Hwar(f8B+a+kM0Egh_5adSMAnq(G^o8I$e_0gq4US}_LYO7 zn%hvB1j@xk&(#mKM12b>K*uiJTDgWT2(+b%{||mh1RBABQtL}qMgh$!0_kHghzJ8X z4#mB<)}cAK;X3DCM(ypLK0se`)asFW7f(&2K%aKWHjRwgXVc7w98`<7R}2M7-n>y% z{P*h_<3FCcopTGGI_fr)B+aFZpo-F{LlsjqqQ*I3I`E(D%RL^qBoEe?DSTEa1gur7 z#A})WB_LR886WE`Xv%r=Kwe0p|9MOZ?a4s1;6^Bi^J8|K(D(9yG`(?5Q#~P{n)Kqo9Y$kJ5R_t zpaM3eBb&5F4oqh?l&lOEkq?2Eu--wd&wlH+Tr#IL5+qgc<)La>;`N-hML?+y&mV{z~JQ6)1Vq@7L zBV0@W@PO23y<&>=mI$MMGDz`73DO=&IL-CD8jz0JX>jQyLZkF63y{g(>FTUNtN7gh zyb86&aJ)nh1S{UwYyRHK41}KPA38!*qpgi^CPmyPD!a`d=h{G#f*9H7 z#?VUapa9MACZxiInjsA}BZMLIf3f%G@l?NGzp$hdN`-_BrDBt5ha_Xz=6TATP}^8S zM5YR5Y_loK%r<05(Lg9=2$_dc2n~jiGLO%?Hr>D9dG5}A&VSE&?(_ZY`}Nwpy+3sr^k*89EQYe@6#%(JsW>@+%z#NjK!rta7Q$>#bHT<=8I{UjcyXmfK}l^y$u^PGN-id9k!Ym;A|zzOT*s;(%MK zZ@Mk$tf%9KrU+C_K^ME)a-T6c)uMwP3p2{T3c+0ENyRAlA@b>f@;O^4AN7Z4t~OWG zA_YITa^GT#<~7jONn_$!TmTMFRFrJRXzqh+6aHzwmcBEc`_k}A4WUf@gABeEKxzn~ z1#M-8??k{Q)Rp8zTq1oCT_%>_JL8ccMy-6>l1bgF^osL$;Y=73VAlu@zh|ixZ%6O3 zGqXY`jC!Z-QIrNILQ0w_Ab^(e@ODPd3@7NPelgvqE>;Hn-C2g#ac%#_c4)>&ni7a{QgBfB zA%R=vL#cdNGj7L(yu`oXpZoFsgyZYr4;ZC>hgp5a@$olmX8&C#+W8!^&YgDlUpm)CTHjDtmE3 zm6De)y}xj3u3*~(=PjA?6qLIHHEf~f5~sz31X^-fyQ47U>HMLT^0y#FGw0fT``8T`9hR+d#De`~NtBnaFGf4opB z&5A_=Rp%2uj6@}2v1{s@N>Dg8elE-I z1PYv5{?|;mT>%tSeB1#9Jr*^ZYZO#mEqKa(n8Z4v&9*?C6p29`CEA8NLiEWiKhD1f z$`;EkL@lnzUx0^sYwKIFZgU1mO*m0BsMUxWfAUcfl(-Icp}Z)f)&fEXTa^BZ`e;n# zM?E*&B*ryy*#cob+YWP(AxPCumf457r!Y9zv*)EsPJXdF9WqoVm9QRPwW>9GFjjRg zAMQ-yy&b#bIqJJwNR4!v@9~n2NNQ^*=<1++3~$P7rl4;zt#7g)t3165Y%WPw?!|lc zhF2Evg-1j3d3+h>fJ99$p=lrupbB6vR&togy_oPX|l7@P#qn*z|4&X07h)B2ihYNYdE_#^rZ|O;{z}t zhQ>7POUm_~G(JCyl;6Wn8%irMtyN?>V7kvJtaf$6vocJ!Hk^gwN$=&4_(#0lN{$J< zY<91%Z4o(!gyvi4TRJ#2$8^7il3MUFx|iU%3Pvot3T2ZcSXCtb7QtAe?XT|>%kmgT z<|wDzbz68wOiqsY$DNujc7DN0*#*|m+paHztnUrKA}1trni6@>1%`*=l@{TQb`fEP)!j;U6x`x;Tc#7W90ldf0oY=k0wdee6ZaiD!{T4I4}(2b zyjsT{CAH_+DN&De>i6h&u^jMrp8|GrTI*DZ?DB!(wXjG?XW$zd3fCqUHKFXXifrRQ2esZ%QJtDWfiP<-@S{4H)Q`Mm7~gIY1v79PtyFt zYK072W$FV04sQx)R4*m&*hCHI8dz~O{jhW(+V}U?*IY^afeA3M?t619DB?W@tk$7hIoY~Q!AMgBhk5iSNqTtTtCrWIE6{dh^-nSY3mrG|0Y&ACSOq!t5)4`dy zEu1JrZ_Xj_K8_ z)8rduraq)X!F^a>7r%#m9J{Lpl9SE2Vpxisn(;N z^I82RcC|*uwWCc>rPf7N(9B?V!^tBoXxUf-1+3Y)?35M~>EO@I*DzpHiC)eo?j$s6v%hrx?xP)H^Jx@j!< zq~+d#(F2fq6swXS!d{10^sb+W0tkbu_exP_BA3Dk&u>V`*pI%ZV%>7e z0s=wqKeigmrwG7{&elVk=(lmTG{7G7HlfHx=UK~<^-ZR!+jHe!=I!y~kU6=RF%-B^ z9_LZCb0`9H!Hxz>(U;KI%x@UEr1<|5BhMUp=-nIqaC*GNdS-Els=Iq@G}A!b6``#w zw%3t$Hc@|p;B}Ze6VHQerMd3s_a2F_ywg)@57zgJqY`IjR&N&;6lXoj$tvYUUIB_g zb1>?cG{*58@F5Q|?-h7Yc< zSJ*)V*Sndw4U40&qe|d`mfme_WDgr=hqW$$ooI^OW||maw;Vxe)AuNy@>4%h)=Ib2 znHtD9S?9%f~{xJOx%Nhs@E=Ze2`*X&fdAc-{K*$ zc&o^pu${b*RUtsAnsTsiLz`*cYa%{)} z+>_UB>?_^l7m|6~`FQB@t~mHf&Y77uo$fwO;1{14LJr>7f|Gx()pP#d6c}9?U1uEq~ec|ZE;8w{<6P<-;aK) z9%bCrahw`vc699w#&#w@^UWuewia-WqqU_7WCV}PNTGBlM*!dTH_y%Bm)Icy_VV_f zHiMs35*rsu4wnJkkVYuKH`s!Vaccd|L8m;7h?%z&JKy!+QwEQ_lF)8LUcl@m__Z?{ znB6F*ZCj6N3a_F@AChB4>~`=sjT>q4WLw-8g}*Ysrn;iP)Yf-P=KBY0*E4T-K8@Ss zhO*SMYsKsSrU(8OeT}Ag6WRoG>u>JuP9OJ#1ov{fmyI+0zz!o*$*trCjD)4%Z>`US zek{%Uo0(`jO?2UiH#+Wrfdjtyo4r;Pc>z{v zmcY(2@f(>J3U@r{zL1^Qn8a9&e;azS;RW+`zei*bhdt4PTw?aw6!~N>*IbWh2Hoo} zustxMrn*fuJS?K7DP$`>8+pTlvy7wN8pSqA&MH4{oU)aByVXqa2#BQ&~5jW4~$KDap&% zbJbwm5(BJ{uYW(64frifm~#68ia;E?3PHjX5|?>9vNt7n54?2v%zKpiL!ha<3!AcL zu+<$laO?VG)~4E#bJ*M)@X|kzNiFw}HUII=f40znqkwc-WF3zN#P%V~t$cXtYyJFh zgr>2HJ5i5*0#y3-&XYfOp|X{*ZQX5v^8$8mF=j#aPoyadW!6Z$3aCh>T2=a4KgfSV z1GQpWsNz{5`1)u%2*KVX{)W;X6?@@jB*a-HfEtS&lEpjA;I9zpCm)W4!@zJe${O~29a5MsJ4updO!^XQdm`8rXF;EOmTzc>S-9^VEa&~vDJ=d={a*RB%TUo zV9q{hz>2`tpkfdjksu6`_QEcoo(W%_{~4uA*!~E{p#FsN(hX2bzkwRDU(EJ6$_F>1 z&Z^yyL8AW#%dvAe;HL^C9KafzKBfFZEA(O))I}@Myl;esS6{9>N+57^L7Dx$#FyI& z*Ln`S*|{l@N*X6#2#zMBBK~k%*r1sZOxQ#fd_2|LU1uBqy_^Gi84DbuSnLm!WZs6E zJM$l_Rdn~>(jX#X^TvC}g;+6IW7`{b9q*yi`Wgx2?+8L;ouLQGIqfD^ml*U5#;0$I z--U{tjl8*ix?lhJ2Lfz23-ZHW75c%WeUn$ko&7D zQYUOeYw)Ku){fr`h1DderOs;~2_^Nv_*KtOmUh?2Ke5fIyBH5RIBJ@!LhV}6rel+7#Gp zv6gXAt3^Ik=DwE{rtAyJXiD z((XVdA1{D4wf#}|bGJK-2v8@^iJIBgO}mFi!PS#qvVKekR)6By5V^z*d3Og`Iiq0L zIJz6Rh-^G&+M|S1=@`lSBLqU)j5_M|7kvIDTEaQ2QKhvAD~DQMxfGPz14KQ(gFDlq znYBl5)%zh_2lWp7Ew(4)7!<$01Lb9o^lB%ueC75l6>BS#wcV&QZ7hZNw<0NWv{h0N zd$!oG9tMN3bX%O5eDRwM7F8=L{N7c+_xKyaA2Fh4}ICVZYx}^^Cf~6AGqU3%`Z0 zt-DUZ7?Si&X=3`J_KIlY@iXYQOj|g3UGZGF2BYRwtC4L}wuS8S0R51w)#|T1L$0Pz zGbIE$%zq`*#Siv>3mG1f!$x;RLeI8@)`_!4AKqcUE}i^W)%5?p%>rv!G2RE8p@TVX zitD;!SVI!8j;P_p8VKGx~y_^Mc)T- zCZl~AhuTaa%Vg15XB6+LUlPNq^)0rND_`)4%3w{&v?>{te~vKV^PpU=NL$CJ@TH!= zFuX>Qa3IY?ueil&jAVZzp(kCLI|@Q&fpc1k0qiZ?){bP*n254Q?aruwyYaNWuRlRG zZHqH-WiHf=oI>rJqAt@^$pG)A&isrj?_%W6?@p`sn9;9%>as`gT*Tg{asc(xM6MIs z)OaVm_hGJ#DUWT$Ty{v?M{d$Ap@q-TPAP(w)tx@X&Vs2IMaa+msBtChF-0Ug4Q!m^ zTi)JzQ-Dn}b3V?~U=Y~5gp~Rrkm(j52}@c|&sw?qmEUSRh;Q_NSy@y%7fRY=KN_*O zR(j&u(AjTlV+LtaZ7K;c~QqoFn?<*~hF7)sno6&lilON}wD2f)feA_!$Dy`yVX%W-=wXFjKgR=vnw@MP zzP}C+jmk1`_rfoooDofe(^6Pwa2|E2ht+8%Nu5B_>w9xYM7+%(?)5K+kWd9|k-bQG z<>Gtmkf5%tVd~qF?J-x7HbRK8m9N--!2DbZ?>Fz1;F2j(rmd~#dJb{n)Y|IGb&m9_ zZ+0u+_zyreD@U=~)mu%+^%KHiKN{@sc8L-oLy?7&V7Shrj6)k{=L56r>0Y~x>@y5= zU(iW_A(b%(htTbD_Kt2s=rhV3kX+7<*YvVdXiQDQUTERmv9dT5b}pFX@KGBZn6fg} zTJ7!hurUIS52$n`7{?z4NdCh7TW!BR-Mj>m^l%3DV_rW$rzr1ixV^ACSIKFGkW^?U zpLwg%O3Ip$0xHYYhz!;O;PwzS`9(9$^y}G30usI_Is1mV85-HagQ{@(1 zJJlRo_M#tRf$Dq!ktve?o4EJa78@;0HUK=i?^Vy51uF7^7R^^v>GSGX~Fq(sZrs z;ECas>s2r=Y4_z%uC;f!r+H}90C1|gvmD0IEDQv0FMqpKlnn z-(XJpE>7Rt+5j3ktN;@y;{))?;w*=s_IW#snu`W0PsXZZfn$Zw$o9qB7$jp2XZ!uv z@Wnq$#|oxaq*t0@1)fCA&D{r+H?!zbZd4D%z?E_^r+iQpbdP{a&-mF4URtFiL62}~ zG|o@(_1YxnsN-GjPPX^Sf>ErZysT{bxZp-6&3A;UG!H|f?yoMh3tI%}9%$^b3q^U_ z4_G{T$Fw@Rc;FF=+R`v6G3AB>837};IzQx^(EPYrhOI&8NvL>iD&1W)&D|bXuR@4V zzie1!LSv$|#G8H|T-v)P%5$S`q6fG}bQ)jrc4Xe~sCL23t;E7(>i-4y#x0fu36hU9 zHA$hQdEPO_vHNJ)f{f8KUIpCE#(nu=X+H>6S9Ng>(Y-yUz zVX(is`30)iXENspL1Kf97X*ei=5wYmzeII}+2(GiLsWopJBfZ-Pr zHNi=r^`!)|copIgSDW(JtF)#SV-FHfqy^c`^hMI`67YU7)SYL-aLm*;h4+CH3kIt< z&Cw|aIsV-P>D@j337wBJFMG}&hotbqy~>YJq%JzIv4b2^DK=ez_3V3-Lyrt4wr;3Q zR=R~dXEC;`?!P8C;nY@H+cvKUj+fuS%Hlj#ljLB^MXsDnL~dFyg&&63)Lug8H2H&X zi$GQvUY$MyADg~jNsUx!*r^Fq>YxYy`L(sw0u?9OpPC1c%Sl zd~A{5Ty=N1LRHJF&3gEqnV~kr?uYGB0aR3T&{{9< z?U(vnQH_E!cK4ZFT{~V3H5BMitbe%c&(nXo7y|_e2R{o-WQ5CG$Ni1p;q|xQ@juXR z{?`ux0fz4I7AQ@Mts@{3J30RL>nH;2!4nOFqpyMEC2Ujxw?Q`kpDTa?a(1w)d1q+@Oo0`D|x1nzv)xuXcf2$U9>7WZ=y^h{4NDiMGY`fIrfdn%w2;bGyH4Z&{=!dnrfWQcZKUA|6BirN#8!2a4mQvJLUSU z2zc{$(N6gL!yk+Yfa%Cy6%t=K(KaLHz@NvEfLpOErKT7>#_UGaAFxJIKRSno`!EXz zPraTvQc!BaQF32Se+Lx#q&E&oFLXdg*ydKLsN!f?H0=A$`#Hca)^#yiYw^b zI_-yTJqWUie5?f4NX5)ZCtLxOjub;ez8*2nD6vQNu!2V-RU6(HWoo0FiPlEMB9NJx z|0!baIQ-HeJ(x!O>4DpuHX;_qUn7>B0+feD4ZKQs1k$0K+&)a?=9lZSmL<9c+%#sHJ8v<%Re4Jif+3up>3Sh9U`%@)?srx`m zF(^h$5(DL0`#S>WcNyt=CWka%`!Z?vo&c00O&5Fd z^UI$A%)D`8)?6?F1@iaMJ@OhVjqE!mhf^ZH7I(EzII17`=&au~`C8?ziF6NCZpay? z96Ndhd8I{Y%|LWucm$&EL2ji8nYmAHuQzSm|5K!z_E@H{F6twGtXlvmD*kTW{u3~l zcBrz*rncq>qk!!LF@0jmuLo8ud<9Vlx$V9+##@K5U#15A`GwQc<3GR~6KB7ya)c8< zF=ed$NmsvULAA+qi@hw>DB@%{)vG**)qcIa@cVTgft%a}#vBvtnl^HxUamC(lhDLH zq_AY0hA(Xb^q51ia{1SMN7CaaN&fv;z7X!^1*Vng*^=-(;`j@m9pvz=VL5+;J5u7H z)^0sti{9dg&QlkD&%J<}xU0q9lJA3ILXjVipK&?X{bMh=6bE<=LxxN#I2QOQBaZ8ZR-ferNzuE6 zwA#H5ysf6ZJ+`Fb64EHXmr*)jnYP+N_X83&=b2d_NR5kgO$}$vk4T#BWCm*%PZhq^}Z(4^i7u5hB!sa_u- zAxm5C-yo;KnVt(ste%)|Js`!o2V1P(?Bx6+X?(v}hz)9dSVk%jD{q-SrAWuvxx zS5}=dSG&T_NN7Ov8j@YVerE?ng<@NFYV)O)ourkqdq6;I$vIN}Z(2EhQkxVz8S0{nKB?$*f8a7O(X zjDNNm1(hU1zAy)Of~CA~$qhv?#`DxmwQAd?^WIa`zXJPB0w&jnjwRq%h^QGAFR(`d z2YU)lfw=o_&^kl7q;`(s8ey>EUeBLXJmB9dz;b+4tQRuekKCYmD~zVVSE0sJiB4th zPw^72XU%WU#-C5x&Y|yf+Ud!$%-j`#qD!PO|GJC1BMyk?EmtYvD5HqT(1sNz=KOlO znf{HKR&iXtrO!B1)+_>JT%DPss?T*co@HlT`E->(sRyFbah~?R$$gk{*CC64dFsE~ z7syVVxJ#gsmIY&M;bADMHT{J{VBdqRxw_LM^+Qpyt-r=E_?L}?d~;i)#17Rc+oV5C zm4Tz;A6ZI%0fkSDg;je;?I5Wu-z%q|@ahnfeq##^h30W7*1C5)6T zh6amZ@a~nCs$$e`dDWqLerjka#^0d0afh_(_sEkIH=|^`T+w6(bD|M#SLFT^k7D+^ zk2eFyIDgBGB+b9^@ZrmTLEmL`^10%Dl$xmJ$bXO_d3cviGW}yI8JXjmuNB5U9d>ua zQE*c5;7LH*9(Ed;RV#BoIQPn=+}i`%^p96TPxE!s&pfM28N}c~r<4f4-PB<90D{!D zS5M4;zCL6VK`-$+AjMwR^r6uavNS}*T|_dF_dsLsts%5dsmelN&B=&?6SnPTBh4$7 zw;I9;w149HIEv?5dXiLuJtzl#EUbsvDc!OJYiJHFL(Cf)r}CMuRw{en-B4+vU@*tx`qO#k`*`H3-ND zhiV`liQ2r~b}@L$(=t~Qo z%b1?*m#xq~(oi4EpFV{JPSr&;=-VldPniImmM6#VJQCKsq#Jsf+{geqwT>~wP!Gd6 zKY1>H$mKov;_8tky69Kjha+fubiyb%FILa?274TocE;Mhg*a%|9FpoS*Ox^s@RlAX{&lHGywf? zSwM}To0fCjRK-YpxHZPR7&8Po91xIL4x)Z))BQ$Ei?icpy1wxkKIO132iFf`j$Jn( zJM^@{g@(rHg1E_MfW2KLwV5FT?*+tNQO;_ESD?RQPdMTt&Hfo@M2$^ir*{ z)5W$*&+PibXKrqCF)#ot)SuGFBNRXi-d^|Bq``i!kMX-6>wK zvR+nY?9x+-a)GvK9n#MK_}91TH~ov)@g@C`K#<(ZM!j~Z`Y-RN2)w*qxHH@05~z91 zaBuKknAHzRmj)6B<>Mv)?br99+m_&ozHNbvsHg%>Rkcul;pqLBe8$F`$M>9h7kByX zlnix8)J7H7@@UZX@YX-~i$5QVzeC~pq*y(2PCmtNqY6897L%cT97x@~`rH1bB49TQ zYIsONwYI4Je}KTMzy6p~#6b&-p5|8Gs3_lrigG&JJ`4vjXY-#jIV_(blz&Il4z1J? zHXiWcUmq}=7OuFZysEO7)|+T0BB}psno&gK`9Jyb{!3`#?s~tvM&`$2pw%DGfBWrt zI3rsD@lSG%@_X%ngBEhKMyL6h)(xCqA9u>zeKU9Ub*u-u@U>B3e;G`nAK~;&&mm!b zVU^zr{LKRb0(Jlr3g4Bm-eF&@>-JXzbDaU2WOXoqCTYFJ-?ih@-`0N<^?QUF52)qX zZ}i}cfzZK!UVjQoad3ct9`An$9T2Jh+gR_vAjf61Yye#k0qAPo5za_o-?71@hH3ULqKjVyujU%U^ecFcVu-_n$fpXHkV_4@Y zrRV-VPNcBFM*Qz(yk0!_SLm4Tc48ws? zIV%VXJu`SZ=+{pV_LzDH3NTdF*8fqLLu_ifEU=QK(wj}v2!lM{qjCk+y-Wo*L+vI~B{@1^tudeV&g`y^t|kN1?c+F}}+>`3=r zYgnu+S7vFEW*7_qIJFBBe;JwZ5!NeJ4gWD2pWt z14eZbo<9Vsr(K11sz?)_gsOBWCe1lC3DKq|l(cXa<|@(w1ao?yw70wY$xL=@Xg|o3 z?)9brQ|U$E@;d*WO9#P%;fZ5I@_?VyG+S=$x1(GO?tb;6uLg)B=c? zv7rJQK50K7Ap_azku=aW)t`Mcvb`f2O|6xJqMS7V2-Mry-E|~jz8_L%6X$rkKwDlo z|8$|>wgq4G0qOD~k-MqpwV(8ByCS82)!{$mE#inzLzpXf`z`xxZ)M@z5PSI#yLq32 zWW`yQrtay8sz{0dWZGf=wMxPx+ber~doNUF|GblYe}DHg)in9T5$~l|yG*mZ->WlC zd8Px*v)@$1h_-Qcta8=SYo^l}4H4!}#nO4~RmB-&pQfhHOPVGU# ziw$=7#&VZxb7}2ndG67o{p1p5Lfhl%e6hD0;~zYivwsGq^91?IiW=zkLFxNu<R9==Q3?{rwB--KIoSR|1&=k&Dq^1bluSxMe}pWNFu z2cWSotlftgd+A&J`I@YGr%RueEAz~r3ci4ukOSbfp|UeRW28kSaL0B18wfYv#MPrp z?~Vi*gZPGrvVY8HUTb?Evgx_G%Wy?zvh6V(Nu9~vH!GF80Xm1iQ)MfJp^eTd+{ z^s%2TWhcXN1pqO6tHriEXJ^&MT&u_I8se&LJYKAh%ys40-83mWtKfA>LVx2cbk+he_EoUYl2Qrwx}jM#GiER4Cu z*kNU7EsEabJZ*XXt(7L;%;idR*A$QWr^%K&=2nOo{7yKwcmC7&tdg@A^R03}U=jkc zpa==#nJ3@uGj}8vmx=LlS9?1|DYk{EuSuFZJ-Jvl?I~TZY-XJ?EOsuG_e&)c&ZH=t z;7x2#t6Jc1IVmkS*Ef))mOq>vx~F1X&eN(dr!#mp2Q%2t;XXHT-$7tFITy?R^nr>k zFP-qatt=p5-l#ZY|DzY2f&yRiKJN>EWAv$FBh@h9MNGQVJ@bdB-VB}}^YKdLgwYT8 z#%v0Hb0^`#TBUcedp}?r?l=45^VHIP(qgsT;^gAo6UuUZ+0wb4a|RAdYX-z&)03t{ z7s`!Z2pr4giRV>%7lGBI*_=KzRhzunkyQnFxDhn5A#@MPi(3Oz!PPZE00xT&)Hk1F z;KfD)vcr(P^DwKo<@3E_Mq8t0UO3Hg2TaVg7oU_qwKma`+gvnk$tm!0Wc%V%1%b7> z6rWYr*^F(&{S4kS-WMRMsPY{93RnR>8oEsGI7i)t7hB}F1*Jzy^1(Q6zw}bc3FuLx zMmQW2sT?%tWA|(nsePk#{y0Y+EwvLafZ@kEIo4nNrNZ2r0sH!&2ax=jgphpy^cKgPbF`x_Es5=U1zh;7-ngM<5m0FWX00i%b|c@jcEHz zCLBt~MbJQBJ9&6_lyuj%RD@qx%~vs3p+W7?sr|YoRl!l3ci6m{`r@^5mP+&fh>7QX z#+qfVm8_A^W3Np-#~3;Lv{srXJtqcFJmoboTrIv{e6gMcZ70*mLS#U=X z2Sw_z>L7XfBc@8UwcWiq;R3WB4lyQ`&-Igz`-Pb!-I2Gr8bJZ+OzrQlo$G2o$t+}O=pE&AC zSK6)JJ~2(bQl{{ruU5QG1#`$3vN(X}Z8K04nDyJ}!}1?^vAVd({Vd&DWYqH1V>m*6 zuYPl%vh+KZ5+1OZFV3tS6b*!-FV5>di19N-oUltj^Cp(i;3Hf5J>N>(ENWHy@ZIlA zttYk=Sf|}eqd3j78KPyl zV``lai@4LisQh=H;d5#Y9?=#cgW2+|O&ntLJhih1(UR~wU(elBA<kiGITF?63ikuu}BSlKnx2os=^4dda zYdfoY=RXB}$f^F%`)7}j{Dj;r2<!yo=>Gu1SW?;{x*;6y{+PFXL8vj{T_d^C7gb z1|Luez5yq(`+KX{BIsZw6VJS7Mo)iRnylmmRb}QUJq{1ZcLF$UgHjbjYOsJ1ezZRE zWBM>xFU>pAtw>EFyS#V|da8%}E7x!TC8DGP4w)8 z{Slc6MvUAXn_iSE`F?r3%6FKr>b(3E1R+w~0^PfhIB(s#f1m8{#UEpC{41uiYs2G( z_dFt4nX9juC)-49^F1fF6?tQ=T3siiWB zoxQYs7FIgGn{<0r&lMLexKDJ?B)NCa6}fjW_;qCK>UvsK+DjF?R7>y`uuqrOce$ER zmPD%!mAHd2z@4G<6*E2+c^*N*i_2rB7X2$%CmQf#HnQlpOuFN|>-Nah=!3#wU|~F#!A~!ii@j9Ysp(!^?815tw zKf07v)4C;QfACHYC{70DijE(MU1KB9tc;bj^&7sXhU-DYFMzPM3#m%izWlamPd7}d zY1Kt9^;2&DHdcM4oBgTqlDoVNL}!VemwP?B=fCD5bHA9XVH*HGPUu<^%wBltKGKE{ zQ1f(N{4{z{%_?c}hr0hrR*ET`w=!MHlJ@t_zVyT9tDmpafpO(FuX)5}2T;dHa{ z?4Xb7A`F>GpYkb7?;Bn&m|TPu?>HL0z5_w%AydyMZyL!g+1?#4)#{&FygK%24RO{e z%=-JWy6)}j-zX|zDj^muD%#8~+?MHyLUxHqnXtij2o?l3lUGly~wQH#f8FJ$40 z=GFIo(MA2u=-Tmc;-aGl{l`NoWqLGT$-u1;zR>pqpHXK{*qtWy5$?Dr?gLtH4XNLh zx*^Z|E=|vaj9&b_NZ5S&tkKUrA;;cSoo}}$oF44oG6+|}h*#6#p4#H4`0FgaS70i1 zFxX#Ut9DQz>7>D8TSxC<7WUF-tS}9w!=pQ^$lxoE?}7*i$F)yd6$^`3-w(^iM5@ zH^{YW7NY*%KGF6C^V;G2g+Dg?HhH{z3RJleP)2bbn*Z!A#QDz5j4$7R^>=cuTWP^$ z7~7F*5y>^jWUR5fK+4!M&`7Y6n<6$_OiuY)wK#m!hRVyRR$-Z(9H|z%vWM_Q+JLW5re7VPdD(l+V z!CjN5^iRBjnORwh6W!-6O_}xQrcJ*MgI34B zHB5ub2Rsb=@ZXK(HSXr5&82&uC;nb23%`(E-$>fD2# zFDfQIT)MS;f3`ih`c7%Fd6X3LXxGfsgWN%57IOi1n_V6uqyDH*bIWvi^bI&hu zdV?}MSEpe~kh3XRnD(rgG`l4>3922xMYC!x4yC*_dPSVj8B^D@%2*uiuoxxJ2+t@x z%&lZq_w>^xb-EO$@P_9ykbb7?rU}{fZqlW(C#dCZ=*CR8C@p@|S1f(+wDZADIRfEe zVkj+-g%q>f*{~c*nuoSGhSSC_tj%6{4Wj+5LQjZ^DJLx!DJx)r`;f?Do8jKqAS}*& zUDzVUK72$lTks2F9Zh;A!u53kiB#CgF;6)8QB6M=n!wdJc6@(0^ZT^97m`|^`kfyz z7G|}4_U;L{b1$C7y}j1w=WWuN@w+ILPxKNj?ytIaazYtj#@AR~*!0p=V$Z~<(^`_?GJ~iILZmd$QJ-JxI=JM=-*7MGH zbiMD}iUoQVb!)Qen0=4q+ZmGwty~jpW-KjBnezp@PIWX7-nUxTlr7+dIqaZe4gT; zuqPug_AE_&5!aNM>Tt4XqmF>VXDyPazBTdg8Kb!+Rm)>rSo0iXMOsY zEcb6CkHp!b^DentiP>A(z+^4!^JyD7T=ut?1|5P)nlIMCeYNZS=?`{5iFz`6;cDM+ zqQxcxX_nVb2Nx&jc%pM(dX+}&iPU{O!xbkorTp%Q@cq{4b}Y@woGC1xjp%ww zFgeJe)Zbo8;wO2IF3GLgl9>F5yoKn?k6*}ik5wflFFsu$v2mByw0l%G?ZBRHhO)-p zTfE~XFbCjWj@XzdM3WKC3;bLuVfjeLgG71RHiMDbNOHjvuHz#2zQQ3HK}#G-Gn0QJ zp7_FJAb$w4gZCA40<%pdUEcGA2caoWaMP9!tiB6W-!6`1-l=)1hO5t`*Ow9VRVRf_ zoEe#HGq!(4p32b9=+1{Lr~O~@!7~NE+_k4EkSWIkFg&nK%61_ zPH_wYbZ~6%I+*VA&Pp}bHa+lI1D8XYv9{8`7NL}Qv1+@>#Zpic8(J`RJTyE}63*!R zZQ;e-4X??Hr^0%~c&>Df@4ZkTJ#p?ZoC=@w+)5v*dQ8N<%J-GzO!1j?=#J@$wO$^x zT9b4rm~=~g?NLtpta7~7U79D+E}PMWcQMCF&)Ix(=7wL-WanIYMYBIt{uE<0lidV_ zSB?E=j9r}%$;{c0eeaYEXB2Ad&}XBBT0#D%%Z_rH8e6usln$k8^4R-sZ)i387Ap8f z^tUz>P7?aS)W8(v}t3;l)4c=hgoR147QT)8pZ0tFbcY z^z9ZX`%^m<=?aKG2s^Ji7PC8YVqvu4cL~X%L8rCDc_csQKDLe+WMrE$y7g{SSDJsR zR(ny;v%Uu7C4rI`&t$rv%f)Bai*|}Ve-t_+o#G~&yd<#Osi_|d`{`oR`8NXupFANF z^p0v}4-tQ5G5op%;d-B(V<*1;6&#NuWZ(LK#OsWkCY3EJd6j(57hdYBx_ZD^7 z>@At4xhETV&_<)?7M8*NsUGQ$pYiQ{T7HYG`~gPi5(FO=pGkCAyx?D+o2$Et_(-?- zTq~N7kY&Y}(p7bBQNuyLX;!~7hE+TCV(l_e`sKj#xyQYt z%aO$at2uK++*i3ls`I%Vi7na3zt`I|o#ZsUH5f~uk?-QJ>e#!l^@P>;&2|}#xP)*b z)NJ33boXQpx%YkdF3eKPnDJw=t(PDaEute}>S{`(f<^zx@4_FVO>_44U1s^OR7M1T zb=(u!p|3SHcmKFJ~IIWh^pC@R`Sfg0N#?vK^hLN?^8BcRC?0>_TSP zQ0L1AoqNiAQmA6v^z%4*J~z#7&uUgLNuG-p z+S*V1@Vjwp7j&|k^0wD;eTXI;WHz&cS`7@9u%>(1B)hhZ&nRxn7vLJU&FdWW*q!y+ z^i@1ZAt~8uWVa6KmSAy-aN@*ib;ryFz8IBF#st-jAP2i>SO$klHQKeB;}yFrZ=M{D zow!l&)!u&1wx0K5yScM~RO2^oIcBxeyU|T;+RUw`Cov34YZ+OhZQeL+IitBf>rKJN z){j<&NnP15O>ps)_Ps&GKAM7(*;{*)4IuoJV&&h12Atc;P>JapbN!SU5k zH0#cFw|zXOCkrOAj8=l?lFgHZriFT~_82qbtEw5xvgvl;WSRDQ!A{S>Mn=0NzQ^X; zb_TOWE=1P~ZDZNyIBO)!UtqfH?sbA5E&Ipu8x?0}Zfv!X7%M1gtAw#E^AVoe#GI5J z+?f_t_Aj(CAM4e0h4X)KYmT&dQ}5*0Xk=w+t(9IAr`X&y~y9F1jBW(IR6y zEIE=qc(tr>Bj~e8oADATpJ7yr&suEi-tnk%xuG(=e;d}=LmHYb=<|JgXv2v~d%mnh zaJR&hRlRm@x3J}BP-0O;_l2PsUKVG@XlFBGv_kbCnSSpW3ybS$zFK>#=>7dhhAr=v zZfA<51YtZv^gMG#GnE&|nu;Sk{q&Z9w_BKOTWsV@A5L&Axb=G|wyH2EE@3dW*xwo3 z$*6bnbm$NXGo-f-B(Umc*ey2(Py&o?9uqf>H=U8I_$U8sYmsOILg4%+-VFRU6A zV^#S!D~0WH>`5Ydy*=&tPU735;Dy8oPSQBFSW(P_-Bgmt^xWl}25n~8Xq0lZy0EPl zj$2xkIj-2=?@0}_?|s*nNYneOja0U;^fL(X!LPO^4OgiJ65&kKz?>1JW3b~;^|vFT zriNBq>(;l+qC(%^Fee-5$E4BX^L^8%{d}L4yyfj4e2wiYw$k?&Ezu6nZL5fVo0fHA z#P_IRV!s5hqpPc6OTuZ>1~vVc#8|G)hRHrFP2VfRElZc|!Yu%nnIeTW4J^_nU#iD& z-ZRb5W3Wqea5atCWS2IKxoXNEGFI+BGw#w5I_ar7()#*f#(1kTCp+_MH>0IplEctq zL7DBCj}%VLwNH7q$RO-Xyn?&$Np9EC|MVMp2??}-=BG87gWt}7_rcnD&&+I zCH&GqiY?EqwMc*Z^sHHd_ST!%tP`XB=gRu4OF~8S71m6j9&}S`uMnM4Fu`Lr8FfZ- zb}oK`35#n!%WR8JqZ^hWuUXOG<=bg!TwRpr(MTkuOgcxf%gim@#zZf%X4A-7yE_(pH`@rX<|=;}RriFN^6r?4i(}6YrnDtGpXe{; zPOEOD`ChhNLw`J7@V;H}@L*1zVygw-H%E=zLT*fz!}LRCog!v2&7NaXukVwN{@$>? zF@`_$HEZ)v7af$5(T|cfN+F%-$yM+?875{NdTAkk>n0Wqs$$-;mheP48{gckAzY>#M^t9GAw%HPF7Yj z_m9=9!8|2jp<$6{SZzC!V`GYsU+JiBcl{x~;8fAB^9#R|p&?SzWAsR(Zg2u=dQ>vS z!y&3*%Ct1xM#JbLy2Nxd(kD)-v*q7Pr4^L!l#j~mBBiJ=GJ6)8 z3B8Ctm`i&hKQkqqJ6>3ATXzu@a|k5YpAA;4*n;Yd(4|o8nsUwPQ~Pd$id31}QC`x>hu+ z4c4=T5%f;z(d5wvcF{U!PK`7e+lhPA>YvdroXLD3k`dd>bWzt@btMRsrPL8`Jr<=4_jE2l# z70Z;dJ1_2Eg8`w+jdCsAB6hd=hjfz7BDQGmmRpOg%PiOSOcQ&jknGbte_NH*T;4Q< z?F5^@dK8YeOUFDmfIYM|r)jq&Pl`_ut7nH~V)IPA;nu1`|CO`q2j&x0q;WYs)Zl?bRwGXh^{2J0BqCha`T$hx&Nn|Yky1n3gc;Z z@j5jfNz-^ryjHqyrdZRmR7jU*Bq~yxDUq6aLzA@D6{BS#1*WC}=H;ttp-vQPiOVh^ zImB?l%AQD3$)+(RbKhER%|Bu1r}uo`^PJCnK0lo2ob#S@t`5iSIOs!DbB}Mkl^@4k z8=arapJynDayF@(fV&QEDG6^$Z5v_}@3K6iSs8?|&GY&grz%WWkZw*KWl@HoZ^x`K zskA+zF5Hco&$!g!aflSGHUeNH;E%-=y2Vx)@8rvI=Xk@_>nEZ>9P@}^ zO+dhm-kRH?WOjV_j$S2aG0+^B!g;iqvH%Zc?6FHU5!w>nH}tNBVJ{HQxeRU{BtI@M z7(*VQn9V~!*}`VurtW2PcRYYy>~_L_(2N8W?_3d3CE|>QSG4f69M;h`zK`5SUcX7> z;1+=kv|6Z6Z_X98b;8%!it_TO%Kyu zq2FW;&q>8w zpGmYixWUa+96`~`B(;bhMpX5|MWyAo5#1Xk*^NT+)FX9y3iTEHG&=*2b)>Iq8?Vr0 zZqjb-z2>D@Ny!w_!YWH*SEKf^)lonXPL*dz0M8}XF`7~d zbn21&@=ej<9L-Bd#gg0-N0f#Sogmg0eS}KQVWr068Ok>kDdn-xP!m##%;eqoPTmgd@B4_F*cT+;ccx5=R#p^+n;9QZTO!&By-d(*77Zqn!p~9@yL06&kfdLN zYs~4q%m5`9h^lRmvd25^Km} zCZ4@{Uber{wOPRJ-qpx+;&tl18+A)#Cms6{UczT#lZZ z8agk_Bc)Ggj&=kb7@d*f3ibL<{Zz0Bu&0QJtJK%^n7~3PM^mq*NT;%ud?lYYuIT8} z=P8MNLhfLPb@k*VTm%(i2Xq{?Y{vm-gE`-a zeXf6U?70lW^~Bnh|1|vD@U*-TWcRx(@{1lmI7X~51UW?iJFAQRuy|$UnnPnWr93i*J ztp77k1H-unXQ>8dm9sflfiQL)9X(qB7_4aZot@NKoc(ZEQIosk0BV*1W=4+T^NHly koLmt8zlr|Nz32_(evssh^WKZ&bHEb3H8h}p^Pb~>0j2Ne>i_@% literal 0 HcmV?d00001 diff --git a/docs/use-cases/general/scheduler.md b/docs/use-cases/general/scheduler.md new file mode 100644 index 0000000000..5b7cccfbfc --- /dev/null +++ b/docs/use-cases/general/scheduler.md @@ -0,0 +1,170 @@ + +# How to Use Scheduler With DeepSparse +In DeepSparse the scheduler determines the Engine's execution strategy. A scheduler ensures that cores are kept busy as long as there is work to do. + +For most synchronous cases, the default `single_stream` is recommended. +For running a model server or parallel inferences, use `multi_stream` for +maximum utilization of hardware. + +The available options are: + +- default: maps to `single_stream` +- `single_stream`: requests from separate threads execute serially +- `multi_stream`: requests from separate threads execute in parallel +- `elastic`: requests from separate threads are distributed across NUMA nodes + +Here are examples of how to schedule work with DeepSparse. + +## Scheduling Work With Context and MultiModelEngine +You can use the `MultiModelEngine` to create multiple instances of DeepSparse on the same machine. Using `MultiModelEngine` means that you'll handle all the preprocessing. + +Here's an example showing how to set up 2 streams using `Context` and `MultiModelEngine`: +```python +from deepsparse import Context +from deepsparse import MultiModelEngine +from deepsparse.engine import Context +import numpy as np + +zoo_stub = "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none" + +context = Context(num_streams=2) + +engine_b64 = MultiModelEngine( + model=zoo_stub, + context=context, + batch_size=64 +) + +random_b64 = np.random.rand(64,3,224,224).astype(np.single) +output_b64 = engine_b64([random_b64]) + +print("\nb64 response:") +print(output_b64[0].shape) +# b64 response: +# (64, 1000) +``` + +## Scheduling Work With Engine +Engine is the lowest-level API for interacting with DeepSparse. As much as possible, we recommended using the Pipeline API but Engine is available if you want to handle pre- or post-processing yourself. + +With Engine, we can compile an ONNX file and run inference on raw tensors. + +DeepSparse also offers a multi-stream inference mode, which allocates CPU resources to multiple inferences at the same time. + +Here's an example, using a 90% pruned-quantized oBERT trained on `sst2` from SparseZoo with the `scheduler` set to `multi_stream`: +```python +from deepsparse import Engine +from deepsparse.utils import generate_random_inputs, model_to_path + +# download onnx from sparsezoo and compile with batchsize 1 +sparsezoo_stub = "zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none" +batch_size = 1 +bert_engine = Engine( + model=sparsezoo_stub, # sparsezoo stub or path to local ONNX + batch_size=batch_size, # defaults to batch size 1, + scheduler="multi_stream" # default: maps to single_stream +) + +# input is raw numpy tensors, output is raw scores for classes +inputs = generate_random_inputs(model_to_path(sparsezoo_stub), batch_size) +output = bert_engine(inputs) +print(output) +# [array([[-0.34614536, 0.09025408]], dtype=float32)] +``` +You can define the maximum number of requests the model can handle concurrently: +```python +from deepsparse import Engine +from deepsparse.utils import generate_random_inputs, model_to_path + +# download onnx from sparsezoo and compile with batchsize 1 +sparsezoo_stub = "zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none" +batch_size = 1 +bert_engine = Engine( + model=sparsezoo_stub, # sparsezoo stub or path to local ONNX + batch_size=batch_size, # defaults to batch size 1, + scheduler="multi_stream", # default: maps to single_stream + num_streams = 12 +) + +# input is raw numpy tensors, output is raw scores for classes +inputs = generate_random_inputs(model_to_path(sparsezoo_stub), batch_size) +output = bert_engine(inputs) +print(output) +# [array([[-0.34614536, 0.09025408]], dtype=float32)] +``` + +## Scheduling Work With Pipeline +Pipeline is the default interface for interacting with DeepSparse. + +Like Hugging Face Pipelines, DeepSparse Pipelines wrap pre- and post-processing around the inference performed by the Engine. This creates a clean API that allows you to pass raw text and images to DeepSparse and receive the post-processed predictions, making it easy to add DeepSparse to your application. + +We can then pass raw text to the `Pipeline` and receive the predictions. All of the pre-processing (such as tokenizing the input) is handled by the `Pipeline`. + +We will use the `Pipeline.create()` constructor to create an instance of a sentiment analysis Pipeline with a 90% pruned-quantized version of oBERT trained on `sst2` with the scheduler set to `single_stream`. + +```python +from deepsparse import Pipeline + +# download onnx from sparsezoo and compile with batch size 1 +sparsezoo_stub = "zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none" +batch_size = 1 +sa_pipeline = Pipeline.create( + task="sentiment-analysis", + model_path=sparsezoo_stub, # sparsezoo stub or path to local ONNX + batch_size=1, # default batch size is 1 + scheduler="single_stream" # default: maps to single_stream +) + +# run inference on image file +prediction = sa_pipeline("The sentiment analysis pipeline is fast and easy to use") +print(prediction) +# labels=['positive'] scores=[0.9955807328224182] +``` +## Scheduling Work With Server +DeepSparse Server is built on top of FastAPI and Uvicorn, enabling you to set up a REST endpoint for serving inferences over HTTP. Since DeepSparse Server wraps the Pipeline API, it inherits all the utilities provided by Pipelines. + + +The first step is to define a configuration file that defines the desired `scheduler`: +```yaml +# sentiment-analysis-config.yaml +endpoints: + - task: sentiment-analysis + model: zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none + scheduler: multi_stream # default: maps to single_stream +``` +The CLI command below launches a sentiment analysis pipeline with a 90% pruned-quantized oBERT model with the configuration file. +```bash +deepsparse.server \ + --config-file sentiment-analysis-config.yaml +``` +You should see Uvicorn report that it is running on http://0.0.0.0:5543. Once launched, a /docs path is created with full endpoint descriptions and support for making sample requests. + +Run inference: +```python +import requests + +# Uvicorn is running on this port +url = 'http://0.0.0.0:5543/predict' + +# send the data +obj = {"sequences": "Sending requests to DeepSparse Server is fast and easy!"} +resp = requests.post(url=url, json=obj) + +# recieve the post-processed output +print(resp.text) +# {"labels":["positive"],"scores":[0.9330279231071472]} +``` diff --git a/docs/use-cases/nlp/question-answering.md b/docs/use-cases/nlp/question-answering.md index ebc5a6cb2e..4d168f4756 100644 --- a/docs/use-cases/nlp/question-answering.md +++ b/docs/use-cases/nlp/question-answering.md @@ -282,6 +282,42 @@ resp = requests.post(url, json=obj) print(resp.text) # {"score":19.74649429321289,"answer":"CPU runtime","start":73,"end":84} ``` + ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to deploy question answering pipelines with custom ONNX files. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download the [DistilBERT - SQuAD page](https://sparsezoo.neuralmagic.com/models/nlp%2Fquestion_answering%2Fdistilbert-none%2Fpytorch%2Fhuggingface%2Fsquad%2Fpruned80_quant-none-vnni) +ONNX model for demonstration: +```python +from sparsezoo import Model +stub = "zoo:nlp/question_answering/distilbert-none/pytorch/huggingface/squad/pruned80_quant-none-vnni" +model = Model(stub) +model_path = f"{model.path}/deployment" +``` +The `deployment` folder contains the following required files: +- `config.json` +- `tokenizer.json` +- `model.onnx` + +Use the folder as the model path to the question answering pipeline: +```python +from deepsparse import Pipeline + +task = "question-answering" + +qa_pipeline = Pipeline.create( + task=task, + model_path=model_path, + ) + +q_context = "DeepSparse is sparsity-aware inference runtime offering GPU-class performance on CPUs and APIs to integrate ML into your application" +question = "What is DeepSparse?" +output = qa_pipeline(question=question, context=q_context) +print(output.answer) +# sparsity-aware +``` diff --git a/docs/use-cases/nlp/sentiment-analysis.md b/docs/use-cases/nlp/sentiment-analysis.md index 9633fd1c6b..c290b73831 100644 --- a/docs/use-cases/nlp/sentiment-analysis.md +++ b/docs/use-cases/nlp/sentiment-analysis.md @@ -307,7 +307,39 @@ resp = requests.post(url=url, json=obj) print(resp.text) # >> {"labels":[["positive","negative"]],"scores":[[0.9330279231071472,0.06697207689285278]]} ``` - ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to deploy sentiment analysis pipelines with custom ONNX files. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download the [oBERT base uncased - sst2](https://sparsezoo.neuralmagic.com/models/nlp%2Fsentiment_analysis%2Fobert-base%2Fpytorch%2Fhuggingface%2Fsst2%2Fpruned90_quant-none) +ONNX model for demonstration: +```bash +sparsezoo.download zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none --save-dir ./sentiment_analysis +``` +The `deployment` folder contains the following required files: +- `config.json` +- `tokenizer.json` +- `model.onnx` + +Use the folder as the model path to the sentiment analysis pipeline: +```python +from deepsparse import Pipeline + +# download onnx from sparsezoo and compile with batch size 1 +batch_size = 1 +sa_pipeline = Pipeline.create( + task="sentiment_analysis", + model_path="sentiment-analysis/deployment", # sparsezoo stub or path to local ONNX + batch_size=1 # default batch size is 1 +) + +# run inference +prediction = sa_pipeline("The sentiment analysis pipeline is fast and easy to use") +print(prediction) +# labels=['positive'] scores=[0.9955807328224182] + +``` diff --git a/docs/use-cases/nlp/text-classification.md b/docs/use-cases/nlp/text-classification.md index f6a1845aaa..c01268402e 100644 --- a/docs/use-cases/nlp/text-classification.md +++ b/docs/use-cases/nlp/text-classification.md @@ -402,6 +402,41 @@ print(resp.text) # {"labels":[["1","0"]],"scores":[[0.9941965341567993,0.005803497973829508]]} ``` + ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to deploy text classification pipelines with custom ONNX files. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download the [BERT base uncased](https://sparsezoo.neuralmagic.com/models/nlp%2Ftext_classification%2Fbert-base%2Fpytorch%2Fhuggingface%2Fsst2%2Fbase-none) +ONNX model for demonstration: +```bash +sparsezoo.download zoo:nlp/text_classification/bert-base/pytorch/huggingface/sst2/base-none --save-dir ./text-classification +``` +The `deployment` folder contains the following required files: +- `config.json` +- `tokenizer.json` +- `model.onnx` + +Use the folder as the model path to the text classification pipeline: +```python +from deepsparse import Pipeline + +from sparsezoo import Model + +# download onnx from sparsezoo and compile with batch size 1 +pipeline = Pipeline.create( + task="text-classification", + model_path="text-classification/deployment", # sparsezoo stub or path to local ONNX + batch_size=1 # default batch size is 1 +) + +# run inference +sequences = ["I think DeepSparse Pipelines are awesome!"] +prediction = pipeline(sequences) +print(prediction) +# labels=['LABEL_1'] scores=[0.9996163845062256] +``` diff --git a/docs/use-cases/nlp/token-classification.md b/docs/use-cases/nlp/token-classification.md index 1d61341bf3..3886ec7802 100644 --- a/docs/use-cases/nlp/token-classification.md +++ b/docs/use-cases/nlp/token-classification.md @@ -275,3 +275,32 @@ print(resp.text) ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. + +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to deploy token classification pipelines with custom ONNX files. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download the [oBERT](https://sparsezoo.neuralmagic.com/models/nlp%2Ftoken_classification%2Fobert-base%2Fpytorch%2Fhuggingface%2Fconll2003%2Fpruned90_quant-none) + ONNX model for demonstration: + +```bash +sparsezoo.download zoo:nlp/token_classification/obert-base/pytorch/huggingface/conll2003/pruned90_quant-none --save-dir ./token_classification +``` + +The `deployment` folder contains the following required files: +- `config.json` +- `tokenizer.json` +- `model.onnx` + +Use the folder as the model path to the token classification pipeline: +```python +from deepsparse import Pipeline +pipeline = Pipeline.create( + task="token_classification", + model_path="token_classification/deployment", + ) +output = pipeline("Mary is flying from Nairobi to New York") +print(output.predictions) +# [[TokenClassificationResult(entity='B-PER', score=0.9971914291381836, word='mary', start=0, end=4, index=1, is_grouped=False), TokenClassificationResult(entity='B-LOC', score=0.9993892312049866, word='nairobi', start=20, end=27, index=5, is_grouped=False), TokenClassificationResult(entity='B-LOC', score=0.9993736147880554, word='new', start=31, end=34, index=7, is_grouped=False), TokenClassificationResult(entity='I-LOC', score=0.997299075126648, word='york', start=35, end=39, index=8, is_grouped=False)]] +``` diff --git a/docs/use-cases/nlp/transformers-embedding-extraction.md b/docs/use-cases/nlp/transformers-embedding-extraction.md index db62e13d2b..bbf5614ddb 100644 --- a/docs/use-cases/nlp/transformers-embedding-extraction.md +++ b/docs/use-cases/nlp/transformers-embedding-extraction.md @@ -209,3 +209,37 @@ print(len(result["embeddings"][0])) ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. + +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to deploy transformer embedding extraction pipelines with custom ONNX files. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download the [DistilBERT - wikipedia_bookcorpus](https://sparsezoo.neuralmagic.com/models/nlp%2Fmasked_language_modeling%2Fdistilbert-none%2Fpytorch%2Fhuggingface%2Fwikipedia_bookcorpus%2Fpruned80_quant-none-vnni) +ONNX model for demonstration: + +```bash +sparsezoo.download zoo:nlp/masked_language_modeling/distilbert-none/pytorch/huggingface/wikipedia_bookcorpus/pruned80_quant-none-vnni --save-dir ./transformers_embedding_extraction +``` + +The `deployment` folder contains the following required files: +- `config.json` +- `tokenizer.json` +- `model.onnx` + +Use the folder as the model path to the transformer embedding extraction pipeline: +```python +from deepsparse import Pipeline + +bert_emb_pipeline = Pipeline.create( + task="transformers_embedding_extraction", + model_path="transformers_embedding_extraction/deployment", + emb_extraction_layer=-1, # (default: detect last layer) + extraction_strategy="per_token" # (default: concat embedding for each token) +) + +input_sequence = "The generalized embedding extraction Pipeline is the best!" +embedding = bert_emb_pipeline(input_sequence) +print(len(embedding.embeddings[0])) +# 98304 +``` diff --git a/docs/use-cases/nlp/zero-shot-text-classification.md b/docs/use-cases/nlp/zero-shot-text-classification.md index 4fe2e7c08b..2461444fa6 100644 --- a/docs/use-cases/nlp/zero-shot-text-classification.md +++ b/docs/use-cases/nlp/zero-shot-text-classification.md @@ -249,6 +249,39 @@ print(resp.text) # {"sequences":["The Boston Red Sox are my favorite baseball team!"],"labels":[["sports","politics","public health"]],"scores":[[0.7818478941917419,0.17189143598079681,0.04626065865159035]]} ``` + ### Cross Use Case Functionality Check out the [Server User Guide](../../user-guide/deepsparse-server.md) for more details on configuring the Server. +## Using a Custom ONNX File +Apart from using models from the SparseZoo, DeepSparse allows you to deploy zero-shot text classification pipelines with custom ONNX files. + +The first step is to obtain the ONNX model. You can obtain the file by converting your model to ONNX after training. + +Download the [BERT base uncased](https://sparsezoo.neuralmagic.com/models/nlp%2Ftext_classification%2Fbert-base%2Fpytorch%2Fhuggingface%2Fsst2%2Fbase-none) +ONNX model for demonstration: +```bash +sparsezoo.download zoo:nlp/text_classification/bert-base/pytorch/huggingface/sst2/base-none --save-dir ./text-classification +``` +The `deployment` folder contains the following required files: +- `config.json` +- `tokenizer.json` +- `model.onnx` + +Use the folder as the model path to the zero-shot text classification pipeline: +```python +from deepsparse import Pipeline + +# download onnx from sparsezoo and compile with batch size 1 +pipeline = Pipeline.create( + task="zero_shot_text_classification", + model_path="text-classification/deployment", # sparsezoo stub or path to local ONNX + batch_size=1, # default batch size is 1 + labels=["poltics", "public health", "sports"] +) + +# run inference +prediction = pipeline("Who are you voting for in the upcoming election") +print(prediction) +# sequences='Who are you voting for in the upcoming election' labels=['sports', 'poltics', 'public health'] scores=[0.35093653202056885, 0.3335352838039398, 0.31552815437316895] +```