Feature/sg 1060 yolo nas pose

LICENSE.YOLONAS-POSE.md

@@ -0,0 +1,16 @@
+# YOLO-NAS-POSE License
+
+These model weights or any components comprising the model and the associated documentation (the "Software") is licensed to you by Deci.AI, Inc. ("Deci") under the following terms:
+© 2023 – Deci.AI, Inc. 
+
+Subject to your full compliance with all of the terms herein, Deci hereby grants you a non-exclusive, revocable, non-sublicensable, non-transferable worldwide and limited right and license to use the Software. If you are using the Deci platform for model optimization, your use of the Software is subject to the Terms of Use available here (the "Terms of Use"). 
+
+You shall not, without Deci's prior written consent: 
+(i) resell, lease, sublicense or distribute the Software to any person; 
+(ii) use the Software to provide third parties with managed services or provide remote access to the Software to any person or compete with Deci in any way; 
+(iii) represent that you possess any proprietary interest in the Software; 
+(iv) directly or indirectly, take any action to contest Deci's intellectual property rights or infringe them in any way; 
+(V) reverse-engineer, decompile, disassemble, alter, enhance, improve, add to, delete from, or otherwise modify, or derive (or attempt to derive) the technology or source code underlying any part of the Software; 
+(vi) use the Software (or any part thereof) in any illegal, indecent, misleading, harmful, abusive, harassing and/or disparaging manner or for any such purposes. Except as provided under the terms of any separate agreement between you and Deci, including the Terms of Use to the extent applicable, you may not use the Software for any commercial use, including in connection with any models used in a production environment. 
+
+DECI PROVIDES THE SOFTWARE "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS OF THE SOFTWARE BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 

Makefile

@@ -32,6 +32,7 @@ sweeper_test:
 
 # Here you define a list of notebooks we want to execute and convert to markdown files
 NOTEBOOKS_TO_RUN := src/super_gradients/examples/model_export/models_export.ipynb
+NOTEBOOKS_TO_RUN += src/super_gradients/examples/model_export/models_export_pose.ipynb
 NOTEBOOKS_TO_RUN += notebooks/what_are_recipes_and_how_to_use.ipynb
 NOTEBOOKS_TO_RUN += notebooks/transfer_learning_classification.ipynb
 NOTEBOOKS_TO_RUN += notebooks/how_to_use_knowledge_distillation_for_classification.ipynb

YOLONAS-POSE.md

@@ -0,0 +1,119 @@
+# YOLO-NAS-POSE
+### A Next-Generation, Pose Estimation Foundational Model generated by Deci’s Neural Architecture Search Technology
+
+Deci is thrilled to announce the release of a new object detection model, YOLO-NAS-POSE - a derivative of [YOLO-NAS](YOLONAS.md), 
+pose estimation architecture, providing superior real-time object detection capabilities and production-ready performance. 
+Deci's mission is to provide AI teams with tools to remove development barriers and attain efficient inference performance more quickly.
+
+![YOLO-NAS-POSE](documentation/source/images/yolo_nas_pose_frontier_t4.png)
+
+The new YOLO-NAS-POSE delivers state-of-the-art (SOTA) performance with the unparalleled accuracy-speed performance, outperforming other models such as YOLOv8-Pose, DEKR and others.
+
+Deci's proprietary Neural Architecture Search technology, [AutoNAC™](https://deci.ai/technology/), generated the architecture of YOLO-NAS-POSE model. 
+The AutoNAC™ engine lets you input any task, data characteristics (access to data is not required), inference environment and performance targets, 
+and then guides you to find the optimal architecture that delivers the best balance between accuracy and inference speed for your specific application. 
+In addition to being data and hardware aware, the AutoNAC engine considers other components in the inference stack, including compilers and quantization.
+
+| Model            | AP    | Latency (ms) |
+|------------------|-------|--------------|
+| YOLO-NAS N       | 59.68 | 2.35 ms      |
+| YOLO-NAS S       | 64.15 | 3.29 ms      |
+| YOLO-NAS M       | 67.87 | 6.87 ms      |
+| YOLO-NAS L       | 68.24 | 8.86 ms      |
+
+AP numbers in table reported for COCO 2017 Val dataset and latency benchmarked for 640x640 images on Nvidia T4 GPU.
+No flip-TTA was used.
+
+Similarly to YOLO-NAS, YOLO-NAS-POSE architecture employs quantization-aware blocks and selective quantization for optimized performance. 
+In fact YOLO-NAS-POSE is a derivative of YOLO-NAS and uses same backbone and neck as YOLO-NAS. 
+Only the head is different and is optimized by AutoNAC for pose estimation task. 
+That enables us to use transfer learning and fine-tune YOLO-NAS-POSE starting from YOLO-NAS weights.
+
+
+## Quickstart
+
+### Extract predicted poses 
+
+```python
+import super_gradients
+
+yolo_nas = super_gradients.training.models.get("yolo_nas_pose_l", pretrained_weights="coco_pose").cuda()
+model_predictions  = yolo_nas.predict("https://deci-pretrained-models.s3.amazonaws.com/sample_images/beatles-abbeyroad.jpg", conf=0.5).show()
+
+prediction = model_predictions[0].prediction # One prediction per image - Here we work with 1 image, so we get the first.
+
+bboxes = prediction.bboxes_xyxy # [Num Instances, 4] List of predicted bounding boxes for each object 
+poses  = prediction.poses       # [Num Instances, Num Joints, 3] list of predicted joints for each detected object (x,y, confidence)
+scores = prediction.scores      # [Num Instances] - Confidence value for each predicted instance
+```
+
+![YOLO-NAS-POSE Predict Demo](documentation/source/images/yolo_nas_pose_predict_demo.jpg)
+
+### Recipes
+
+We provide training recipies for training YOLO-NAS-POSE on COCO, CrowdPose and AnimalPose datasets. 
+
+#### COCO 2017
+
+* [super_gradients/recipes/coco2017_yolo_nas_pose_n.yaml](src/super_gradients/recipes/coco2017_yolo_nas_pose_n.yaml)
+* [super_gradients/recipes/coco2017_yolo_nas_pose_s.yaml](src/super_gradients/recipes/coco2017_yolo_nas_pose_s.yaml)
+* [super_gradients/recipes/coco2017_yolo_nas_pose_m.yaml](src/super_gradients/recipes/coco2017_yolo_nas_pose_m.yaml)
+* [super_gradients/recipes/coco2017_yolo_nas_pose_l.yaml](src/super_gradients/recipes/coco2017_yolo_nas_pose_l.yaml)
+
+
+## Additional resources
+
+<table>
+<tr>
+    <td>   
+        <a target="_blank" href="https://colab.research.google.com/drive/1O4N5Vbzv0rfkT81LQidPktX8RtoS5A40">
+            <img src="./documentation/assets/SG_img/colab_logo.png" /> Predict poses with YoloNAS Pose Model
+        </a>
+    </td>
+</tr>
+<tr>
+    <td>   
+        <a target="_blank" href="https://colab.research.google.com/drive/1agLj0aGx48C_rZPrTkeA18kuncack6lF">
+            <img src="./documentation/assets/SG_img/colab_logo.png" /> Fine-Tune YoloNAS Pose on AnimalPose dataset Notebook
+        </a>
+    </td>
+</tr>
+<tr>
+    <td>   
+        <a target="_blank" href="documentation/source/YoloNASPoseQuickstart.md"> 
+            Documentation: YOLO-NAS-POSE Quickstart 
+        </a>
+    </td>
+</tr>
+<tr>
+    <td>   
+        <a target="_blank" href="documentation/source/Recipes_Training.md"> 
+            Documentation: Recipies
+        </a>
+    </td>
+</tr>
+<tr>
+    <td>   
+        <a target="_blank" href="documentation/source/models_export_pose.md"> 
+            Documentation: YOLO-NAS-POSE Export 
+        </a>
+    </td>
+</tr>
+
+
+<tr>
+    <td>   
+        Join our <a target="_blank" href="https://discord.gg/2v6cEGMREN">
+             Discord Community
+        </a>
+    </td>
+</tr>
+</table>
+
+
+## LICENSE
+
+The YOLO-NAS-POSE model is available under an open-source license with pre-trained weights available for non-commercial use on SuperGradients, Deci's PyTorch-based, open-source, computer vision training library. 
+With SuperGradients, users can train models from scratch or fine-tune existing ones, leveraging advanced built-in training techniques like Distributed Data Parallel, Exponential Moving Average, Automatic mixed precision, and Quantization Aware Training.
+
+License file is available here: [YOLO-NAS-POSE WEIGHTS LICENSE](LICENSE.YOLONAS-POSE.md)

documentation/source/YoloNASPoseQuickstart.md

@@ -0,0 +1,45 @@
+# YOLO-NAS-POSE Quickstart
+<div>
+<img src="images/yolo_nas_pose_frontier_t4.png" width="750">
+</div>
+
+Deci’s leveraged its proprietary Neural Architecture Search engine (AutoNAC) to generate YOLO-NAS-POSE - a new object 
+detection architecture that delivers the world’s best accuracy-latency performance. 
+
+The YOLO-NAS-POSE model incorporates quantization-aware RepVGG blocks to ensure compatibility with post-training 
+quantization,  making it very flexible and usable for different hardware configurations.
+
+In this tutorial, we will go over the basic functionality of the YOLO-NAS-POSE model. 
+
+
+## Instantiate a YOLO-NAS-POSE Model
+
+```python
+from super_gradients.training import models
+from super_gradients.common.object_names import Models
+
+yolo_nas_pose = models.get(Models.YOLO_NAS_POSE_L, pretrained_weights="coco_pose")
+```
+
+## Predict
+
+```python
+prediction = yolo_nas_pose.predict("https://deci-pretrained-models.s3.amazonaws.com/sample_images/beatles-abbeyroad.jpg")
+prediction.show()
+```
+<div>
+<img src="images/yolo_nas_pose_predict_demo.jpg" width="750">
+</div>
+
+## Export to ONNX & TensorRT
+
+```python
+yolo_nas_pose.export("yolo_nas_pose.onnx")
+```
+
+Please follow our [Pose Estimation Models Export](models_export_pose.md) tutorial for more details.
+
+## Evaluation using pycocotools
+
+We provide example notebook to evaluate YOLO-NAS POSE using COCO protocol.
+Please check [Pose Estimation Models Export](https://github.com/Deci-AI/super-gradients/blob/master/notebooks/yolo_nas_pose_eval_with_pycocotools.ipynb) tutorial for more details.

documentation/source/model_zoo.md

@@ -92,9 +92,13 @@ All the available models are listed in the column `Model name`.
 
 ### Pretrained Pose Estimation PyTorch Checkpoints
 
-| Model           | Model Name      | Dataset     | Resolution | AP (No TTA / H-Flip TTA / H-Flip TTA+Rescoring) | Latency b1<sub>T4</sub> | Latency b1<sub>T4</sub> including IO | Latency (Production)**<sub>Jetson Xavier NX</sub> | 
-|-----------------|-----------------|-------------|------------|-------------------------------------------------|-------------------------|--------------------------------------|:-------------------------------------------------:|
-| DEKR_W32_NO_DC  | dekr_w32_no_dc  | COCO2017 PE | 640x640    | 63.08 / 64.96 / 67.32                           | 13.29 ms                | 15.31 ms                             | 75.99 ms                                          |
+| Model          | Model Name      | Dataset     | Resolution | AP (No TTA / H-Flip TTA / H-Flip TTA+Rescoring) | Latency b1<sub>T4</sub> | Latency b1<sub>T4</sub> including IO | Latency (Production)**<sub>Jetson Xavier NX</sub> | 
+|----------------|-----------------|-------------|------------|-------------------------------------------------|-------------------------|--------------------------------------|:-------------------------------------------------:|
+| DEKR_W32_NO_DC | dekr_w32_no_dc  | COCO2017 PE | 640x640    | 63.08 / 64.96 / 67.32                           | 13.29 ms                | 15.31 ms                             |                     75.99 ms                      |
+| YoloNAS POSE N | yolo_nas_pose_n | COCO2017 PE | 640x640    | 59.68 / N/A / N/A                               | N/A                     | 2.35 ms                              |                     15.99 ms                      |
+| YoloNAS POSE S | yolo_nas_pose_s | COCO2017 PE | 640x640    | 64.15 / N/A / N/A                               | N/A                     | 3.29 ms                              |                     21.01 ms                      |
+| YoloNAS POSE M | yolo_nas_pose_m | COCO2017 PE | 640x640    | 67.87 / N/A / N/A                               | N/A                     | 6.87 ms                              |                     38.40 ms                      |
+| YoloNAS POSE L | yolo_nas_pose_l | COCO2017 PE | 640x640    | 68.24 / N/A / N/A                               | N/A                     | 8.86 ms                              |                     49.34 ms                      |
 
 
 ## Implemented Model Architectures 
@@ -141,4 +145,6 @@ Devices[https://arxiv.org/pdf/1807.11164](https://arxiv.org/pdf/1807.11164)
 
 
 ### Pose Estimation
+
 - [HRNet DEKR](https://github.com/HRNet/HigherHRNet-Human-Pose-Estimation) - Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression [https://arxiv.org/pdf/2104.02300.pdf](https://arxiv.org/pdf/2104.02300.pdf)
+- YoloNAS Pose