Feature/sg 1060 yolo nas pose release pr to add datasets and metric #1506

BloodAxe · 2023-10-05T14:32:48Z

Also pretty big PR:

Introduce PoseEstimationPredictions that holds model's decoded predictions in a strongly-typed and documented 😎 dataclass
Introduce a clear API interface - AbstractPoseEstimationPostPredictionCallback which sets the signature for all pose estimation post-prediction implementations (DEKR for now) and for YoloNASPose (To be merged next)
Updates to predict API for pose estimation models to reflect these changes above
New abstract class AbstractPoseEstimationDataset
PoseEstimationMetrics now can work with both old-style way of passing gt targets and using samples
Improvement to visualization of poses. Added PoseVisualization class with static methods similary to DetectionVisualization

src/super_gradients/training/metrics/pose_estimation_metrics.py

Louis-Dupont · 2023-10-07T15:02:13Z

src/super_gradients/training/metrics/pose_estimation_metrics.py

+        gt_joints,
+        gt_bboxes,
+        gt_areas,
+        gt_iscrowd,


Typing missing. I think it's this:

gt_joints: List[np.ndarray], gt_bboxes: Optional[List[np.ndarray]], gt_areas: Optional[List[np.ndarray]], gt_iscrowd: Optional[List[np.ndarray]],

Thanks. Fixed in d27f624

src/super_gradients/training/metrics/pose_estimation_metrics.py

Louis-Dupont · 2023-10-07T15:07:14Z

src/super_gradients/training/pipelines/pipelines.py

+        result = self.post_prediction_callback(model_output)
        predictions = []
-        for poses, scores, image in zip(all_poses, all_scores, model_input):
+        for r, image in zip(result, model_input):


Not important, you can skip, but other ways of naming could be

result -> batch_results or batch_predictions

r -> sample_results or sample_predictions

Fixed in 86b515f, thanks :)

Louis-Dupont · 2023-10-07T15:35:56Z

src/super_gradients/training/utils/visualization/pose_estimation.py

+class PoseVisualization:
+    @classmethod
+    def draw_poses(


Why not simply a function draw_poses ?
Not fundamentally against the class + class_method, but I'm not really sure it adds clarity

Did it for the purpose to stay in line with DetectionVisualization class.

src/super_gradients/training/datasets/pose_estimation_datasets/coco_utils.py

Louis-Dupont

Looks good, I like especially the new AbstractPoseEstimationPostPredictionCallback interface with PoseEstimationPredictions , it strongly adds clarity in what is hapenning.
I'm just slightly concerned about risk of this introducing an internal breaking change. Are you sure our tests cover well what has been changed ?

BloodAxe · 2023-10-07T17:07:57Z

Looks good, I like especially the new AbstractPoseEstimationPostPredictionCallback interface with PoseEstimationPredictions , it strongly adds clarity in what is hapenning.
I'm just slightly concerned about risk of this introducing an internal breaking change. Are you sure our tests cover well what has been changed ?

Pretty much sure. New class does not change existing base class for pose estimation, so it does not change existing API.

…ded a bunch of docstrings

* Introduce sample-centric keypoint transforms * Cleanup leftovers * Fixed way of checking transforms that require additional samples * Docstrings * :attr -> :param * Added docs clarifying behavior of mosaic & mixup * Added docs clarifying behavior of mosaic & mixup * Improved tests * Additional docstrings & typing annotations * Added missing additional_samples_count field * KeypointsRemoveSmallObjects * KeypointsRemoveSmallObjects * Feature/sg 1060 yolo nas pose release pr to add datasets and metric (#1506) * Cherry pick changes to post-prediction, visualization and metric * Remove unwanted references to new datasets * Remove YoloNASPoseCollateFN * Fixed unit test * Improve clarify of bbox format by giving it more explicit name and added a bunch of docstrings * Improve variable names

* Introduce sample-centric keypoint transforms * Cleanup leftovers * Fixed way of checking transforms that require additional samples * Docstrings * :attr -> :param * Added docs clarifying behavior of mosaic & mixup * Added docs clarifying behavior of mosaic & mixup * Improved tests * Additional docstrings & typing annotations * Added missing additional_samples_count field * KeypointsRemoveSmallObjects * KeypointsRemoveSmallObjects * Feature/sg 1060 yolo nas pose release pr to add datasets and metric (#1506) * Cherry pick changes to post-prediction, visualization and metric * Remove unwanted references to new datasets * Remove YoloNASPoseCollateFN * Fixed unit test * Improve clarify of bbox format by giving it more explicit name and added a bunch of docstrings * Improve variable names * Squashed changes with YoloNASPose & Loss * Remove print statement * Fixed attribute name that was not renamed * Improve docstrings to use 'Num Keypoints' instead of magic number 17 * Fixed PoseNMS export to work with custom number of keypoints * Added docstrings * Simplify forward/forward_eval * Simplify forward/forward_eval * _insert_heads_list_params * Added tests * Refactor the way we generate usage instructions. Should be easier to follow and update * Improve docstrings * Improved docstrings * Improved docstrings * Improved docstrings * Improved docstrings * Rename bboxes -> bboxes_xyxy * Fixed instructions text * Improve efficiency of training --------- Co-authored-by: Shay Aharon <80472096+shaydeci@users.noreply.github.com>

* Introduce sample-centric keypoint transforms * Cleanup leftovers * Fixed way of checking transforms that require additional samples * Docstrings * :attr -> :param * Added docs clarifying behavior of mosaic & mixup * Added docs clarifying behavior of mosaic & mixup * Improved tests * Additional docstrings & typing annotations * Added missing additional_samples_count field * KeypointsRemoveSmallObjects * KeypointsRemoveSmallObjects * Feature/sg 1060 yolo nas pose release pr to add datasets and metric (#1506) * Cherry pick changes to post-prediction, visualization and metric * Remove unwanted references to new datasets * Remove YoloNASPoseCollateFN * Fixed unit test * Improve clarify of bbox format by giving it more explicit name and added a bunch of docstrings * Improve variable names * Squashed changes with YoloNASPose & Loss * Remove print statement * Fixed attribute name that was not renamed * Improve docstrings to use 'Num Keypoints' instead of magic number 17 * Fixed PoseNMS export to work with custom number of keypoints * Added docstrings * Simplify forward/forward_eval * Simplify forward/forward_eval * _insert_heads_list_params * Added tests * Refactor the way we generate usage instructions. Should be easier to follow and update * Dataset & Visualization callback * Improve docstrings * Improved docstrings * Improved docstrings * Improved docstrings * Improved docstrings * Rename bboxes -> bboxes_xyxy * Put missing parameter include_empty_samples: True * Clarify in docs that COCO annotation should have one category and used TypeFactory to resolve enum type * Clarify remove_duplicate_annotations --------- Co-authored-by: Shay Aharon <80472096+shaydeci@users.noreply.github.com>

* crowdpose_yolo_nas_pose_s * crowdpose_yolo_nas_pose_s * crowdpose_yolo_nas_pose_s * crowdpose_yolo_nas_pose_s * coco2017_yolo_nas_pose_s_ema_less_mosaic * coco2017_yolo_nas_pose_s_less_mosaic * coco2017_yolo_nas_pose_s_ema_less_mosaic_higher_final_lr_fp32 * coco2017_yolo_nas_pose_s_ema_less_mosaic_higher_final_lr_fp32 * coco2017_yolo_nas_pose_s_ema_less_mosaic_lr_focal * shared head * YoloNASPoseBoxesPostPredictionCallback * New head design * Another recipe with less zoom out, no crowd images * Another recipe with less zoom out, no crowd images * Another recipe with less zoom out, no crowd images * coco2017_yolo_nas_pose_shared_s_ema_less_mosaic_lr_bce_local * coco2017_yolo_nas_pose_shared_s_ema_less_mosaic_lr_bce_local * Update scores * Cleanup old configs, keep one config that gives best AP score * Shortened recipe * coco2017_yolo_nas_pose_shared_s_384_short * Tune short recipe * Tune short recipe * Tune short recipe * Tune short recipe * coco2017_yolo_nas_pose_s_local * Update settings of crowd_annotations_action to mask_as_normal since this is the setting we got the best result with * coco2017_yolo_nas_pose_shared_s_local * Update default params * Update default params * Update DEKR recipe * M variant * M variant * Put more correct min_deltha * Put more correct min_deltha * Put more correct min_deltha * Adding placeholders for YOLO-NAS-POSE * Rename detection model export test file * Adding export API support for pose estimation * Adding export API support for pose estimation * Added tmp hack * multiply_by_pose_oks * assigner_multiply_by_pose_oks * ExperimentImprove visualization * Update CrowdPose dataset * Crowdpose * Crowdpose * Crowdpose * crowdpose_yolo_nas_pose_s_no_crowd_no_ema_local * Lower LR * Proxy recipe * crowdpose_yolo_nas_pose_s_proxy * crowdpose_yolo_nas_pose_s_proxy * crowdpose_yolo_nas_pose_s_proxy * New architectures * Fix WANDB params * Fix WANDB params * Fix WANDB params * Fix WANDB params * New architectures * M * L * M * coco2017_yolo_nas_pose_l_resume * coco2017_yolo_nas_pose_m_resume * Added fix to _is_more_extreme which would ensure callback would not crash if observed loss/metric is NaN/Inf * Reduce LR * Reduce LR * Change EMA paramass * Export and scores * Export * Fix bug of not saving simplified model * Optimize head return types for better inference efficiency * Metrics * Yolo NAS Pose N * Only EarlyStop no batch visualization * coco2017_yolo_nas_pose_l_no_ema * Only EarlyStop no batch visualization * Removing old architectures * Notebook for evaluation on COCO * Remove unnecessary recipies * Simplify the metric -> pass entire sample to the metric * Simplify recipe * coco2017_yolo_nas_pose_n_resume * Simplify recipe * Transforms overhaul & refactoring * Transforms overhaul & refactoring * Remove KeypointsImageToTensor transform - this will be done in collate fn * Fix collate fn to do image layout change HWC->CHW * Attempt to optimize efficiency * Attempt to optimize efficiency * Attempt to optimize efficiency * Attempt to optimize efficiency * Attempt to optimize efficiency * Attempt to optimize efficiency * Attempt to optimize efficiency * Attempt to optimize efficiency * Attempt to optimize efficiency * Attempt to optimize efficiency * Refactor sample * Simplify recipe * New keypoint transform * Lower dropout rates & heavy augs * crowdpose_yolo_nas_pose_s * Improve visualization of pose gt by showing whether it is crowd target or not * Make convert_to_tensor a bit more efficient by avoiding creating a tensor on cpu and then moving it to target device. * Compute metric on CPU (Surprisingly it is faster, since amount of data & compute is not that big so data transfer takes more time than compute) * Improve speed of computing focal loss * New batch of training experiments * New batch of training experiments * Introduce sample-centric keypoint transforms * Cleanup leftovers * Update numbers * Add benchmark results * Fixed way of checking transforms that require additional samples * Docstrings * :attr -> :param * Added docs clarifying behavior of mosaic & mixup * Added docs clarifying behavior of mosaic & mixup * Improved tests * Additional docstrings & typing annotations * Focal-EIOU loss * Added missing additional_samples_count field * Fixed predict implementation for pose * Added docstrings * KeypointsRemoveSmallObjects * KeypointsRemoveSmallObjects * Metric class to use data samples * New dataset classes * Reverting back old files to keep & update dataset recipies * Simplified rescoring dataset params YAML file by using coco_pose_common_dataset_params defaults * Removed old docs * Remove space * Introduce AbstractPoseEstimationPostPredictionCallback interface and move PoseEstimationPredictions to a proper place * Cherry pick changes to post-prediction, visualization and metric * Remove unwanted references to new datasets * Remove YoloNASPoseCollateFN * Make heavy augs a default training param for M & L * Remove dropout * Fixed unit test * Update YoloNAS-M score * Feature/sg 1060 yolo nas pose release pr to add datasets and metric (#1506) * Cherry pick changes to post-prediction, visualization and metric * Remove unwanted references to new datasets * Remove YoloNASPoseCollateFN * Fixed unit test * Improve clarify of bbox format by giving it more explicit name and added a bunch of docstrings * Improve variable names * Document YoloNASPose loss * Squashed changes with YoloNASPose & Loss * Fixed attribute name that was not renamed * Remove print statement * Remove print statement * Fixed attribute name that was not renamed * Improve docstrings to use 'Num Keypoints' instead of magic number 17 * Fixed PoseNMS export to work with custom number of keypoints * Remove outdated test * Update recipes * Added docstrings * Simplify forward/forward_eval * Simplify forward/forward_eval * Remove any2device_no_grad * _insert_heads_list_params * _insert_heads_list_params * Update ExtremeBatchCaseVisualizationCallback * Document visualization callback better * Added YoloNASPose test * Added tests * Refactor the way we generate usage instructions. Should be easier to follow and update * Revert rename * Dataset & Visualization callback * Improve docstrings * Improved docstrings * Improved docstrings * Improved docstrings * Improved docstrings * Rename bboxes -> bboxes_xyxy * Example colab for evaluation of ONNX model * Rename bboxes -> bboxes_xyxy * Fixed instructions text * Improve efficiency of training * Remove files * Update numbers * Update animal pose * Added integration tests for YoloNASPose * Fix bug in replace head * Add pretrain weights * Added export notebook example * Update integration test * Updating branch for merge * Updating branch for merge * Remove AnimalPoseDataset * Update markdown text * Update markdown text * Revert * Added license * Improve debug text in transfer_weights * Revert * Cleanup recipes * Revert * Cleanup recipes * Revert * Update notebooks * Update mkdocs to include pose estimation * Update docs * Revert test * Put back YAML file * Added check to print license for YoloNAS-POSE * Fixed bug in _pad_image that did not support pad_value=(R,B,G) input * Added images & updated links to notebooks * Added images & updated links to notebooks * Added pop dataset_class from dataloader params * Update quickstart * Added missing rgb2bgr conversion * Added missing rgb2bgr conversion * Disable visualization of samples by default * Added docstrings * Updated additional resoruces section with link to recipies docs * successor -> derivative * Re-run notebook * Fixed recipe to code test * Re-run notebook --------- Co-authored-by: Shay Aharon <80472096+shaydeci@users.noreply.github.com>

BloodAxe added 3 commits October 5, 2023 16:55

Cherry pick changes to post-prediction, visualization and metric

ccf553e

Remove unwanted references to new datasets

9a87609

Remove YoloNASPoseCollateFN

fd30331

BloodAxe requested review from shaydeci, ofrimasad and Louis-Dupont as code owners October 5, 2023 14:32

Fixed unit test

4311ca1

Louis-Dupont reviewed Oct 7, 2023

View reviewed changes

src/super_gradients/training/metrics/pose_estimation_metrics.py Outdated Show resolved Hide resolved

Louis-Dupont reviewed Oct 7, 2023

View reviewed changes

src/super_gradients/training/metrics/pose_estimation_metrics.py Show resolved Hide resolved

Louis-Dupont reviewed Oct 7, 2023

View reviewed changes

src/super_gradients/training/datasets/pose_estimation_datasets/coco_utils.py Show resolved Hide resolved

Louis-Dupont reviewed Oct 7, 2023

View reviewed changes

BloodAxe added 2 commits October 8, 2023 22:53

Improve clarify of bbox format by giving it more explicit name and ad…

d27f624

…ded a bunch of docstrings

Improve variable names

86b515f

BloodAxe merged commit fd41b71 into feature/SG-1060-yolo-nas-pose-release Oct 9, 2023
4 of 5 checks passed

BloodAxe deleted the feature/SG-1060-yolo-nas-pose-release-pr-to-add-datasets-and-metric branch October 9, 2023 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/sg 1060 yolo nas pose release pr to add datasets and metric #1506

Feature/sg 1060 yolo nas pose release pr to add datasets and metric #1506

BloodAxe commented Oct 5, 2023

Louis-Dupont Oct 7, 2023 •

edited

Loading

BloodAxe Oct 8, 2023

Louis-Dupont Oct 7, 2023 •

edited

Loading

BloodAxe Oct 8, 2023

Louis-Dupont Oct 7, 2023

BloodAxe Oct 8, 2023

Louis-Dupont left a comment

BloodAxe commented Oct 7, 2023

Feature/sg 1060 yolo nas pose release pr to add datasets and metric #1506

Feature/sg 1060 yolo nas pose release pr to add datasets and metric #1506

Conversation

BloodAxe commented Oct 5, 2023

Louis-Dupont Oct 7, 2023 • edited Loading

Choose a reason for hiding this comment

BloodAxe Oct 8, 2023

Choose a reason for hiding this comment

Louis-Dupont Oct 7, 2023 • edited Loading

Choose a reason for hiding this comment

BloodAxe Oct 8, 2023

Choose a reason for hiding this comment

Louis-Dupont Oct 7, 2023

Choose a reason for hiding this comment

BloodAxe Oct 8, 2023

Choose a reason for hiding this comment

Louis-Dupont left a comment

Choose a reason for hiding this comment

BloodAxe commented Oct 7, 2023

Louis-Dupont Oct 7, 2023 •

edited

Loading

Louis-Dupont Oct 7, 2023 •

edited

Loading