PtndArrayEx.multiboxDetection() implementation

[juliangamble]

Description

The big goal here is to implement PtNDArrayEx.multiBoxDetection() (currently throws NotImplementException) so that you can run PikachuTraining with Pytorch and so you can run SSD Training on M1 Macs.

The implementation for this is derived from:
https://github.com/apache/mxnet/blob/master/src/operator/contrib/multibox_detection.cc
and
https://github.com/apache/mxnet/blob/master/src/operator/contrib/multibox_detection-inl.h

This implementation is based on implementing the unit test on the C++ side and comparing the steps internally.

[lanking520]

Is that possible to achieve below code under matrix calculation through using existing NDArray functions?

[juliangamble]

Thanks, that’s helpful. I’ll take a look.

…

On Thu, 7 Sep 2023 at 2:34 am, Qing Lan ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In engines/pytorch/pytorch-engine/src/main/java/ai/djl/pytorch/engine/PtNDArrayEx.java <#2769 (comment)>: > + float temp = pClsProb[j * numAnchors + i]; + if (temp > score) { + score = temp; + id = j; + } + } + + if (id > 0 && score < threshold) { + id = 0; + } + + // [id, prob, xmin, ymin, xmax, ymax] + outputs[i][0] = id - 1; + outputs[i][1] = score; + int offset = i * 4; + float[] pAnchorRow4 = new float[4]; Is that possible to achieve below code under matrix calculation through using existing NDArray functions? — Reply to this email directly, view it on GitHub <#2769 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB734FANB5BWQTUINR4WDDXZCQXPANCNFSM6AAAAAA4KH6HHQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[zachgk]

I made a few updates to merge this in for our next release. This includes some cleanup and a few bug fixes (including with freezing parameters in PtModel loading outside the scope of this PR).

One thing to note is I got the tests running in SingleShotDetectionTest. Right now, it passes the SingleShotDetectionTest.testLoadPredict using the new code. However, it does not fully support SSD. It won't pass SingleShotDetectionTest.testLoadTrain or other SSD training without also implementing PtNDArrayEx.multiBoxTarget

[codecov-commenter]

Codecov Report

Attention: 1375 lines in your changes are missing coverage. Please review.

Comparison is base (bb5073f) 72.08% compared to head (61597e5) 72.21%.
Report is 885 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2769      +/-   ##
============================================
+ Coverage     72.08%   72.21%   +0.12%     
- Complexity     5126     7111    +1985     
============================================
  Files           473      702     +229     
  Lines         21970    31680    +9710     
  Branches       2351     3284     +933     
============================================
+ Hits          15838    22879    +7041     
- Misses         4925     7237    +2312     
- Partials       1207     1564     +357     

Files	Coverage Δ
...ava/ai/djl/inference/streaming/StreamingBlock.java	100.00% <100.00%> (ø)
api/src/main/java/ai/djl/metric/Dimension.java	100.00% <100.00%> (ø)
api/src/main/java/ai/djl/metric/Unit.java	100.00% <100.00%> (ø)
api/src/main/java/ai/djl/modality/audio/Audio.java	100.00% <100.00%> (ø)
api/src/main/java/ai/djl/modality/cv/Image.java	69.23% <ø> (-4.11%)	⬇️
...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java	76.00% <ø> (ø)
...ava/ai/djl/modality/cv/output/DetectedObjects.java	96.29% <100.00%> (+1.29%)	⬆️
...rc/main/java/ai/djl/modality/cv/output/Joints.java	71.42% <100.00%> (ø)
.../main/java/ai/djl/modality/cv/output/Landmark.java	100.00% <ø> (ø)
...i/djl/modality/cv/transform/RandomResizedCrop.java	94.11% <100.00%> (+5.22%)	⬆️
... and 226 more

... and 372 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[juliangamble]

Thank you Zach. I’ll give this a try. JG

…

On Thu, 28 Sep 2023 at 3:04 pm, Zach Kimberg ***@***.***> wrote: I made a few updates to merge this in for our next release. This includes some cleanup and a few bug fixes (including with freezing parameters in PtModel loading outside the scope of this PR). One thing to note is I got the tests running in SingleShotDetectionTest. Right now, it passes the SingleShotDetectionTest.testLoadPredict using the new code. However, it does not fully support SSD. It won't pass SingleShotDetectionTest.testLoadTrain or other SSD training without also implementing PtNDArrayEx.multiBoxTarget — Reply to this email directly, view it on GitHub <#2769 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB734ACDRJVE7UU63H2KBDX4UAPDANCNFSM6AAAAAA4KH6HHQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>