Add multi-animal support to DLCLive #72

jeylau · 2022-06-09T19:12:49Z

Check DLCRNet model freezing
Add dependency on DLC??

MultiAnimalDLCLive can be instantiated e.g. as:

from dlclive.dlclive import MultiAnimalDLCLive

live = MultiAnimalDLCLive(
    'path_to_snapshot',
    n_animals=3,
    n_multibodyparts=12,
    track_method="box",
    sim_threshold=0.3,
)
live.init_inference(allow_growth=True)
animals, unique = live.get_pose(frame)

Fixes #57

sneakers-the-rat

Minor revisions, mostly to do with keeping API consistent between DLCLive and MultiAnimalDLCLive. To me it would be nice to have these in a flat hierarchy: root DLCLive object, then subclasses that implement the different variations of inference -- as is it's getting a bit messy with the root object having a bunch of if switches that are sort of mutually incompatible with the subclass. Seems like a headache to maintain, but I ain't doing it so i'm not going to sit here and demand you to do it for me ;)

The stuff that I think should be changed are

the things that match the signatures between the two classes,
matching behavior for cropping and dynamic_cropping (by splitting them out to a separate method in the parent class rather than duplicating the code in the child class)
adding documentation (in __init__ and any changes to get_pose (like if pose is a different shape or something)

the rest are 'would be nice' and might be helpful for maintenance/readability.

sneakers-the-rat · 2022-06-10T23:23:09Z

dlclive/dlclive.py

+        pcutoff: float = 0.5,
+        display_radius: int = 3,
+        display_cmap: str = "bmy",
+    ):


needs docstring! Since inherits from DLCLive but the signature differs, need to also document changes in calling convention. Also need to document any new attrs

sneakers-the-rat · 2022-06-10T23:24:20Z

dlclive/dlclive.py

+class MultiAnimalDLCLive(DLCLive):
+    def __init__(
+        self,
+        model_path,


type hint!
model_path: Union[Path, str]
n_animals: int
n_multibodyparts: int

sneakers-the-rat · 2022-06-10T23:24:40Z

dlclive/dlclive.py

+    def __init__(
+        self,
+        model_path,
+        n_animals,


are these not specified in the config file?

sneakers-the-rat · 2022-06-10T23:25:12Z

dlclive/dlclive.py

+        model_path,
+        n_animals,
+        n_multibodyparts,
+        track_method: str = "box",


this could be a literal
track_method: Literal['box', 'ellipse']

sneakers-the-rat · 2022-06-10T23:30:45Z

dlclive/dlclive.py

+        display: typing.Union[bool, Display] = False,
+        pcutoff: float = 0.5,
+        display_radius: int = 3,
+        display_cmap: str = "bmy",


missing arguments from parent signature:

model_type

precision

tf_config

cropping

dynamic

Any reason? We should try and make uniform API if possible.

Since these are all in the DLCLive.PARAMETERS tuple, this will cause .parameterization to error. This also removes control over the process_frame method.

Two approaches: add the missing arguments, or else remove all of them and accept a **kwargs that gets passed to the parent class.

sneakers-the-rat · 2022-06-10T23:47:25Z

dlclive/dlclive.py

+            self.display.display_frame(frame, self.pose)
+
+        if self.resize is not None:
+            self.pose[0][..., :2] *= 1 / self.resize


seems like this should happen for n poses, right? rather than a hard 2? but i'm not sure what the structure of pose is here.

sneakers-the-rat · 2022-06-10T23:47:52Z

dlclive/dlclive.py

+            self.pose[1][:, :2] *= 1 / self.resize
+
+        if self.processor:
+            self.pose = self.processor.process(self.pose, **kwargs)


does the processor need to be different for maDLC pose?

sneakers-the-rat · 2022-06-10T23:48:30Z

dlclive/dlclive.py

+        if self.processor:
+            self.pose = self.processor.process(self.pose, **kwargs)
+
+        return self.pose


does the close method need to be updated at all? again not sure how the maDLC stuff works.

sneakers-the-rat · 2022-06-10T23:49:43Z

dlclive/dlclive.py

+            raise DLCLiveError("No frame provided for live pose estimation")
+
+        frame = self.process_frame(frame)
+        data_dict = predict_multianimal.predict_batched_peaks_and_costs(


not sure what happens from here to L627, some comments would be nice!

sneakers-the-rat · 2022-06-10T23:52:41Z

dlclive/dlclive.py

+
+        return pose
+
+    def get_pose(self, frame=None, **kwargs):


now might also be a good time to add docstrings to these methods too -- their function is straightforward enough, but documenting what goes on in them, etc. either here or documenting the arguments/attrs in the __init__ docstring that change the function here.

sneakers-the-rat · 2022-06-11T01:38:44Z

Hell ya. thanks for handling this. I can't push to your fork, so I made a few initial hopefully uncontroversial amendments in madlc branch. We can either use that as the PR source or else you can just merge that into yours.

Tests fail because scipy and deeplabcut are now imported but not added to package requirements in pyproject.toml, added (deeplabcut^2.2.0 and scipy^1.8.0 for python>=3.8 and >=1.7.0,<1.8.0 for python>=3.7.1,<3.8)
Version not incremented per semver, incremented to 1.1.0 because of addition of new backwards-compatible feature
Added CHANGELOG.md to describe changes to library since we don't have docs to speak of. Would you mind filling that out/adding some usage notes to the README?
Added CONTRIBUTING.md to formalize the update process a tiny tiny bit: just stuff like increment version, write docs, tests, etc.
Can we get a test for this? it can be as barebones as the existing one (just checking that it runs, rather than confirming the values of the run) imo. This will save us headaches down the line and prevent accidental regressions in any future changes. The test action just runs dlc-live-test (dlclive.check_install.check_install.main), so you could add it as an additional function there called within main(). Like the existing test, that would require
- Downloading a pretrained model & project
- downloading a test video
- running inference on the test video

thomasilmer · 2022-07-15T12:55:20Z

@jeylau Are you still working on this? Otherwise I could attempt to make the requested changes.

AlexEMG · 2022-08-01T11:32:56Z

Also consider: https://forum.image.sc/t/dlc-live-multianimal-what-is-affinity-in-inferenceutil-py-empty-assemblies-bug-or-bad-model/69972

jeylau · 2022-11-10T10:53:13Z

Hey @thomasilmer, I'll update this PR soon, but please feel free to contribute too; you'd be very welcome!

MMathisLab · 2023-01-08T15:10:46Z

@jeylau let's finish this :)

thomasilmer · 2023-01-18T14:22:17Z

Hey @jeylau, I took a look at the requested changes and must admit that I quickly realized that I am not familiar enough with the internal workings of DeepLabCut to contribute anything meaningful at this moment.

My apologies.

frankiechang123 · 2023-02-28T04:07:31Z

Hi @jeylau @MMathisLab !I'm currently trying to apply maDLC models on live videos. Just wondering what is the status of supporting maDLC models in DLCLive? I'm also willing to contribute to help the development!

thomasilmer · 2023-08-16T13:22:40Z

I have tried running this code locally and there were some changes I had to make to prevent exceptions from happening when indexes went out of bounds. What is the preferred method for contributing these changes @jeylau ?

Additionally, I noticed that track_method='ellipse' seems to give more consistent poses than track_method='box'. Is there any reason for this or is this heavily dependent on the type of data you process?

Finally, there is some strange behavior when analysing videos where animals enter the frame from one side and exit on the other side (have not tested on other footage). The animals are lost after some time and we need to run 'init_inference' again to get better results again.

MMathisLab · 2023-08-16T14:40:17Z

Very happy to accept a PR - so please just edit in your forked copy and open a pull request to us and tag us for review, thanks!!

Cc @n-poulsen

itisomar · 2023-08-17T07:22:03Z

Hi guys @gkane26 @jeylau @MMathisLab @sneakers-the-rat , first things first, a huge thanks and shout out to the DLC team and the people who are contributing to this wonderful project in any way whatsoever.

As I implemented the MultiAnimalDLCLive class, I encountered several issues that are known to you guys and the great DLC community.

I'll list the issues which I have encountered below:

1 - init_inference() : When you call init_inference(allow_growth=True) before get_pose(frame), the pose estimation seems to work well. However, in the long run, the pose estimation drops drastically in frames. In some cases, it fails to detect any body parts. One workaround is to call init_inference(allow_growth=True) whenever that happens, which asks for a lot of memory, causing the frame rate to drop to zero.”
As some have mentioned in the following forum: https://forum.image.sc/t/live-inference-for-multi-animal-deeplabcut/73509.diff

2 - max_age : It needs to be set as high as ‘100’ for my project, in order for MultiAnimalDLCLive to be able to detect body parts. Otherwise, MultiAnimalDLCLive is unable to detect body parts and the detection rate drops drastically, near to zero detections.

3 - track_method : The ‘Box’ tracking method detects far fewer body parts in frames compared to the ‘Ellipse’ tracking method

jeylau added 2 commits June 9, 2022 21:06

Fix typo

b5391df

Draft MultiAnimalDLCLive

8500867

jeylau added the enhancement New feature or request label Jun 9, 2022

jeylau requested review from sneakers-the-rat, gkane26 and MMathisLab June 9, 2022 19:12

jeylau marked this pull request as draft June 9, 2022 19:12

jeylau changed the title ~~Add multi-animal support to DLCLive~~ WIP: Add multi-animal support to DLCLive Jun 9, 2022

jeylau mentioned this pull request Jun 9, 2022

Unable to retrieve multiple predictions from maDLC trained network #57

Open

Fix resizing

5b223d2

jeylau changed the title ~~WIP: Add multi-animal support to DLCLive~~ Add multi-animal support to DLCLive Jun 10, 2022

jeylau marked this pull request as ready for review June 10, 2022 07:18

sneakers-the-rat reviewed Jun 10, 2022

View reviewed changes

jeylau marked this pull request as draft August 29, 2022 09:14

Minor fixes

79cb784

Handle absence of assemblies

61e3dc1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multi-animal support to DLCLive #72

Add multi-animal support to DLCLive #72

jeylau commented Jun 9, 2022 •

edited

Loading

sneakers-the-rat left a comment

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat Jun 10, 2022

sneakers-the-rat commented Jun 11, 2022

thomasilmer commented Jul 15, 2022 •

edited

Loading

AlexEMG commented Aug 1, 2022

jeylau commented Nov 10, 2022

MMathisLab commented Jan 8, 2023

thomasilmer commented Jan 18, 2023

frankiechang123 commented Feb 28, 2023

thomasilmer commented Aug 16, 2023 •

edited

Loading

MMathisLab commented Aug 16, 2023

itisomar commented Aug 17, 2023

Add multi-animal support to DLCLive #72

Are you sure you want to change the base?

Add multi-animal support to DLCLive #72

Conversation

jeylau commented Jun 9, 2022 • edited Loading

sneakers-the-rat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sneakers-the-rat commented Jun 11, 2022

thomasilmer commented Jul 15, 2022 • edited Loading

AlexEMG commented Aug 1, 2022

jeylau commented Nov 10, 2022

MMathisLab commented Jan 8, 2023

thomasilmer commented Jan 18, 2023

frankiechang123 commented Feb 28, 2023

thomasilmer commented Aug 16, 2023 • edited Loading

MMathisLab commented Aug 16, 2023

itisomar commented Aug 17, 2023

jeylau commented Jun 9, 2022 •

edited

Loading

thomasilmer commented Jul 15, 2022 •

edited

Loading

thomasilmer commented Aug 16, 2023 •

edited

Loading