-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nix export of tracking results #1068
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #1068 +/- ##
===========================================
+ Coverage 69.94% 70.19% +0.25%
===========================================
Files 130 131 +1
Lines 22496 22751 +255
===========================================
+ Hits 15735 15971 +236
- Misses 6761 6780 +19
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for creating the adaptor. I know this PR is still listed as a draft, but I had a few comments. We will also need to add a test in test_formats.py. I really like that you kept nix
as an optional import. Let me know if there's anything I can help with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a vulnerability in how the track_map
and skeleton_map
are defined.
The track_map
is a bigger concern as tracks often use non-unique names and are instead identified by the object reference. Another way of identifying Track
s is by their index in the Labels.tracks
list.
The comments pertaining to skeleton_map
are less critical as SLEAP does not currently support multi-Skeleton
projects (although we have some infrastructure written to scale SLEAP in that direction). Like Track
s, Skeleton
s are not required to have unique names and can instead be identified by the object's index in Labels.skeletons
.
Thanks again for creating the adaptor (and writing a test)! Let me know if there is anything I can help with.
sleap/io/format/nix.py
Outdated
def can_write_filename(cls, filename: str) -> bool: | ||
"""Returns whether this adaptor can write format of this filename.""" | ||
raise NotImplementedError |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A problem we have run into is people trying to handle .pkg.slp
s as .slp
s. Can we add a quick check here to check that the file
ends in .slp
instead of .pkg.slp
? While the .pkg.slp
contains all the pose information, the references to the videos are stored within the actual .pkg.slp
and no longer reference the original video - this would result in an inaccurate nix file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure, I follow. I understood this method as checking whether or not the adaptor can write to a file with the specified extension. The sleap_analysis adaptor seems to delegate to the respective function in adaptor.py that does this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, I double checked this and was interpreting it the wrong way (i.e. filename being the name of the loaded labels object to convert). No action needed.
sleap/io/format/nix.py
Outdated
def skeleton_map(source: Labels): | ||
skel_map = {} | ||
for s in source.skeletons: | ||
if s.name in skel_map: | ||
continue | ||
skel_map[s.name] = len(skel_map) | ||
return skel_map | ||
|
||
def node_map(source: Labels): | ||
n_map = {} | ||
for skeleton in source.skeletons: | ||
for n in skeleton.nodes: | ||
if n.name in n_map: | ||
continue | ||
n_map[n.name] = len(n_map) | ||
return n_map |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seeing as you only keep skeletons with unique names
if s.name in skel_map:
continue
should we also only be considering the nodes of skeletons that are listed in skel_map
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed this to work with skeletons that are not unique regarding their name only. Nodes now also store the skeleton they are part of, if they are part of one. Accepting now all nodes irrespective of their assignment to a skeleton
sleap/io/format/nix.py
Outdated
for t in source.tracks: | ||
if t.name in track_map: | ||
continue | ||
track_map[t.name] = len(track_map) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While we hope that users will uniquely label their Track
s, it is not a requirement. Assuming the name is the unique identifier could lead to a lot of missed data.
sleap/io/format/nix.py
Outdated
if inst.track is not None: | ||
track[index] = track_map[inst.track.name] | ||
else: | ||
track[index] = -1 | ||
skeleton[index] = skeleton_map[inst.skeleton.name] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on the way track_map
was created (assuming Track.name
is a unique identifier), it is possible that inst.track.name
is not in the track_map
. Likewise for skeleton_map
.
sleap/io/format/nix.py
Outdated
fnames = set([n.name for n in inst.nodes]) | ||
nnames = set(list(node_map.keys())) | ||
missing = nnames.difference(fnames) | ||
|
||
for n, p in zip(inst.nodes, inst.points): | ||
positions[index, :, node_map[n.name]] = np.array([p.x, p.y]) | ||
for m in missing: | ||
positions[index, :, node_map[m]] = np.array([np.nan, np.nan]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Flagging this because bit of code assumes that all the nodes used by inst
are listed in node_map
. No action required, just ensure node_map
continues to contain all Node
s from any Skeleton
used in the project.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch! Since we now store all nodes whether or not they are part of a skeleton, this should no longer be an issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know you haven't re-requested a review yet, but I peaked and...
The dictionary maps are looking great! Two main concerns:
- Open reference and writing to file outside of
try/except
(orwith open ___as __
) - Changing
Skeleton.__eq__
behavior
sleap/nn/inference.py
Outdated
dists = tf.sqrt(tf.reduce_sum(tf.math.square(dists), axis=-1)) # reduce over xy | ||
dists = tf.sqrt(tf.reduce_sum(dists**2, axis=-1)) # reduce over xy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Local black reformats this to to dists**2
, but then Github Actions' black expects it as dists ** 2
. So I just changed it to tf.math.square
to get rid of this annoying miscommunication between the linters.
dists = tf.sqrt(tf.reduce_sum(tf.math.square(dists), axis=-1)) # reduce over xy | |
dists = tf.sqrt(tf.reduce_sum(dists**2, axis=-1)) # reduce over xy | |
dists = tf.sqrt(tf.reduce_sum(tf.math.square(dists), axis=-1)) # reduce over xy |
dev_requirements.txt
Outdated
click==8.0.4 | ||
nixio>=1.5.3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove, no need to add if already in requirements.txt
sleap/skeleton.py
Outdated
def __eq__(self, other): | ||
if not isinstance(other, Skeleton): | ||
return False | ||
return self.name == other.name | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this method. There are already countless SLEAP projects out there with same-name skeletons which are not equal to each other... For equality, use the Skeleton.matches
method instead
I may have missed it, but do you use the ==
on Skeleton
in your PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is actually something I wanted to ask. Tried to use matches
instead but got an error saying that other
does not have a _graph
member.
It is in the dictionaries, that the eq is needed. My first attempt was to implement __eq__
by calling matches
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry, my bad, apparently cannot reproduce my issue anymore, removed the __eq__
sleap/io/format/nix.py
Outdated
nix_file = create_file(filename, source_path, video) | ||
write_data(nix_file.blocks[0], source_object, video) | ||
try: | ||
print(" done") | ||
except Exception as e: | ||
print(f"\n\t Writing failed with error message {e}!") | ||
finally: | ||
nix_file.close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be safer to have all these lines in the try
block:
try:
nix_file = create_file(filename, source_path, video)
write_data(nix_file.blocks[0], source_object, video)
print(" done")
but that assumes that no error occurred in the create_file
function and a nix_file
was returned. So maybe something like this:
nix_file = create_file(filename, source_path, video) | |
write_data(nix_file.blocks[0], source_object, video) | |
try: | |
print(" done") | |
except Exception as e: | |
print(f"\n\t Writing failed with error message {e}!") | |
finally: | |
nix_file.close() | |
nix_file = None | |
try: | |
nix_file = create_file(filename, source_path, video) | |
write_data(nix_file.blocks[0], source_object, video) | |
print(" done") | |
except Exception as e: | |
print(f"\n\t Writing failed with error message {e}!") | |
finally: | |
if nix_file is not None: | |
nix_file.close() |
or something else that safely closes the file while references are open.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch, moved the write outside the try for testing, will go into the try again.
@roomrys ready for review again, I guess. Thank you for your comments! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi! Jumping in here with a review since this seems almost done.
Just requested some minor changes.
Nix seems like a really cool project btw! Have you guys interacted with NWB at all?
dev_requirements.txt
Outdated
click==8.0.4 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to pin click to a specific version? It's a pretty common dependency which might break others so we'd rather not pin it unless necessary (or at a minimum use a more permissive version fence like click>=8.0.4,<9.0.0
).
Also this should probably be in requirements.txt
not dev_requirements.txt
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the click requirement was not added by me. At some point I added nixio
to the dev_requirements for the tests to pass during the CI checks. I guess I left the line-break when removing nixio again, reverted to the state without line-break
Nix seems like a really cool project btw! Have you guys interacted with NWB at all?
Thanks, and yes we have been in touch with them and before NWB version 1, NIX was one of the candidate formats to be used. The NIX data model is very generic and that makes it harder to read than domain specific names for the data structures you want to store. Otoh, it keeps the number of different structures low. Eventually, NWB decided to go a slightly different way ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We specified the version of click in dev_requirements when it broke the linting workflow in github actions. Off the topic of this PR, but I can check if we can remove the versioning now that a new release of click is out (>8.1.0).
not sure why this occurred. The nix adaptor cannot save instances that are not related to a frame.
make Node.name and Node.weight instance variables instead of class variables
allow for non-unique entries in node, track and skeleton. Extend node map to store the skeleton it is part of
Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com>
Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com>
Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com>
Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. I am making the change in my comment, and then we will be good to merge.
@roomrys thanks a lot for all your work on sleap and this PR. it would be totally fine with me if you decide to squash it. It has grown to be quite a few commits ;) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Feeling confident we covered our basis. Changes since last review:
|
* GUI Training: Use hidden params from loaded config (#1053) * Add optional unragging arg to model export (#1054) * Fix config option to `split_by_inds` (#1060) * Convert training, validation, and test to Labels object * Add test for split_by_inds * Use Labels.extract instead of Labels(List[LabeledFrames]) * Tracking: robust assignment of the best score to an instance (#1062) * Set max instances for top down models (#1070) * Add optional unragging arg to model export * Add option to set max instances for multi-instance models * Fix test * Don't create instances during inference if no points were found (#1073) * Don't create instances during inference if no points were found * Add points check for all predictors * Fix single instance predictor logic and test * Add tests for all predictors Co-authored-by: roomrys <38435167+roomrys@users.noreply.github.com> * Add one-line fix to VideoWriterSkyvideo (#1082) * Fix parser for sleap-export (#1085) * Refactor commands to load project as `AppCommand`s (#1098) * Add working Proof of Concept * Create command class for loading project * Split `LoadProjectFile` as a subclass of `LoadLabelsObject` * Reroute last existing reference * Remove debugging code * Flexibly resize input layer of `tf.keras.Model` upon loading trained model (#1084) * Add initial implementation (auto output stride problematic) * Add to load_predictor test (error when auto-compute output stride) * Use output stride from config instead of auto-computing * Fix output-stride/padding modulo error and do not resize on export * Fix resizing bug in multi-class predictors * Non-functional clean-up * Rename new input layer to original name * Add inference integration test * Minimize config surgery, generalize layer iteration Co-authored-by: Talmo Pereira <talmo@salk.edu> * Add Option to Make Trail Shade Darker/Lighter (#1103) * Make trails 60% darker than track color * Add menu option for shade of trails * Remove unexpected indent (fat-fingered) * Create signal that updates plot instead of removing and replotting items (#1134) * Create signal that updates plot instead of redrawing * Remove debug code * Non-functional self-review changes * Fix symmetric skeletons (via table input) (#1136) Ensure variable initialized before calling it * Nix export of tracking results (#1068) * [io] export tracking results to NIX file * [io] nix added to export filter only if available * [nixio] refactor, add scores link data as mtag * [nixio] speeding up export by chunked writing * [nixio] rename point score to node score * [nixio] fix missing dimension descriptor for node scores * [export analysis] support multiple formats also for bulk export * [nixio] export centroid, some documentation * [nixio] fix double dot before filename suffix * [nixio] fix bug when not all nodes were found * [nixio] housekeeping * [nix] add nix analyis output format to convert * [nix] tiny fix, catch file write error and properly close file * [inference] main takes optional args. Can be imported to run inference form scripts * [convert] simplify if else structure and outfile handling for analysis export * [nix] use pathlib instead of os * [nix] catch if there are instances with a None frame_idx ... not sure why this occurred. The nix adaptor cannot save instances that are not related to a frame. * [nix] move checks to top of write function * [nix] use absolute imports * [nix] use black to reformat * [commands] revert qtpy import and apply code style * [convert] use absolute imports, apply code style * [commands]fix imports * [inference/nix]fix linter complaint, adjust nix types for scores * [nix] add test case for nix export format * [nix] extended testing, some modifications of adaptor * [skeleton] add __eq__ to Skeleton ... make Node.name and Node.weight instance variables instead of class variables * [nix] add nixio to requirements, remove unused nix_available, ... allow for non-unique entries in node, track and skeleton. Extend node map to store the skeleton it is part of * [nix] make the linter happy * [Node] force definition of a name Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * [nix] use getattr for getting grayscale information Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * [nix] fixes according to review * [convert] break out of loop upon finding the video Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * [commands.py] use pathilb instead of splitting filename Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * [dev requirements] remove linebreak at last line * [skeleton] revert attribute creation back to original * [nix] break lines in class documentation * Ensure all file references are closed * Make the linter happy * Add tests for ExportAnalysis and (docs for) sleap-convert Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Fix body vs symmetry subgraph filtering (#1142) Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Handle changing backbones in training editor GUI (#1140) Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Added scaling functionality for both the instances and bounding box. (#1133) * Create VisibleBoundingBox class. * Added instance scaling functionality in addition to bounding box scaling functionality. * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Update sleap/gui/widgets/video.py Co-authored-by: Talmo Pereira <talmo@salk.edu> * Added new testing for scaling operation and simplified VisibleBoundingBox class code. * Added type hinting to the scaling padding and removed erroneous bounding rect initialization. Co-authored-by: Talmo Pereira <talmo@salk.edu> Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Add better error message for top down (#1121) * Add better error message for top down * Add test for error message * Raise different error, fix test * Hotfix for video save #1098 (#1148) * Add a hotfix for #1098 * WIP: Add test for detecting changes on load * Finialize change on load test * Remove unused imports * Skip test if on windows since files are being used in parallel * Add central padding to SizeMatcher (#1129) * add center padding to size matcher * add test for center padding * add ensure_float option to inference layer * reformat resizing and test_resizing * Remove redundant operation * Replace existing constants with fixtures --------- Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Added MoveNet as an external model reference (#1141) * add center padding to size matcher * add test for center padding * add ensure_float option to inference layer * reformat resizing and test_resizing * add MoveNet as an external model inference * add the movenet to the from_model_paths * add tests * add comments to movenet predictor * add tensorflow_hub to the requirements.txt * modified default video path * resolved most of the comments except expanding the predictor * expanded Predictor.from_model_paths function to include any pre-trained models. * add test_load_model * added from_trained_models in class Predictor and modified test_load_model for it. * modified test_load_model to be more generalized. * moved pretrained model from Predictor.from_trained_model to Predictor.from_model_paths and added a test for it. * Fix Predictor.from_model_paths and tests * Rename load_movenet_model to make_model_movenet * minor clean-up * Remove redundant operation * Replace existing constants with fixtures * Handle loading movenet models via load_model API * Clean-up doc strings --------- Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Resumable Training (#1130) * add resume training functionality * add testing function for resume training functionality * linting black * Resumable Training 2 - CLI Options (#1131) * add cli options for resumable training * add test for cli resume training * black linting for cli resumable training * simplify resumable checkpoint CLI fn to a single CLI arg (#1132) * simplify resumable checkpoint CLI fn to a single CLI arg * Adam/resumable training 3 (#1150) * correct path of labels_path for test_training * add resume training to gui * add train from scratch message * Add finishing touches to resumable training PR (#1150) (#1168) * Refactor/update 'use trained' and 'resume training' checkbox logic * Simplify checkbox logic and reset model field when resume training * Reset checkboxes upon changing config selection * Handle case for updating TrainingEditor when sender is not a checkbox * Add complete state space GUI test for checkboxes * Finish combobox test * Test that form is reset * Remove straggling TODO --------- Co-authored-by: roomrys <38435167+roomrys@users.noreply.github.com> Co-authored-by: jimzers <jimzersml@gmail.com> --------- Co-authored-by: jimzers <jimzersml@gmail.com> Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Return trainer from sleap-train and check that trainer configured correctly * Add CLI documentation for website --------- Co-authored-by: jimzers <jimzersml@gmail.com> Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Small (final?) revisions and fix test * Revert changes to fixture --------- Co-authored-by: jimzers <jimzersml@gmail.com> Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * GenericTableModel/View improvements (#1163) * [dataviews] GenericTableModel/View improvements ... * GenericTableView got a new argument specifying whether the ellipsis for long cell content should be right (old behavior, default) or left useful for long content such as the filenames in the video table. * GenericTableView uses all the space that available to the table. * The model's data function returns the full cell content to be shown as tool tip text. * [gui/app] set the ellipsis to be on the left for long table contents --------- Co-authored-by: Liezl Maree <38435167+roomrys@users.noreply.github.com> * Add Skeleton Templates (#1122) * Update docs: change 'M1' to 'Apple Silicon' (#1188) * Bump to 1.3.0a0 (#1189) --------- Co-authored-by: sheridana <asheridan@salk.edu> Co-authored-by: getzze <getzze@gmail.com> Co-authored-by: Talmo Pereira <talmo@salk.edu> Co-authored-by: Jan Grewe <jngrewe@googlemail.com> Co-authored-by: Sean Afshar <84047864+sean-afshar@users.noreply.github.com> Co-authored-by: Jiaying Hsu <72051972+jiayinghsu@users.noreply.github.com> Co-authored-by: Adam Lee <38634441+jimzers@users.noreply.github.com> Co-authored-by: jimzers <jimzersml@gmail.com> Co-authored-by: Jan Grewe <jan.grewe@g-node.org> Co-authored-by: Aaditya Prasad <78439225+aaprasad@users.noreply.github.com>
Description
Dear SLEAP team, thanks for developing sleap. This pr
inference.py
usable from custom code line by simply passing the args tomain
.For 1 I added a new
NixAdaptor
class insleep.io
and support using it from the GUI as well as theconvert
CLI.Reading nix-exported files works nicely using either the generic nixio python package or via the more domain specific nixtrack package (work in progress). Exporting to NIX is fully optional, the option is not even offered in the file export dialog if
nixio
is not installed in the environment. Hence, nixio is not added to the requirements.Types of changes
Does this address any currently open issues?
no
Outside contributors checklist
Thank you for contributing to SLEAP!
❤️