Spikesorting quality of life helpers #910

samuelbray32 · 2024-04-03T16:22:51Z

Description

Helper to find spikesorting data

Adds the function SpikeSortingOutput().get_restricted_merge_ids()
Does the needed linking and traversing to allow user to get merge ids corresponding to restrictions from any stage of the spikesorting pipeline
Allows results from the v1 pipeline to be more easily restricted by common keys like nwb_file_name or interval_list_name without user knowledge of table depemdencies
Inelegant but functional address of searchability in Artifact detection enforcement in the v1 spikesorting pipeline #909

Resolves #221

Adds the functions SpikesortingOutput.get_sort_group_info()
Returns a table linking a spikesorting merge id to a Electrode and BrainRegion entry

Fixes #938

makes existing clusterless sorter parameter entries compatable with spikeinterface>=0.99.1

Checklist:

This PR should be accompanied by a release: no
(If release) I have updated the CITATION.cff
I have updated the CHANGELOG.md
I have added/edited docs/notebooks to reflect the changes

…rge id

edeno · 2024-04-10T17:58:12Z

@khl02007 and @CBroz1 could you review?

src/spyglass/spikesorting/spikesorting_merge.py

CBroz1 · 2024-04-10T22:49:50Z

src/spyglass/spikesorting/spikesorting_merge.py

+        sort_group_info : Table
+            Table linking a merge id to information about the electrode group.
+        """
+        source_table = source_class_dict[


source_class_dict was later added as a property of the Merge class such that this method could cls.source_class_dict and remove the variable at the top of the schema

src/spyglass/spikesorting/v0/spikesorting_curation.py

src/spyglass/spikesorting/v1/curation.py

CBroz1

The only change I would request is adding a demo of get_sort_group_info to the notebooks, v1 especially

samuelbray32 · 2024-04-11T18:27:32Z

New todo: Implement a similar helper function for FigUrlCuration in the v1 spikesorting pipeline

…he restriction

review-notebook-app · 2024-04-19T20:41:10Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

samuelbray32 · 2024-04-19T20:42:58Z

New todo: Implement a similar helper function for FigUrlCuration in the v1 spikesorting pipeline

Leaving this to a case for general restrict_upstream_merge (see #930)

CBroz1

Thanks, Sam. One super minor comment, then we're good to go. Thanks for taking the time to run these notebooks

notebooks/py_scripts/10_Spike_SortingV1.py

* Add spyglass version to created analysis nwb files (#897) * Add sg version to created analysis nwb files * update changelog * Change existing source script to spyglass version (#900) * Add pynapple support (#898) * Preliminary code * Add retrieval of file names * Add get_nwb_table function * Update docstrings * Update CHANGELOG.md * Hot fixes for clusterless `get_ahead_behind_distance` and `get_spike_times` (#904) * Squeeze results * Make method and not class method * Update CHANGELOG.md * fix bugs in fetch_nwb (#913) * Check for entry in merge part table prior to insert (#922) * check for entry in merge part table prior to insert * update changelog * Kachery fixes (#918) * Prioritize datajoint filepath for getting analysis file abs_path * remove deprecated kachery tables * update changelog * fix lint --------- Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu> Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com> * remove old tables from init (#925) * Fix improper uses of strip (#929) Strip will remove leading characters * Update CHANGELOG.md * Misc Issues (#903) * #892 * #885 * #879 * Partial address of #860 * Update Changelog * Partial solve of #886 - Ask import * Fix failing tests * Add note on order of inheritace * #933 * Could not replicate fill_nan error. Reverting except clause * Export logger (#875) * WIP: rebase Export process * WIP: revise doc * ✅ : Generate working export script * Cleanup: Expand notebook, migrate export process from graph class to export * Revert dj_chains related edits * Update changelog * Revise doc * Address review comments #875 * Remove walrus in eval * prevent log on preview * Fix arg order on fetch, iterate over restr * Add upstream analysis files during cascade. Address false positive fetch * Avoid regen file list on revisit node * Bump Export.Table.restr to mediumblob * Revise Export.Table uniqueness to include export_id * Spikesorting quality of life helpers (#910) * add utitlity function for finding spikesorting merge ids * add option to select v1 sorts that didn't go through artifact detection * add option to return merge keys as dicts for future restrictions * Add tool to get brain region and electrode info for a spikesorting merge id * update changelog * style cleanup * style cleanup * fix restriction bug for curation_id * account for change or radiu_um argument name in spikeinterface * only do joins with metric curastion tables if have relevant keys in the restriction * Update tutorial to use spikesorting merge table helper functions * fix spelling * Add logging of AnalysisNwbfile creation time and file size (#937) * Add logging for any func that creates AnalysisNwbfile * Migrate create to top of respective funcs * Use pathlib for file size. Bump creation time to top of in spikesort * Clear pre_create_time on create * get/del -> pop * Log when file accessed (#941) * Add logging for any func that creates AnalysisNwbfile * Fix bug on empty delete in merge table (#940) * fix bug on empty delete in merge table * update changelog * fix spelling --------- Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu> * Remove master restriction * Part delete takes restriction from self --------- Co-authored-by: Samuel Bray <sam.bray@ucsf.edu> Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com> Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu> Co-authored-by: Eric Denovellis <edeno@bu.edu>

* Create class for group parts to help propagate deletes * spelling * update changelog * Part delete edits (#946) * Add spyglass version to created analysis nwb files (#897) * Add sg version to created analysis nwb files * update changelog * Change existing source script to spyglass version (#900) * Add pynapple support (#898) * Preliminary code * Add retrieval of file names * Add get_nwb_table function * Update docstrings * Update CHANGELOG.md * Hot fixes for clusterless `get_ahead_behind_distance` and `get_spike_times` (#904) * Squeeze results * Make method and not class method * Update CHANGELOG.md * fix bugs in fetch_nwb (#913) * Check for entry in merge part table prior to insert (#922) * check for entry in merge part table prior to insert * update changelog * Kachery fixes (#918) * Prioritize datajoint filepath for getting analysis file abs_path * remove deprecated kachery tables * update changelog * fix lint --------- Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu> Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com> * remove old tables from init (#925) * Fix improper uses of strip (#929) Strip will remove leading characters * Update CHANGELOG.md * Misc Issues (#903) * #892 * #885 * #879 * Partial address of #860 * Update Changelog * Partial solve of #886 - Ask import * Fix failing tests * Add note on order of inheritace * #933 * Could not replicate fill_nan error. Reverting except clause * Export logger (#875) * WIP: rebase Export process * WIP: revise doc * ✅ : Generate working export script * Cleanup: Expand notebook, migrate export process from graph class to export * Revert dj_chains related edits * Update changelog * Revise doc * Address review comments #875 * Remove walrus in eval * prevent log on preview * Fix arg order on fetch, iterate over restr * Add upstream analysis files during cascade. Address false positive fetch * Avoid regen file list on revisit node * Bump Export.Table.restr to mediumblob * Revise Export.Table uniqueness to include export_id * Spikesorting quality of life helpers (#910) * add utitlity function for finding spikesorting merge ids * add option to select v1 sorts that didn't go through artifact detection * add option to return merge keys as dicts for future restrictions * Add tool to get brain region and electrode info for a spikesorting merge id * update changelog * style cleanup * style cleanup * fix restriction bug for curation_id * account for change or radiu_um argument name in spikeinterface * only do joins with metric curastion tables if have relevant keys in the restriction * Update tutorial to use spikesorting merge table helper functions * fix spelling * Add logging of AnalysisNwbfile creation time and file size (#937) * Add logging for any func that creates AnalysisNwbfile * Migrate create to top of respective funcs * Use pathlib for file size. Bump creation time to top of in spikesort * Clear pre_create_time on create * get/del -> pop * Log when file accessed (#941) * Add logging for any func that creates AnalysisNwbfile * Fix bug on empty delete in merge table (#940) * fix bug on empty delete in merge table * update changelog * fix spelling --------- Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu> * Remove master restriction * Part delete takes restriction from self --------- Co-authored-by: Samuel Bray <sam.bray@ucsf.edu> Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com> Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu> Co-authored-by: Eric Denovellis <edeno@bu.edu> * Fix linting --------- Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu> Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com> Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu> Co-authored-by: Eric Denovellis <edeno@bu.edu>

samuelbray32 added 5 commits April 2, 2024 11:51

add utitlity function for finding spikesorting merge ids

18791dd

add option to select v1 sorts that didn't go through artifact detection

651b506

add option to return merge keys as dicts for future restrictions

cc60dd2

Add tool to get brain region and electrode info for a spikesorting me…

612f817

…rge id

update changelog

d41f2cc

edeno requested review from CBroz1 and khl02007 April 3, 2024 22:25

CBroz1 reviewed Apr 10, 2024

View reviewed changes

CBroz1 requested changes Apr 10, 2024

View reviewed changes

CBroz1 mentioned this pull request Apr 10, 2024

General-purporse key lookup tool(s) for buried primary keys #923

Closed

samuelbray32 added 6 commits April 12, 2024 11:50

style cleanup

f5bf802

style cleanup

b178e56

fix restriction bug for curation_id

5eb298b

account for change or radiu_um argument name in spikeinterface

e57701b

only do joins with metric curastion tables if have relevant keys in t…

1fbf93c

…he restriction

Update tutorial to use spikesorting merge table helper functions

59c4f70

samuelbray32 requested a review from CBroz1 April 19, 2024 20:43

edeno added the spike sorting label Apr 19, 2024

CBroz1 approved these changes Apr 19, 2024

View reviewed changes

notebooks/py_scripts/10_Spike_SortingV1.py Outdated Show resolved Hide resolved

fix spelling

2507499

CBroz1 merged commit 696dee7 into master Apr 19, 2024
7 checks passed

CBroz1 deleted the spikesorting_helpers branch April 19, 2024 23:12

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spikesorting quality of life helpers #910

Spikesorting quality of life helpers #910

samuelbray32 commented Apr 3, 2024 •

edited

Loading

edeno commented Apr 10, 2024

CBroz1 Apr 10, 2024

CBroz1 left a comment

samuelbray32 commented Apr 11, 2024

review-notebook-app bot commented Apr 19, 2024

samuelbray32 commented Apr 19, 2024

CBroz1 left a comment

Spikesorting quality of life helpers #910

Spikesorting quality of life helpers #910

Conversation

samuelbray32 commented Apr 3, 2024 • edited Loading

Description

Checklist:

edeno commented Apr 10, 2024

CBroz1 Apr 10, 2024

Choose a reason for hiding this comment

CBroz1 left a comment

Choose a reason for hiding this comment

samuelbray32 commented Apr 11, 2024

review-notebook-app bot commented Apr 19, 2024

samuelbray32 commented Apr 19, 2024

CBroz1 left a comment

Choose a reason for hiding this comment

samuelbray32 commented Apr 3, 2024 •

edited

Loading