Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix voxel offset for Neuroglancer Precomputed datasets #7019

Merged
merged 11 commits into from
May 2, 2023
Merged

Fix voxel offset for Neuroglancer Precomputed datasets #7019

merged 11 commits into from
May 2, 2023

Conversation

fm3
Copy link
Member

@fm3 fm3 commented Apr 25, 2023 •
edited
Loading

  • use neuroglancer precomputed “voxel offset” field to set bounding box when exploring
  • use the same voxel offset when loading data
  • adapts computeChunkIndices to handle negative total offsets
  • note that the offset in chunk can still become negative, but copyRange already handles that 😎
  • change some of the error handling from exceptions to Fox.failure
  • catch exceptions during chunk decompression (related: Error handling in DatasetArray: move away from Exceptions, use Fox/flatMap #6906)

Steps to test:

  • Explore minnie dataset (segmentation and em, talk to us for dataset uri)
  • Load data, should look plausible, layers should match

Issues:

  • Updated changelog
  • Needs datastore update after deployment

@fm3 fm3 assigned fm3 and frcroth Apr 25, 2023
@fm3 fm3 requested a review from frcroth April 26, 2023 12:16
@fm3 fm3 marked this pull request as ready for review April 26, 2023 12:16
@fm3 fm3 changed the title WIP: Fix voxel offset precomputed Fix voxel offset for Neuroglancer Precomputed datasets Apr 26, 2023
frcroth
frcroth reviewed Apr 28, 2023
View reviewed changes
webknossos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/ChunkUtils.scala Outdated
Comment on lines 19 to 20
val toIndexRaw = (selectedOffset(dim) + selectedShape(dim) - 1) / arrayChunkSize(dim)
val toIndexClampedToBbox = Math.max(smallestPossibleIndex, Math.min(largestPossibleIndex, toIndexRaw))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know if to matches start, perhaps use end in names here?

webknossos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/ChunkUtils.scala Outdated
currentIdx(depthIdx) = start(depthIdx)
depthIdx -= 1
chunkIndices(i) = currentIndex.clone
var dimIdx = nDims - 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
var dimIdx = nDims - 1
var dimIndex = nDims - 1

Is also changed above, should be changed here as well

...nossos-datastore/app/com/scalableminds/webknossos/datastore/services/BinaryDataService.scala Outdated
Comment on lines 95 to 96
/*logger.warn(
s"Bucket loading for layer ${request.dataLayer.name} of dataset ${request.dataSource.id} at ${readInstruction.bucket} failed with error: $msg")*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove

@fm3 fm3 requested a review from frcroth May 2, 2023 07:15
@fm3 fm3 enabled auto-merge (squash) May 2, 2023 07:42
@fm3 fm3 merged commit eab96fe into master May 2, 2023
@fm3 fm3 deleted the fix-voxel-offset-precomputed branch May 2, 2023 07:43
@frcroth frcroth mentioned this pull request May 9, 2023
Merged
2 tasks
hotzenklotz added a commit that referenced this pull request May 17, 2023
@hotzenklotz
Merge branch 'master' of github.com:scalableminds/webknossos into emp…
c9ee334 
…ty-list-drawings

* 'master' of github.com:scalableminds/webknossos: (25 commits)
  Fix issues with styling in dark mode on login page (#7052)
  Fix nightly by setting missing token (#7048)
  Release 23.05.1 (#7042)
  DRY types in update_actions.ts (#7036)
  Remove some spammy logging from backend (#7039)
  Use zarr string fill values (#7017)
  Fix voxel offset for Neuroglancer Precomputed datasets (#7019)
  Log when user is activated (#7027)
  Fix exception in applying UpdateTreeGroupVisibility skeleton action (#7037)
  Fix organization storage layouting (#7034)
  Update docker compose commands + dev install readme (#7002)
  Add segment groups (#6966)
  Add screenshot nightly test for wkorg (#7030)
  Workaround for WebGL crash for datasets with many segmentation layers (#6995)
  Fix download of public annotation, include access ctx in user cache key (#7025)
  Fix that changing a segment color could lead to a crash (#7000)
  Add more error chaining to annotation download (#7023)
  Guard against NaNs in shader (#7018)
  Store editable mappings in multiple fossildb columns+keys (#6903)
  Context action to move tree to group (#7005)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@frcroth frcroth frcroth approved these changes

Assignees

@fm3 fm3

@frcroth frcroth

Labels
backend bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Neuroglancer Precomputed voxel_offset is not respected for sharded datasets
2 participants
@fm3 @frcroth