Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Updates to the alignment entity #298

Merged
merged 6 commits into from
Oct 2, 2024
Merged

feat: Updates to the alignment entity #298

merged 6 commits into from
Oct 2, 2024

Conversation

manasaV3
Copy link
Contributor

@manasaV3 manasaV3 commented Oct 1, 2024 •
edited
Loading

Relates to

Description

Changes to the config schema:

  • Updates the is_canonical -> is_portal_standard
  • Adds the method_type
  • Updates the spelling of volume_dimension

Changes to the configs:

  • Adds migration handler for alignment.method_type in the config. It follows the following conditions:
Everything that is FIDUCIAL is fiducial_based
Everything that mentions AreTomo as reconstruction software is projection matching
If reconstruction_software=IMOD and NON_FIDUCIAL is patch_tracking
Else undefined
  • Removes trailling whitespaces in the configs

S3 Workflows:

  • Adds "files" field to the alignment metadata. This lists all the files associated to this alignment
  • Adds get_name_and_path class method to the AlignmentImporter. This allows the DestinationFilterFinder to reference an existing Alignment.

@manasaV3 manasaV3 changed the title Updates to the alignment entity feat: Updates to the alignment entity Oct 1, 2024
jgadling
jgadling reviewed Oct 1, 2024
View reviewed changes
ingestion_tools/scripts/importers/alignment.py Outdated
return None
@classmethod
def get_name_and_path(cls, metadata: dict, name: str, path: str, results: dict[str, str]) -> [str, str, dict]:
paths = {filname: filname for filname in metadata.get("files", [])}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: filename vs filname

Bento007
Bento007 approved these changes Oct 1, 2024
View reviewed changes
ingestion_tools/dataset_configs/template.yaml
format: OPTIONAL ENUM (IMOD, ARETOMO)
method_type: OPTIONAL ENUM (fiducial_based, patch_tracking, projection_matching)
is_portal_standard: OPTIONAL, BOOLEAN (DEFAULT FALSE)
offset: OPTIONAL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for adding these

ingestion_tools/scripts/schema_migration/upgrade.py
@@ -34,7 +36,7 @@ def upgrade_config(data: dict[str, Any]) -> dict[str, Any]:


def has_changes(file, config):
with open(file, 'r') as file:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think our formatters disagree

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wdym? 😅

schema/core/v1.1.0/metadata.yaml
@@ -817,7 +817,7 @@ classes:
offset:
description: *desc_alignment_offset
range: AlignmentOffset
volume_dimesion:
volume_dimension:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

jgadling
jgadling approved these changes Oct 1, 2024
View reviewed changes
Copy link
Contributor

@jgadling jgadling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after the typo is fixed

@manasaV3 manasaV3 merged commit 604635f into main Oct 2, 2024
8 checks passed
@manasaV3 manasaV3 deleted the mvenkatakrishnan/align_updates branch October 2, 2024 17:19
manasaV3 added a commit that referenced this pull request Oct 4, 2024
@manasaV3
feat: Updates to the alignment entity (#298)
c5770ea 
* Updates to schema and template

* Fixing tests

* Adding files to alignments metadata

* Update the alignment configs method type

* Fixing typo

* Adding undefined alignment_method_type
manasaV3 added a commit that referenced this pull request Oct 14, 2024
@manasaV3 @jgadling
feat: Adding multi tomogram relationships (#302)
7204736 
* Updating tomograms

* updates to tomogram processing

* updates to tomogram key photo filename

* updates to tomogram neuroglance filepath

* cleaning up tomogram and related entities

* Updating tiltseries with identifiers

* Updating alignment for tomogram relations

* Adding alignment metadatapath to annotations and tomograms

* Updating the paths

* Updating the paths

* Updating failed tests

* Working tests clean up needed

* fix: workaround for docker compose bugs. (#295)

* fix: support filtering entities by related object id's (#296)

* fix: support filtering entities by related object id's

* chore: add seed script. (#297)

* feat: Updates to the alignment entity (#298)

* Updates to schema and template

* Fixing tests

* Adding files to alignments metadata

* Update the alignment configs method type

* Fixing typo

* Adding undefined alignment_method_type

* chore: Documenting the Jensen config generation (#293)

* Adds comments to jensen config generation

* Adding guardrails

* Clean up

* fix: method_name

* chore: Update footer of release please PR's. (#299)

* Adding tests

* cleaning up the paths

* Cleaning up paths

* fix tests

* Updating tomogram id generation

* Updating viz_config generation

* Making id directories

* Making id directories

* Making annotation id directory

* Cleaning up paths with id

* Updating neuroglancer precompute for dir structure

* Updating neuroglancer config for dir structure

* Updating raw tilt import to tiltseries_id directory

* Fixing dependency on existing metadata

* Adding documentation comment

---------

Co-authored-by: Jessica Gadling <jgadling@chanzuckerberg.com>
jgadling added a commit that referenced this pull request Oct 21, 2024
@manasaV3 @jgadling
feat: Script to migrate the directory structure (#311)
d061960 
* Updating tomograms

* updates to tomogram processing

* updates to tomogram key photo filename

* updates to tomogram neuroglance filepath

* cleaning up tomogram and related entities

* Updating tiltseries with identifiers

* Updating alignment for tomogram relations

* Adding alignment metadatapath to annotations and tomograms

* Updating the paths

* Updating the paths

* Updating failed tests

* Working tests clean up needed

* fix: workaround for docker compose bugs. (#295)

* fix: support filtering entities by related object id's (#296)

* fix: support filtering entities by related object id's

* chore: add seed script. (#297)

* feat: Updates to the alignment entity (#298)

* Updates to schema and template

* Fixing tests

* Adding files to alignments metadata

* Update the alignment configs method type

* Fixing typo

* Adding undefined alignment_method_type

* chore: Documenting the Jensen config generation (#293)

* Adds comments to jensen config generation

* Adding guardrails

* Clean up

* fix: method_name

* chore: Update footer of release please PR's. (#299)

* Adding tests

* cleaning up the paths

* Cleaning up paths

* fix tests

* Updating tomogram id generation

* Updating viz_config generation

* Making id directories

* Making id directories

* Making annotation id directory

* Cleaning up paths with id

* Updating neuroglancer precompute for dir structure

* Updating neuroglancer config for dir structure

* Updating raw tilt import to tiltseries_id directory

* Migration script

* Update for tomograms

* Adding support for alignments

* Migrating annotation precompute

* Migrating collection_metadata

* Migrating rawtilt and gains

* Clean up

* Clean up

* Clean up

* Enabling the move

* Adding metadata updates

* Uncommenting deletion

* Adding check before move

* Adding wdl

* minor fixes

* Updating enqueue script

* Updating annotation file names

* Updating the tomogram key photo path

* Fixing tests

* Fix key_photo migration

* Handling default alignments

* fix path standardization when bucket name not in path

* Lint.

---------

Co-authored-by: Jessica Gadling <jgadling@chanzuckerberg.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jgadling jgadling jgadling approved these changes

@Bento007 Bento007 Bento007 approved these changes

@uermel uermel Awaiting requested review from uermel

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@manasaV3 @jgadling @Bento007