feat: Adding multi tomogram relationships #302

manasaV3 · 2024-10-04T18:42:55Z

Relates to chanzuckerberg/cryoet-data-portal#1053

Description

There are general updates to the file paths. More on this can be found here.

New file structure:
.
├── DatasetID/
│   ├── ExperimentalRun/
│   │   ├── Alignment/
│   │   │   └── <alignment-id>/
│   │   │       ├── alignment_metadata.json
│   │   │       └── <name-in-source>.(aln|xf|tlt|json|xtilt|com)
│   │   ├── Frames/
│   │   │   ├── <name-in-source>.(eer|mrc|mrc.bz|??)
│   │   │   ├── <name-in-source>.mdoc
│   │   │   └── frames_metadata.json
│   │   ├── Gains/
│   │   │   └── <name-in-source>.(mrc|gain|??)
│   │   ├── Reconstructions/
│   │   │   └── VoxelSpacingXX.XX/
│   │   │       ├── Annotations/
│   │   │       │   └── <anno-id>/
│   │   │       │       ├── annotation_files
│   │   │       │       └── <annotation_key>.json (metadata file)
│   │   │       ├── Images/
│   │   │       │   ├── <tomo-id>-expanded.png
│   │   │       │   ├── <tomo-id>-original.png
│   │   │       │   ├── <tomo-id>-thumbnail.png
│   │   │       │   └── <tomo-id>-snapshot.png
│   │   │       ├── NeuroglancerPrecompute/
│   │   │       │   ├── <anno-id>-shape_precompute
│   │   │       │   └── <tomo-id>-neuroglancer-config.json
│   │   │       └── Tomograms/
│   │   │           └── <tomo-id>/
│   │   │               ├── tomogram-metadata.json
│   │   │               ├── <run_name>.zarr 
│   │   │               └── <run_name>.mrc
│   │   ├── TiltSeries/
│   │   │   └── <ts-id>/
│   │   │       ├── <run_name>.mrc
│   │   │       ├── <run_name>.zarr
│   │   │       ├── tiltseries_metadata.json
│   │   │       └── <name-in-source>.rawtlt
│   │   └── run_metadata.json
│   └── Images
└── DepositionMetadata/
    └── <deposition-id>/
        ├── deposition_metadata.json
        └── Images

Alignments

The path is updated, to consolidate all the related alignment files into a directory with alignment_id name inside the Alignment folder
In the identifier generation, container key now has a prefix of "alignments". This is to prevent key clashes between any other identifier that is generated at a run level.

Annotations

The annotations are being moved from Tomograms/VoxelSpacingX.XXX/Annotations -> Reconstructions/VoxelSpacingX.XXX/Annotations/{annotation_id}
The annotation_id is now a directory inside the Annotations folder. All the annotation files and metadata json are inside the annotation_id folder.
alignment_metadata_path is being added to Annotations metadata at the root level.
In the identifier generation, container key now has a prefix of "annotation". This is to prevent key clashes between any other identifier that is generated at a voxel spacing level level.
Annotation id hash generation components now include an optional annotation_object.state and related alignment_metadata_path

Key Photos

The tomogram key photos are being moved from Tomograms/VoxelSpacingX.XXX/KeyPhotos -> Reconstructions/VoxelSpacingX.XXX/Images/
The key photo file names will have a prefix of the tomogram_id they are related to.

Raw Tilts

The raw tilts are now being added as a child to tiltseries in the importer dependency. This means, they will be imported only if there is a related tiltseries in the config.
The raw tilt files are being moved from TiltSeries -> TiltSeries/{tiltseries_id}/

Tilt Series

The tiltseries files are being moved from TiltSeries -> TiltSeries/{tiltseries_id}/
The id for a tiltseries is being generated using deposition_id (ie) There can only be only tiltseries for a deposition.

Tomograms

The tomogram files are being moved from Tomograms/VoxelSpacingX.XXX/CanonicalTomogram -> Reconstructions/VoxelSpacingX.XXX/Tomograms/{tomogram_id}
The id for a tomogram is being generated based on the values of voxel_spacing, alignment, reconstruction_method, processing, deposition_id
The tomogram metadata now fields for alignment_metadata_path and neuroglancer_config_path.
The neuroglancer_config_path is only set to a non-null value if is_visualization_default=true

Visualization Config

The neuroglancer config file is being moved from Tomograms/VoxelSpacingX.XXX/CanonicalTomogram -> Reconstructions/VoxelSpacingX.XXX/NeuroglancerPrecompute/
It will have a prefix of the related tomogram id
It will be generated only for the tomograms that have is_visualization_default = true
The annotations included in the config will have to share the same alignment as the tomogram.

Annotation Precompute

The files are being moved from Tomograms/VoxelSpacingX.XXX/NeuroglancerPrecompute -> Reconstructions/VoxelSpacingX.XXX/NeuroglancerPrecompute/

* fix: support filtering entities by related object id's

* Updates to schema and template * Fixing tests * Adding files to alignments metadata * Update the alignment configs method type * Fixing typo * Adding undefined alignment_method_type

* Adds comments to jensen config generation * Adding guardrails * Clean up * fix: method_name

manasaV3 · 2024-10-09T19:01:14Z

ingestion_tools/scripts/importers/annotation.py

+        self.local_metadata = {
+            "object_count": 0,
+            "files": [],
+            "alignment_metadata_path": alignment_metadata_path,


There can only be one alignment associated to an annotation.

manasaV3 · 2024-10-09T19:02:21Z

ingestion_tools/scripts/common/metadata.py

@@ -78,7 +78,6 @@ def get_filename_prefix(self, output_dir: str, identifier: int) -> str:
            output_dir,
            "-".join(
                [
-                    str(identifier),


@uermel Should the annotation object state also be included in the file name?

Yes please. This was something I asked for.

To confirm my understanding, this has already been included in annotation hash used for generating the identifier (ref here). We also want this in the filename?

manasaV3 · 2024-10-09T19:04:20Z

ingestion_tools/scripts/importers/base_importer.py

+        if hasattr(self, "identifier") and self.identifier:
+            glob_vars[f"{self.type_key}_id"] = self.identifier


Handles for the introduction of ids in the file paths.

manasaV3 · 2024-10-09T19:15:34Z

ingestion_tools/scripts/importers/utils.py

+                TiltSeriesImporter: {
+                    RawTiltImporter: {},
+                },


The raw tilts are now being added as a child to tiltseries. This means aw tilts files will be imported only if there is a related tiltseries in the config.
@uermel Would this break any expected patterns?

That sounds good to me. I don't expect anything to be broken.

uermel · 2024-10-09T19:52:31Z

.
├── DatasetID/
│   ├── ExperimentalRun/
│   │   ├── Alignment/
│   │   │   └── <alignment-id>/
│   │   │       ├── alignment_metadata.json
│   │   │       └── <name-in-source>.(aln|xf|tlt|json)

This needs to include .xtilt, and .com.

│   │   ├── TiltSeries/
│   │   │   └── <ts-id>/
│   │   │       ├── <run_name>.mrc
│   │   │       ├── <run_name>.zarr
│   │   │       ├── tiltseries_metadata.json
│   │   │       └── <name-in-source>.raw_tlt

This should be .rawtlt.

uermel

Looks good to me, slight updates to the description may be useful for future reference.

jgadling · 2024-10-09T20:19:38Z

ingestion_tools/scripts/importers/alignment.py

-        )
+        return all(volume_dim.get(dim) for dim in "xyz") or self.get_tomogram() is not None
+
+    def get_tomogram(self) -> Optional["TomogramImporter"]:


We should leave a comment as to why this is safe -- basically that we only allow one alignment & voxel spacing per ingestion config, and that we assume all tomograms imported for a single run have the same dimensions.

jgadling · 2024-10-09T20:26:44Z

ingestion_tools/scripts/importers/visualization_config.py

@@ -24,23 +24,32 @@ class VisualizationConfigImporter(BaseImporter):
    finder_factory = DefaultImporterFactory
    has_metadata = False
    dir_path = (
-        "{dataset_name}/{run_name}/Tomograms/VoxelSpacing{voxel_spacing_name}/CanonicalTomogram/"
-        "neuroglancer_config.json"
+        "{dataset_name}/{run_name}/Reconstructions/VoxelSpacing{voxel_spacing_name}/NeuroglancerPrecompute/"


Out of curiosity, why aren't we putting neuroglancer precompute data in a subdirectory of the tomogram, sort of like this?

{dataset_id}/{run_name}/Reconstructions/Tomograms/{tomo_id}/NeuroglancerPrecompute

The annotations precompute should be shareable across different tomograms, if they have the same alignments. So, I didn't want to tie it to a tomogram.

jgadling

Looks good to me once @uermel 's requests are handled.

manasaV3 and others added 25 commits September 25, 2024 16:28

Updating tomograms

7607584

updates to tomogram processing

076c3b5

updates to tomogram key photo filename

b3ab2d3

updates to tomogram neuroglance filepath

8aee0ee

cleaning up tomogram and related entities

15b7a40

Updating tiltseries with identifiers

5125982

Updating alignment for tomogram relations

f1002b0

Adding alignment metadatapath to annotations and tomograms

c79936b

Merge remote-tracking branch 'origin' into multi_tomo

74a3247

Merge remote-tracking branch 'origin' into multi_tomo

a42c297

Updating the paths

c1fc0a2

Updating the paths

60cd302

Updating failed tests

7d01008

Working tests clean up needed

98e9163

fix: workaround for docker compose bugs. (#295)

08f0bed

fix: support filtering entities by related object id's (#296)

b89d2c6

* fix: support filtering entities by related object id's

chore: add seed script. (#297)

dd5f547

feat: Updates to the alignment entity (#298)

c5770ea

* Updates to schema and template * Fixing tests * Adding files to alignments metadata * Update the alignment configs method type * Fixing typo * Adding undefined alignment_method_type

chore: Documenting the Jensen config generation (#293)

92d45ec

* Adds comments to jensen config generation * Adding guardrails * Clean up * fix: method_name

chore: Update footer of release please PR's. (#299)

d3c0007

Adding tests

04bedde

Merge remote-tracking branch 'origin' into multi_tomo

8f6fe81

Merge remote-tracking branch 'origin' into multi_tomo

06d3a7c

cleaning up the paths

807f782

Cleaning up paths

ca54b91

manasaV3 changed the title ~~Adding multi tomogram relationships~~ feat: Adding multi tomogram relationships Oct 7, 2024

manasaV3 added 3 commits October 7, 2024 09:34

fix tests

25df3a4

Updating tomogram id generation

84a58d5

Updating viz_config generation

a656aad

manasaV3 marked this pull request as ready for review October 7, 2024 20:02

manasaV3 marked this pull request as draft October 7, 2024 20:02

manasaV3 added 7 commits October 7, 2024 14:27

Making id directories

065cf49

Making id directories

f677d01

Making annotation id directory

785d9c2

Cleaning up paths with id

8791a7d

Updating neuroglancer precompute for dir structure

b0efcb2

Updating neuroglancer config for dir structure

a6314f5

Updating raw tilt import to tiltseries_id directory

c1afde9

manasaV3 requested a review from jgadling October 9, 2024 18:54

manasaV3 marked this pull request as ready for review October 9, 2024 18:59

manasaV3 commented Oct 9, 2024

View reviewed changes

manasaV3 requested review from Bento007 and uermel October 9, 2024 19:19

uermel approved these changes Oct 9, 2024

View reviewed changes

jgadling reviewed Oct 9, 2024

View reviewed changes

jgadling approved these changes Oct 9, 2024

View reviewed changes

manasaV3 added 2 commits October 11, 2024 13:13

Fixing dependency on existing metadata

b02eb0c

Adding documentation comment

98bbde6

manasaV3 merged commit 7204736 into main Oct 14, 2024
6 checks passed

manasaV3 deleted the mvenkatakrishnan/multi_tomo branch October 14, 2024 20:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Adding multi tomogram relationships #302

feat: Adding multi tomogram relationships #302

manasaV3 commented Oct 4, 2024 •

edited

Loading

manasaV3 Oct 9, 2024

manasaV3 Oct 9, 2024

uermel Oct 9, 2024

manasaV3 Oct 9, 2024

manasaV3 Oct 9, 2024

manasaV3 Oct 9, 2024

uermel Oct 9, 2024

uermel commented Oct 9, 2024

uermel left a comment

jgadling Oct 9, 2024

jgadling Oct 9, 2024

manasaV3 Oct 9, 2024

jgadling left a comment

		if hasattr(self, "identifier") and self.identifier:
		glob_vars[f"{self.type_key}_id"] = self.identifier

feat: Adding multi tomogram relationships #302

feat: Adding multi tomogram relationships #302

Conversation

manasaV3 commented Oct 4, 2024 • edited Loading

Description

Alignments

Annotations

Key Photos

Raw Tilts

Tilt Series

Tomograms

Visualization Config

Annotation Precompute

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uermel commented Oct 9, 2024

uermel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgadling left a comment

Choose a reason for hiding this comment

manasaV3 commented Oct 4, 2024 •

edited

Loading