-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Script to migrate the directory structure (#311)
* Updating tomograms * updates to tomogram processing * updates to tomogram key photo filename * updates to tomogram neuroglance filepath * cleaning up tomogram and related entities * Updating tiltseries with identifiers * Updating alignment for tomogram relations * Adding alignment metadatapath to annotations and tomograms * Updating the paths * Updating the paths * Updating failed tests * Working tests clean up needed * fix: workaround for docker compose bugs. (#295) * fix: support filtering entities by related object id's (#296) * fix: support filtering entities by related object id's * chore: add seed script. (#297) * feat: Updates to the alignment entity (#298) * Updates to schema and template * Fixing tests * Adding files to alignments metadata * Update the alignment configs method type * Fixing typo * Adding undefined alignment_method_type * chore: Documenting the Jensen config generation (#293) * Adds comments to jensen config generation * Adding guardrails * Clean up * fix: method_name * chore: Update footer of release please PR's. (#299) * Adding tests * cleaning up the paths * Cleaning up paths * fix tests * Updating tomogram id generation * Updating viz_config generation * Making id directories * Making id directories * Making annotation id directory * Cleaning up paths with id * Updating neuroglancer precompute for dir structure * Updating neuroglancer config for dir structure * Updating raw tilt import to tiltseries_id directory * Migration script * Update for tomograms * Adding support for alignments * Migrating annotation precompute * Migrating collection_metadata * Migrating rawtilt and gains * Clean up * Clean up * Clean up * Enabling the move * Adding metadata updates * Uncommenting deletion * Adding check before move * Adding wdl * minor fixes * Updating enqueue script * Updating annotation file names * Updating the tomogram key photo path * Fixing tests * Fix key_photo migration * Handling default alignments * fix path standardization when bucket name not in path * Lint. --------- Co-authored-by: Jessica Gadling <jgadling@chanzuckerberg.com>
- Loading branch information
Showing
6 changed files
with
553 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
version 1.0 | ||
|
||
|
||
task cryoet_folder_migration_workflow { | ||
input { | ||
String docker_image_id | ||
String aws_region | ||
String config_file | ||
String output_bucket | ||
String flags | ||
} | ||
|
||
command <<< | ||
set -euxo pipefail | ||
export PYTHONUNBUFFERED=1 | ||
python --version 1>&2 | ||
ls -l 1>&2 | ||
pwd 1>&2 | ||
cd /usr/src/app/ingestion_tools/scripts | ||
set +x | ||
echo python migrate_folder_structure.py migrate ~{config_file} ~{output_bucket} ~{flags} 1>&2 | ||
python migrate_folder_structure.py migrate ~{config_file} ~{output_bucket} ~{flags} 1>&2 | ||
>>> | ||
|
||
runtime { | ||
docker: docker_image_id | ||
} | ||
} | ||
|
||
workflow cryoet_folder_migration { | ||
input { | ||
String docker_image_id = "cryoet_data_ingestion:latest" | ||
String aws_region = "us-west-2" | ||
String config_file | ||
String output_bucket | ||
String flags | ||
} | ||
|
||
call cryoet_folder_migration_workflow { | ||
input: | ||
docker_image_id = docker_image_id, | ||
aws_region = aws_region, | ||
config_file = config_file, | ||
output_bucket = output_bucket, | ||
flags = flags | ||
} | ||
|
||
output { | ||
File log = "output.txt" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.