Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate existing ingested data to new folder structure #899

Open
Tracked by #780
manasaV3 opened this issue Jul 22, 2024 · 0 comments
Open
Tracked by #780

Migrate existing ingested data to new folder structure #899

manasaV3 opened this issue Jul 22, 2024 · 0 comments
Assignees
Labels
backend P0 Must have

Comments

@manasaV3
Copy link
Contributor

manasaV3 commented Jul 22, 2024

Motivation

To support the multiple tomograms, there needs to be restructuring of how the data is store. While it is easy for us to do this using the ingestion process for new data, it will be expensive to re ingest all the information. So, for the data that already exists, we should migrate them using a scripting. This will not apply to metadata json files, as they are cheaper to create.

Definition of Done

All the existing datasets are updated to follow the new structure.

Tasks

  • Move the data the would remain the same, but would be renamed or relocated.
  • Re-ingest all the relevant metadata.

Note

This should not be synced to production until the work for #897 is complete.

Relevant Resources

New directory structure: https://docs.google.com/document/d/1InjJKrx78Sv44QguMXR0p1o9Y34oPnMOY9J3aj375ks/edit#bookmark=id.oj7rrczgok3c

@manasaV3 manasaV3 mentioned this issue Jul 22, 2024
19 tasks
@manasaV3 manasaV3 added P0 Must have backend labels Jul 22, 2024
@manasaV3 manasaV3 changed the title Update existing ingested data to follow new structure Update existing ingested data to new structure Jul 22, 2024
@manasaV3 manasaV3 changed the title Update existing ingested data to new structure Update existing ingested data to new folder structure Aug 6, 2024
@manasaV3 manasaV3 changed the title Update existing ingested data to new folder structure Migrate existing ingested data to new folder structure Aug 29, 2024
@junxini junxini added this to the sprint 4 (phase 4) milestone Sep 5, 2024
@manasaV3 manasaV3 removed their assignment Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend P0 Must have
Projects
None yet
Development

No branches or pull requests

3 participants