-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Updating paths for Gains and CollectionMetadata #271
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
cda2142
Updating gain file paths
manasaV3 57e29ab
Updating collection metadata paths
manasaV3 c6ee638
Merge branch 'main' into mvenkatakrishnan/multi_tomo_paths1
manasaV3 30f3907
Adding collections metadata tests
manasaV3 e70fd24
Adding validation for collection metadata output file
manasaV3 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
ingestion_tools/scripts/tests/s3_import/test_collection_metadata.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
import os.path | ||
from os.path import basename | ||
|
||
from importers.collection_metadata import CollectionMetadataImporter | ||
from mypy_boto3_s3 import S3Client | ||
|
||
from common.fs import FileSystemApi | ||
from tests.s3_import.util import create_config, get_data_from_s3, get_dataset_and_run, list_dir | ||
|
||
|
||
def test_collection_metadata_import(s3_fs: FileSystemApi, test_output_bucket: str, s3_client: S3Client) -> None: | ||
config = create_config(s3_fs, test_output_bucket) | ||
dataset, run = get_dataset_and_run(config) | ||
collection_metadata = list(CollectionMetadataImporter.finder(config, dataset=dataset, run=run)) | ||
for item in collection_metadata: | ||
item.import_item() | ||
|
||
run_name = run.name | ||
prefix = f"output/{dataset.name}/{run_name}/Frames" | ||
actual_files = [basename(item) for item in list_dir(s3_client, test_output_bucket, prefix)] | ||
source_filename = os.path.basename(collection_metadata[0].path) | ||
assert source_filename in actual_files | ||
|
||
actual = get_data_from_s3(s3_client, test_output_bucket, os.path.join(prefix, source_filename)).readlines() | ||
source_file_path = "/".join(collection_metadata[0].path.split("/")[1:]) | ||
expected = get_data_from_s3(s3_client, "test-public-bucket", source_file_path).readlines() | ||
assert actual == expected |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
35 changes: 35 additions & 0 deletions
35
test_infra/test_files/input_bucket/10001_input/metadata/mdocs_modified/foo-TS_run1.mdoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
PixelSpacing = 3.3702 | ||
Voltage = 300 | ||
ImageFile = TS_run1.mrc | ||
ImageSize = 3708 3838 | ||
DataMode = 1 | ||
|
||
[T = SerialEM: Digitized on EMBL Krios 28-Apr-18 11:36:19 ] | ||
|
||
[T = Tilt axis angle = 84.7, binning = 1 spot = 6 camera = 0] | ||
|
||
[ZValue = 0] | ||
TiltAngle = -0.00499939 | ||
StagePosition = 181.087 446.897 | ||
StageZ = -26.7037 | ||
Magnification = 42000 | ||
Intensity = 0.116505 | ||
ExposureDose = 2.35936 | ||
PixelSpacing = 3.3702 | ||
SpotSize = 6 | ||
Defocus = -0.279296 | ||
ImageShift = -0.05244 -0.0632113 | ||
RotationAngle = 174.7 | ||
ExposureTime = 1 | ||
Binning = 1 | ||
CameraIndex = 0 | ||
DividedBy2 = 1 | ||
MagIndex = 27 | ||
CountsPerElectron = 17.25 | ||
MinMaxMean = -55 1151 220.207 | ||
TargetDefocus = -2 | ||
SubFramePath = TS_run1_01.mrc | ||
NumSubFrames = 10 | ||
FrameDosesAndNumber = 0.23594 10 | ||
DateTime = 28-Apr-18 11:38:54 | ||
NavigatorLabel = 112 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is the code for collection_metadata support not part of this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! the collection_metadata test file got missed in being tracked.