Skip to content

Commit

Permalink
Multiscale (#47)
Browse files Browse the repository at this point in the history 
* Update submodules. Fix DM bugs in test_basic.py

* Update submodules.

* Update submodules.

* Expand StorageProperties to support chunking and compression parameters.

* Update acquire-driver-zarr submodule. Add chunking test.

* Squashed commit of the following:

commit 750f6f6
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Wed May 10 10:37:05 2023 -0400

    Address PR comments.

commit 5a256f7
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Tue May 9 13:29:49 2023 -0400

    Clean up build script

commit a88e12a
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Tue May 9 13:12:43 2023 -0400

    add GH token to environment variables in typing test.

commit df82566
Merge: 652c401 b07cbb3
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Tue May 9 12:56:25 2023 -0400

    Merge remote-tracking branch 'nclack/testing' into submodule-updates

commit 652c401
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Tue May 9 12:22:45 2023 -0400

    update dcam and egrabber test yamls

commit 3cc7cdd
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Tue May 9 12:10:00 2023 -0400

    remove driver submodules

commit e4b9e01
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Tue May 9 11:51:14 2023 -0400

    build and test passes

commit 0b27539
Author: Nathan Clack <nclack@gmail.com>
Date:   Mon May 8 16:41:49 2023 -0700

    fix packaging

commit b07cbb3
Author: Nathan Clack <nclack@gmail.com>
Date:   Mon May 8 17:09:18 2023 -0700

    get dcam test to pass

commit bc54ff2
Author: Nathan Clack <nclack@gmail.com>
Date:   Mon May 8 16:57:21 2023 -0700

    packaging: remove old setuptools options

commit afba01d
Author: Nathan Clack <nclack@gmail.com>
Date:   Mon May 8 16:41:49 2023 -0700

    fix packaging

commit a887683
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Mon May 8 17:22:09 2023 -0400

    update build and test yamls

commit 8e08001
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Mon May 8 17:19:31 2023 -0400

    wip

commit 7b3180f
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Mon May 8 16:56:02 2023 -0400

    update submodules

commit 6e41aff
Author: Nathan Clack <nclack@gmail.com>
Date:   Mon May 8 10:00:42 2023 -0700

    fix trigger assignment

commit 9cd2915
Author: Nathan Clack <nclack@gmail.com>
Date:   Fri May 5 15:38:23 2023 -0700

    testing (wip)

commit b5bf4f9
Author: Nathan Clack <nclack@gmail.com>
Date:   Fri May 5 15:29:32 2023 -0700

    update deps, tests

commit 20ea3a0
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Thu May 4 12:24:23 2023 -0400

    update acquire-driver-zarr submodule.

commit 941027c
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Thu May 4 11:38:06 2023 -0400

    Update acquire-driver-common and acquire-video-runtime submodules

commit 5fcd566
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Thu May 4 10:55:14 2023 -0400

    Update typing and dcam tests

commit d68d40f
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Wed May 3 15:47:45 2023 -0400

    Try again to enable long paths in CI.

commit 34647a7
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Wed May 3 15:41:40 2023 -0400

    Try to enable long paths in CI.

commit 1d79e24
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Wed May 3 15:24:57 2023 -0400

    Restore capital-T Tiff.

commit 5b317b9
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Wed May 3 15:23:54 2023 -0400

    Update acquire-driver-common and acquire-driver-hdcam. Restore recursive submodule checkout in CI.

commit 5d170dd
Merge: f989feb 51a7ea1
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Wed May 3 13:40:46 2023 -0400

    Merge branch 'main' into submodule-updates

commit 51a7ea1
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Wed May 3 13:38:25 2023 -0400

    remove uses of PAT from workflow files, set submodules from recursive to true

* Update driver hashes.

* Squashed commit of the following:

commit 6909668
Author: Alan Liddell <aliddell@chanzuckerberg.com>
Date:   Wed May 10 19:08:01 2023 -0400

    Use artifact downloads instead of submodules (#2)

    * Update submodules. Fix DM bugs in test_basic.py

    * Update acquire-driver-common and acquire-driver-hdcam. Restore recursive submodule checkout in CI.

    * Restore capital-T Tiff.

    * Try to enable long paths in CI.

    * Try again to enable long paths in CI.

    * Update typing and dcam tests

    * Update acquire-driver-common and acquire-video-runtime submodules

    * update acquire-driver-zarr submodule.

    * update deps, tests

    * testing (wip)

    * fix trigger assignment

    * update submodules

    * wip

    * update build and test yamls

    * fix packaging

    * packaging: remove old setuptools options

    * get dcam test to pass

    * fix packaging

    * build and test passes

    * remove driver submodules

    * update dcam and egrabber test yamls

    * add GH token to environment variables in typing test.

    * Clean up build script

    * Address PR comments.

    * update zarr sha

    * update shas

    ---------

    Co-authored-by: Nathan Clack <nclack@gmail.com>

* update acquire-video-runtime

* wip: update zarr driver (?)

* wip

* Update drivers, fix bindings, fix tests.

* Misc fixes

* Ran black and flake8

* fix readme

* wip

* add multiscale test

* Update drivers list.

* Update storage properties.

* Update multiscale test.

* Fix chunking tests by specifying u8 sample type.
  • Loading branch information
@aliddell
aliddell authored Jul 28, 2023
1 parent c139a96 commit 8616428
Show file tree
Hide file tree
Showing 5 changed files with 109 additions and 5 deletions.
1 change: 1 addition & 0 deletions build.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ fn main() {
.expect("Failed to parse drivers.json");

let out = std::path::PathBuf::from(std::env::var("OUT_DIR").unwrap());

fetch_acquire_driver(
&out,
"acquire-driver-common",
Expand Down
8 changes: 4 additions & 4 deletions drivers.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"acquire-driver-common": "0.1.3",
"acquire-driver-zarr": "0.1.2",
"acquire-driver-egrabber": "0.1.2",
"acquire-driver-hdcam": "0.1.4"
"acquire-driver-common": "0.1.4",
"acquire-driver-zarr": "0.1.3",
"acquire-driver-egrabber": "0.1.3",
"acquire-driver-hdcam": "0.1.5"
}
2 changes: 1 addition & 1 deletion python/acquire/acquire.pyi
View file
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,6 @@ class SampleType:
def __lt__(self, other: object) -> bool: ...
def __ne__(self, other: object) -> bool: ...


@final
class SignalIOKind:
Input: ClassVar[SignalIOKind] = SignalIOKind.Input
Expand Down Expand Up @@ -228,6 +227,7 @@ class StorageProperties:
first_frame_id: int
pixel_scale_um: Tuple[float, float]
chunking: ChunkingProperties
enable_multiscale: bool
def dict(self) -> Dict[str, Any]: ...

@final
Expand Down
42 changes: 42 additions & 0 deletions src/storage.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ pub struct StorageProperties {

#[pyo3(get, set)]
pub(crate) chunking: Py<ChunkingProperties>,

#[pyo3(get, set)]
pub(crate) enable_multiscale: bool,
}

impl_plain_old_dict!(StorageProperties);
Expand All @@ -86,6 +89,7 @@ impl Default for StorageProperties {
first_frame_id: Default::default(),
pixel_scale_um: Default::default(),
chunking,
enable_multiscale: Default::default(),
}
}
}
Expand Down Expand Up @@ -131,6 +135,7 @@ impl TryFrom<capi::StorageProperties> for StorageProperties {
external_metadata_json,
pixel_scale_um: (value.pixel_scale_um.x, value.pixel_scale_um.y),
chunking,
enable_multiscale: (value.enable_multiscale == 1),
})
}
}
Expand Down Expand Up @@ -199,6 +204,12 @@ impl TryFrom<&StorageProperties> for capi::StorageProperties {
) == 1
} {
Err(anyhow::anyhow!("Failed acquire api status check"))
} else if !unsafe {
capi::storage_properties_set_enable_multiscale(&mut out,
value.enable_multiscale as u8,
) == 1
} {
Err(anyhow::anyhow!("Failed acquire api status check"))
} else {
Ok(out)
}
Expand All @@ -213,6 +224,7 @@ impl Default for capi::StorageProperties {
external_metadata_json: Default::default(),
pixel_scale_um: Default::default(),
chunking: Default::default(),
enable_multiscale: Default::default(),
}
}
}
Expand Down Expand Up @@ -246,6 +258,20 @@ impl Default for capi::StorageProperties_storage_properties_chunking_s_storage_p
}
}

impl TryFrom<&TileShape> for capi::StorageProperties_storage_properties_chunking_s_storage_properties_chunking_tile_s {
type Error = anyhow::Error;

fn try_from(value: &TileShape) -> Result<Self, Self::Error> {
let mut out: capi::StorageProperties_storage_properties_chunking_s_storage_properties_chunking_tile_s = unsafe { std::mem::zeroed() };

out.height = value.height;
out.width = value.width;
out.planes = value.planes;

Ok(out)
}
}

impl Default for capi::StorageProperties_storage_properties_chunking_s {
fn default() -> Self {
Self {
Expand All @@ -255,6 +281,22 @@ impl Default for capi::StorageProperties_storage_properties_chunking_s {
}
}

impl TryFrom<&ChunkingProperties> for capi::StorageProperties_storage_properties_chunking_s {
type Error = anyhow::Error;

fn try_from(value: &ChunkingProperties) -> Result<Self, Self::Error> {
let tile = Python::with_gil(|py| -> PyResult<_> {
let tile: TileShape = value.tile.extract(py)?;
Ok(tile)
})?;

Ok(capi::StorageProperties_storage_properties_chunking_s {
max_bytes_per_chunk: value.max_bytes_per_chunk,
tile: (&tile).try_into()?,
})
}
}

impl Default for capi::ImageShape_image_dims_s {
fn default() -> Self {
Self {
Expand Down
61 changes: 61 additions & 0 deletions tests/test_basic.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
from acquire.acquire import DeviceKind, DeviceState, Runtime, Trigger
from ome_zarr.io import parse_url
from ome_zarr.reader import Reader
from skimage.transform import downscale_local_mean
import numpy as np


@pytest.fixture(scope="module")
Expand Down Expand Up @@ -428,6 +430,7 @@ def test_write_zarr_with_chunking(
)
p.video[0].camera.settings.shape = (1920, 1080)
p.video[0].camera.settings.exposure_time_us = 1e4
p.video[0].camera.settings.pixel_type = acquire.SampleType.U8
p.video[0].storage.identifier = dm.select(
DeviceKind.Storage,
"Zarr",
Expand Down Expand Up @@ -459,6 +462,64 @@ def test_write_zarr_with_chunking(
assert data.nchunks == expected_number_of_chunks


def test_write_zarr_multiscale(
runtime: acquire.Runtime,
request: pytest.FixtureRequest,
):
filename = f"{request.node.name}.zarr"
filename = filename.replace("[", "_").replace("]", "_")

dm = runtime.device_manager()

p = runtime.get_configuration()
p.video[0].camera.identifier = dm.select(
DeviceKind.Camera, "simulated.*empty.*"
)
p.video[0].camera.settings.shape = (1920, 1080)
p.video[0].camera.settings.exposure_time_us = 1e4
p.video[0].camera.settings.pixel_type = acquire.SampleType.U8
p.video[0].storage.identifier = dm.select(
DeviceKind.Storage,
"Zarr",
)
p.video[0].storage.settings.filename = filename
p.video[0].storage.settings.pixel_scale_um = (1, 1)
p.video[0].max_frame_count = 100

p.video[0].storage.settings.chunking.max_bytes_per_chunk = 16 * 2**20
p.video[0].storage.settings.chunking.tile.width = (
p.video[0].camera.settings.shape[0] // 3
)
p.video[0].storage.settings.chunking.tile.height = (
p.video[0].camera.settings.shape[1] // 3
)
p.video[0].storage.settings.chunking.tile.planes = 1

p.video[0].storage.settings.enable_multiscale = True

runtime.set_configuration(p)

runtime.start()
runtime.stop()

reader = Reader(parse_url(filename))
zgroup = list(reader())[0]
# loads each layer as a dask array from the Zarr dataset
data = [
da.from_zarr(filename, component=str(i))
for i in range(len(zgroup.data))
]
assert len(data) == 3

image = data[0][0, 0, :, :].compute() # convert dask array to numpy array

for d in data:
assert (
np.linalg.norm(image - d[0, 0, :, :].compute()) == 0
) # validate against the same method from scikit-image
image = downscale_local_mean(image, (2, 2)).astype(np.uint8)


@pytest.mark.skip(
reason="Runs into memory limitations on github ci."
+ " See https://github.com/acquire-project/cpx/issues/147"
Expand Down

0 comments on commit 8616428

Please sign in to comment.