Nontrivial chunking (#8)

* Update submodules. Fix DM bugs in test_basic.py * Update submodules. * Update submodules. * Expand StorageProperties to support chunking and compression parameters. * Update acquire-driver-zarr submodule. Add chunking test. * Squashed commit of the following: commit 750f6f6 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Wed May 10 10:37:05 2023 -0400 Address PR comments. commit 5a256f7 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Tue May 9 13:29:49 2023 -0400 Clean up build script commit a88e12a Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Tue May 9 13:12:43 2023 -0400 add GH token to environment variables in typing test. commit df82566 Merge: 652c401 b07cbb3 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Tue May 9 12:56:25 2023 -0400 Merge remote-tracking branch 'nclack/testing' into submodule-updates commit 652c401 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Tue May 9 12:22:45 2023 -0400 update dcam and egrabber test yamls commit 3cc7cdd Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Tue May 9 12:10:00 2023 -0400 remove driver submodules commit e4b9e01 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Tue May 9 11:51:14 2023 -0400 build and test passes commit 0b27539 Author: Nathan Clack <nclack@gmail.com> Date: Mon May 8 16:41:49 2023 -0700 fix packaging commit b07cbb3 Author: Nathan Clack <nclack@gmail.com> Date: Mon May 8 17:09:18 2023 -0700 get dcam test to pass commit bc54ff2 Author: Nathan Clack <nclack@gmail.com> Date: Mon May 8 16:57:21 2023 -0700 packaging: remove old setuptools options commit afba01d Author: Nathan Clack <nclack@gmail.com> Date: Mon May 8 16:41:49 2023 -0700 fix packaging commit a887683 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Mon May 8 17:22:09 2023 -0400 update build and test yamls commit 8e08001 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Mon May 8 17:19:31 2023 -0400 wip commit 7b3180f Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Mon May 8 16:56:02 2023 -0400 update submodules commit 6e41aff Author: Nathan Clack <nclack@gmail.com> Date: Mon May 8 10:00:42 2023 -0700 fix trigger assignment commit 9cd2915 Author: Nathan Clack <nclack@gmail.com> Date: Fri May 5 15:38:23 2023 -0700 testing (wip) commit b5bf4f9 Author: Nathan Clack <nclack@gmail.com> Date: Fri May 5 15:29:32 2023 -0700 update deps, tests commit 20ea3a0 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Thu May 4 12:24:23 2023 -0400 update acquire-driver-zarr submodule. commit 941027c Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Thu May 4 11:38:06 2023 -0400 Update acquire-driver-common and acquire-video-runtime submodules commit 5fcd566 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Thu May 4 10:55:14 2023 -0400 Update typing and dcam tests commit d68d40f Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Wed May 3 15:47:45 2023 -0400 Try again to enable long paths in CI. commit 34647a7 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Wed May 3 15:41:40 2023 -0400 Try to enable long paths in CI. commit 1d79e24 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Wed May 3 15:24:57 2023 -0400 Restore capital-T Tiff. commit 5b317b9 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Wed May 3 15:23:54 2023 -0400 Update acquire-driver-common and acquire-driver-hdcam. Restore recursive submodule checkout in CI. commit 5d170dd Merge: f989feb 51a7ea1 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Wed May 3 13:40:46 2023 -0400 Merge branch 'main' into submodule-updates commit 51a7ea1 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Wed May 3 13:38:25 2023 -0400 remove uses of PAT from workflow files, set submodules from recursive to true * Update driver hashes. * Squashed commit of the following: commit 6909668 Author: Alan Liddell <aliddell@chanzuckerberg.com> Date: Wed May 10 19:08:01 2023 -0400 Use artifact downloads instead of submodules (#2) * Update submodules. Fix DM bugs in test_basic.py * Update acquire-driver-common and acquire-driver-hdcam. Restore recursive submodule checkout in CI. * Restore capital-T Tiff. * Try to enable long paths in CI. * Try again to enable long paths in CI. * Update typing and dcam tests * Update acquire-driver-common and acquire-video-runtime submodules * update acquire-driver-zarr submodule. * update deps, tests * testing (wip) * fix trigger assignment * update submodules * wip * update build and test yamls * fix packaging * packaging: remove old setuptools options * get dcam test to pass * fix packaging * build and test passes * remove driver submodules * update dcam and egrabber test yamls * add GH token to environment variables in typing test. * Clean up build script * Address PR comments. * update zarr sha * update shas --------- Co-authored-by: Nathan Clack <nclack@gmail.com> * update acquire-video-runtime * wip: update zarr driver (?) * wip * Update drivers, fix bindings, fix tests. * Misc fixes * Ran black and flake8 * fix readme
acquire-project · Jul 6, 2023 · 06fd9be · 06fd9be
1 parent 238cd35
commit 06fd9be
Show file tree

Hide file tree

Showing 10 changed files with 212 additions and 56 deletions.
diff --git a/Cargo.toml b/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "acquire-imaging"
 authors = ["Nathan Clack <nclack@chanzuckerberg.com>"]
-version = "0.1.3"
+version = "0.1.4"
 edition = "2021"
 
 [lib]

diff --git a/README.md b/README.md
@@ -138,7 +138,7 @@ This project uses [`pre-commit`](https://pre-commit.com/) to run required
 checks as git hooks.
 
 ```bash
-pip install precommit
+pip install pre-commit
 pre-commit install
 ```
 

diff --git a/acquire-video-runtime b/acquire-video-runtime
diff --git a/build.rs b/build.rs
@@ -95,7 +95,7 @@ fn fetch_acquire_driver(dst: &std::path::PathBuf, name: &str, tag: &str) {
     } else {
         format!("v{tag}")
     };
-    let uri = format!("https://github.com/acquire-project/{name}/releases/download/{vstring}/{name}-{tag}-{build}.zip");
+    let uri = format!("https://github.com/acquire-project/{name}/releases/download/{vstring}/{name}-{vstring}-{build}.zip");
     let request = client
         .get(uri)
         .header("Accept", "application/vnd.github+json")

diff --git a/drivers.json b/drivers.json
@@ -1,6 +1,6 @@
 {
-  "acquire-driver-common": "0.1.1",
-  "acquire-driver-zarr": "0.1.1",
-  "acquire-driver-egrabber": "0.1.1",
-  "acquire-driver-hdcam": "0.1.2"
+  "acquire-driver-common": "0.1.3",
+  "acquire-driver-zarr": "0.1.2",
+  "acquire-driver-egrabber": "0.1.2",
+  "acquire-driver-hdcam": "0.1.4"
 }
diff --git a/python/acquire/acquire.pyi b/python/acquire/acquire.pyi
@@ -208,13 +208,26 @@ class Storage:
     settings: StorageProperties
     def dict(self) -> Dict[str, Any]: ...
 
+@final
+class TileShape:
+    width: int
+    height: int
+    planes: int
+    def dict(self) -> Dict[str, Any]: ...
+
+@final
+class ChunkingProperties:
+    max_bytes_per_chunk: int
+    tile: TileShape
+    def dict(self) -> Dict[str, Any]: ...
+
 @final
 class StorageProperties:
-    bytes_per_chunk: int
     external_metadata_json: Optional[str]
     filename: Optional[str]
     first_frame_id: int
     pixel_scale_um: Tuple[float, float]
+    chunking: ChunkingProperties
     def dict(self) -> Dict[str, Any]: ...
 
 @final

diff --git a/src/lib.rs b/src/lib.rs
@@ -73,6 +73,8 @@ fn acquire(py: Python, m: &PyModule) -> PyResult<()> {
     m.add_class::<camera::CameraProperties>()?;
     m.add_class::<camera::InputTriggers>()?;
     m.add_class::<camera::OutputTriggers>()?;
+    m.add_class::<storage::TileShape>()?;
+    m.add_class::<storage::ChunkingProperties>()?;
     m.add_class::<storage::StorageProperties>()?;
 
     m.add_class::<components::Direction>()?;

diff --git a/src/storage.rs b/src/storage.rs
@@ -9,6 +9,49 @@ use std::{
 
 #[pyclass]
 #[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct TileShape {
+    #[pyo3(get, set)]
+    #[serde(default)]
+    pub(crate) width: u32,
+
+    #[pyo3(get, set)]
+    #[serde(default)]
+    pub(crate) height: u32,
+
+    #[pyo3(get, set)]
+    #[serde(default)]
+    pub(crate) planes: u32,
+}
+
+impl_plain_old_dict!(TileShape);
+
+#[pyclass]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ChunkingProperties {
+    #[pyo3(get, set)]
+    #[serde(default)]
+    pub(crate) max_bytes_per_chunk: u64,
+
+    #[pyo3(get, set)]
+    pub(crate) tile: Py<TileShape>,
+}
+
+impl_plain_old_dict!(ChunkingProperties);
+
+impl Default for ChunkingProperties {
+    fn default() -> Self {
+        let tile = Python::with_gil(|py| {
+            Py::new(py, TileShape::default()).unwrap()
+        });
+        Self {
+            max_bytes_per_chunk: Default::default(),
+            tile,
+        }
+    }
+}
+
+#[pyclass]
+#[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct StorageProperties {
     #[pyo3(get, set)]
     #[serde(default)]
@@ -27,11 +70,26 @@ pub struct StorageProperties {
     pub(crate) pixel_scale_um: (f64, f64),
 
     #[pyo3(get, set)]
-    pub(crate) bytes_per_chunk: u32,
+    pub(crate) chunking: Py<ChunkingProperties>,
 }
 
 impl_plain_old_dict!(StorageProperties);
 
+impl Default for StorageProperties {
+    fn default() -> Self {
+        let chunking = Python::with_gil(|py| {
+            Py::new(py, ChunkingProperties::default()).unwrap()
+        });
+        Self {
+            filename: Default::default(),
+            external_metadata_json: Default::default(),
+            first_frame_id: Default::default(),
+            pixel_scale_um: Default::default(),
+            chunking,
+        }
+    }
+}
+
 impl TryFrom<capi::StorageProperties> for StorageProperties {
     type Error = anyhow::Error;
 
@@ -54,12 +112,25 @@ impl TryFrom<capi::StorageProperties> for StorageProperties {
                     .to_owned(),
             )
         };
+
+        let chunking = Python::with_gil(|py| {
+            let tile = Py::new(py, TileShape {
+                width: value.chunking.tile.width,
+                height: value.chunking.tile.height,
+                planes: value.chunking.tile.planes,
+            }).unwrap();
+            Py::new(py, ChunkingProperties {
+                max_bytes_per_chunk: value.chunking.max_bytes_per_chunk,
+                tile,
+            }).unwrap()
+        });
+
         Ok(Self {
             filename,
             first_frame_id: value.first_frame_id,
             external_metadata_json,
             pixel_scale_um: (value.pixel_scale_um.x, value.pixel_scale_um.y),
-            bytes_per_chunk: value.bytes_per_chunk,
+            chunking,
         })
     }
 }
@@ -93,6 +164,16 @@ impl TryFrom<&StorageProperties> for capi::StorageProperties {
             (null(), 0)
         };
 
+        let chunking_props = Python::with_gil(|py| -> PyResult<_> {
+            let chunking_props: ChunkingProperties = value.chunking.extract(py)?;
+            Ok(chunking_props)
+        })?;
+
+        let tile_shape = Python::with_gil(|py| -> PyResult<_> {
+            let tile_shape: TileShape = chunking_props.tile.extract(py)?;
+            Ok(tile_shape)
+        })?;
+
         // This copies the string into a buffer owned by the return value.
         if !unsafe {
             capi::storage_properties_init(
@@ -106,7 +187,15 @@ impl TryFrom<&StorageProperties> for capi::StorageProperties {
                     x: value.pixel_scale_um.0,
                     y: value.pixel_scale_um.1,
                 },
-                value.bytes_per_chunk,
+            ) == 1
+        } {
+            Err(anyhow::anyhow!("Failed acquire api status check"))
+        } else if !unsafe {
+            capi::storage_properties_set_chunking_props(&mut out,
+                                                        tile_shape.width,
+                                                        tile_shape.height,
+                                                        tile_shape.planes,
+                                                        chunking_props.max_bytes_per_chunk,
             ) == 1
         } {
             Err(anyhow::anyhow!("Failed acquire api status check"))
@@ -123,7 +212,7 @@ impl Default for capi::StorageProperties {
             first_frame_id: Default::default(),
             external_metadata_json: Default::default(),
             pixel_scale_um: Default::default(),
-            bytes_per_chunk: Default::default(),
+            chunking: Default::default(),
         }
     }
 }
@@ -147,6 +236,57 @@ impl Default for capi::PixelScale {
     }
 }
 
+impl Default for capi::StorageProperties_storage_properties_chunking_s_storage_properties_chunking_tile_s {
+    fn default() -> Self {
+        Self {
+            width: Default::default(),
+            height: Default::default(),
+            planes: Default::default(),
+        }
+    }
+}
+
+impl Default for capi::StorageProperties_storage_properties_chunking_s {
+    fn default() -> Self {
+        Self {
+            max_bytes_per_chunk: Default::default(),
+            tile: Default::default(),
+        }
+    }
+}
+
+impl Default for capi::ImageShape_image_dims_s {
+    fn default() -> Self {
+        Self {
+            channels: Default::default(),
+            width: Default::default(),
+            height: Default::default(),
+            planes: Default::default(),
+        }
+    }
+}
+
+impl Default for capi::ImageShape_image_strides_s {
+    fn default() -> Self {
+        Self {
+            channels: Default::default(),
+            width: Default::default(),
+            height: Default::default(),
+            planes: Default::default(),
+        }
+    }
+}
+
+impl Default for capi::ImageShape {
+    fn default() -> Self {
+        Self {
+            dims: Default::default(),
+            strides: Default::default(),
+            type_: Default::default(),
+        }
+    }
+}
+
 impl Display for capi::String {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         let s = unsafe { CStr::from_ptr(self.str_) }.to_string_lossy();