Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Be flexible with Zarr 3 index codecs #7574

Open
frcroth opened this issue Jan 22, 2024 · 0 comments
Open

Be flexible with Zarr 3 index codecs #7574

frcroth opened this issue Jan 22, 2024 · 0 comments

Comments

@frcroth
Copy link
Member

frcroth commented Jan 22, 2024

Currently, only bytes and crc32c are supported. Other codecs will have an impact on the compressed size of the index, which needs to be calculated based on the codecs present.

Raised in #7553

I guess this works for most datasets, but not for all. Although, it doesn't make a lot of sense, it is spec-compliant to have multiple crc32c checksums. In the future, there might be other codecs that change the size of the shard index.

A more correct way would be to compute the size of the index based on the entire index_codecs pipeline. zarrita uses for that the compute_encoded_size methods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant