use src layout and use `hatch` for packaging #1592

d-v-b · 2023-12-07T20:02:44Z

This PR changes the layout of the repo by putting the python source code in the src directory at the root of the repo. The logic for this layout can be found in lots of places online, but the scientific python packaging guide expresses it fairly concisely. Basically putting the source code in src helps prevent running tests with the local code instead of the installed code.

This change broke some tests, specifically all the tests that were trying to treat zarr.tests as a module, which was simple to fix. Oddly, there's code in main that imports from the test suite: see here which seems pretty undesirable, if not a bug. I fixed this by reversing the direction of the imports: the relevant class is now defined in meta.py and the tests that use this class import it from zarr.

Something else I noticed is that the test suite creates a persistent directory called fixture that contains a bunch of data that outlives the tests. These tests can be found here in main. The tests put this directory in the same directory as the root of the module, which means after running tests we end up with a src/fixture directory that has a bunch of stuff in it. This seems like something we should not be doing. For example, perhaps we could create temporary directories scoped to the test session. Maybe someone who understands these tests better can shed some light on how we ended up here.

In addition to changing the layout of the source code, I modified pyproject.toml to support use of the hatch project manager. I also considered PDM and Poetry (which I have the most experience with). I chose hatch because it was used in the scientific python packaging guide, and I've never heard anyone complain about it, so it seems safe. I think a modern project manager could make development a lot easier. We can replace the requirements_dev_*.txt files with entries in pyproject.toml, define environment variables for workflows in project.toml, and manage separate environments more easily than with our current setup.

This PR is WIP because

It's a work in progress. I'm not done folding all our build / test / ci configuration into pyproject.toml
I anticipate that lots of people will want to weigh in on how packaging for the repo should be configured.

I'm very interested to hear what people think!

d-v-b · 2023-12-07T21:13:17Z

I think this is blocked until I can figure out how to preserve our git history.

d-v-b · 2023-12-08T09:39:59Z

Git history has been restored via me using git mv instead of vanilla mv.

I opened #1601 to separate the test importing issues from the actual layout change.

Things I would appreciate help with:

getting CI to use hatch environments / commands
Setting up hatch environments for our different test contexts
getting dynamic versioning working
expressing the doc generation process via hatch

Saransh-cpp

Saw the help wanted label in my GitHub feed 😄

tests/__about__.py

Saransh-cpp · 2023-12-08T12:01:28Z

pyproject.toml

@@ -1,42 +1,38 @@
 [build-system]
-requires = ["setuptools>=64.0.0", "setuptools-scm>1.5.4"]
-build-backend = "setuptools.build_meta"
+requires = ["hatchling"]


There is an equivalent setuptools-scm for hatch - hatch-vcs. I feel that going from automated versioning to manual versioning would be a downgrade and an extra overhead for the maintainers.

that makes sense! We definitely want to avoid manual versioning. I will look into adding hatch-vcs

Looks like this is referred to in the scientific packaging guide too https://learn.scientific-python.org/development/guides/packaging-simple/#versioning

pyproject.toml

Saransh-cpp · 2023-12-08T12:06:03Z

pyproject.toml

 [tool.black]
+target-version = ["py38"]
+line-length = 100
+skip-string-normalization = true


black can now be replaced with ruff format but maybe not in this PR

Saransh-cpp · 2023-12-08T12:06:44Z

src/zarr/__about__.py

+# SPDX-FileCopyrightText: 2023-present Zarr Developers
+#
+# SPDX-License-Identifier: MIT
+__version__ = "0.0.1"


See the message about hatch-vcs above.

pyproject.toml

clbarnes · 2023-12-08T15:42:48Z

What's the rationale behind the __about__.py files? Seems like an extra place to store a different version.

d-v-b · 2023-12-08T15:47:27Z

What's the rationale behind the __about__.py files? Seems like an extra place to store a different version.

I honestly have no clue -- when I ran hatch new zarr, they were automatically generated, and I figured there must be some logic to it.

after searching, it seems like the hatch dev answered this question here. He doesn't want to clutter the root namespace with extra importables. This seems reasonable to me, but I don't feel strongly either way.

clbarnes · 2023-12-08T16:22:43Z

__version__ is something you have a cause to look for (feature switches) and having it at least importable at the root is a pretty standard place that you'd expect to find it. But if we switch to hatch-vcs then the version will come from a different file anyway.

clbarnes · 2023-12-12T10:10:25Z

Hatch has just released a new version, 1.8, with some nice features https://hatch.pypa.io/latest/blog/2023/12/11/hatch-v180/

jhamman · 2024-01-24T18:06:08Z

@d-v-b - would love to see this move forward. Is the list of help-needed here still relevant?

d-v-b · 2024-01-24T19:18:55Z

Yes! Help is definitely still wanted here. I'm not sure when I will have time to advance this.

Saransh-cpp · 2024-01-24T22:04:59Z

@d-v-b would it be possible for you to give me access to your fork? I might get some time this weekend to work on the hatch-vcs part. Otherwise, if you have time to work on it yourself, I could answer your questions or help you with the issues you are facing with hatch-vcs.

getting CI to use hatch environments / commands
Setting up hatch environments for our different test contexts
expressing the doc generation process via hatch

These are add-ons. Everything working in the CI with setuptools should work with hatch. I would suggest just migrating to hatch and hatch-vcs in this PR and then switching to hatch commands in the CI in the follow-up PRs.

d-v-b · 2024-01-24T22:13:17Z

@Saransh-cpp thank you! I added you as a collaborator to my fork of zarr-python.

d-v-b · 2024-01-24T22:13:36Z

These are add-ons. Everything working in the CI with setuptools should work with hatch. I would suggest just migrating to hatch and hatch-vcs in this PR and then switching to hatch commands in the CI in the follow-up PRs.

+1 to this approach

Saransh-cpp · 2024-01-24T22:14:50Z

Thanks! Will work on this on weekend.

src/zarr/_storage/v3.py

docs/contributing.rst

Saransh-cpp · 2024-01-26T18:06:41Z

src/zarr/meta.py

@@ -459,7 +459,7 @@ def _encode_storage_transformer_metadata(

    @classmethod
    def _decode_storage_transformer_metadata(cls, meta: Mapping) -> "StorageTransformer":
-        from zarr.tests.test_storage_v3 import DummyStorageTransfomer
+        from tests.test_storage_v3 import DummyStorageTransfomer


Why is a class from tests imported in user-facing functionality?

we don't know! see #1601

Oh, I see, haha! No worries, I'll rebase this once your PR is merged!

Saransh-cpp · 2024-01-26T18:10:41Z

This should be ready for a review. The tests pass for me locally and the package builds correctly.

The failing mypy test looks unrelated to this PR.

I have also force-pushed because it was much easier to migrate from scratch than resolving the conflicts in this PR.

@d-v-b please feel free to remove [WIP] from the PR title.

MSanKeys963 · 2024-01-26T20:02:08Z

Thanks a lot, @Saransh-cpp.

Saransh-cpp · 2024-01-30T21:19:36Z

Should be ready now

jhamman

Two small questions around the edges here. I think we should be good to go here soon though.

jhamman · 2024-02-03T02:31:51Z

.git_archival.txt

+node: $Format:%H$
+node-date: $Format:%cI$
+describe-name: $Format:%(describe:tags=true,match=*[0-9]*)$
+ref-names: $Format:%D$


can you explain why we need this file? Looking at hatch-vcs, I don't see any requirement of this configuration.

This file allows users to install Zarr from Git archives without encountering version issues. This is not a hatch-vcs thing, instead this file should've been introduced when setuptools-scm was introduced in Zarr - https://setuptools-scm.readthedocs.io/en/latest/usage/#builtin-mechanisms-for-obtaining-version-numbers. hatch-vcs uses setuptools_scm underneath.

Scientific Python also recommends adding git archival support -

You should also add these two files: .git_archival.txt: node: $Format:%H$ node-date: $Format:%cI$ describe-name: $Format:%(describe:tags=true,match=*[0-9]*)$ ref-names: $Format:%D$ And .gitattributes (or add this line if you are already using this file): .git_archival.txt export-subst This will allow git archives (including the ones generated from GitHub) to also support versioning.

Makes sense. Thanks for the explanation.

jhamman · 2024-02-03T02:37:10Z

tests/test_attrs.py

@@ -8,7 +8,7 @@
 from zarr.attrs import Attributes
 from zarr.storage import KVStore, DirectoryStore
 from zarr._storage.v3 import KVStoreV3
-from .util import CountingDict, CountingDictV3
+from tests.util import CountingDict, CountingDictV3


this is a bit of surprising change. Is it required? Is this what hatch suggests packages do? Generally, I prefer a relative imports within the tests directory.

Oh yes, this was just a personal preference. Happy to revert it!

Let's revert it. Happy to discuss this change elsewhere but let's keep this pr scope as small as possible.

joshmoore · 2024-02-05T21:24:20Z

There have been a number of breakages in conda land related to the fixtures/ directory. Considering the scale of the rework, this might be a time to address the root issue (if it doesn't just do it by chance!)

see:

jhamman · 2024-02-06T03:42:22Z

@joshmoore - let's sort out CI separately. Hatch is going to help us manage our various envs so let's get right on that after merging this.

Saransh-cpp · 2024-02-06T12:34:13Z

This will require some minor changes in the feedstock. I'll keep a watch, or if I forget, please ping me on the next version update PR in Zarr's feedstock.

d-v-b added the V3 Affects the v3 branch label Dec 7, 2023

d-v-b mentioned this pull request Dec 8, 2023

Don't import from tests #1601

Merged

d-v-b added the help wanted Issue could use help from someone with familiarity on the topic label Dec 8, 2023

Saransh-cpp suggested changes Dec 8, 2023

View reviewed changes

d-v-b requested a review from Saransh-cpp January 24, 2024 22:09

Saransh-cpp added 2 commits January 26, 2024 19:02

src layout + migrate to hatch

4577ed3

feat: git archival support

a5981a3

Saransh-cpp force-pushed the use_hatch branch from 3e3636e to a5981a3 Compare January 26, 2024 18:04

rm test generated files

2e823a5

Saransh-cpp reviewed Jan 26, 2024

View reviewed changes

Apply suggestions from code review

b9bc08f

d-v-b changed the title ~~[WIP] use src layout and use hatch for packaging~~ use src layout and use hatch for packaging Jan 26, 2024

Resolve conflicts

4c04117

jhamman approved these changes Feb 3, 2024

View reviewed changes

jhamman mentioned this pull request Feb 3, 2024

add py.typted file to zarr package d-v-b/zarr-python#108

Closed

Use relative imports

3ec8164

jhamman merged commit f8874bf into zarr-developers:v3 Feb 6, 2024
5 of 6 checks passed

jhamman mentioned this pull request Feb 6, 2024

[v3] Test environments and CI #1648

Open

6 tasks

jhamman added this to the 3.0.0.alpha milestone Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use src layout and use `hatch` for packaging #1592

use src layout and use `hatch` for packaging #1592

d-v-b commented Dec 7, 2023

d-v-b commented Dec 7, 2023

d-v-b commented Dec 8, 2023

Saransh-cpp left a comment •

edited

Loading

Saransh-cpp Dec 8, 2023

d-v-b Dec 8, 2023

clbarnes Dec 8, 2023

Saransh-cpp Dec 8, 2023

Saransh-cpp Dec 8, 2023

clbarnes commented Dec 8, 2023 •

edited

Loading

d-v-b commented Dec 8, 2023

clbarnes commented Dec 8, 2023

clbarnes commented Dec 12, 2023

jhamman commented Jan 24, 2024

d-v-b commented Jan 24, 2024

Saransh-cpp commented Jan 24, 2024

d-v-b commented Jan 24, 2024

d-v-b commented Jan 24, 2024

Saransh-cpp commented Jan 24, 2024

Saransh-cpp Jan 26, 2024

d-v-b Jan 26, 2024

Saransh-cpp Jan 26, 2024

Saransh-cpp commented Jan 26, 2024 •

edited

Loading

MSanKeys963 commented Jan 26, 2024

Saransh-cpp commented Jan 30, 2024

jhamman left a comment

jhamman Feb 3, 2024

Saransh-cpp Feb 3, 2024

jhamman Feb 3, 2024

jhamman Feb 3, 2024

Saransh-cpp Feb 3, 2024

jhamman Feb 3, 2024

joshmoore commented Feb 5, 2024

jhamman commented Feb 6, 2024

Saransh-cpp commented Feb 6, 2024

use src layout and use hatch for packaging #1592

use src layout and use hatch for packaging #1592

Conversation

d-v-b commented Dec 7, 2023

d-v-b commented Dec 7, 2023

d-v-b commented Dec 8, 2023

Saransh-cpp left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clbarnes commented Dec 8, 2023 • edited Loading

d-v-b commented Dec 8, 2023

clbarnes commented Dec 8, 2023

clbarnes commented Dec 12, 2023

jhamman commented Jan 24, 2024

d-v-b commented Jan 24, 2024

Saransh-cpp commented Jan 24, 2024

d-v-b commented Jan 24, 2024

d-v-b commented Jan 24, 2024

Saransh-cpp commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Saransh-cpp commented Jan 26, 2024 • edited Loading

MSanKeys963 commented Jan 26, 2024

Saransh-cpp commented Jan 30, 2024

jhamman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshmoore commented Feb 5, 2024

jhamman commented Feb 6, 2024

Saransh-cpp commented Feb 6, 2024

use src layout and use `hatch` for packaging #1592

use src layout and use `hatch` for packaging #1592

Saransh-cpp left a comment •

edited

Loading

clbarnes commented Dec 8, 2023 •

edited

Loading

Saransh-cpp commented Jan 26, 2024 •

edited

Loading