-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add new backend api documentation #4810
Merged
Merged
Changes from 2 commits
Commits
Show all changes
58 commits
Select commit
Hold shift + click to select a range
7e33010
documentation first draft
aurghs 75544c9
documentation update
aurghs c6f64cc
documentation update
aurghs 11fc283
update documentation
aurghs 1b8ac13
update backend documentation
aurghs e471f3a
incompletre draft: update Backend Documentation
aurghs 65c339d
fix
aurghs 113197d
fix syle
aurghs f9ed1d4
Update doc/internals.rst
aurghs 6b2ecd5
Update doc/internals.rst
aurghs 1958163
Update doc/internals.rst
aurghs bba32e4
Update doc/internals.rst
aurghs cb8d716
Update doc/internals.rst
aurghs 2adf355
Update doc/internals.rst
aurghs 7ec3238
Update doc/internals.rst
aurghs fb03493
Update doc/internals.rst
aurghs 22794d2
Update doc/internals.rst
aurghs 67d2c1f
Update doc/internals.rst
aurghs f58f16b
Update doc/internals.rst
aurghs 6a07a7c
Update doc/internals.rst
aurghs 1285874
Update doc/internals.rst
aurghs 112837d
Update doc/internals.rst
aurghs bdc46aa
Update doc/internals.rst
aurghs fe22048
Update doc/internals.rst
aurghs f492136
update section lazy laoding
aurghs fa7f212
Merge branch 'documentation-draft' of github.com:bopen/xarray into do…
aurghs b74c803
Merge branch 'master' into documentation-draft
aurghs de9432f
Merge remote-tracking branch 'origin/master' into documentation-draft
aurghs c50a95c
Update doc/internals.rst
aurghs ab62beb
Update doc/internals.rst
aurghs e470f36
Update doc/internals.rst
aurghs a777445
update internals.rst backend
aurghs 87ed0fa
Merge branch 'documentation-draft' of github.com:bopen/xarray into do…
aurghs 1f0870e
add lazy loading documentation
aurghs 885a6bd
update example on indexing type
aurghs 1381336
style
aurghs 0ef410a
fix
aurghs dc36138
modify backend indexing doc
aurghs 23e2423
fix
aurghs 99ca49e
removed LazilyVectorizedIndexedArray from backend doc
aurghs b1eb077
small fix in doc
aurghs 39bf16b
small fixes in backend doc
aurghs 121c060
removed exmple vectorized indexing
aurghs e838d40
update documentation
aurghs 8633e08
update documentation
aurghs 3281345
Merge branch 'documentation-draft' of github.com:bopen/xarray into do…
aurghs 992d47d
Merge remote-tracking branch 'origin/master' into documentation-draft
aurghs e3eb56d
isort
aurghs a456478
rename store_spec in filename_or_obj in guess_can_open
aurghs abf60e0
small update in backend documentation
aurghs e72ce9b
small update in backend documentation
aurghs 7108f80
Update doc/internals.rst
aurghs e8499cd
Update doc/internals.rst
aurghs 0955c16
fix backend documentation
aurghs 54c202c
replace LazilyOuterIndexedArray with LazilyIndexedArray
aurghs 3cc18d5
Update doc/internals.rst
aurghs 9faf5e6
Update doc/internals.rst
alexamici 06371df
Fix broken doc merge
alexamici File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -231,3 +231,101 @@ re-open it directly with Zarr: | |||||
zgroup = zarr.open("rasm.zarr") | ||||||
print(zgroup.tree()) | ||||||
dict(zgroup["Tair"].attrs) | ||||||
|
||||||
|
||||||
How to add a new backend | ||||||
------------------------ | ||||||
|
||||||
Adding a new backend for read support to Xarray is easy, and you don't need to integrate your code in Xarray. | ||||||
All you need to do is to: | ||||||
|
||||||
- Implement a function that returns an instance :py:class:``Dataset`` | ||||||
|
||||||
- Create a `BackendEntrypoint`` instance with your function as input. | ||||||
|
||||||
- Declare such instance as external plugin in your setup.py. | ||||||
keewis marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
``BackendEntrypoint` class is the main interface with Xarray, | ||||||
it's a container of attributes and functions to be implemented by the backend: | ||||||
|
||||||
- ``open_dataset`` | ||||||
- [``open_dataset_parameters``] | ||||||
- [``guess_can_open``] | ||||||
|
||||||
While ``open_dataset`` is mandatory, ``open_dataset_parameters`` and ``guess_can_open`` are optional. | ||||||
|
||||||
|
||||||
BackendEntrypoint.open_dataset | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider documenting the interface as a class? That might make it easier to document (e.g., in the form of docstrings). |
||||||
++++++++++++++++++++++++++++++ | ||||||
|
||||||
**Inputs** | ||||||
|
||||||
``BackendEntrypoint.open_dataset`` function shall take in input one argument, ``filename`` and one keyword argument ``drop_variables``: | ||||||
|
||||||
- ``filename`` may be a string containg a relative path, or the an instance of ``pathlib.Path``. | ||||||
- ``drop_variables`` may be `None` or a iterable containing the variables names to be dropped in reading the data. | ||||||
|
||||||
It may also take in input a set of keyword arguments, that will be passed from Xarray :py:func:`open_dataset` | ||||||
directly to the backend ``BackendEntrypoint.open_dataset``. | ||||||
Currently in Xarray :py:func:`open_dataset` there are two group of arguments will be passed to the backend. | ||||||
The first one are the **decoders**, explicity defined in Xarray :py:func:`open_dataset` signature: | ||||||
|
||||||
- ``mask_and_scale=None`` | ||||||
aurghs marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- ``decode_times=None`` | ||||||
- ``decode_timedelta=None`` | ||||||
- ``use_cftime=None`` | ||||||
- ``concat_characters=None`` | ||||||
- ``decode_coords=None`` | ||||||
|
||||||
They will be passed to the backend only if the user will pass explicity a value different from `None`. | ||||||
These parameters can be enabled/disabled by by the User, setting the keyword ``decode_cf`` managed by Xarray. | ||||||
The backend can implement these specific decoders keywords arguments, | ||||||
and it is desiderable if this makes sense for the specific backend. For more details see **decoders** sub-section. | ||||||
|
||||||
|
||||||
The second one can be passed by the user in a dictionary inside ``backend_kwargs`` or explicity as keyword arguments ``**kwargs``. | ||||||
They will be grouped together and passed to the backend as keyword arguments. | ||||||
|
||||||
**Output** | ||||||
|
||||||
```BackendEntrypoint`.open_dataset`` output shall be an instance of Xarray :py:class:`Dataset` | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
that implements an additional method ``close``, used by Xarray to ensure that the related files are closed. | ||||||
If don't want to support the lazy loading, then the :py:class:`Dataset` shall contain numpy.arrays and your work is almost done. | ||||||
|
||||||
BackendEntrypoint.open_dataset_parameters | ||||||
+++++++++++++++++++++++++++++++++++++++++ | ||||||
``open_dataset_parameters``is the list of ``BackendEntrypoint.open_dataset`` parameters. | ||||||
It is needed to enable/disable the decoders supported by the backend when the User set explicity ``decode_cf``. For this | ||||||
reason all the decoders supported by the backend must be explicity declared in the signature. | ||||||
``open_dataset_parameters`` it is no mandatory and if it is not provided xarray will inspect the signature of | ||||||
``BackendEntrypoint.open_dataset` and it will create ``open_dataset_parameters``. | ||||||
However, the signature inspection will not support `**kwargs` and `*args` are in the signature and in this case it will | ||||||
raise an error. | ||||||
|
||||||
BackendEntrypoint.guess_can_open | ||||||
+++++++++++++++++++++++++++++++++++++++++ | ||||||
|
||||||
|
||||||
|
||||||
How to support Lazy Loading | ||||||
+++++++++++++++++++++++++++ | ||||||
|
||||||
Decoders | ||||||
++++++++ | ||||||
- strings.CharacterArrayCoder() | ||||||
aurghs marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- strings.EncodedStringCoder() | ||||||
- variables.UnsignedIntegerCoder() | ||||||
- variables.CFMaskCoder() | ||||||
- variables.CFScaleOffsetCoder() | ||||||
- times.CFTimedeltaCoder() | ||||||
- times.CFDatetimeCoder(use_cftime=use_cftime) | ||||||
|
||||||
How to register a backend | ||||||
+++++++++++++++++++++++++ | ||||||
Define in your setup.py (or setup.cfg) an new entrypoint with: | ||||||
|
||||||
- group: ``xarray.backend`` | ||||||
- name: the name to be passed to :py:func:`open_dataset` as `engine``.` | ||||||
- object reference: the reference to the instance of ``BackendEntrypoint`` | ||||||
|
||||||
See https://packaging.python.org/specifications/entry-points/#data-model for more information. |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.