-
Notifications
You must be signed in to change notification settings - Fork 44
open_mfdatatree #51
Comments
In the case of
Extending a little on the second option, it could be a nice functionality to be able to extract any level in the tree without losing information. It could be a layer before actually exporting to a netCDF. If I have a DataTree object |
I think this is what I was imagining. That's the most direct and simple mapping between an in-memory datatree and a set of folders and
I'm hesistant to do anything that introduces "inheritance" from nodes above like this. The problem is that different group-supporting formats have different hierarchical behaviours, and so something that follows netCDF might be weird with Zarr. Ultimately the in-memory
There is no specific method for flattening parts of the tree, but we can make one! (xref #79) I'm not quite sure what you want it to do though - what type would you want |
Sounds wise preserving the structure. I have two suggestions on that:
On the flattening, I was thinking on something like I have no strong opinion on any of those, but just ideas. |
Just found this: https://gitlab.eumetsat.int/open-source/netcdf-flattener/ |
Our team is interested in |
closing in favor of pydata/xarray#9351 |
Currently we have an
open_datatree
function which opens a single netcdf file (or zarr store). We could imagine anopen_mfdatatree
function which is analogous toopen_mfdataset
, which can open multiple files at once.As
DataTree
has a structure essentially the same as that of a filesystem, I'm imagining a use case where the user has a bunch of data files stored in nested directories, e.g.We could look through all of these folders recursively, open any files found of the correct format, and store them in a single tree.
We could even allow for multiple data files in each folder if we called
open_mfdataset
on all the files found in each folder.EDIT: We could also save a tree out to multiple folders like this using a
save_mfdatatree
method.This might be particularly useful for users who want the benefit of a tree-like structure but are using a file format that doesn't support groups.
The text was updated successfully, but these errors were encountered: