-
-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
S3 support #125
S3 support #125
Conversation
…nda-forge-pinning 2021.05.07.21.54.12
Hi! This is the friendly automated conda-forge-linting service. I just wanted to let you know that I linted all conda-recipes in your PR ( |
Ping @DennisHeimbigner @WardF Is there something we're missing here in the build config? I tried digging into the CI builds in Unidata/netcdf-c, but I didn't see anything obvious that was setting up the S3 testing. |
This enables for zarr only. But, the byterange support should work for S3 (as long as it is publically accessible) for netcdf-3 and netcdf-4 files stored as objects in S3. |
We currently do not do Zarr S3 testing on a routine basis since our tests use the NCAR |
@DennisHeimbigner Any pointers on what this build needs in order to properly build against 1.7 of the AWS SDK? We should also do something about the testing on netCDF-C; this is captured in Unidata/netcdf-c#1994. |
I have had a lot of trouble getting aws-sdk-cpp to compile. At the moment, the only |
If it is not in a stable URL the build reproducibility is questionable at best. However, getting that version does not seems to be the issue, building it is. I was not able to make the old version build properly. |
I finally got the latest version of aws-sdk-cpp to build for me on ubuntu 21.
|
@DennisHeimbigner That's great to hear. Have you successfully built netcdf-c with that version? |
Yes, I was able to use it to build netcdf-c with S3 support and successfully run |
@DennisHeimbigner Great. Can you list specifically what version of aws-sdk-cpp. They release often, so "latest release" is quite likely to be incorrect soon. |
Version 1.9 |
AWS releases a new 1.9.N for each business day. 😆 So which one of those did you use? |
To be precise, 1.9.96. |
What is the current status of this PR? Is this something that is possible now? I see some of the referenced issues in netcdf-c (with latest aws cpp) are still open. |
At this point, aws-sdk-cpp appears to have reached a reasonably stable point. |
@DennisHeimbigner So you're saying libnetcdf seems to support modern aws-sdk-cpp? So theoretically if this branch was updated with the feedstock main and newest libnetcdf version and newest aws-sdk-cpp it might "just work"? |
I would suggest that any new build changes be based on 4.9.1 (I've got the rc building in #154). |
Yes. It should just work, at least for linux. |
Given current schedule/TODO/life things I'm not sure I can guarantee working on this. I also know very little about enabling S3 support in NetCDF C apart from what @ocefpaf did in this PR already. If someone gets bored then please open a PR. Otherwise I'll have to wait until I'm avoiding other work I'm supposed to be doing. |
Local blizzard means kid stayed home from daycare means I work on what I want... |
@dopplershift I could not get this to compile locally even with aws-sdk 1.7 as recommended in Unidata/netcdf-c#1981. Do mind taking a look on how upstream do this?
Another question is: will this enable S3 support for all formats or just zarr? Is not how, if possible, can we do that for netcdf too?
Pinging @rsignell-usgs who is interested in this.