Skip to content

Commit

Permalink
🌱 Initial conda environment (#1)
Browse files Browse the repository at this point in the history
Add conda dependency specification file and getting started instructions in the main README.md. The conda environment.yml is paired with a conda-lock.yml lockfile for full reproducibility. Main README.md contains setup instructions to install NVIDIA GPU Direct Storage and the conda virtual environment locally.
  • Loading branch information
weiji14 authored Oct 1, 2023
1 parent 9e71258 commit 69f1ae8
Show file tree
Hide file tree
Showing 3 changed files with 6,544 additions and 0 deletions.
75 changes: 75 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,81 @@ data pipelines. Learn more as the Pangeo community develops tutorials at
about the challenges and ideas on scaling machine learning in the geosciences
with the [Pangeo ML Working Group](https://pangeo.io/meeting-notes.html#working-group-meetings).


# Getting started

## Installation

### NVIDIA GPU Direct Storage

Follow instructions at
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#install-gpudirect-storage
to install NVIDIA GPU Direct Storage (GDS).

Verify that NVIDIA GDS has been installed properly following
https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html#verify-suc-install.
E.g. if you are on Linux and have CUDA 12.2 installed, run:

/usr/local/cuda-12.2/gds/tools/gdscheck.py -p

Alternatively, if you have your conda environment setup below, follow
https://xarray.dev/blog/xarray-kvikio#appendix-ii--making-sure-gds-is-working
and run:

mamba activate foss4g2023oceania
curl -s https://github.com/raw/rapidsai/kvikio/branch-23.08/python/benchmarks/single-node-io.py | python

### Basic

To help out with development, start by cloning this [repo-url](/../../)

git clone <repo-url>

Then I recommend [using mamba](https://mamba.readthedocs.io/en/latest/installation.html)
to install the dependencies.
A virtual environment will also be created with Python and
[JupyterLab](https://github.com/jupyterlab/jupyterlab) installed.

cd foss4g2023oceania
mamba env create --file environment.yml

Activate the virtual environment first.

mamba activate foss4g2023oceania

Finally, double-check that the libraries have been installed.

mamba list

### Advanced

This is for those who want full reproducibility of the virtual environment.
Create a virtual environment with just Python and conda-lock installed first.

mamba create --name foss4g2023oceania python=3.10 conda-lock=2.3.0
mamba activate foss4g2023oceania

Generate a unified [`conda-lock.yml`](https://github.com/conda/conda-lock) file
based on the dependency specification in `environment.yml`. Use only when
creating a new `conda-lock.yml` file or refreshing an existing one.

conda-lock lock --mamba --file environment.yml --platform linux-64 --with-cuda=11.8

Installing/Updating a virtual environment from a lockile. Use this to sync your
dependencies to the exact versions in the `conda-lock.yml` file.

conda-lock install --mamba --name foss4g2023oceania conda-lock.yml

See also https://conda.github.io/conda-lock/output/#unified-lockfile for more
usage details.


# References

## Links

- https://xarray.dev/blog/xarray-kvikio

## License

All code in this repository is licensed under
Expand Down
Loading

0 comments on commit 69f1ae8

Please sign in to comment.