Add conda environment to run on personal computers #393
Conversation
|
Formatted documentation can be read here : |
There was a problem hiding this comment.
I will give it a try too. This looks great. A couple questions.
First, the documentation says the model is always run interactively (./cice.run) but the launch scripts suggest otherwise. The launch script runs with mpirun in mpi model and interactively in serial mode.
Second, how does the conda environment.yml file really work? It doesn't seem to specify any particular compiler, netcdf, or openmpi version or even vendor (gnu, intel, etc). In general, I think we like to specify the versions, so we avoid using problematic combinations of software or even buggy versions of software. Can we be a little more specific with respect to versioning?
|
Here are a little more details about how it works under the hood:
I mean interactively as not batch (there is no job scheduler involved). Even in MPI mode the model will run interactively, i.e. the user types
The environment file lists the required packages and a conda channel (
It is important to download all packages from the same channel to keep ABI compatibility between the packages.
How conda-forge works is that the whole stack is built using GCC on linux and the LLVM toolchain on macOS (with one exception: Gfortran is the Fortran compiler). There is no other choice with conda. The Regarding version of dependencies: since no versions are specified in the environment.yml file, the dependency solver of the conda package manager will do the job to choose mutually compatible versions of all listed packages. At the moment it installs (at least on my Linux machine):
It installs the same versions on mac, expect Clang instead of GCC as the C compiler. Conda will refuse to create an environment with mutually incompatible packages. What I had in mind is that if ever there is a dependency conflict we can adjust the environment file accordingly by pinning some packages to specific versions. |
|
@phil-blain Thanks for the clarifications. That makes sense. I guess I'd prefer to be able to explicitly specify versions so we have complete control. I worry that a nice suite of compatible versions as determined by conda could, at some point, have a compiler bug that affects us and that it could be hard to avoid it. The other issue is that if baselines are generated at different times by different compiler versions, answers could change. Again, we can control that by controlling when compiler versions are updated. Just a quick followup. When is the local conda environment created/updated. Obviously, first install is important. Then does each case, each build, each run potentially update the conda environment with new versions? Or is the conda environment static until the user does an update? Maybe some of this information should be included in the documentation? |
|
The environment is created when the user runs So you are right that we want to tell users to not touch the environment after it is created. I'll add it to the doc. |
|
@apcraig Another thought: I think it's good that the model gets tested with different versions of its dependencies. If ever we have a report at some point in time that the environmment installs versions of dependencies that CICE is not compatible with, then we can go back and edit the environment file to pin our dependencies to a last known working version. What do you think ? |
|
@phil-blain I agree that's a good plan. I still need to test and will try to do so in the next few days. Thanks! |
|
I just edited the PR description to add another positive point of conda:
|
|
I tried to build CICE on my mac laptop and there are a few problems. I'm not sure everything needs to be revised, but this is where I ran into problems. My laptop doesn't have wget by default, so I ended up just pointing to https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh in my browser and downloading that way. Once I installed the conda download, "which conda" returned nothing. I had to formally "source $HOME/miniconda3/etc/profile.d/conda.csh" in order to get conda started. Thereafter, I have to do that source in order to use the conda environment whenever I fire up a terminal window. I have created an alias for that. It certainly would be possible to automatically source conda whenever someone opens a terminal window. We probably need to make a recommendation. When I tried to build the case, I ran into an error,
That's where I stopped for now. Any thoughts on how to resolve the stdio.h error? I tried a few thing like manually doing "conda activate cice" but haven't figured it out yet. |
|
Hi Tony ! thanks for giving it a try.
I guess you mean https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh since you are on mac ?
Basically the installation instructions for macOS say:
If you follow the installation prompts, the installer will asks you if you want to initialize your shell to use conda. This should add the conda initialization sequence to your You mention that you need to "
Ok, tha's a bigger problem. What version of macOS are you running ? Apple is making it harder and harder for open-source developer in each release... I'm doing some research and I think I'm missing some steps if I want the procedure to work on macOS >= 10.13... I can reproduce your error on both 10.13 and 10.14:: |
|
Also: |
|
Just a quick followup. My default shell on my mac is csh/tcsh. I am running 10.14.2 on my laptop. I am also running 10.11.6 on my desktop mac which I will try to do an install on as well at some point just to check the procedure. In general, I haven't download a bunch of "software tools" like Xcode or whatever it's called now. I think we should assume users have not done that. I did logout of my terminal and log back in. I also tried working with bash instead of csh and I still think it didn't know about conda. You also have this command in your documentation "conda config --set auto_activate_base false". Does that turn off the auto activate? I'm not sure we should recommend auto activation anyway. And whether people use it or not, we should let them know they may need to manually source the conda profile file if it doesn't exist. I'm think conda should work with both bash and csh/tcsh. I agree that the bigger problem is the compiler problem for now. The other stuff is just settling on process and documentation. Thanks for working on this. If we can get something working, that would be awesome. One really nice thing is that users don't even need to "port" if this works as proposed. Just setup the env with conda and then the CICE port is already there. That's cool. |
Recent versions of Xcode and macOS do not install the system C header files to /usr/include, but keep them in the MacOSX SDK. Thus we need to explicitely pass their location to the compiler.
The CFLAG_HOST variable was added in CICE-Consortium#307, and some Makefile logic was added to define the variable to be blank if the included Macros file does not define it. However, the Make syntax is wrong, as 'ifndef' takes a variable name and not a reference to a variable [1], so ifndef $(CFLAGS_HOST) should have been written ifndef CFLAGS_HOST The effect of this error is to invert the logic of the check (!) since if CFLAGS_HOST is defined, Make will check if a variable with name equal to whatever CFLAGS_HOST is defined to be exists, which will most probably be false, and the conditional will then evaluate to true and CFLAG_HOSTS will be redefined to be blank, defeating the whole purpose of the flag. Since there's no harm in Make referencing and empty variable, fix this by just removing the conditional check. [1] https://www.gnu.org/software/make/manual/html_node/Conditional-Syntax.html
Ok. I did not check that case as I guess most users will not ever change their login shell, so they get Bash (or Zsh starting with macOS 10.15). source $HOME/miniconda3/etc/profile.d/conda.csh
conda init tcshfor your Then you would have access to the
That would be super. We want the procedure to work on as many OS versions as possible. I created a small project here: https://github.com/phil-blain/CICE-conda that gets tested using TravisCI (on 10.10 - 10.14) and GitHub actions (on 10.15). With the commits I just pushed all versions work.
It's true that we should not assume that our users have Xcode or the command line tools installed, but I think it will be required to have at least the command line tools on macOS >= 10.14. I could not test it yet as I don't have access to a Mac running either 10.14 or 10.15 that do not have Xcode installed (all macOS CI images either on Travis or GitHub of course have Xcode installed...)
With the two commits I just pushed it should work on 10.14 and 10.15 also (with the caveat that maybe Xcode/the command line tools needs to be installed). Once this is confirmed I will add a mention in the doc as to how to install the command line tools (
If you installed miniconda and answered 'yes'' to
Then I think that the miniconda installer should write it's initialization sequence to
This command turns off activation of the base environment on shell startup.
If people follow the installation instructions (and they did not change their default shell!) then it works. On macOS 10.15 they might have to |
|
I got it to build and run, but there were a few problems. First, this line in env.conda_macos cannot work, "source Second, I believe when you do "conda activate cice" in the env file as part of the cice.build, that also doesn't seem to work, although I'm not sure why. If I source the conda profile file manually first and then run cice.build, that fails. If I source the conda profile file manually and then source env.conda_macos then run cice.build, it works. If I manually source the conda profile file and manually run "conda activate cice" then run the cice.build script, that works. It seems doing "conda activate cice" as part of the cice.build script does not load the conda cice stuff permanently. Also, I played around with the "conda init tcsh" and that didn't seem to do anything. I think it's not clear whether the .tcshrc is sourced under csh and/or tcsh. Of course, lots of our scripts also have #!/bin/csh -f so that may also be creating a problem. This is what I recommend. I suggest we remove all of the "conda" stuff from the scripts. I think that's just in the env file. Then I propose the documentation say that to build/run with conda, the user needs to manually execute source $HOME/miniconda3/etc/profile.d/conda.csh (or conda.sh) and that those could easily be aliased via something like alias loadcondacice 'source $HOME/miniconda3/etc/profile.d/conda.csh; conda activate cice' What that does is remove all the complexity of the scripts/setup with regard to csh/tcsh/bash, whether the user has does something in their default environment to turn on conda or otherwise setup some defaults that may or may not be correct, and whether it all works together under various hardware/software versions. This may not be the most sophisticated implementation but I think it's the simplest and easiest to understand. Expert users can do the automation part on their own if they want and in the way that is most comfortable for them, but for new users, what I propose is probably cleaner and more robust. If we wanted, we could probably add something to the env script to check that the conda cice is active. This probably works set chk = but there are many ways it could probably be done. The above should fail if conda is not found or if the cice env is not loaded. It's probably worth having a check like that in any case. Anyway, the main thing is that I have been able to build and run a cice case on my Mac Laptop with 10.14.2, but I had to manually do the conda part and that is how I suggest we implement and document usage going forward. Excellent! Once the scripts are updated, I will check again on my laptop and then try on my desktop. |
wget is not installed by default
This variable is always defined to the path to the conda executable when the login shell is properly initialized. Calling `conda info --base` works if the shell initialization procedure puts the conda executable in the $PATH, which is the case for Bash and Zsh but not for tcsh.
The Miniconda installer only initializes Bash on macOS and Linux, so additional steps needs to be taken if your login shell is different. Add instructions for tcsh, zsh, fish and xonsh.
Let's check if the conda executable is found and if the cice conda environment exists.
|
I understand what you are saying, although I really think it's cleaner to not require to activate manually. The method I used does work if the shell initialization procedure puts the conda executable in the $PATH, which is the case for Bash and zsh but not for tcsh. I just pushed a commit that uses a more robust method that should work for all possible login shells as long as it's correctly configured to use conda (as we show in the doc). I think it's reasonable to assume that the user correctly read and executed the installation instructions. I also added some documentation as to how to correctly initialize conda in different login shells. I also added error checking to both env file, first checking if the conda executable exists, then if the "cice" conda environment can be correctly activated. What do you think of the new instructions ? |
|
Maybe some other members of the team could also test the updated procedure ? |
|
I think the documentation is better and we're almost there. The main thing is that it seems to work which is awesome! I will say that overall, I do not like that we are telling people that they have to change their default environment. I have learned the hard way that this always has unintended consequences. If this were an HPC environment, I would say it's unacceptable. We have the "env" file so we can setup the local environment; modules, env variables, etc, automatically for the model at hand. My preferences for handling the environment in general is to implement by
So again, overall, I don't think setting stuff in the default environment is the best approach. Also, I am unclear about what is going on in the env file. You have this line,
Isn't that what the default initialization is doing? Why does it have to be sourced again? Also, if the env file does the source, why can't the "default initialization" just be to
if the env file is sourcing the conda env anyway, how does having this in your default scripts add any value? It seems there is some redundancy that might mean the conda default stuff is not needed. In fact, I tested that with the current implementation, I can delete the conda init stuff in my .tcshrc file, simply set the CONDA_EXE env variable manually (it could also be in my login), and everything works. Personally, I think this is a much better way to go. Just set one env variable and not have to set anything about conda by default. In fact, I'd change the env file so it is just if ! $?CONDA_EXE then Then the documentation does not have to say anything about installing conda in the default environment and it can just say, "if you did not install miniconda3 under your $HOME directory, you need to set an env variable, CONDA_EXE, in order to build and run CICE on your laptop. ...". But assuming we are going to keep things as they are, I have a few other comments.
|
|
I implemented the changes proposed above in the documentation. I just noticed I added a new section "Porting to Laptop or Personal Computers" after "Porting" instead of under it. However I think it's maybe more useful that way as it limits the number of subsubsections. I could also change it. |
|
@phil-blain this is AWESOME! I followed the directions in the documentation and managed to get it to work, stumbling over a few things. My only complaint about this is that the instructions force the input data to be in ~/cice-dirs. I'd rather not have cice-dirs in my home directory, and so I tried to put it where I want it, but that didn't work, maybe because I'd set up conda from my home directory (?) or maybe because the locations are hardwired somewhere in the scripts. Now that it's there, am I stuck with it? I'm not sure that these need to be documented, but for the record, here are the things that gave me problems (macOS 10.14.6):
So I downloaded it manually.
This was because I hadn't cloned your branch yet... :)
I suspended my VPN session and this then worked. This was likely the problem with 1 above.
The only thing I suggest is that the conda environment and/or cice-dirs be installed where the user wants them. For other machines, there are standard names for the directories (CICE_RUNS, CICE_BASELINE or something like that, I think), and the user can choose where to put them by modifying the machine files. Since there are a number of "do one time" instructions for the conda environment, I wonder if the user should be given some of these choices early on? It would be good to be consistent, anyhow. THANK YOU. This is fantastic. |
|
I'll just add a couple thoughts based on @eclare108213 comments. The ability to set your own input data, baseline, and run directory is an important one. But what it means is that each user would have to create their own machine port. What @phil-blain has done is create a single "conda" port which requires leveraging the paths set there. We do not want users to customize the conda port and then push back to master as that would break the port for others. But there are a couple options. First, we could suggest users can actually do a machine port, taking the conda port as a starting point and then creating a new set of machine files. The second option would be to let people know they can create links from the hardwired conda paths to wherever they want. So, if you want to put the input data somewhere else, just use the conda port and manually create a permanent link from $HOME/cice_dirs/cice_input (or whatever it's called) to $WORK/cice_input (or wherever the data actually exists on your computer). I think the link is a reasonable workaround in this case and it should be "one-time". But that's not to say a machine port isn't also an option folks can do as well or instead. I think we probably just need to add a little documentation about both scenarios. |
|
I agree that we need to strike a balance between "works out-of-the-box" and "I don't want this Personally, I would not want them there either :P But as @apcraig says then the user would have to modify the env file. @apcraig I agree that at least mentioning that creating symlinks is possible is a good idea. We can also mention that if they want their paths elsewhere they can port to their own machine by customizing the env file based on the conda one. That's also a good idea. Another thing we could do is modify the env file so that all paths are defined using a @apcraig @eclare108213 what do you think of that approach ? |
|
The CICE_HOME env variable is an interesting one. I worry a little it could drive that same implementation into other machine files or introduce some confusion when users switch between their mac and other machines. So my initial instinct is to not implement it. But it does raise some interesting questions about whether we could move away from machine ports and move to a more env driven configuration system where the machine dependent differences are addressed by defining some local env variables and largely reusing the same env, Macros, launch and batch scripts across machines. I'm not convinced it's viable or better. But an interesting idea to ponder moving forward. |
|
I wondered the same thing, @apcraig. Would this procedure also work on something bigger than a desktop, like a linux cluster with a few hundred processors? It's a headache to have to rely on admins to keep the software on some bigger, local machines up to date (especially netCDF). |
|
@eclare108213 it would work in theory but for optimal performance the MPI implementation must be the one from the machine vendor, or OpenMPI/MPICH must be compiled on the machine itself... and with the conda environment there is not job scheduler so all runs are interactive. Altough it might be possible to use a similar approach for the env files for that cluser to install say just NetCDF through conda... but then there's the complication that each user must have access to the conda environment in their interactive and batch sessions... so it's not easy |
|
I see. This is very cool for laptops, anyhow! A number of years ago, someone sent me (or pointed me to) a video of CICE output scrolling on their hotel TV screen, being run from a raspberry pi... I thought that was hilarious. Pre iPhone days. I guess someone will make an app sooner or later. |
|
@apcraig @eclare108213 I added a note about symlinks and also about creating their own port. I think this is ready for final review now. |
|
the RTD build is failing with some cryptic Python error... probably a change of config on their end. I'll try to re-trigger it later. |
|
RTD was failing intermittently last week too. Just force push or retrigger when you think it's time. |
|
We need to trigger RTD again, still not working. Last week it took me a couple tries on one of the PRs too. Odd because other PRs are working. Saw this last week too. I'm pretty sure it's not the PR but RTD. It would be good to figure out how to retrigger RTD without a push. I know how to do it for travis but not RTD. |
|
@apcraig @eclare108213 RTD worked. Formatted documentation is here: For the record a simple way to trigger a new build is just to amend the last commit in the branch and force push: git checkout <branch>
git commit --amend --no-edit
git push --forcethis doesn't change anything in the last commit apart from the commiter date. |
|
Documentation looks good. Thanks @phil-blain, this is an awesome capability! Once travis is done, I think we can merge. |
|
One other thing, it would be nice if we could retrigger RTD without having to push. We can do that with Travis (you just go into travis and click the retrigger build link). Have looked around RTD, but have not found a retrigger capability. The requirement to push means only users with write access to the development branch can retrigger which is not ideal. If anyone has any ideas how to deal with this, lets discuss (maybe we can move to email). |
|
@apcraig I googled the python error and found this: They suggest wiping the build environment in RTD: Someone with admin rights need to do that. It might not stick though until this RTD PR is merged, according to this comment |
|
OK, I wiped the master and latest versions. I don't know if it'll help with the current PRs and there is no version associated with them that I can wipe. Hopefully this will stop being a problem as we generate new PRs, we'll see. |
…CE-Consortium#393) * machines: add env and Macro files for conda on Linux and macOS * machines: add environment.yml conda specification * doc: add section on laptop computers Uses the conda environment. * doc: recommend not to touch the 'cice' conda environment * machines: add CFLAGS_HOST to conda_macos for recent macOS Recent versions of Xcode and macOS do not install the system C header files to /usr/include, but keep them in the MacOSX SDK. Thus we need to explicitely pass their location to the compiler. * Makefile: remove uneeded (and faulty) logic around CFLAGS_HOST The CFLAG_HOST variable was added in CICE-Consortium#307, and some Makefile logic was added to define the variable to be blank if the included Macros file does not define it. However, the Make syntax is wrong, as 'ifndef' takes a variable name and not a reference to a variable [1], so ifndef $(CFLAGS_HOST) should have been written ifndef CFLAGS_HOST The effect of this error is to invert the logic of the check (!) since if CFLAGS_HOST is defined, Make will check if a variable with name equal to whatever CFLAGS_HOST is defined to be exists, which will most probably be false, and the conditional will then evaluate to true and CFLAG_HOSTS will be redefined to be blank, defeating the whole purpose of the flag. Since there's no harm in Make referencing and empty variable, fix this by just removing the conditional check. [1] https://www.gnu.org/software/make/manual/html_node/Conditional-Syntax.html * doc: use curl instead of wget on macOS wget is not installed by default * machines: conda: invoke 'conda info --base' using $CONDA_EXE This variable is always defined to the path to the conda executable when the login shell is properly initialized. Calling `conda info --base` works if the shell initialization procedure puts the conda executable in the $PATH, which is the case for Bash and Zsh but not for tcsh. * doc: conda: add initialization instructions for alternative login shells The Miniconda installer only initializes Bash on macOS and Linux, so additional steps needs to be taken if your login shell is different. Add instructions for tcsh, zsh, fish and xonsh. * machines: conda: add error checking to env files Let's check if the conda executable is found and if the cice conda environment exists. * doc: separate Miniconda installation and conda initialization - Move the documentation for the conda configuration under "Porting" - Move the instructions for initializing the user's shell for use with conda to a speparate section - Add a new section for manually initializing the user's shell if they do not want to modify their startup files - Emphasize which steps are "first-time setup" steps - Add details about how the conda environment is activated during build/run * doc: conda: mention symlinks and creating a complete port
…CE-Consortium#393) * machines: add env and Macro files for conda on Linux and macOS * machines: add environment.yml conda specification * doc: add section on laptop computers Uses the conda environment. * doc: recommend not to touch the 'cice' conda environment * machines: add CFLAGS_HOST to conda_macos for recent macOS Recent versions of Xcode and macOS do not install the system C header files to /usr/include, but keep them in the MacOSX SDK. Thus we need to explicitely pass their location to the compiler. * Makefile: remove uneeded (and faulty) logic around CFLAGS_HOST The CFLAG_HOST variable was added in CICE-Consortium#307, and some Makefile logic was added to define the variable to be blank if the included Macros file does not define it. However, the Make syntax is wrong, as 'ifndef' takes a variable name and not a reference to a variable [1], so ifndef $(CFLAGS_HOST) should have been written ifndef CFLAGS_HOST The effect of this error is to invert the logic of the check (!) since if CFLAGS_HOST is defined, Make will check if a variable with name equal to whatever CFLAGS_HOST is defined to be exists, which will most probably be false, and the conditional will then evaluate to true and CFLAG_HOSTS will be redefined to be blank, defeating the whole purpose of the flag. Since there's no harm in Make referencing and empty variable, fix this by just removing the conditional check. [1] https://www.gnu.org/software/make/manual/html_node/Conditional-Syntax.html * doc: use curl instead of wget on macOS wget is not installed by default * machines: conda: invoke 'conda info --base' using $CONDA_EXE This variable is always defined to the path to the conda executable when the login shell is properly initialized. Calling `conda info --base` works if the shell initialization procedure puts the conda executable in the $PATH, which is the case for Bash and Zsh but not for tcsh. * doc: conda: add initialization instructions for alternative login shells The Miniconda installer only initializes Bash on macOS and Linux, so additional steps needs to be taken if your login shell is different. Add instructions for tcsh, zsh, fish and xonsh. * machines: conda: add error checking to env files Let's check if the conda executable is found and if the cice conda environment exists. * doc: separate Miniconda installation and conda initialization - Move the documentation for the conda configuration under "Porting" - Move the instructions for initializing the user's shell for use with conda to a speparate section - Add a new section for manually initializing the user's shell if they do not want to modify their startup files - Emphasize which steps are "first-time setup" steps - Add details about how the conda environment is activated during build/run * doc: conda: mention symlinks and creating a complete port
PR checklist
Add a conda environment spec file describing the needed dependencies to compile and run CICE on personal computers. Also update the documentation with instructions to use this environment.
Philippe Blain
@eclare108213 @CICE-Consortium/devteam (anyone interested can try it!)
No tests.
This PR adds a "environment.yml" file that describes the required software needed to compile and run CICE under a conda environment. I chose to use the conda package and environment manager because:
@eclare108213 please test the procedure and tell me if there is any hiccups. I will update the instructions accordingly.