Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EnvironmentManager class for Workflows to extend Environment class functionality #701

Open
tclose opened this issue Sep 13, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@tclose
Copy link
Contributor

tclose commented Sep 13, 2023

What would you like changed/added and why?

Extending on the concepts introduced by #516, I would like to add similar concept for workflows, which selects appropriate environments for each of the nodes in the workflow instead of running the whole workflow in single environment. Essentially, the environment manager provides a mapping from tool interfaces used in the workflow to appropriate images/environment modules to use in the corresponding nodes.

We could provide a basic DockerHub-based environment manager for neuroimaging tools, which uses all available official Docker images available on DockerHub. But if someone is running on HPC with access to a range of singularity/apptainer containers, they could design their own manager to use them instead.

Another example is the analysis environment, NeuroDesk provides a range of apptainer images for different neuroimaging tools, e.g. FSL, MRtrix, Freesurfer, etc..., loaded/unloaded using environment modules. In this case, I would like to be able to define a NeuroDesk environment manager somewhere, which could be imported to enable any workflow be run using the appropriate software versions.

What would be the benefit?

  • Would make workflows using Environment classes portable between different execution environments
  • Provide standardised environments to use with workflows, that could be updated as new versions are released
@tclose tclose added the enhancement New feature or request label Sep 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant